An Update on Spark on Kubernetes [Jun 23]

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

An Update on Spark on Kubernetes [Jun 23]

Anirudh Ramanathan
Project Description: Kubernetes cluster manager integration that enables native support for submitting Spark applications to a kubernetes cluster. The submitted applications can make use of Kubernetes native constructs.

JIRA: 18278
Upstream Kubernetes Issue: 34377

Summary of activity in the last 2 weeks:
  • PySpark support being added by ifilolenko (Bloomberg)
  • HDFS locality fixes by kimoonkim (PepperData)
  • Minor bug fixes and performance improvement (Tencent)
  • Planning of work around Spark and HDFS on Kubernetes for Q3 2017
  • Weekly meeting notes and discussions (link)
Recent talks and updates:
  • Spark on Kubernetes (Spark Summit 2017) - link
  • HDFS on Kubernetes - Lessons Learned (Spark Summit 2017) - link

For more details and to contribute to our discussions/efforts, please attend our weekly public meeting, or join our slack/mailing list. (link)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: An Update on Spark on Kubernetes [Jun 23]

rxin
Thanks, Anirudh. This is super helpful!

On Fri, Jun 23, 2017 at 9:50 AM, Anirudh Ramanathan <[hidden email]> wrote:
Project Description: Kubernetes cluster manager integration that enables native support for submitting Spark applications to a kubernetes cluster. The submitted applications can make use of Kubernetes native constructs.

JIRA: 18278
Upstream Kubernetes Issue: 34377

Summary of activity in the last 2 weeks:
  • PySpark support being added by ifilolenko (Bloomberg)
  • HDFS locality fixes by kimoonkim (PepperData)
  • Minor bug fixes and performance improvement (Tencent)
  • Planning of work around Spark and HDFS on Kubernetes for Q3 2017
  • Weekly meeting notes and discussions (link)
Recent talks and updates:
  • Spark on Kubernetes (Spark Summit 2017) - link
  • HDFS on Kubernetes - Lessons Learned (Spark Summit 2017) - link

For more details and to contribute to our discussions/efforts, please attend our weekly public meeting, or join our slack/mailing list. (link)

Loading...