[Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

Matt Cheah

Hi everyone,

 

I filed SPARK-25299 to promote discussion on how we can improve the shuffle operation in Spark. The basic premise is to discuss the ways we can leverage distributed storage to improve the reliability and isolation of Spark’s shuffle architecture.

 

A few designs and a full problem statement are outlined in this architecture discussion document.

 

This is a complex problem and it would be great to get feedback from the community about the right direction to take this work in. Note that we have not yet committed to a specific implementation and architecture – there’s a lot that needs to be discussed for this improvement, so we hope to get as much input as possible before moving forward with a design.

 

Please feel free to leave comments and suggestions on the JIRA ticket or on the discussion document.

 

Thank you!

 

-Matt Cheah


smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

Yuanjian Li
Hi Matt,
     Thanks for the great document and proposal, I want to +1 for the reliable shuffle data and give some feedback.
     I think a reliable shuffle service based on DFS is necessary on Spark, especially running Spark job over unstable environment. For example, while mixed deploying Spark with online service, Spark executor will be killed any time. Current stage retry strategy will make the job many times slower than normal job.
     Actually we(Baidu inc) solved this problem by stable shuffle service over Hadoop, and we are now docking Spark to this shuffle service. The POC work will be done at October as expect. We'll post more benchmark and detailed work at that time. I'm still reading your discussion document and happy to give more feedback in the doc.

Thanks,
Yuanjian Li

Matt Cheah <[hidden email]> 于2018年9月1日周六 上午8:42写道:

Hi everyone,

 

I filed SPARK-25299 to promote discussion on how we can improve the shuffle operation in Spark. The basic premise is to discuss the ways we can leverage distributed storage to improve the reliability and isolation of Spark’s shuffle architecture.

 

A few designs and a full problem statement are outlined in this architecture discussion document.

 

This is a complex problem and it would be great to get feedback from the community about the right direction to take this work in. Note that we have not yet committed to a specific implementation and architecture – there’s a lot that needs to be discussed for this improvement, so we hope to get as much input as possible before moving forward with a design.

 

Please feel free to leave comments and suggestions on the JIRA ticket or on the discussion document.

 

Thank you!

 

-Matt Cheah

Reply | Threaded
Open this post in threaded view
|

Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

Matt Cheah

Yuanjian, Thanks for sharing your progress! I was wondering if there was any prototype code that we could read to get an idea of what the implementation looks like? We can evaluate the design together and also benchmark workloads from across the community – that is, we can collect more data from more Spark users.

 

The experience would be greatly appreciated in the discussion.

 

-Matt Cheah

 

From: Yuanjian Li <[hidden email]>
Date: Friday, August 31, 2018 at 8:29 PM
To: Matt Cheah <[hidden email]>
Cc: Spark dev list <[hidden email]>
Subject: Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

 

Hi Matt,

     Thanks for the great document and proposal, I want to +1 for the reliable shuffle data and give some feedback.

     I think a reliable shuffle service based on DFS is necessary on Spark, especially running Spark job over unstable environment. For example, while mixed deploying Spark with online service, Spark executor will be killed any time. Current stage retry strategy will make the job many times slower than normal job.

     Actually we(Baidu inc) solved this problem by stable shuffle service over Hadoop, and we are now docking Spark to this shuffle service. The POC work will be done at October as expect. We'll post more benchmark and detailed work at that time. I'm still reading your discussion document and happy to give more feedback in the doc.

 

Thanks,

Yuanjian Li

 

Matt Cheah <[hidden email]> 201891日周六 上午8:42写道:

Hi everyone,

 

I filed SPARK-25299 [issues.apache.org] to promote discussion on how we can improve the shuffle operation in Spark. The basic premise is to discuss the ways we can leverage distributed storage to improve the reliability and isolation of Spark’s shuffle architecture.

 

A few designs and a full problem statement are outlined in this architecture discussion document [docs.google.com].

 

This is a complex problem and it would be great to get feedback from the community about the right direction to take this work in. Note that we have not yet committed to a specific implementation and architecture – there’s a lot that needs to be discussed for this improvement, so we hope to get as much input as possible before moving forward with a design.

 

Please feel free to leave comments and suggestions on the JIRA ticket or on the discussion document.

 

Thank you!

 

-Matt Cheah


smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

Felix Cheung
Hi
+baibing3
+huangtao6

Came across your presentation on Alluxio - including shuffling - would you be interested in this?



From: Matt Cheah <[hidden email]>
Sent: Tuesday, September 4, 2018 2:54 PM
To: Yuanjian Li
Cc: Spark dev list
Subject: Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data
 

Yuanjian, Thanks for sharing your progress! I was wondering if there was any prototype code that we could read to get an idea of what the implementation looks like? We can evaluate the design together and also benchmark workloads from across the community �C that is, we can collect more data from more Spark users.

 

The experience would be greatly appreciated in the discussion.

 

-Matt Cheah

 

From: Yuanjian Li <[hidden email]>
Date: Friday, August 31, 2018 at 8:29 PM
To: Matt Cheah <[hidden email]>
Cc: Spark dev list <[hidden email]>
Subject: Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

 

Hi Matt,

     Thanks for the great document and proposal, I want to +1 for the reliable shuffle data and give some feedback.

     I think a reliable shuffle service based on DFS is necessary on Spark, especially running Spark job over unstable environment. For example, while mixed deploying Spark with online service, Spark executor will be killed any time. Current stage retry strategy will make the job many times slower than normal job.

     Actually we(Baidu inc) solved this problem by stable shuffle service over Hadoop, and we are now docking Spark to this shuffle service. The POC work will be done at October as expect. We'll post more benchmark and detailed work at that time. I'm still reading your discussion document and happy to give more feedback in the doc.

 

Thanks,

Yuanjian Li

 

Matt Cheah <[hidden email]>201891日周六上午8:42写道:

Hi everyone,

 

I filed SPARK-25299 [issues.apache.org] to promote discussion on how we can improve the shuffle operation in Spark. The basic premise is to discuss the ways we can leverage distributed storage to improve the reliability and isolation of Spark’s shuffle architecture.

 

A few designs and a full problem statement are outlined in thisarchitecture discussion document [docs.google.com].

 

This is a complex problem and it would be great to get feedback from the community about the right direction to take this work in. Note that we have not yet committed to a specific implementation and architecture �C there’s a lot that needs to be discussed for this improvement, so we hope to get as much input as possible before moving forward with a design.

 

Please feel free to leave comments and suggestions on the JIRA ticket or on the discussion document.

 

Thank you!

 

-Matt Cheah