Initial Decom PR for Spark 3?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Initial Decom PR for Spark 3?

Holden Karau
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Erik Erlandson-2
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Hyukjin Kwon
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Holden Karau
For follow up while I've backported this in some internal releases I'm not considering a candidate for backporting to Spark 3 anymore. I should have updated the thread with that. The design doc is linked in the PR.

On Thu, Jun 18, 2020 at 6:05 PM Hyukjin Kwon <[hidden email]> wrote:
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Stephen Boesch
In reply to this post by Hyukjin Kwon
Second paragraph of the PR lists the design doc.


On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <[hidden email]> wrote:
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Stephen Boesch
In reply to this post by Hyukjin Kwon
Hi given there is a design doc (contrary to that common) - is this going to move forward?

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <[hidden email]> wrote:
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Holden Karau
I believe so, however since Hyukjin is a committer and has asked for an SPIP I'll be making an SPIP for this next week. I hope to send out the draft for comment by the end of Spark summit. I'll be using the same design document for the design component, so if anyone has input on the design document feel free to start leaving comments there now.

On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <[hidden email]> wrote:
Hi given there is a design doc (contrary to that common) - is this going to move forward?

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <[hidden email]> wrote:
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Hyukjin Kwon
Yeah, I believe the community decided to do a SPIP for such significant changes. It would be best if we stick to the standard approaches.

2020년 6월 21일 (일) 오전 8:52, Holden Karau <[hidden email]>님이 작성:
I believe so, however since Hyukjin is a committer and has asked for an SPIP I'll be making an SPIP for this next week. I hope to send out the draft for comment by the end of Spark summit. I'll be using the same design document for the design component, so if anyone has input on the design document feel free to start leaving comments there now.

On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <[hidden email]> wrote:
Hi given there is a design doc (contrary to that common) - is this going to move forward?

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <[hidden email]> wrote:
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Stephen Boesch
I guess I missed that "community decision" where the existing design document that had been reviewed was put aside and a new SPIP  document was required.  

On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon <[hidden email]> wrote:
Yeah, I believe the community decided to do a SPIP for such significant changes. It would be best if we stick to the standard approaches.

2020년 6월 21일 (일) 오전 8:52, Holden Karau <[hidden email]>님이 작성:
I believe so, however since Hyukjin is a committer and has asked for an SPIP I'll be making an SPIP for this next week. I hope to send out the draft for comment by the end of Spark summit. I'll be using the same design document for the design component, so if anyone has input on the design document feel free to start leaving comments there now.

On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <[hidden email]> wrote:
Hi given there is a design doc (contrary to that common) - is this going to move forward?

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <[hidden email]> wrote:
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: Initial Decom PR for Spark 3?

Hyukjin Kwon
See https://spark.apache.org/improvement-proposals.html

On Tue, 23 Jun 2020, 07:01 Stephen Boesch, <[hidden email]> wrote:
I guess I missed that "community decision" where the existing design document that had been reviewed was put aside and a new SPIP  document was required.  

On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon <[hidden email]> wrote:
Yeah, I believe the community decided to do a SPIP for such significant changes. It would be best if we stick to the standard approaches.

2020년 6월 21일 (일) 오전 8:52, Holden Karau <[hidden email]>님이 작성:
I believe so, however since Hyukjin is a committer and has asked for an SPIP I'll be making an SPIP for this next week. I hope to send out the draft for comment by the end of Spark summit. I'll be using the same design document for the design component, so if anyone has input on the design document feel free to start leaving comments there now.

On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch <[hidden email]> wrote:
Hi given there is a design doc (contrary to that common) - is this going to move forward?

On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon <[hidden email]> wrote:
Looks it had to be with SPIP and a proper design doc to discuss.

2020년 2월 9일 (일) 오전 1:23, Erik Erlandson <[hidden email]>님이 작성:
I'd be willing to pull this in, unless others have concerns post branch-cut.

On Tue, Feb 4, 2020 at 2:51 PM Holden Karau <[hidden email]> wrote:
Hi Y’all,

I’ve got a K8s graceful decom PR ( ) I’d love to try and get in for Spark 3, but I don’t want to push on it if folks don’t think it’s worth it. I’ve been working on it since 2017 and it was really close in November but then I had the crash and had to step back for awhile.

It’s effectiveness is behind a feature flag and it’s been outstanding for awhile so those points are in its favour. It does however change things in core which is not great.

Cheers,

Holden
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9