[DISCUSS][SPIP] Graceful Decommissioning

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS][SPIP] Graceful Decommissioning

Holden Karau
At the recommendation of Hyukjin, I'm converting the graceful decommissioning work to an SPIP. The SPIP document is at https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing and the associated JIRA is at https://issues.apache.org/jira/browse/SPARK-20624. This work dates back to 2017 when an earlier design was brought up. Now in 2019 I've updated the design https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing.

From the SPIP requirements, I am willing to act as the shepherd & committer on this proposal, and have already been reviewing and creating PRs here. There are several folks who have contributed to design (you can see the discussion in the 2019 & 2017 documents) and I'm very thankful for those contributions, as well as the other code reviewers & PR authors who have been participating from a variety of vendors/users.

Given the existence of multiple vendors' proprietary implementations (Databricks & AWS) of this feature, I think the user need is very clear.

This begins the discussion phase of the SPIP process where the goal is to determine the need for the change and the general design outline. Once the discussion settles, we move on the voting stage. While there are WIP PRS attached to the design document to illustrate the implementation, approval of the SPIP does not necessarily mean those are the specific implementations that we will use.

For more information about SPIPs in general see https://spark.apache.org/improvement-proposals.html

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

Hyukjin Kwon
Thank you so much, Holden.

PS: I cc'ed some people who might be interested in this too FYI.

2020년 6월 26일 (금) 오전 11:26, Holden Karau <[hidden email]>님이 작성:
At the recommendation of Hyukjin, I'm converting the graceful decommissioning work to an SPIP. The SPIP document is at https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing and the associated JIRA is at https://issues.apache.org/jira/browse/SPARK-20624. This work dates back to 2017 when an earlier design was brought up. Now in 2019 I've updated the design https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing.

From the SPIP requirements, I am willing to act as the shepherd & committer on this proposal, and have already been reviewing and creating PRs here. There are several folks who have contributed to design (you can see the discussion in the 2019 & 2017 documents) and I'm very thankful for those contributions, as well as the other code reviewers & PR authors who have been participating from a variety of vendors/users.

Given the existence of multiple vendors' proprietary implementations (Databricks & AWS) of this feature, I think the user need is very clear.

This begins the discussion phase of the SPIP process where the goal is to determine the need for the change and the general design outline. Once the discussion settles, we move on the voting stage. While there are WIP PRS attached to the design document to illustrate the implementation, approval of the SPIP does not necessarily mean those are the specific implementations that we will use.

For more information about SPIPs in general see https://spark.apache.org/improvement-proposals.html

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

Holden Karau
Thanks for looping in more folks :)

On Thu, Jun 25, 2020 at 7:41 PM Hyukjin Kwon <[hidden email]> wrote:
Thank you so much, Holden.

PS: I cc'ed some people who might be interested in this too FYI.

2020년 6월 26일 (금) 오전 11:26, Holden Karau <[hidden email]>님이 작성:
At the recommendation of Hyukjin, I'm converting the graceful decommissioning work to an SPIP. The SPIP document is at https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing and the associated JIRA is at https://issues.apache.org/jira/browse/SPARK-20624. This work dates back to 2017 when an earlier design was brought up. Now in 2019 I've updated the design https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing.

From the SPIP requirements, I am willing to act as the shepherd & committer on this proposal, and have already been reviewing and creating PRs here. There are several folks who have contributed to design (you can see the discussion in the 2019 & 2017 documents) and I'm very thankful for those contributions, as well as the other code reviewers & PR authors who have been participating from a variety of vendors/users.

Given the existence of multiple vendors' proprietary implementations (Databricks & AWS) of this feature, I think the user need is very clear.

This begins the discussion phase of the SPIP process where the goal is to determine the need for the change and the general design outline. Once the discussion settles, we move on the voting stage. While there are WIP PRS attached to the design document to illustrate the implementation, approval of the SPIP does not necessarily mean those are the specific implementations that we will use.

For more information about SPIPs in general see https://spark.apache.org/improvement-proposals.html

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

Holden Karau
There’s been some comments & a few additions in the doc, but it seems like the folks taking a look generally agree on the design. If there are no other issues I will bring this to a vote late next week.

On Thu, Jun 25, 2020 at 7:43 PM Holden Karau <[hidden email]> wrote:
Thanks for looping in more folks :)

On Thu, Jun 25, 2020 at 7:41 PM Hyukjin Kwon <[hidden email]> wrote:
Thank you so much, Holden.

PS: I cc'ed some people who might be interested in this too FYI.

2020년 6월 26일 (금) 오전 11:26, Holden Karau <[hidden email]>님이 작성:
At the recommendation of Hyukjin, I'm converting the graceful decommissioning work to an SPIP. The SPIP document is at https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing and the associated JIRA is at https://issues.apache.org/jira/browse/SPARK-20624. This work dates back to 2017 when an earlier design was brought up. Now in 2019 I've updated the design https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing.

From the SPIP requirements, I am willing to act as the shepherd & committer on this proposal, and have already been reviewing and creating PRs here. There are several folks who have contributed to design (you can see the discussion in the 2019 & 2017 documents) and I'm very thankful for those contributions, as well as the other code reviewers & PR authors who have been participating from a variety of vendors/users.

Given the existence of multiple vendors' proprietary implementations (Databricks & AWS) of this feature, I think the user need is very clear.

This begins the discussion phase of the SPIP process where the goal is to determine the need for the change and the general design outline. Once the discussion settles, we move on the voting stage. While there are WIP PRS attached to the design document to illustrate the implementation, approval of the SPIP does not necessarily mean those are the specific implementations that we will use.

For more information about SPIPs in general see https://spark.apache.org/improvement-proposals.html

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

Mridul Muralidharan

  Thanks for shepherding this Holden !
I left a few comments, but overall it looks good to me.

Regards,
Mridul


On Sat, Jun 27, 2020 at 9:34 PM Holden Karau <[hidden email]> wrote:
There’s been some comments & a few additions in the doc, but it seems like the folks taking a look generally agree on the design. If there are no other issues I will bring this to a vote late next week.

On Thu, Jun 25, 2020 at 7:43 PM Holden Karau <[hidden email]> wrote:
Thanks for looping in more folks :)

On Thu, Jun 25, 2020 at 7:41 PM Hyukjin Kwon <[hidden email]> wrote:
Thank you so much, Holden.

PS: I cc'ed some people who might be interested in this too FYI.

2020년 6월 26일 (금) 오전 11:26, Holden Karau <[hidden email]>님이 작성:
At the recommendation of Hyukjin, I'm converting the graceful decommissioning work to an SPIP. The SPIP document is at https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing and the associated JIRA is at https://issues.apache.org/jira/browse/SPARK-20624. This work dates back to 2017 when an earlier design was brought up. Now in 2019 I've updated the design https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing.

From the SPIP requirements, I am willing to act as the shepherd & committer on this proposal, and have already been reviewing and creating PRs here. There are several folks who have contributed to design (you can see the discussion in the 2019 & 2017 documents) and I'm very thankful for those contributions, as well as the other code reviewers & PR authors who have been participating from a variety of vendors/users.

Given the existence of multiple vendors' proprietary implementations (Databricks & AWS) of this feature, I think the user need is very clear.

This begins the discussion phase of the SPIP process where the goal is to determine the need for the change and the general design outline. Once the discussion settles, we move on the voting stage. While there are WIP PRS attached to the design document to illustrate the implementation, approval of the SPIP does not necessarily mean those are the specific implementations that we will use.

For more information about SPIPs in general see https://spark.apache.org/improvement-proposals.html

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

wuyi
In reply to this post by Holden Karau
Thank you for your effort, Holden.

I left a few comments in SPIP. I asked for some details, though I know some
contents have been include in the design doc. I'm not very clear about
difference between the design doc and SPIP. But from what I saw at the SPIP
template questions, I think some details maybe still needed.


--
Yi





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

Holden Karau
So from the template I believe the SPIP is supposed to be more high level and then design goes into the linked “design sketch.” What sort of detail would you like to see added?

On Mon, Jun 29, 2020 at 1:38 AM wuyi <[hidden email]> wrote:
Thank you for your effort, Holden.

I left a few comments in SPIP. I asked for some details, though I know some
contents have been include in the design doc. I'm not very clear about
difference between the design doc and SPIP. But from what I saw at the SPIP
template questions, I think some details maybe still needed.


--
Yi





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

wuyi
I've left the comments in SPIP, so let's discuss there.


Holden Karau wrote
> So from the template I believe the SPIP is supposed to be more high level
> and then design goes into the linked “design sketch.” What sort of detail
> would you like to see added?
>
> On Mon, Jun 29, 2020 at 1:38 AM wuyi &lt;

> yi.wu@

> &gt; wrote:
>
>> Thank you for your effort, Holden.
>>
>> I left a few comments in SPIP. I asked for some details, though I know
>> some
>> contents have been include in the design doc. I'm not very clear about
>> difference between the design doc and SPIP. But from what I saw at the
>> SPIP
>> template questions, I think some details maybe still needed.
>>
>>
>> --
>> Yi
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail:

> dev-unsubscribe@.apache

>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  &lt;https://amzn.to/2MaRAG9&gt;
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Graceful Decommissioning

Holden Karau
Ah, I had thought there was a larger issue given the scope of the comments. Excited to hear that is not the case. I'll respond in the doc :)

On Mon, Jun 29, 2020 at 8:03 AM wuyi <[hidden email]> wrote:
I've left the comments in SPIP, so let's discuss there.


Holden Karau wrote
> So from the template I believe the SPIP is supposed to be more high level
> and then design goes into the linked “design sketch.” What sort of detail
> would you like to see added?
>
> On Mon, Jun 29, 2020 at 1:38 AM wuyi &lt;

> yi.wu@

> &gt; wrote:
>
>> Thank you for your effort, Holden.
>>
>> I left a few comments in SPIP. I asked for some details, though I know
>> some
>> contents have been include in the design doc. I'm not very clear about
>> difference between the design doc and SPIP. But from what I saw at the
>> SPIP
>> template questions, I think some details maybe still needed.
>>
>>
>> --
>> Yi
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail:

> dev-unsubscribe@.apache

>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  &lt;https://amzn.to/2MaRAG9&gt;
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9