[VOTE] Release Spark 3.1.0 (RC1)

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Takeshi Yamamuro
> I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.

+1

Thanks for working on this, Hyukjin!

Bests,
Takeshi

On Thu, Jan 7, 2021 at 2:13 PM Hyukjin Kwon <[hidden email]> wrote:
Thank you Holden and Wenchen!

Let me:
- prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST)
- and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts



2021년 1월 7일 (목) 오후 2:04, Holden Karau <[hidden email]>님이 작성:
I think that posting the 3.1.0 maven release was an accident and we're going to 3.1.1 RCs is the right step forward.
I'd ask for maybe a day before cutting the 3.1.1 release, I think https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at first I thought it was just a test issue, but Dongjoon pointed out the NPE happens in prod too).

I'd also like to echo the: it's totally ok we all make mistakes especially in partially manual & partially automated environments, I've created a bunch of RCs labels without recognizing they were getting pushed automatically.

On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan <[hidden email]> wrote:
I agree with Jungtaek that people are likely to be biased when testing 3.1.0. At least this will not be the same community-blessed release as previous ones, because the voting is already affected by the fact that 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon <[hidden email]> wrote:
Okay, let me just start to prepare 3.1.1. I think that will address all concerns except that 3.1.0 will remain in Maven as incomplete.
By right, removal in the Maven repo is disallowed. Overwrite is possible as far as I know but other mirrors that maintain cache will get affected.
Maven is one of the downstream publish channels, and we haven't officially announced and published it to Apache repo anyway.
I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.
Are we all good with this?



2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon <[hidden email]>님이 작성:
I think that It would be great though if we have a clear blocker that makes the release pointless if we want to drop this RC practically given that we will schedule 3.1.1 faster - non-regression bug fixes will be delivered to end users relatively fast.
That would make it clear which option we should take. I personally don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's response anyway.


2021년 1월 7일 (목) 오후 1:03, Sean Owen <[hidden email]>님이 작성:
I don't agree the first two are blockers for reasons I gave earlier.
Those two do look like important issues - are they regressions from 3.0.1?
I do agree we'd probably cut a new RC for those in any event, so agree with the plan to drop 3.1.0 (if the Maven release can't be overwritten)

On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <[hidden email]> wrote:
Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin already made two blockers shared here.
IIUC, it meant implicitly RC1 failure at that time.

In addition to that, there are two correctness issues. So, I made up my mind to cast -1 for this RC1 before joining this thread.

SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache (committed after tagging)
SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR is under review)

Although the above issues are not regression, those are enough for me to give -1 for 3.1.0 RC1.

On Wed, Jan 6, 2021 at 3:52 PM Sean Owen <[hidden email]> wrote:
I just don't see a reason to believe there's a rush? just test it as normal? I did, you can too, etc.
Or specifically what blocks the current RC?


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Gengliang Wang-2
> - prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST)
> - and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts

+1

On Thu, Jan 7, 2021 at 7:54 PM Takeshi Yamamuro <[hidden email]> wrote:
> I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.

+1

Thanks for working on this, Hyukjin!

Bests,
Takeshi

On Thu, Jan 7, 2021 at 2:13 PM Hyukjin Kwon <[hidden email]> wrote:
Thank you Holden and Wenchen!

Let me:
- prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST)
- and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts



2021년 1월 7일 (목) 오후 2:04, Holden Karau <[hidden email]>님이 작성:
I think that posting the 3.1.0 maven release was an accident and we're going to 3.1.1 RCs is the right step forward.
I'd ask for maybe a day before cutting the 3.1.1 release, I think https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at first I thought it was just a test issue, but Dongjoon pointed out the NPE happens in prod too).

I'd also like to echo the: it's totally ok we all make mistakes especially in partially manual & partially automated environments, I've created a bunch of RCs labels without recognizing they were getting pushed automatically.

On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan <[hidden email]> wrote:
I agree with Jungtaek that people are likely to be biased when testing 3.1.0. At least this will not be the same community-blessed release as previous ones, because the voting is already affected by the fact that 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon <[hidden email]> wrote:
Okay, let me just start to prepare 3.1.1. I think that will address all concerns except that 3.1.0 will remain in Maven as incomplete.
By right, removal in the Maven repo is disallowed. Overwrite is possible as far as I know but other mirrors that maintain cache will get affected.
Maven is one of the downstream publish channels, and we haven't officially announced and published it to Apache repo anyway.
I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.
Are we all good with this?



2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon <[hidden email]>님이 작성:
I think that It would be great though if we have a clear blocker that makes the release pointless if we want to drop this RC practically given that we will schedule 3.1.1 faster - non-regression bug fixes will be delivered to end users relatively fast.
That would make it clear which option we should take. I personally don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's response anyway.


2021년 1월 7일 (목) 오후 1:03, Sean Owen <[hidden email]>님이 작성:
I don't agree the first two are blockers for reasons I gave earlier.
Those two do look like important issues - are they regressions from 3.0.1?
I do agree we'd probably cut a new RC for those in any event, so agree with the plan to drop 3.1.0 (if the Maven release can't be overwritten)

On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <[hidden email]> wrote:
Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin already made two blockers shared here.
IIUC, it meant implicitly RC1 failure at that time.

In addition to that, there are two correctness issues. So, I made up my mind to cast -1 for this RC1 before joining this thread.

SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache (committed after tagging)
SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR is under review)

Although the above issues are not regression, those are enough for me to give -1 for 3.1.0 RC1.

On Wed, Jan 6, 2021 at 3:52 PM Sean Owen <[hidden email]> wrote:
I just don't see a reason to believe there's a rush? just test it as normal? I did, you can too, etc.
Or specifically what blocks the current RC?


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Jacek Laskowski
In reply to this post by Hyukjin Kwon
Hi,

I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that we're skipping that particular release. Gonna be more fun! :)

On Thu, Jan 7, 2021 at 6:13 AM Hyukjin Kwon <[hidden email]> wrote:
Thank you Holden and Wenchen!

Let me:
- prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST)
- and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts



2021년 1월 7일 (목) 오후 2:04, Holden Karau <[hidden email]>님이 작성:
I think that posting the 3.1.0 maven release was an accident and we're going to 3.1.1 RCs is the right step forward.
I'd ask for maybe a day before cutting the 3.1.1 release, I think https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at first I thought it was just a test issue, but Dongjoon pointed out the NPE happens in prod too).

I'd also like to echo the: it's totally ok we all make mistakes especially in partially manual & partially automated environments, I've created a bunch of RCs labels without recognizing they were getting pushed automatically.

On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan <[hidden email]> wrote:
I agree with Jungtaek that people are likely to be biased when testing 3.1.0. At least this will not be the same community-blessed release as previous ones, because the voting is already affected by the fact that 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon <[hidden email]> wrote:
Okay, let me just start to prepare 3.1.1. I think that will address all concerns except that 3.1.0 will remain in Maven as incomplete.
By right, removal in the Maven repo is disallowed. Overwrite is possible as far as I know but other mirrors that maintain cache will get affected.
Maven is one of the downstream publish channels, and we haven't officially announced and published it to Apache repo anyway.
I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.
Are we all good with this?



2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon <[hidden email]>님이 작성:
I think that It would be great though if we have a clear blocker that makes the release pointless if we want to drop this RC practically given that we will schedule 3.1.1 faster - non-regression bug fixes will be delivered to end users relatively fast.
That would make it clear which option we should take. I personally don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's response anyway.


2021년 1월 7일 (목) 오후 1:03, Sean Owen <[hidden email]>님이 작성:
I don't agree the first two are blockers for reasons I gave earlier.
Those two do look like important issues - are they regressions from 3.0.1?
I do agree we'd probably cut a new RC for those in any event, so agree with the plan to drop 3.1.0 (if the Maven release can't be overwritten)

On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <[hidden email]> wrote:
Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin already made two blockers shared here.
IIUC, it meant implicitly RC1 failure at that time.

In addition to that, there are two correctness issues. So, I made up my mind to cast -1 for this RC1 before joining this thread.

SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache (committed after tagging)
SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR is under review)

Although the above issues are not regression, those are enough for me to give -1 for 3.1.0 RC1.

On Wed, Jan 6, 2021 at 3:52 PM Sean Owen <[hidden email]> wrote:
I just don't see a reason to believe there's a rush? just test it as normal? I did, you can too, etc.
Or specifically what blocks the current RC?


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Jacek Laskowski
Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?


On Thu, Jan 7, 2021 at 3:12 PM Jacek Laskowski <[hidden email]> wrote:
Hi,

I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that we're skipping that particular release. Gonna be more fun! :)

On Thu, Jan 7, 2021 at 6:13 AM Hyukjin Kwon <[hidden email]> wrote:
Thank you Holden and Wenchen!

Let me:
- prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST)
- and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts



2021년 1월 7일 (목) 오후 2:04, Holden Karau <[hidden email]>님이 작성:
I think that posting the 3.1.0 maven release was an accident and we're going to 3.1.1 RCs is the right step forward.
I'd ask for maybe a day before cutting the 3.1.1 release, I think https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at first I thought it was just a test issue, but Dongjoon pointed out the NPE happens in prod too).

I'd also like to echo the: it's totally ok we all make mistakes especially in partially manual & partially automated environments, I've created a bunch of RCs labels without recognizing they were getting pushed automatically.

On Wed, Jan 6, 2021 at 8:57 PM Wenchen Fan <[hidden email]> wrote:
I agree with Jungtaek that people are likely to be biased when testing 3.1.0. At least this will not be the same community-blessed release as previous ones, because the voting is already affected by the fact that 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me.

On Thu, Jan 7, 2021 at 12:54 PM Hyukjin Kwon <[hidden email]> wrote:
Okay, let me just start to prepare 3.1.1. I think that will address all concerns except that 3.1.0 will remain in Maven as incomplete.
By right, removal in the Maven repo is disallowed. Overwrite is possible as far as I know but other mirrors that maintain cache will get affected.
Maven is one of the downstream publish channels, and we haven't officially announced and published it to Apache repo anyway.
I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.
Are we all good with this?



2021년 1월 7일 (목) 오후 1:11, Hyukjin Kwon <[hidden email]>님이 작성:
I think that It would be great though if we have a clear blocker that makes the release pointless if we want to drop this RC practically given that we will schedule 3.1.1 faster - non-regression bug fixes will be delivered to end users relatively fast.
That would make it clear which option we should take. I personally don't mind dropping 3.1.0 as well; we'll have to wait for the INFRA team's response anyway.


2021년 1월 7일 (목) 오후 1:03, Sean Owen <[hidden email]>님이 작성:
I don't agree the first two are blockers for reasons I gave earlier.
Those two do look like important issues - are they regressions from 3.0.1?
I do agree we'd probably cut a new RC for those in any event, so agree with the plan to drop 3.1.0 (if the Maven release can't be overwritten)

On Wed, Jan 6, 2021 at 9:38 PM Dongjoon Hyun <[hidden email]> wrote:
Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin already made two blockers shared here.
IIUC, it meant implicitly RC1 failure at that time.

In addition to that, there are two correctness issues. So, I made up my mind to cast -1 for this RC1 before joining this thread.

SPARK-34011 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache (committed after tagging)
SPARK-34027 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache (PR is under review)

Although the above issues are not regression, those are enough for me to give -1 for 3.1.0 RC1.

On Wed, Jan 6, 2021 at 3:52 PM Sean Owen <[hidden email]> wrote:
I just don't see a reason to believe there's a rush? just test it as normal? I did, you can too, etc.
Or specifically what blocks the current RC?


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Sean Owen-2
While we can delete the tag, maybe just leave it. As a general rule we would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Xiao Li
> I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away.

+1

Sean Owen <[hidden email]> 于2021年1月7日周四 上午6:44写道:
While we can delete the tag, maybe just leave it. As a general rule we would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Jacek Laskowski
In reply to this post by Sean Owen-2
Hi Sean,

+1 to leave it. Makes so much more sense (as that's what really happened and the history of Apache Spark is...irreversible).

On Thu, Jan 7, 2021 at 3:44 PM Sean Owen <[hidden email]> wrote:
While we can delete the tag, maybe just leave it. As a general rule we would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Hyukjin Kwon
I had a response from the INFRA team and Sonatype. Just to share, the removal is possible as an exception, but it's best to go ahead for 3.1.1 for safety as we all discussed.
There are several non-regression but correctness issues being tracked under https://issues.apache.org/jira/browse/SPARK-33507 that contains few tickets Dongjoon mentioned.
I will monitor this ticket and proceed for 3.1.1 RC1 when there are no more standing issues under this ticket.
In addition, I made a PR to share this to Spark users and community (https://github.com/apache/spark-website/pull/301), and it's merged and visible now (https://spark.apache.org/news/next-official-release-spark-3.1.1.html)

Thank you all for your support and feedback!

2021년 1월 8일 (금) 오전 2:07, Jacek Laskowski <[hidden email]>님이 작성:
Hi Sean,

+1 to leave it. Makes so much more sense (as that's what really happened and the history of Apache Spark is...irreversible).

On Thu, Jan 7, 2021 at 3:44 PM Sean Owen <[hidden email]> wrote:
While we can delete the tag, maybe just leave it. As a general rule we would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Dongjoon Hyun-2
Thank you, Hyukjin!

Bests,
Dongjoon.

On Mon, Jan 11, 2021 at 7:24 AM Hyukjin Kwon <[hidden email]> wrote:
I had a response from the INFRA team and Sonatype. Just to share, the removal is possible as an exception, but it's best to go ahead for 3.1.1 for safety as we all discussed.
There are several non-regression but correctness issues being tracked under https://issues.apache.org/jira/browse/SPARK-33507 that contains few tickets Dongjoon mentioned.
I will monitor this ticket and proceed for 3.1.1 RC1 when there are no more standing issues under this ticket.
In addition, I made a PR to share this to Spark users and community (https://github.com/apache/spark-website/pull/301), and it's merged and visible now (https://spark.apache.org/news/next-official-release-spark-3.1.1.html)

Thank you all for your support and feedback!

2021년 1월 8일 (금) 오전 2:07, Jacek Laskowski <[hidden email]>님이 작성:
Hi Sean,

+1 to leave it. Makes so much more sense (as that's what really happened and the history of Apache Spark is...irreversible).

On Thu, Jan 7, 2021 at 3:44 PM Sean Owen <[hidden email]> wrote:
While we can delete the tag, maybe just leave it. As a general rule we would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.0 (RC1)

Hyukjin Kwon
I plan to cut RC 1 for Spark 3.1.1 this friday or next monday. Let me know if there are any blocker.

Thanks guys.

2021년 1월 12일 (화) 오전 1:22, Dongjoon Hyun <[hidden email]>님이 작성:
Thank you, Hyukjin!

Bests,
Dongjoon.

On Mon, Jan 11, 2021 at 7:24 AM Hyukjin Kwon <[hidden email]> wrote:
I had a response from the INFRA team and Sonatype. Just to share, the removal is possible as an exception, but it's best to go ahead for 3.1.1 for safety as we all discussed.
There are several non-regression but correctness issues being tracked under https://issues.apache.org/jira/browse/SPARK-33507 that contains few tickets Dongjoon mentioned.
I will monitor this ticket and proceed for 3.1.1 RC1 when there are no more standing issues under this ticket.
In addition, I made a PR to share this to Spark users and community (https://github.com/apache/spark-website/pull/301), and it's merged and visible now (https://spark.apache.org/news/next-official-release-spark-3.1.1.html)

Thank you all for your support and feedback!

2021년 1월 8일 (금) 오전 2:07, Jacek Laskowski <[hidden email]>님이 작성:
Hi Sean,

+1 to leave it. Makes so much more sense (as that's what really happened and the history of Apache Spark is...irreversible).

On Thu, Jan 7, 2021 at 3:44 PM Sean Owen <[hidden email]> wrote:
While we can delete the tag, maybe just leave it. As a general rule we would not remove anything pushed to the main git repo.

On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1?

12