[VOTE] Spark 2.3.0 (RC1)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] Spark 2.3.0 (RC1)

Sameer Agarwal-2
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.3.0 (RC1)

Anirudh Ramanathan-3
Felix just pointed out to me that staging is missing the spark-kubernetes package.
I think we missed updating release-build.sh, which is why staging and the binary release are missing spark-kubernetes. 
Created SPARK-23063 to track.

On Fri, Jan 12, 2018 at 2:42 PM, Sameer Agarwal <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer



--
Anirudh Ramanathan
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.3.0 (RC1)

Sean Owen
In reply to this post by Sameer Agarwal-2
The signatures and licenses look OK. Except for the missing k8s package, the contents look OK. Tests look pretty good with "-Phive -Phadoop-2.7 -Pyarn" on Ubuntu 17.10, except that KafkaContinuousSourceSuite seems to hang forever. That was just fixed and needs to get into an RC?

Aside from the Blockers just filed for R docs, etc., we have:

Blocker:
SPARK-23000 Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3
SPARK-23020 Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher
SPARK-23051 job description in Spark UI is broken

Critical:
SPARK-22739 Additional Expression Support for Objects

I actually don't think any of those Blockers should be Blockers; not sure if the last one is really critical either.

I think this release will have to be re-rolled so I'd say -1 to RC1.

On Fri, Jan 12, 2018 at 4:42 PM Sameer Agarwal <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.3.0 (RC1)

Ted Yu
Is there going to be another RC ?

With KafkaContinuousSourceSuite hanging, it is hard to get the rest of the tests going.

Cheers

On Sat, Jan 13, 2018 at 7:29 AM, Sean Owen <[hidden email]> wrote:
The signatures and licenses look OK. Except for the missing k8s package, the contents look OK. Tests look pretty good with "-Phive -Phadoop-2.7 -Pyarn" on Ubuntu 17.10, except that KafkaContinuousSourceSuite seems to hang forever. That was just fixed and needs to get into an RC?

Aside from the Blockers just filed for R docs, etc., we have:

Blocker:
SPARK-23000 Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3
SPARK-23020 Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher
SPARK-23051 job description in Spark UI is broken

Critical:
SPARK-22739 Additional Expression Support for Objects

I actually don't think any of those Blockers should be Blockers; not sure if the last one is really critical either.

I think this release will have to be re-rolled so I'd say -1 to RC1.

On Fri, Jan 12, 2018 at 4:42 PM Sameer Agarwal <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.3.0 (RC1)

Sameer Agarwal-3
Yes, I'll cut an RC2 as soon as the remaining blockers are resolved. In the meantime, please continue to report any other issues here.

Here's a quick update on progress towards the next RC:

- SPARK-22908 (KafkaContiniousSourceSuite) has been reverted
- SPARK-23051 (Spark UI), SPARK-23063 (k8s packaging) and SPARK-23065 (R API docs) have all been resolved
- A fix for SPARK-23020 (SparkLauncherSuite) has been merged. We're monitoring the builds to make sure that the flakiness has been resolved.



On 16 January 2018 at 13:21, Ted Yu <[hidden email]> wrote:
Is there going to be another RC ?

With KafkaContinuousSourceSuite hanging, it is hard to get the rest of the tests going.

Cheers

On Sat, Jan 13, 2018 at 7:29 AM, Sean Owen <[hidden email]> wrote:
The signatures and licenses look OK. Except for the missing k8s package, the contents look OK. Tests look pretty good with "-Phive -Phadoop-2.7 -Pyarn" on Ubuntu 17.10, except that KafkaContinuousSourceSuite seems to hang forever. That was just fixed and needs to get into an RC?

Aside from the Blockers just filed for R docs, etc., we have:

Blocker:
SPARK-23000 Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3
SPARK-23020 Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher
SPARK-23051 job description in Spark UI is broken

Critical:
SPARK-22739 Additional Expression Support for Objects

I actually don't think any of those Blockers should be Blockers; not sure if the last one is really critical either.

I think this release will have to be re-rolled so I'd say -1 to RC1.

On Fri, Jan 12, 2018 at 4:42 PM Sameer Agarwal <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer




--
Sameer Agarwal
Computer Science | UC Berkeley
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.3.0 (RC1)

Holden Karau
So looking at http://pgp.mit.edu/pks/lookup?op=vindex&search=0xA1CEDBA8AD0C022A it seems like Sameer's key isn't in the Apache web of trust yet. This shouldn't block RC process but before we publish it's important to get the key in the Apache web of trust.

On Tue, Jan 16, 2018 at 3:00 PM, Sameer Agarwal <[hidden email]> wrote:
Yes, I'll cut an RC2 as soon as the remaining blockers are resolved. In the meantime, please continue to report any other issues here.

Here's a quick update on progress towards the next RC:

- SPARK-22908 (KafkaContiniousSourceSuite) has been reverted
- SPARK-23051 (Spark UI), SPARK-23063 (k8s packaging) and SPARK-23065 (R API docs) have all been resolved
- A fix for SPARK-23020 (SparkLauncherSuite) has been merged. We're monitoring the builds to make sure that the flakiness has been resolved.



On 16 January 2018 at 13:21, Ted Yu <[hidden email]> wrote:
Is there going to be another RC ?

With KafkaContinuousSourceSuite hanging, it is hard to get the rest of the tests going.

Cheers

On Sat, Jan 13, 2018 at 7:29 AM, Sean Owen <[hidden email]> wrote:
The signatures and licenses look OK. Except for the missing k8s package, the contents look OK. Tests look pretty good with "-Phive -Phadoop-2.7 -Pyarn" on Ubuntu 17.10, except that KafkaContinuousSourceSuite seems to hang forever. That was just fixed and needs to get into an RC?

Aside from the Blockers just filed for R docs, etc., we have:

Blocker:
SPARK-23000 Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3
SPARK-23020 Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher
SPARK-23051 job description in Spark UI is broken

Critical:
SPARK-22739 Additional Expression Support for Objects

I actually don't think any of those Blockers should be Blockers; not sure if the last one is really critical either.

I think this release will have to be re-rolled so I'd say -1 to RC1.

On Fri, Jan 12, 2018 at 4:42 PM Sameer Agarwal <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer




--
Sameer Agarwal
Computer Science | UC Berkeley



--
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.3.0 (RC1)

Sameer Agarwal-3
Thanks, will do!

On 16 January 2018 at 22:09, Holden Karau <[hidden email]> wrote:
So looking at http://pgp.mit.edu/pks/lookup?op=vindex&search=0xA1CEDBA8AD0C022A it seems like Sameer's key isn't in the Apache web of trust yet. This shouldn't block RC process but before we publish it's important to get the key in the Apache web of trust.

On Tue, Jan 16, 2018 at 3:00 PM, Sameer Agarwal <[hidden email]> wrote:
Yes, I'll cut an RC2 as soon as the remaining blockers are resolved. In the meantime, please continue to report any other issues here.

Here's a quick update on progress towards the next RC:

- SPARK-22908 (KafkaContiniousSourceSuite) has been reverted
- SPARK-23051 (Spark UI), SPARK-23063 (k8s packaging) and SPARK-23065 (R API docs) have all been resolved
- A fix for SPARK-23020 (SparkLauncherSuite) has been merged. We're monitoring the builds to make sure that the flakiness has been resolved.



On 16 January 2018 at 13:21, Ted Yu <[hidden email]> wrote:
Is there going to be another RC ?

With KafkaContinuousSourceSuite hanging, it is hard to get the rest of the tests going.

Cheers

On Sat, Jan 13, 2018 at 7:29 AM, Sean Owen <[hidden email]> wrote:
The signatures and licenses look OK. Except for the missing k8s package, the contents look OK. Tests look pretty good with "-Phive -Phadoop-2.7 -Pyarn" on Ubuntu 17.10, except that KafkaContinuousSourceSuite seems to hang forever. That was just fixed and needs to get into an RC?

Aside from the Blockers just filed for R docs, etc., we have:

Blocker:
SPARK-23000 Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3
SPARK-23020 Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher
SPARK-23051 job description in Spark UI is broken

Critical:
SPARK-22739 Additional Expression Support for Objects

I actually don't think any of those Blockers should be Blockers; not sure if the last one is really critical either.

I think this release will have to be re-rolled so I'd say -1 to RC1.

On Fri, Jan 12, 2018 at 4:42 PM Sameer Agarwal <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer




--
Sameer Agarwal
Computer Science | UC Berkeley



--



--
Sameer Agarwal
Computer Science | UC Berkeley
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.3.0 (RC1)

Sameer Agarwal-2
This vote has failed in favor of a new RC. I'll follow up with a new RC2 as soon as the 3 remaining test/UI blockers are resolved. 

On 17 January 2018 at 16:38, Sameer Agarwal <[hidden email]> wrote:
Thanks, will do!

On 16 January 2018 at 22:09, Holden Karau <[hidden email]> wrote:
So looking at http://pgp.mit.edu/pks/lookup?op=vindex&search=0xA1CEDBA8AD0C022A it seems like Sameer's key isn't in the Apache web of trust yet. This shouldn't block RC process but before we publish it's important to get the key in the Apache web of trust.

On Tue, Jan 16, 2018 at 3:00 PM, Sameer Agarwal <[hidden email]> wrote:
Yes, I'll cut an RC2 as soon as the remaining blockers are resolved. In the meantime, please continue to report any other issues here.

Here's a quick update on progress towards the next RC:

- SPARK-22908 (KafkaContiniousSourceSuite) has been reverted
- SPARK-23051 (Spark UI), SPARK-23063 (k8s packaging) and SPARK-23065 (R API docs) have all been resolved
- A fix for SPARK-23020 (SparkLauncherSuite) has been merged. We're monitoring the builds to make sure that the flakiness has been resolved.



On 16 January 2018 at 13:21, Ted Yu <[hidden email]> wrote:
Is there going to be another RC ?

With KafkaContinuousSourceSuite hanging, it is hard to get the rest of the tests going.

Cheers

On Sat, Jan 13, 2018 at 7:29 AM, Sean Owen <[hidden email]> wrote:
The signatures and licenses look OK. Except for the missing k8s package, the contents look OK. Tests look pretty good with "-Phive -Phadoop-2.7 -Pyarn" on Ubuntu 17.10, except that KafkaContinuousSourceSuite seems to hang forever. That was just fixed and needs to get into an RC?

Aside from the Blockers just filed for R docs, etc., we have:

Blocker:
SPARK-23000 Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3
SPARK-23020 Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher
SPARK-23051 job description in Spark UI is broken

Critical:
SPARK-22739 Additional Expression Support for Objects

I actually don't think any of those Blockers should be Blockers; not sure if the last one is really critical either.

I think this release will have to be re-rolled so I'd say -1 to RC1.

On Fri, Jan 12, 2018 at 4:42 PM Sameer Agarwal <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Thursday January 18, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast.


[ ] +1 Release this package as Apache Spark 2.3.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.3.0-rc1: https://github.com/apache/spark/tree/v2.3.0-rc1 (964cc2e31b2862bca0bd968b3e9e2cbf8d3ba5ea)

List of JIRA tickets resolved in this release can be found here: https://issues.apache.org/jira/projects/SPARK/versions/12339551

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.3.0?
===========================================

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.3.1 or 2.3.0 as appropriate.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.2.0. That being said, if there is something which is a regression from 2.2.0 that has not been correctly targeted please ping me or a committer to help target the issue (you can see the open issues listed as impacting Spark 2.3.0 at https://s.apache.org/WmoI).

=======================================
What are the unresolved issues targeted for 2.3.0?
=======================================

Please see https://s.apache.org/oXKi. At the time of the writing, there are 19 JIRA issues targeting 2.3.0 tracking various QA/audit tasks, test failures and other feature/bugs. In particular, we've currently marked 3 JIRAs as release blockers that are being actively worked on: 

1. SPARK-23051 that tracks a regression in the Spark UI
2. SPARK-23020 and SPARK-23000 that track a couple of flaky tests that are responsible for build failures. Additionally, https://github.com/apache/spark/pull/20242 fixes a few Java linter errors in RC1.

Given that these blockers are fairly isolated, in the sprit of starting a thorough QA early, this RC1 aims to serve as a good approximation of the functionality of final release.

Regards,
Sameer




--
Sameer Agarwal
Computer Science | UC Berkeley



--



--
Sameer Agarwal
Computer Science | UC Berkeley