[VOTE] Release Spark 3.1.1 (RC1)

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[VOTE] Release Spark 3.1.1 (RC1)

Hyukjin Kwon
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Hyukjin Kwon
I forgot to say :). I'll start with my +1.

On Mon, 18 Jan 2021, 21:06 Hyukjin Kwon, <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Yuming Wang
+1.

On Tue, Jan 19, 2021 at 7:54 AM Hyukjin Kwon <[hidden email]> wrote:
I forgot to say :). I'll start with my +1.

On Mon, 18 Jan 2021, 21:06 Hyukjin Kwon, <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Jungtaek Lim-2
+1 (non-binding)

* verified signature and sha for all files (there's a glitch which I'll describe in below)
* built source (DISCLAIMER: didn't run tests) and made custom distribution, and built a docker image based on the distribution
  - used profiles: kubernetes, hadoop-3.2, hadoop-cloud
* ran some SS PySpark queries (Rate to Kafka, Kafka to Kafka) with Spark on k8s (used MinIO - s3 compatible - as checkpoint location)
  - for Kafka reader, tested both approaches: newer (offset via admin client) and older (offset via consumer)
* ran simple batch query with magic committer against MinIO storage & dynamic volume provisioning (with NFS)
* verified DataStreamReader.table & DataStreamWriter.toTable works in PySpark (which also verifies on Scala API as well)
* ran test stateful SS queries and checked the new additions of SS UI (state store & watermark information)

A glitch from verifying sha; the file format of sha512 is different between source targz and others. My tool succeeded with others and failed with source targz, though I confirmed sha itself is the same. Not a blocker but would be ideal if we can make it be consistent.

Thanks for driving the release process!

On Tue, Jan 19, 2021 at 2:25 PM Yuming Wang <[hidden email]> wrote:
+1.

On Tue, Jan 19, 2021 at 7:54 AM Hyukjin Kwon <[hidden email]> wrote:
I forgot to say :). I'll start with my +1.

On Mon, 18 Jan 2021, 21:06 Hyukjin Kwon, <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Gengliang Wang-2
+1 (non-binding)


On Tue, Jan 19, 2021 at 2:05 PM Jungtaek Lim <[hidden email]> wrote:
+1 (non-binding)

* verified signature and sha for all files (there's a glitch which I'll describe in below)
* built source (DISCLAIMER: didn't run tests) and made custom distribution, and built a docker image based on the distribution
  - used profiles: kubernetes, hadoop-3.2, hadoop-cloud
* ran some SS PySpark queries (Rate to Kafka, Kafka to Kafka) with Spark on k8s (used MinIO - s3 compatible - as checkpoint location)
  - for Kafka reader, tested both approaches: newer (offset via admin client) and older (offset via consumer)
* ran simple batch query with magic committer against MinIO storage & dynamic volume provisioning (with NFS)
* verified DataStreamReader.table & DataStreamWriter.toTable works in PySpark (which also verifies on Scala API as well)
* ran test stateful SS queries and checked the new additions of SS UI (state store & watermark information)

A glitch from verifying sha; the file format of sha512 is different between source targz and others. My tool succeeded with others and failed with source targz, though I confirmed sha itself is the same. Not a blocker but would be ideal if we can make it be consistent.

Thanks for driving the release process!

On Tue, Jan 19, 2021 at 2:25 PM Yuming Wang <[hidden email]> wrote:
+1.

On Tue, Jan 19, 2021 at 7:54 AM Hyukjin Kwon <[hidden email]> wrote:
I forgot to say :). I'll start with my +1.

On Mon, 18 Jan 2021, 21:06 Hyukjin Kwon, <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Yang,Jie(INF)

+1

 

发件人: Gengliang Wang <[hidden email]>
日期: 2021119 星期二 下午3:04
收件人: Jungtaek Lim <[hidden email]>
抄送: Yuming Wang <[hidden email]>, Hyukjin Kwon <[hidden email]>, dev <[hidden email]>
主题: Re: [VOTE] Release Spark 3.1.1 (RC1)

 

+1 (non-binding)

 

 

On Tue, Jan 19, 2021 at 2:05 PM Jungtaek Lim <[hidden email]> wrote:

+1 (non-binding)

 

* verified signature and sha for all files (there's a glitch which I'll describe in below)

* built source (DISCLAIMER: didn't run tests) and made custom distribution, and built a docker image based on the distribution

  - used profiles: kubernetes, hadoop-3.2, hadoop-cloud

* ran some SS PySpark queries (Rate to Kafka, Kafka to Kafka) with Spark on k8s (used MinIO - s3 compatible - as checkpoint location)

  - for Kafka reader, tested both approaches: newer (offset via admin client) and older (offset via consumer)

* ran simple batch query with magic committer against MinIO storage & dynamic volume provisioning (with NFS)

* verified DataStreamReader.table & DataStreamWriter.toTable works in PySpark (which also verifies on Scala API as well)

* ran test stateful SS queries and checked the new additions of SS UI (state store & watermark information)

 

A glitch from verifying sha; the file format of sha512 is different between source targz and others. My tool succeeded with others and failed with source targz, though I confirmed sha itself is the same. Not a blocker but would be ideal if we can make it be consistent.

 

Thanks for driving the release process!

 

On Tue, Jan 19, 2021 at 2:25 PM Yuming Wang <[hidden email]> wrote:

+1.

 

On Tue, Jan 19, 2021 at 7:54 AM Hyukjin Kwon <[hidden email]> wrote:

I forgot to say :). I'll start with my +1.

 

On Mon, 18 Jan 2021, 21:06 Hyukjin Kwon, <[hidden email]> wrote:

Please vote on releasing the following candidate as Apache Spark version 3.1.1.

 

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

 

[ ] +1 Release this package as Apache Spark 3.1.0

[ ] -1 Do not release this package because ...

 

To learn more about Apache Spark, please see http://spark.apache.org/

 

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

 

The release files, including signatures, digests, etc. can be found at:

 

Signatures used for Spark RCs can be found in this file:

 

The staging repository for this release can be found at:

 

The documentation corresponding to this release can be found at:

 

The list of bug fixes going into 3.1.1 can be found at the following URL:

 

This release is using the release script of the tag v3.1.1-rc1.

 

FAQ

 

===================
What happened to 3.1.0?

===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================

How can I help test this release?

=========================

 

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.

 

If you're working in PySpark you can set up a virtual env and install

the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.

In the Java/Scala, you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with an out of date RC going forward).

 

===========================================

What should happen to JIRA tickets still targeting 3.1.1?

===========================================

 

The current list of open tickets targeted at 3.1.1 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

 

Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.

 

==================

But my bug isn't fixed?

==================

 

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.

 

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Prashant Sharma
+1

On Tue, Jan 19, 2021 at 4:38 PM Yang,Jie(INF) <[hidden email]> wrote:

+1

 

发件人: Gengliang Wang <[hidden email]>
日期: 2021119 星期二 下午3:04
收件人: Jungtaek Lim <[hidden email]>
抄送: Yuming Wang <[hidden email]>, Hyukjin Kwon <[hidden email]>, dev <[hidden email]>
主题: Re: [VOTE] Release Spark 3.1.1 (RC1)

 

+1 (non-binding)

 

 

On Tue, Jan 19, 2021 at 2:05 PM Jungtaek Lim <[hidden email]> wrote:

+1 (non-binding)

 

* verified signature and sha for all files (there's a glitch which I'll describe in below)

* built source (DISCLAIMER: didn't run tests) and made custom distribution, and built a docker image based on the distribution

  - used profiles: kubernetes, hadoop-3.2, hadoop-cloud

* ran some SS PySpark queries (Rate to Kafka, Kafka to Kafka) with Spark on k8s (used MinIO - s3 compatible - as checkpoint location)

  - for Kafka reader, tested both approaches: newer (offset via admin client) and older (offset via consumer)

* ran simple batch query with magic committer against MinIO storage & dynamic volume provisioning (with NFS)

* verified DataStreamReader.table & DataStreamWriter.toTable works in PySpark (which also verifies on Scala API as well)

* ran test stateful SS queries and checked the new additions of SS UI (state store & watermark information)

 

A glitch from verifying sha; the file format of sha512 is different between source targz and others. My tool succeeded with others and failed with source targz, though I confirmed sha itself is the same. Not a blocker but would be ideal if we can make it be consistent.

 

Thanks for driving the release process!

 

On Tue, Jan 19, 2021 at 2:25 PM Yuming Wang <[hidden email]> wrote:

+1.

 

On Tue, Jan 19, 2021 at 7:54 AM Hyukjin Kwon <[hidden email]> wrote:

I forgot to say :). I'll start with my +1.

 

On Mon, 18 Jan 2021, 21:06 Hyukjin Kwon, <[hidden email]> wrote:

Please vote on releasing the following candidate as Apache Spark version 3.1.1.

 

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

 

[ ] +1 Release this package as Apache Spark 3.1.0

[ ] -1 Do not release this package because ...

 

To learn more about Apache Spark, please see http://spark.apache.org/

 

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

 

The release files, including signatures, digests, etc. can be found at:

 

Signatures used for Spark RCs can be found in this file:

 

The staging repository for this release can be found at:

 

The documentation corresponding to this release can be found at:

 

The list of bug fixes going into 3.1.1 can be found at the following URL:

 

This release is using the release script of the tag v3.1.1-rc1.

 

FAQ

 

===================
What happened to 3.1.0?

===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================

How can I help test this release?

=========================

 

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.

 

If you're working in PySpark you can set up a virtual env and install

the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.

In the Java/Scala, you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with an out of date RC going forward).

 

===========================================

What should happen to JIRA tickets still targeting 3.1.1?

===========================================

 

The current list of open tickets targeted at 3.1.1 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

 

Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.

 

==================

But my bug isn't fixed?

==================

 

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.

 

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

JackyLee
+1



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

John Zhuge
+1 (non-binding)

On Tue, Jan 19, 2021 at 4:08 AM JackyLee <[hidden email]> wrote:
+1



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
John Zhuge
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Sean Owen-2
In reply to this post by Hyukjin Kwon
+1 from me. Same results as in 3.1.0 testing.

On Mon, Jan 18, 2021 at 6:06 AM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Holden Karau
+1, pip installs on Python 3.8

One potential thing we might want to consider if there ends up being another RC is that the error message for installing with Python2 could be clearer.

Processing ./pyspark-3.1.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /tmp/py3.1/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-W1BsIL
         cwd: /tmp/pip-req-build-lmlitE/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-lmlitE/setup.py", line 31
        file=sys.stderr)
            ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.



On Tue, Jan 19, 2021 at 10:26 AM Sean Owen <[hidden email]> wrote:
+1 from me. Same results as in 3.1.0 testing.

On Mon, Jan 18, 2021 at 6:06 AM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Dongjoon Hyun-2
+1

I additionally
- Ran JDBC integration test
- Ran with AWS EKS 1.16
- Ran unit tests with Python 3.9.1 combination (numpy 1.19.5, pandas 1.2.0, scipy 1.6.0)
  (PyArrow is not tested because it's not supported in Python 3.9.x. This is documented via SPARK-34162)

There exists some on-going work in the umbrella JIRA (SPARK-33507: Improve and fix cache behavior in v1 and v2).
I believe it can be achieved at 3.2.0 and we can add some comments on the release note at 3.1.0.

Thank you, Hyukjin and all.

Bests,
Dongjoon.

On Tue, Jan 19, 2021 at 10:49 AM Holden Karau <[hidden email]> wrote:
+1, pip installs on Python 3.8

One potential thing we might want to consider if there ends up being another RC is that the error message for installing with Python2 could be clearer.

Processing ./pyspark-3.1.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /tmp/py3.1/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-W1BsIL
         cwd: /tmp/pip-req-build-lmlitE/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-lmlitE/setup.py", line 31
        file=sys.stderr)
            ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.



On Tue, Jan 19, 2021 at 10:26 AM Sean Owen <[hidden email]> wrote:
+1 from me. Same results as in 3.1.0 testing.

On Mon, Jan 18, 2021 at 6:06 AM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Jacek Laskowski
In reply to this post by Hyukjin Kwon
Hi,

+1 (non-binding)

1. Built locally using AdoptOpenJDK (build 11.0.9+11) with -Pyarn,kubernetes,hive-thriftserver,scala-2.12 -DskipTests
2. Ran batch and streaming demos using Spark on Kubernetes (minikube) using spark-shell (client deploy mode) and spark-submit --deploy-mode cluster

I reported a non-blocking issue with "the only developer Matei" (https://issues.apache.org/jira/browse/SPARK-34158)

Found a minor non-blocking (but annoying) issue in Spark on k8s that's different from 3.0.1 that should really be silenced as the other debug message in ExecutorPodsAllocator:

21/01/19 12:23:26 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.
21/01/19 12:23:27 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.
21/01/19 12:23:28 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.
21/01/19 12:23:29 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.


On Mon, Jan 18, 2021 at 1:06 PM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Terry Kim
+1 (non-binding)

(Also ran .NET for Apache Spark E2E tests, which touch many of DataFrame, Function APIs)

Thanks,
Terry

On Wed, Jan 20, 2021 at 6:01 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

+1 (non-binding)

1. Built locally using AdoptOpenJDK (build 11.0.9+11) with -Pyarn,kubernetes,hive-thriftserver,scala-2.12 -DskipTests
2. Ran batch and streaming demos using Spark on Kubernetes (minikube) using spark-shell (client deploy mode) and spark-submit --deploy-mode cluster

I reported a non-blocking issue with "the only developer Matei" (https://issues.apache.org/jira/browse/SPARK-34158)

Found a minor non-blocking (but annoying) issue in Spark on k8s that's different from 3.0.1 that should really be silenced as the other debug message in ExecutorPodsAllocator:

21/01/19 12:23:26 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.
21/01/19 12:23:27 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.
21/01/19 12:23:28 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.
21/01/19 12:23:29 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 0 pending. 0 unacknowledged.


On Mon, Jan 18, 2021 at 1:06 PM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

回复: [VOTE] Release Spark 3.1.1 (RC1)

ruifengz
In reply to this post by Dongjoon Hyun-2
+1 (non-binding)

Thank you, Hyukjin

Bests,
Ruifeng

------------------ 原始邮件 ------------------
发件人: "Dongjoon Hyun" <[hidden email]>;
发送时间: 2021年1月20日(星期三) 中午1:57
收件人: "Holden Karau"<[hidden email]>;
抄送: "Sean Owen"<[hidden email]>;"Hyukjin Kwon"<[hidden email]>;"dev"<[hidden email]>;
主题: Re: [VOTE] Release Spark 3.1.1 (RC1)

+1

I additionally
- Ran JDBC integration test
- Ran with AWS EKS 1.16
- Ran unit tests with Python 3.9.1 combination (numpy 1.19.5, pandas 1.2.0, scipy 1.6.0)
  (PyArrow is not tested because it's not supported in Python 3.9.x. This is documented via SPARK-34162)

There exists some on-going work in the umbrella JIRA (SPARK-33507: Improve and fix cache behavior in v1 and v2).
I believe it can be achieved at 3.2.0 and we can add some comments on the release note at 3.1.0.

Thank you, Hyukjin and all.

Bests,
Dongjoon.

On Tue, Jan 19, 2021 at 10:49 AM Holden Karau <[hidden email]> wrote:
+1, pip installs on Python 3.8

One potential thing we might want to consider if there ends up being another RC is that the error message for installing with Python2 could be clearer.

Processing ./pyspark-3.1.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /tmp/py3.1/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-W1BsIL
         cwd: /tmp/pip-req-build-lmlitE/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-lmlitE/setup.py", line 31
        file=sys.stderr)
            ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.



On Tue, Jan 19, 2021 at 10:26 AM Sean Owen <[hidden email]> wrote:
+1 from me. Same results as in 3.1.0 testing.

On Mon, Jan 18, 2021 at 6:06 AM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Mridul Muralidharan
+1

Signatures, digests, etc check out fine.
Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes

The sha512 signature for spark-3.1.1.tgz tripped up my scripts :-)


Regards,
Mridul


On Wed, Jan 20, 2021 at 8:17 PM 郑瑞峰 <[hidden email]> wrote:
+1 (non-binding)

Thank you, Hyukjin

Bests,
Ruifeng

------------------ 原始邮件 ------------------
发件人: "Dongjoon Hyun" <[hidden email]>;
发送时间: 2021年1月20日(星期三) 中午1:57
收件人: "Holden Karau"<[hidden email]>;
抄送: "Sean Owen"<[hidden email]>;"Hyukjin Kwon"<[hidden email]>;"dev"<[hidden email]>;
主题: Re: [VOTE] Release Spark 3.1.1 (RC1)

+1

I additionally
- Ran JDBC integration test
- Ran with AWS EKS 1.16
- Ran unit tests with Python 3.9.1 combination (numpy 1.19.5, pandas 1.2.0, scipy 1.6.0)
  (PyArrow is not tested because it's not supported in Python 3.9.x. This is documented via SPARK-34162)

There exists some on-going work in the umbrella JIRA (SPARK-33507: Improve and fix cache behavior in v1 and v2).
I believe it can be achieved at 3.2.0 and we can add some comments on the release note at 3.1.0.

Thank you, Hyukjin and all.

Bests,
Dongjoon.

On Tue, Jan 19, 2021 at 10:49 AM Holden Karau <[hidden email]> wrote:
+1, pip installs on Python 3.8

One potential thing we might want to consider if there ends up being another RC is that the error message for installing with Python2 could be clearer.

Processing ./pyspark-3.1.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /tmp/py3.1/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-W1BsIL
         cwd: /tmp/pip-req-build-lmlitE/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-lmlitE/setup.py", line 31
        file=sys.stderr)
            ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.



On Tue, Jan 19, 2021 at 10:26 AM Sean Owen <[hidden email]> wrote:
+1 from me. Same results as in 3.1.0 testing.

On Mon, Jan 18, 2021 at 6:06 AM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Holden Karau
Hi folks,

Just an FYI -- I've found a potential race condition in this RC with block manager decommissioning and the torrent broadcast factory  ( https://issues.apache.org/jira/browse/SPARK-34193 ).

I don't think this should block the release (it's not a regression), so my +1 stands as is and I haven't triggered it more than once.

Cheers,

Holden

On Wed, Jan 20, 2021 at 9:05 PM Mridul Muralidharan <[hidden email]> wrote:
+1

Signatures, digests, etc check out fine.
Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes

The sha512 signature for spark-3.1.1.tgz tripped up my scripts :-)


Regards,
Mridul


On Wed, Jan 20, 2021 at 8:17 PM 郑瑞峰 <[hidden email]> wrote:
+1 (non-binding)

Thank you, Hyukjin

Bests,
Ruifeng

------------------ 原始邮件 ------------------
发件人: "Dongjoon Hyun" <[hidden email]>;
发送时间: 2021年1月20日(星期三) 中午1:57
收件人: "Holden Karau"<[hidden email]>;
抄送: "Sean Owen"<[hidden email]>;"Hyukjin Kwon"<[hidden email]>;"dev"<[hidden email]>;
主题: Re: [VOTE] Release Spark 3.1.1 (RC1)

+1

I additionally
- Ran JDBC integration test
- Ran with AWS EKS 1.16
- Ran unit tests with Python 3.9.1 combination (numpy 1.19.5, pandas 1.2.0, scipy 1.6.0)
  (PyArrow is not tested because it's not supported in Python 3.9.x. This is documented via SPARK-34162)

There exists some on-going work in the umbrella JIRA (SPARK-33507: Improve and fix cache behavior in v1 and v2).
I believe it can be achieved at 3.2.0 and we can add some comments on the release note at 3.1.0.

Thank you, Hyukjin and all.

Bests,
Dongjoon.

On Tue, Jan 19, 2021 at 10:49 AM Holden Karau <[hidden email]> wrote:
+1, pip installs on Python 3.8

One potential thing we might want to consider if there ends up being another RC is that the error message for installing with Python2 could be clearer.

Processing ./pyspark-3.1.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /tmp/py3.1/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-W1BsIL
         cwd: /tmp/pip-req-build-lmlitE/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-lmlitE/setup.py", line 31
        file=sys.stderr)
            ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.



On Tue, Jan 19, 2021 at 10:26 AM Sean Owen <[hidden email]> wrote:
+1 from me. Same results as in 3.1.0 testing.

On Mon, Jan 18, 2021 at 6:06 AM Hyukjin Kwon <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Tom Graves-2
In reply to this post by Hyukjin Kwon
+1

built from tarball, verified sha and regular CI and tests all pass.

Tom

On Monday, January 18, 2021, 06:06:42 AM CST, Hyukjin Kwon <[hidden email]> wrote:


Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

cloud0fan
-1 as I just found a regression in 3.1. A self-join query works well in 3.0 but fails in 3.1. It's being fixed at https://github.com/apache/spark/pull/31287

On Fri, Jan 22, 2021 at 4:34 AM Tom Graves <[hidden email]> wrote:
+1

built from tarball, verified sha and regular CI and tests all pass.

Tom

On Monday, January 18, 2021, 06:06:42 AM CST, Hyukjin Kwon <[hidden email]> wrote:


Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 3.1.1 (RC1)

Maxim Gekk
Also I am investigating a performance regression in some TPC-DS queries (q88 for instance) that is caused by a recent commit in 3.1, highly likely in the period from 19th November, 2020 to 18th December, 2020.

Maxim Gekk

Software Engineer

Databricks, Inc.



On Fri, Jan 22, 2021 at 10:45 AM Wenchen Fan <[hidden email]> wrote:
-1 as I just found a regression in 3.1. A self-join query works well in 3.0 but fails in 3.1. It's being fixed at https://github.com/apache/spark/pull/31287

On Fri, Jan 22, 2021 at 4:34 AM Tom Graves <[hidden email]> wrote:
+1

built from tarball, verified sha and regular CI and tests all pass.

Tom

On Monday, January 18, 2021, 06:06:42 AM CST, Hyukjin Kwon <[hidden email]> wrote:


Please vote on releasing the following candidate as Apache Spark version 3.1.1.

The vote is open until January 22nd 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v3.1.1-rc1 (commit 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 3.1.1 can be found at the following URL:

This release is using the release script of the tag v3.1.1-rc1.

FAQ

===================
What happened to 3.1.0?
===================

There was a technical issue during Apache Spark 3.1.0 preparation, and it was discussed and decided to skip 3.1.0.
Please see https://spark.apache.org/news/next-official-release-spark-3.1.1.html for more details.

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC via "pip install https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz"
and see if anything important breaks.
In the Java/Scala, you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 3.1.1?
===========================================

The current list of open tickets targeted at 3.1.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 3.1.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

12