[VOTE] Release Spark 2.4.7 (RC1)

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] Release Spark 2.4.7 (RC1)

Prashant Sharma
Please vote on releasing the following candidate as Apache Spark version 2.4.7.

The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.7
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))

The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1352/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/

The list of bug fixes going into 2.4.7 can be found at the following URL:
https://s.apache.org/spark-v2.4.7-rc1

This release is using the release script of the tag v2.4.7-rc1.

FAQ


=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with an out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.7?
===========================================

The current list of open tickets targeted at 2.4.7 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Sean Owen-3
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:

>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Dongjoon Hyun-2
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal correctness.

Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Holden Karau
I'm going to go ahead and vote -0 then based on that then.

On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal correctness.

Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Dongjoon Hyun-2
Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
    https://github.com/apache/spark/pull/29383

It seems that the patch is already ready in this case.
I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

Bests,
Dongjoon.


On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <[hidden email]> wrote:
I'm going to go ahead and vote -0 then based on that then.

On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal correctness.

Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Takeshi Yamamuro
Thanks for letting us know about the two issues above, Dongjoon.

----
I've checked the release materials (signatures, tag, ...) and it looks fine, too.
Also, I run the tests on my local Mac (java 1.8.0) with the options
`-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
and they passed.

Bests,
Takeshi



On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <[hidden email]> wrote:
Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
    https://github.com/apache/spark/pull/29383

It seems that the patch is already ready in this case.
I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

Bests,
Dongjoon.


On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <[hidden email]> wrote:
I'm going to go ahead and vote -0 then based on that then.

On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal correctness.

Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Prashant Sharma
Thanks for letting us know. So this vote is cancelled in favor of RC2.



On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <[hidden email]> wrote:
Thanks for letting us know about the two issues above, Dongjoon.

----
I've checked the release materials (signatures, tag, ...) and it looks fine, too.
Also, I run the tests on my local Mac (java 1.8.0) with the options
`-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
and they passed.

Bests,
Takeshi



On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <[hidden email]> wrote:
Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
    https://github.com/apache/spark/pull/29383

It seems that the patch is already ready in this case.
I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

Bests,
Dongjoon.


On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <[hidden email]> wrote:
I'm going to go ahead and vote -0 then based on that then.

On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal correctness.

Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Xiao Li-2
https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release. 

Thanks,

Xiao

On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <[hidden email]> wrote:
Thanks for letting us know. So this vote is cancelled in favor of RC2.



On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <[hidden email]> wrote:
Thanks for letting us know about the two issues above, Dongjoon.

----
I've checked the release materials (signatures, tag, ...) and it looks fine, too.
Also, I run the tests on my local Mac (java 1.8.0) with the options
`-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
and they passed.

Bests,
Takeshi



On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <[hidden email]> wrote:
Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

    [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
    https://github.com/apache/spark/pull/29383

It seems that the patch is already ready in this case.
I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

Bests,
Dongjoon.


On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <[hidden email]> wrote:
I'm going to go ahead and vote -0 then based on that then.

On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

Unfortunately, there is an on-going discussion about the new decimal correctness.

Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

    [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

I also confirmed that 2.4.7 RC1 is affected.

Bests,
Dongjoon.


On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
+1 from me. The same as usual. Licenses and sigs look OK, builds and
passes tests on a standard selection of profiles.

On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.7.
>
> The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.7
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
> https://github.com/apache/spark/tree/v2.4.7-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1352/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
>
> The list of bug fixes going into 2.4.7 can be found at the following URL:
> https://s.apache.org/spark-v2.4.7-rc1
>
> This release is using the release script of the tag v2.4.7-rc1.
>
> FAQ
>
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.7?
> ===========================================
>
> The current list of open tickets targeted at 2.4.7 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Reply | Threaded
Open this post in threaded view
|

RE: [VOTE] Release Spark 2.4.7 (RC1)

Nicholas Marion

It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


Regards,

NICHOLAS T. MARION
IBM Open Data Analytics for z/OS -
CPO and Service Team Lead

Phone: 1-845-433-5010 | Tie-Line: 293-5010
E-mail:
[hidden email]
Find me on:


2455 South Rd
Poughkeepie, New York 12601-5400
United States



Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

From: Xiao Li <[hidden email]>
To: Prashant Sharma <[hidden email]>
Cc: Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
Date: 08/17/2020 11:33 AM
Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)




https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release. 

Thanks,

Xiao

On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <[hidden email]> wrote:
    Thanks for letting us know. So this vote is cancelled in favor of RC2.



    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <[hidden email]> wrote:
    Thanks for letting us know about the two issues above, Dongjoon.

    ----
    I've checked the release materials (signatures, tag, ...) and it looks fine, too.
    Also, I run the tests on my local Mac (java 1.8.0) with the options
    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
    and they passed.

    Bests,
    Takeshi



    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <[hidden email]> wrote:
      Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

      It seems that the patch is already ready in this case.
      I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

      Bests,
      Dongjoon.


      On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <[hidden email]> wrote:
      I'm going to go ahead and vote -0 then based on that then.

      On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
        Hi, All.

        Unfortunately, there is an on-going discussion about the new decimal correctness.

        Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

        Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

            [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

        I also confirmed that 2.4.7 RC1 is affected.

        Bests,
        Dongjoon.


        On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
        +1 from me. The same as usual. Licenses and sigs look OK, builds and
        passes tests on a standard selection of profiles.

        On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:

        >
        > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
        >
        > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
        >
        > [ ] +1 Release this package as Apache Spark 2.4.7
        > [ ] -1 Do not release this package because ...
        >
        > To learn more about Apache Spark, please see http://spark.apache.org/
        >
        > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
        >
        > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
        > https://github.com/apache/spark/tree/v2.4.7-rc1
        >
        > The release files, including signatures, digests, etc. can be found at:
        > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
        >
        > Signatures used for Spark RCs can be found in this file:
        > https://dist.apache.org/repos/dist/dev/spark/KEYS
        >
        > The staging repository for this release can be found at:
        > https://repository.apache.org/content/repositories/orgapachespark-1352/
        >
        > The documentation corresponding to this release can be found at:
        > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
        >
        > The list of bug fixes going into 2.4.7 can be found at the following URL:
        > https://s.apache.org/spark-v2.4.7-rc1
        >
        > This release is using the release script of the tag v2.4.7-rc1.
        >
        > FAQ
        >
        >
        > =========================
        > How can I help test this release?
        > =========================
        >
        > If you are a Spark user, you can help us test this release by taking
        > an existing Spark workload and running on this release candidate, then
        > reporting any regressions.
        >
        > If you're working in PySpark you can set up a virtual env and install
        > the current RC and see if anything important breaks, in the Java/Scala
        > you can add the staging repository to your projects resolvers and test
        > with the RC (make sure to clean up the artifact cache before/after so
        > you don't end up building with an out of date RC going forward).
        >
        > ===========================================
        > What should happen to JIRA tickets still targeting 2.4.7?
        > ===========================================
        >
        > The current list of open tickets targeted at 2.4.7 can be found at:
        > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
        >
        > Committers should look at those and triage. Extremely important bug
        > fixes, documentation, and API tweaks that impact compatibility should
        > be worked on immediately. Everything else please retarget to an
        > appropriate release.
        >
        > ==================
        > But my bug isn't fixed?
        > ==================
        >
        > In order to make timely releases, we will typically not hold the
        > release unless the bug in question is a regression from the previous
        > release. That being said, if there is something which is a regression
        > that has not been correctly targeted please ping me or a committer to
        > help target the issue.

        ---------------------------------------------------------------------
        To unsubscribe e-mail: [hidden email]


      --
      Twitter: https://twitter.com/holdenkarau
      Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
      YouTube Live Streams: https://www.youtube.com/user/holdenkarau


    --
    ---
    Takeshi Yamamuro


--



Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

cloud0fan
I think so. I don't see other bug reports for 2.4.

On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <[hidden email]> wrote:

It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


Regards,

NICHOLAS T. MARION
IBM Open Data Analytics for z/OS -
CPO and Service Team Lead

Phone: 1-845-433-5010 | Tie-Line: 293-5010
E-mail:
[hidden email]
Find me on:


2455 South Rd
Poughkeepie, New York 12601-5400
United States



Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

From: Xiao Li <[hidden email]>
To: Prashant Sharma <[hidden email]>
Cc: Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
Date: 08/17/2020 11:33 AM
Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)





https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release. 

Thanks,

Xiao

On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <[hidden email]> wrote:
    Thanks for letting us know. So this vote is cancelled in favor of RC2.



    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <[hidden email]> wrote:
    Thanks for letting us know about the two issues above, Dongjoon.

    ----
    I've checked the release materials (signatures, tag, ...) and it looks fine, too.
    Also, I run the tests on my local Mac (java 1.8.0) with the options
    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
    and they passed.

    Bests,
    Takeshi



    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <[hidden email]> wrote:
      Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

      It seems that the patch is already ready in this case.
      I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

      Bests,
      Dongjoon.


      On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <[hidden email]> wrote:
      I'm going to go ahead and vote -0 then based on that then.

      On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
        Hi, All.

        Unfortunately, there is an on-going discussion about the new decimal correctness.

        Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

        Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

            [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

        I also confirmed that 2.4.7 RC1 is affected.

        Bests,
        Dongjoon.


        On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
        +1 from me. The same as usual. Licenses and sigs look OK, builds and
        passes tests on a standard selection of profiles.

        On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:

        >
        > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
        >
        > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
        >
        > [ ] +1 Release this package as Apache Spark 2.4.7
        > [ ] -1 Do not release this package because ...
        >
        > To learn more about Apache Spark, please see http://spark.apache.org/
        >
        > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
        >
        > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
        > https://github.com/apache/spark/tree/v2.4.7-rc1
        >
        > The release files, including signatures, digests, etc. can be found at:
        > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
        >
        > Signatures used for Spark RCs can be found in this file:
        > https://dist.apache.org/repos/dist/dev/spark/KEYS
        >
        > The staging repository for this release can be found at:
        > https://repository.apache.org/content/repositories/orgapachespark-1352/
        >
        > The documentation corresponding to this release can be found at:
        > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
        >
        > The list of bug fixes going into 2.4.7 can be found at the following URL:
        > https://s.apache.org/spark-v2.4.7-rc1
        >
        > This release is using the release script of the tag v2.4.7-rc1.
        >
        > FAQ
        >
        >
        > =========================
        > How can I help test this release?
        > =========================
        >
        > If you are a Spark user, you can help us test this release by taking
        > an existing Spark workload and running on this release candidate, then
        > reporting any regressions.
        >
        > If you're working in PySpark you can set up a virtual env and install
        > the current RC and see if anything important breaks, in the Java/Scala
        > you can add the staging repository to your projects resolvers and test
        > with the RC (make sure to clean up the artifact cache before/after so
        > you don't end up building with an out of date RC going forward).
        >
        > ===========================================
        > What should happen to JIRA tickets still targeting 2.4.7?
        > ===========================================
        >
        > The current list of open tickets targeted at 2.4.7 can be found at:
        > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
        >
        > Committers should look at those and triage. Extremely important bug
        > fixes, documentation, and API tweaks that impact compatibility should
        > be worked on immediately. Everything else please retarget to an
        > appropriate release.
        >
        > ==================
        > But my bug isn't fixed?
        > ==================
        >
        > In order to make timely releases, we will typically not hold the
        > release unless the bug in question is a regression from the previous
        > release. That being said, if there is something which is a regression
        > that has not been correctly targeted please ping me or a committer to
        > help target the issue.

        ---------------------------------------------------------------------
        To unsubscribe e-mail: [hidden email]


      --
      Twitter: https://twitter.com/holdenkarau
      Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
      YouTube Live Streams: https://www.youtube.com/user/holdenkarau


    --
    ---
    Takeshi Yamamuro


--



Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Spark 2.4.7 (RC1)

Tom Graves-2
There is a correctness issue with caching that should go into this if possible: https://github.com/apache/spark/pull/29506

Tom

On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <[hidden email]> wrote:


I think so. I don't see other bug reports for 2.4.

On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <[hidden email]> wrote:

It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


Regards,

NICHOLAS T. MARION
IBM Open Data Analytics for z/OS -
CPO and Service Team Lead

Phone: 1-845-433-5010 | Tie-Line: 293-5010
E-mail:
[hidden email]
Find me on:


2455 South Rd
Poughkeepie, New York 12601-5400
United States



Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

From: Xiao Li <[hidden email]>
To: Prashant Sharma <[hidden email]>
Cc: Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
Date: 08/17/2020 11:33 AM
Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)





https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release. 

Thanks,

Xiao

On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <[hidden email]> wrote:
    Thanks for letting us know. So this vote is cancelled in favor of RC2.



    On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <[hidden email]> wrote:
    Thanks for letting us know about the two issues above, Dongjoon.

    ----
    I've checked the release materials (signatures, tag, ...) and it looks fine, too.
    Also, I run the tests on my local Mac (java 1.8.0) with the options
    `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
    and they passed.

    Bests,
    Takeshi



    On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <[hidden email]> wrote:
      Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

      It seems that the patch is already ready in this case.
      I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

      Bests,
      Dongjoon.


      On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <[hidden email]> wrote:
      I'm going to go ahead and vote -0 then based on that then.

      On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <[hidden email]> wrote:
        Hi, All.

        Unfortunately, there is an on-going discussion about the new decimal correctness.

        Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

        Please see https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

            [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

        I also confirmed that 2.4.7 RC1 is affected.

        Bests,
        Dongjoon.


        On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <[hidden email]> wrote:
        +1 from me. The same as usual. Licenses and sigs look OK, builds and
        passes tests on a standard selection of profiles.

        On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <[hidden email]> wrote:

        >
        > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
        >
        > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
        >
        > [ ] +1 Release this package as Apache Spark 2.4.7
        > [ ] -1 Do not release this package because ...
        >
        > To learn more about Apache Spark, please see http://spark.apache.org/
        >
        > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
        >
        > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
        > https://github.com/apache/spark/tree/v2.4.7-rc1
        >
        > The release files, including signatures, digests, etc. can be found at:
        > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
        >
        > Signatures used for Spark RCs can be found in this file:
        > https://dist.apache.org/repos/dist/dev/spark/KEYS
        >
        > The staging repository for this release can be found at:
        > https://repository.apache.org/content/repositories/orgapachespark-1352/
        >
        > The documentation corresponding to this release can be found at:
        > https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
        >
        > The list of bug fixes going into 2.4.7 can be found at the following URL:
        > https://s.apache.org/spark-v2.4.7-rc1
        >
        > This release is using the release script of the tag v2.4.7-rc1.
        >
        > FAQ
        >
        >
        > =========================
        > How can I help test this release?
        > =========================
        >
        > If you are a Spark user, you can help us test this release by taking
        > an existing Spark workload and running on this release candidate, then
        > reporting any regressions.
        >
        > If you're working in PySpark you can set up a virtual env and install
        > the current RC and see if anything important breaks, in the Java/Scala
        > you can add the staging repository to your projects resolvers and test
        > with the RC (make sure to clean up the artifact cache before/after so
        > you don't end up building with an out of date RC going forward).
        >
        > ===========================================
        > What should happen to JIRA tickets still targeting 2.4.7?
        > ===========================================
        >
        > The current list of open tickets targeted at 2.4.7 can be found at:
        > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
        >
        > Committers should look at those and triage. Extremely important bug
        > fixes, documentation, and API tweaks that impact compatibility should
        > be worked on immediately. Everything else please retarget to an
        > appropriate release.
        >
        > ==================
        > But my bug isn't fixed?
        > ==================
        >
        > In order to make timely releases, we will typically not hold the
        > release unless the bug in question is a regression from the previous
        > release. That being said, if there is something which is a regression
        > that has not been correctly targeted please ping me or a committer to
        > help target the issue.

        ---------------------------------------------------------------------
        To unsubscribe e-mail: [hidden email]


        --
        Twitter: https://twitter.com/holdenkarau
        Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
        YouTube Live Streams: https://www.youtube.com/user/holdenkarau

        --
        ---
        Takeshi Yamamuro

        --





        ---------------------------------------------------------------------
        To unsubscribe e-mail: [hidden email]
        Reply | Threaded
        Open this post in threaded view
        |

        RE: [VOTE] Release Spark 2.4.7 (RC1)

        Nicholas Marion

        Hello,

        I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there is an active vote going through for it?



        Regards,

        NICHOLAS T. MARION
        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:
        LinkedIn: http://www.linkedin.com/in/nicholasmarion
        IBM

        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States
        IBM Redbooks Silver AuthorData Science Foundations - Level 1



        Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go intoTom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go into this if possible: https://urldefense.

        From: Tom Graves <[hidden email]>
        To: Nicholas Marion <[hidden email]>, Wenchen Fan <[hidden email]>
        Cc: Xiao Li <[hidden email]>, dev <[hidden email]>, Takeshi Yamamuro <[hidden email]>, Prashant Sharma <[hidden email]>
        Date: 08/21/2020 03:09 PM
        Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)




        There is a correctness issue with caching that should go into this if possible: https://github.com/apache/spark/pull/29506

        Tom

        On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <[hidden email]> wrote:


        I think so. I don't see other bug reports for 2.4.

        On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <[hidden email]> wrote:
          It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


        Regards,

        NICHOLAS T. MARION

        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:


        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States



          Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

          From:
          Xiao Li <[hidden email]>
          To:
          Prashant Sharma <[hidden email]>
          Cc:
          Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
          Date:
          08/17/2020 11:33 AM
          Subject:
          [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)




          https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release.

          Thanks,

          Xiao

          On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
          [hidden email]> wrote:
          Thanks for letting us know. So this vote is cancelled in favor of RC2.



          On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
          [hidden email]> wrote:
          Thanks for letting us know about the two issues above, Dongjoon.

          ----
          I've checked the release materials (signatures, tag, ...) and it looks fine, too.
          Also, I run the tests on my local Mac (java 1.8.0) with the options
          `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
          and they passed.

          Bests,
          Takeshi



          On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
          [hidden email]> wrote:
          Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

          It seems that the patch is already ready in this case.
          I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

          Bests,
          Dongjoon.


          On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <
          [hidden email]> wrote:
          I'm going to go ahead and vote -0 then based on that then.

          On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
          [hidden email]> wrote:
          Hi, All.

          Unfortunately, there is an on-going discussion about the new decimal correctness.

          Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

          Please see
          https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

          [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

          I also confirmed that 2.4.7 RC1 is affected.

          Bests,
          Dongjoon.


          On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <
          [hidden email]> wrote:
          +1 from me. The same as usual. Licenses and sigs look OK, builds and
          passes tests on a standard selection of profiles.

          On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
          [hidden email]> wrote:

          >
          > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
          >
          > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
          >
          > [ ] +1 Release this package as Apache Spark 2.4.7
          > [ ] -1 Do not release this package because ...
          >
          > To learn more about Apache Spark, please see
          http://spark.apache.org/
          >
          > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
          >
          > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
          >
          https://github.com/apache/spark/tree/v2.4.7-rc1
          >
          > The release files, including signatures, digests, etc. can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
          >
          > Signatures used for Spark RCs can be found in this file:
          >
          https://dist.apache.org/repos/dist/dev/spark/KEYS
          >
          > The staging repository for this release can be found at:
          >
          https://repository.apache.org/content/repositories/orgapachespark-1352/
          >
          > The documentation corresponding to this release can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
          >
          > The list of bug fixes going into 2.4.7 can be found at the following URL:
          >
          https://s.apache.org/spark-v2.4.7-rc1
          >
          > This release is using the release script of the tag v2.4.7-rc1.
          >
          > FAQ
          >
          >
          > =========================
          > How can I help test this release?
          > =========================
          >
          > If you are a Spark user, you can help us test this release by taking
          > an existing Spark workload and running on this release candidate, then
          > reporting any regressions.
          >
          > If you're working in PySpark you can set up a virtual env and install
          > the current RC and see if anything important breaks, in the Java/Scala
          > you can add the staging repository to your projects resolvers and test
          > with the RC (make sure to clean up the artifact cache before/after so
          > you don't end up building with an out of date RC going forward).
          >
          > ===========================================
          > What should happen to JIRA tickets still targeting 2.4.7?
          > ===========================================
          >
          > The current list of open tickets targeted at 2.4.7 can be found at:
          >
          https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
          >
          > Committers should look at those and triage. Extremely important bug
          > fixes, documentation, and API tweaks that impact compatibility should
          > be worked on immediately. Everything else please retarget to an
          > appropriate release.
          >
          > ==================
          > But my bug isn't fixed?
          > ==================
          >
          > In order to make timely releases, we will typically not hold the
          > release unless the bug in question is a regression from the previous
          > release. That being said, if there is something which is a regression
          > that has not been correctly targeted please ping me or a committer to
          > help target the issue.

          ---------------------------------------------------------------------
          To unsubscribe e-mail:
          [hidden email]


          --
          Twitter:
          https://twitter.com/holdenkarau
          Books (Learning Spark, High Performance Spark, etc.):
          https://amzn.to/2MaRAG9
          YouTube Live Streams:
          https://www.youtube.com/user/holdenkarau

          --
          ---
          Takeshi Yamamuro

          --




          ---------------------------------------------------------------------
          To unsubscribe e-mail: [hidden email]



        Reply | Threaded
        Open this post in threaded view
        |

        Re: [VOTE] Release Spark 2.4.7 (RC1)

        Prashant Sharma
        Hi All,

        FYI, currently there are no blockers to release 2.4.7 at the moment. Soon there will be RC3, I am awaiting some correctness bugs to be merged. 

        Thanks,

        On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <[hidden email]> wrote:

        Hello,

        I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there is an active vote going through for it?



        Regards,

        NICHOLAS T. MARION
        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:
        LinkedIn: http://www.linkedin.com/in/nicholasmarion
        IBM

        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States
        IBM Redbooks Silver AuthorData Science Foundations - Level 1



        Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go intoTom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go into this if possible: https://urldefense.

        From: Tom Graves <[hidden email]>
        To: Nicholas Marion <[hidden email]>, Wenchen Fan <[hidden email]>
        Cc: Xiao Li <[hidden email]>, dev <[hidden email]>, Takeshi Yamamuro <[hidden email]>, Prashant Sharma <[hidden email]>
        Date: 08/21/2020 03:09 PM
        Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)





        There is a correctness issue with caching that should go into this if possible: https://github.com/apache/spark/pull/29506

        Tom

        On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <[hidden email]> wrote:


        I think so. I don't see other bug reports for 2.4.

        On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <[hidden email]> wrote:
          It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


        Regards,

        NICHOLAS T. MARION

        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:


        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States



          Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

          From:
          Xiao Li <[hidden email]>
          To:
          Prashant Sharma <[hidden email]>
          Cc:
          Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
          Date:
          08/17/2020 11:33 AM
          Subject:
          [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)




          https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release.

          Thanks,

          Xiao

          On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
          [hidden email]> wrote:
          Thanks for letting us know. So this vote is cancelled in favor of RC2.



          On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
          [hidden email]> wrote:
          Thanks for letting us know about the two issues above, Dongjoon.

          ----
          I've checked the release materials (signatures, tag, ...) and it looks fine, too.
          Also, I run the tests on my local Mac (java 1.8.0) with the options
          `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
          and they passed.

          Bests,
          Takeshi



          On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
          [hidden email]> wrote:
          Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

          It seems that the patch is already ready in this case.
          I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

          Bests,
          Dongjoon.


          On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <
          [hidden email]> wrote:
          I'm going to go ahead and vote -0 then based on that then.

          On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
          [hidden email]> wrote:
          Hi, All.

          Unfortunately, there is an on-going discussion about the new decimal correctness.

          Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

          Please see
          https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

          [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

          I also confirmed that 2.4.7 RC1 is affected.

          Bests,
          Dongjoon.


          On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <
          [hidden email]> wrote:
          +1 from me. The same as usual. Licenses and sigs look OK, builds and
          passes tests on a standard selection of profiles.

          On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
          [hidden email]> wrote:

          >
          > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
          >
          > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
          >
          > [ ] +1 Release this package as Apache Spark 2.4.7
          > [ ] -1 Do not release this package because ...
          >
          > To learn more about Apache Spark, please see
          http://spark.apache.org/
          >
          > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
          >
          > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
          >
          https://github.com/apache/spark/tree/v2.4.7-rc1
          >
          > The release files, including signatures, digests, etc. can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
          >
          > Signatures used for Spark RCs can be found in this file:
          >
          https://dist.apache.org/repos/dist/dev/spark/KEYS
          >
          > The staging repository for this release can be found at:
          >
          https://repository.apache.org/content/repositories/orgapachespark-1352/
          >
          > The documentation corresponding to this release can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
          >
          > The list of bug fixes going into 2.4.7 can be found at the following URL:
          >
          https://s.apache.org/spark-v2.4.7-rc1
          >
          > This release is using the release script of the tag v2.4.7-rc1.
          >
          > FAQ
          >
          >
          > =========================
          > How can I help test this release?
          > =========================
          >
          > If you are a Spark user, you can help us test this release by taking
          > an existing Spark workload and running on this release candidate, then
          > reporting any regressions.
          >
          > If you're working in PySpark you can set up a virtual env and install
          > the current RC and see if anything important breaks, in the Java/Scala
          > you can add the staging repository to your projects resolvers and test
          > with the RC (make sure to clean up the artifact cache before/after so
          > you don't end up building with an out of date RC going forward).
          >
          > ===========================================
          > What should happen to JIRA tickets still targeting 2.4.7?
          > ===========================================
          >
          > The current list of open tickets targeted at 2.4.7 can be found at:
          >
          https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
          >
          > Committers should look at those and triage. Extremely important bug
          > fixes, documentation, and API tweaks that impact compatibility should
          > be worked on immediately. Everything else please retarget to an
          > appropriate release.
          >
          > ==================
          > But my bug isn't fixed?
          > ==================
          >
          > In order to make timely releases, we will typically not hold the
          > release unless the bug in question is a regression from the previous
          > release. That being said, if there is something which is a regression
          > that has not been correctly targeted please ping me or a committer to
          > help target the issue.

          ---------------------------------------------------------------------
          To unsubscribe e-mail:
          [hidden email]


          --
          Twitter:
          https://twitter.com/holdenkarau
          Books (Learning Spark, High Performance Spark, etc.):
          https://amzn.to/2MaRAG9
          YouTube Live Streams:
          https://www.youtube.com/user/holdenkarau

          --
          ---
          Takeshi Yamamuro

          --




          ---------------------------------------------------------------------
          To unsubscribe e-mail: [hidden email]



        Reply | Threaded
        Open this post in threaded view
        |

        Re: [VOTE] Release Spark 2.4.7 (RC1)

        Micah Kornfield
        Just curious what is the policy for releases off of the 2.4 branch?  

        It would be nice if a resolution to SPARK-32708 (Reusing exchanges for DataSourceV2) [1] could make it into the release.  But if it can't what would determine the need for a new release?

        Thanks,

        On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <[hidden email]> wrote:
        Hi All,

        FYI, currently there are no blockers to release 2.4.7 at the moment. Soon there will be RC3, I am awaiting some correctness bugs to be merged. 

        Thanks,

        On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <[hidden email]> wrote:

        Hello,

        I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there is an active vote going through for it?



        Regards,

        NICHOLAS T. MARION
        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:
        LinkedIn: http://www.linkedin.com/in/nicholasmarion
        IBM

        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States
        IBM Redbooks Silver AuthorData Science Foundations - Level 1



        Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go intoTom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go into this if possible: https://urldefense.

        From: Tom Graves <[hidden email]>
        To: Nicholas Marion <[hidden email]>, Wenchen Fan <[hidden email]>
        Cc: Xiao Li <[hidden email]>, dev <[hidden email]>, Takeshi Yamamuro <[hidden email]>, Prashant Sharma <[hidden email]>
        Date: 08/21/2020 03:09 PM
        Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)





        There is a correctness issue with caching that should go into this if possible: https://github.com/apache/spark/pull/29506

        Tom

        On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <[hidden email]> wrote:


        I think so. I don't see other bug reports for 2.4.

        On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <[hidden email]> wrote:
          It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


        Regards,

        NICHOLAS T. MARION

        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:


        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States



          Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

          From:
          Xiao Li <[hidden email]>
          To:
          Prashant Sharma <[hidden email]>
          Cc:
          Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
          Date:
          08/17/2020 11:33 AM
          Subject:
          [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)




          https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release.

          Thanks,

          Xiao

          On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
          [hidden email]> wrote:
          Thanks for letting us know. So this vote is cancelled in favor of RC2.



          On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
          [hidden email]> wrote:
          Thanks for letting us know about the two issues above, Dongjoon.

          ----
          I've checked the release materials (signatures, tag, ...) and it looks fine, too.
          Also, I run the tests on my local Mac (java 1.8.0) with the options
          `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
          and they passed.

          Bests,
          Takeshi



          On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
          [hidden email]> wrote:
          Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

          It seems that the patch is already ready in this case.
          I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

          Bests,
          Dongjoon.


          On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <
          [hidden email]> wrote:
          I'm going to go ahead and vote -0 then based on that then.

          On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
          [hidden email]> wrote:
          Hi, All.

          Unfortunately, there is an on-going discussion about the new decimal correctness.

          Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

          Please see
          https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

          [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

          I also confirmed that 2.4.7 RC1 is affected.

          Bests,
          Dongjoon.


          On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <
          [hidden email]> wrote:
          +1 from me. The same as usual. Licenses and sigs look OK, builds and
          passes tests on a standard selection of profiles.

          On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
          [hidden email]> wrote:

          >
          > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
          >
          > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
          >
          > [ ] +1 Release this package as Apache Spark 2.4.7
          > [ ] -1 Do not release this package because ...
          >
          > To learn more about Apache Spark, please see
          http://spark.apache.org/
          >
          > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
          >
          > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
          >
          https://github.com/apache/spark/tree/v2.4.7-rc1
          >
          > The release files, including signatures, digests, etc. can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
          >
          > Signatures used for Spark RCs can be found in this file:
          >
          https://dist.apache.org/repos/dist/dev/spark/KEYS
          >
          > The staging repository for this release can be found at:
          >
          https://repository.apache.org/content/repositories/orgapachespark-1352/
          >
          > The documentation corresponding to this release can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
          >
          > The list of bug fixes going into 2.4.7 can be found at the following URL:
          >
          https://s.apache.org/spark-v2.4.7-rc1
          >
          > This release is using the release script of the tag v2.4.7-rc1.
          >
          > FAQ
          >
          >
          > =========================
          > How can I help test this release?
          > =========================
          >
          > If you are a Spark user, you can help us test this release by taking
          > an existing Spark workload and running on this release candidate, then
          > reporting any regressions.
          >
          > If you're working in PySpark you can set up a virtual env and install
          > the current RC and see if anything important breaks, in the Java/Scala
          > you can add the staging repository to your projects resolvers and test
          > with the RC (make sure to clean up the artifact cache before/after so
          > you don't end up building with an out of date RC going forward).
          >
          > ===========================================
          > What should happen to JIRA tickets still targeting 2.4.7?
          > ===========================================
          >
          > The current list of open tickets targeted at 2.4.7 can be found at:
          >
          https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
          >
          > Committers should look at those and triage. Extremely important bug
          > fixes, documentation, and API tweaks that impact compatibility should
          > be worked on immediately. Everything else please retarget to an
          > appropriate release.
          >
          > ==================
          > But my bug isn't fixed?
          > ==================
          >
          > In order to make timely releases, we will typically not hold the
          > release unless the bug in question is a regression from the previous
          > release. That being said, if there is something which is a regression
          > that has not been correctly targeted please ping me or a committer to
          > help target the issue.

          ---------------------------------------------------------------------
          To unsubscribe e-mail:
          [hidden email]


          --
          Twitter:
          https://twitter.com/holdenkarau
          Books (Learning Spark, High Performance Spark, etc.):
          https://amzn.to/2MaRAG9
          YouTube Live Streams:
          https://www.youtube.com/user/holdenkarau

          --
          ---
          Takeshi Yamamuro

          --




          ---------------------------------------------------------------------
          To unsubscribe e-mail: [hidden email]



        Reply | Threaded
        Open this post in threaded view
        |

        Re: [VOTE] Release Spark 2.4.7 (RC1)

        Sean Owen-2
        If you mean, will there be another 2.4.8 release? yes eventually. I am sure we'll do maintenance releases into early next year.
        What is the policy for holding a release for more changes? generally for correctness issues and/or regressions vs the previous maintenance release. Those entail an element of judgment call; other really important changes might hold it up.
        That looks like a change that should go into 2.4.x but not necessarily block 2.4.7. It doesn't look ready to commit, even.

        On Wed, Sep 2, 2020 at 1:29 AM Micah Kornfield <[hidden email]> wrote:
        Just curious what is the policy for releases off of the 2.4 branch?  

        It would be nice if a resolution to SPARK-32708 (Reusing exchanges for DataSourceV2) [1] could make it into the release.  But if it can't what would determine the need for a new release?

        Thanks,

        On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <[hidden email]> wrote:
        Hi All,

        FYI, currently there are no blockers to release 2.4.7 at the moment. Soon there will be RC3, I am awaiting some correctness bugs to be merged. 

        Thanks,

        On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <[hidden email]> wrote:

        Hello,

        I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there is an active vote going through for it?



        Regards,

        NICHOLAS T. MARION
        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:
        LinkedIn: http://www.linkedin.com/in/nicholasmarion
        IBM

        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States
        IBM Redbooks Silver AuthorData Science Foundations - Level 1



        Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go intoTom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go into this if possible: https://urldefense.

        From: Tom Graves <[hidden email]>
        To: Nicholas Marion <[hidden email]>, Wenchen Fan <[hidden email]>
        Cc: Xiao Li <[hidden email]>, dev <[hidden email]>, Takeshi Yamamuro <[hidden email]>, Prashant Sharma <[hidden email]>
        Date: 08/21/2020 03:09 PM
        Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)





        There is a correctness issue with caching that should go into this if possible: https://github.com/apache/spark/pull/29506

        Tom

        On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <[hidden email]> wrote:


        I think so. I don't see other bug reports for 2.4.

        On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <[hidden email]> wrote:
          It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


        Regards,

        NICHOLAS T. MARION

        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:


        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States



          Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

          From:
          Xiao Li <[hidden email]>
          To:
          Prashant Sharma <[hidden email]>
          Cc:
          Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
          Date:
          08/17/2020 11:33 AM
          Subject:
          [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)




          https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release.

          Thanks,

          Xiao

          On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
          [hidden email]> wrote:
          Thanks for letting us know. So this vote is cancelled in favor of RC2.



          On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
          [hidden email]> wrote:
          Thanks for letting us know about the two issues above, Dongjoon.

          ----
          I've checked the release materials (signatures, tag, ...) and it looks fine, too.
          Also, I run the tests on my local Mac (java 1.8.0) with the options
          `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
          and they passed.

          Bests,
          Takeshi



          On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
          [hidden email]> wrote:
          Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

          It seems that the patch is already ready in this case.
          I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

          Bests,
          Dongjoon.


          On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <
          [hidden email]> wrote:
          I'm going to go ahead and vote -0 then based on that then.

          On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
          [hidden email]> wrote:
          Hi, All.

          Unfortunately, there is an on-going discussion about the new decimal correctness.

          Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

          Please see
          https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

          [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

          I also confirmed that 2.4.7 RC1 is affected.

          Bests,
          Dongjoon.


          On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <
          [hidden email]> wrote:
          +1 from me. The same as usual. Licenses and sigs look OK, builds and
          passes tests on a standard selection of profiles.

          On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
          [hidden email]> wrote:

          >
          > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
          >
          > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
          >
          > [ ] +1 Release this package as Apache Spark 2.4.7
          > [ ] -1 Do not release this package because ...
          >
          > To learn more about Apache Spark, please see
          http://spark.apache.org/
          >
          > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
          >
          > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
          >
          https://github.com/apache/spark/tree/v2.4.7-rc1
          >
          > The release files, including signatures, digests, etc. can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
          >
          > Signatures used for Spark RCs can be found in this file:
          >
          https://dist.apache.org/repos/dist/dev/spark/KEYS
          >
          > The staging repository for this release can be found at:
          >
          https://repository.apache.org/content/repositories/orgapachespark-1352/
          >
          > The documentation corresponding to this release can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
          >
          > The list of bug fixes going into 2.4.7 can be found at the following URL:
          >
          https://s.apache.org/spark-v2.4.7-rc1
          >
          > This release is using the release script of the tag v2.4.7-rc1.
          >
          > FAQ
          >
          >
          > =========================
          > How can I help test this release?
          > =========================
          >
          > If you are a Spark user, you can help us test this release by taking
          > an existing Spark workload and running on this release candidate, then
          > reporting any regressions.
          >
          > If you're working in PySpark you can set up a virtual env and install
          > the current RC and see if anything important breaks, in the Java/Scala
          > you can add the staging repository to your projects resolvers and test
          > with the RC (make sure to clean up the artifact cache before/after so
          > you don't end up building with an out of date RC going forward).
          >
          > ===========================================
          > What should happen to JIRA tickets still targeting 2.4.7?
          > ===========================================
          >
          > The current list of open tickets targeted at 2.4.7 can be found at:
          >
          https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
          >
          > Committers should look at those and triage. Extremely important bug
          > fixes, documentation, and API tweaks that impact compatibility should
          > be worked on immediately. Everything else please retarget to an
          > appropriate release.
          >
          > ==================
          > But my bug isn't fixed?
          > ==================
          >
          > In order to make timely releases, we will typically not hold the
          > release unless the bug in question is a regression from the previous
          > release. That being said, if there is something which is a regression
          > that has not been correctly targeted please ping me or a committer to
          > help target the issue.

          ---------------------------------------------------------------------
          To unsubscribe e-mail:
          [hidden email]


          --
          Twitter:
          https://twitter.com/holdenkarau
          Books (Learning Spark, High Performance Spark, etc.):
          https://amzn.to/2MaRAG9
          YouTube Live Streams:
          https://www.youtube.com/user/holdenkarau

          --
          ---
          Takeshi Yamamuro

          --




          ---------------------------------------------------------------------
          To unsubscribe e-mail: [hidden email]



        Reply | Threaded
        Open this post in threaded view
        |

        Re: [VOTE] Release Spark 2.4.7 (RC1)

        Micah Kornfield
        Thank you for the clarification.

        On Wed, Sep 2, 2020 at 6:18 AM Sean Owen <[hidden email]> wrote:
        If you mean, will there be another 2.4.8 release? yes eventually. I am sure we'll do maintenance releases into early next year.
        What is the policy for holding a release for more changes? generally for correctness issues and/or regressions vs the previous maintenance release. Those entail an element of judgment call; other really important changes might hold it up.
        That looks like a change that should go into 2.4.x but not necessarily block 2.4.7. It doesn't look ready to commit, even.

        On Wed, Sep 2, 2020 at 1:29 AM Micah Kornfield <[hidden email]> wrote:
        Just curious what is the policy for releases off of the 2.4 branch?  

        It would be nice if a resolution to SPARK-32708 (Reusing exchanges for DataSourceV2) [1] could make it into the release.  But if it can't what would determine the need for a new release?

        Thanks,

        On Tue, Sep 1, 2020 at 11:19 PM Prashant Sharma <[hidden email]> wrote:
        Hi All,

        FYI, currently there are no blockers to release 2.4.7 at the moment. Soon there will be RC3, I am awaiting some correctness bugs to be merged. 

        Thanks,

        On Tue, Sep 1, 2020 at 6:43 PM Nicholas Marion <[hidden email]> wrote:

        Hello,

        I see that a 2.4.7-rc2 tag was created over a week ago, wondering if there is an active vote going through for it?



        Regards,

        NICHOLAS T. MARION
        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:
        LinkedIn: http://www.linkedin.com/in/nicholasmarion
        IBM

        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States
        IBM Redbooks Silver AuthorData Science Foundations - Level 1



        Inactive hide details for Tom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go intoTom Graves ---08/21/2020 03:09:49 PM--- There is a correctness issue with caching that should go into this if possible: https://urldefense.

        From: Tom Graves <[hidden email]>
        To: Nicholas Marion <[hidden email]>, Wenchen Fan <[hidden email]>
        Cc: Xiao Li <[hidden email]>, dev <[hidden email]>, Takeshi Yamamuro <[hidden email]>, Prashant Sharma <[hidden email]>
        Date: 08/21/2020 03:09 PM
        Subject: [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)





        There is a correctness issue with caching that should go into this if possible: https://github.com/apache/spark/pull/29506

        Tom

        On Wednesday, August 19, 2020, 11:18:37 AM CDT, Wenchen Fan <[hidden email]> wrote:


        I think so. I don't see other bug reports for 2.4.

        On Thu, Aug 20, 2020 at 12:11 AM Nicholas Marion <[hidden email]> wrote:
          It appears all 3 issues slated for Spark 2.4.7 have been merged. Should we be looking at getting RC2 ready?


        Regards,

        NICHOLAS T. MARION

        IBM Open Data Analytics for z/OS -
        CPO and Service Team Lead

        Phone: 1-845-433-5010 | Tie-Line: 293-5010
        E-mail:
        [hidden email]
        Find me on:


        2455 South Rd
        Poughkeepie, New York 12601-5400
        United States



          Inactive hide details for Xiao Li ---08/17/2020 11:33:30 AM---INVALID URI REMOVEDXiao Li ---08/17/2020 11:33:30 AM---https://issues.apache.org/jira/browse/SPARK-32609

          From:
          Xiao Li <[hidden email]>
          To:
          Prashant Sharma <[hidden email]>
          Cc:
          Takeshi Yamamuro <[hidden email]>, dev <[hidden email]>
          Date:
          08/17/2020 11:33 AM
          Subject:
          [EXTERNAL] Re: [VOTE] Release Spark 2.4.7 (RC1)




          https://issues.apache.org/jira/browse/SPARK-32609 got merged. This is to fix a correctness bug in DSV2 of Spark 2.4. Please include it in the upcoming Spark 2.4.7 release.

          Thanks,

          Xiao

          On Sun, Aug 9, 2020 at 10:26 PM Prashant Sharma <
          [hidden email]> wrote:
          Thanks for letting us know. So this vote is cancelled in favor of RC2.



          On Sun, Aug 9, 2020 at 8:31 AM Takeshi Yamamuro <
          [hidden email]> wrote:
          Thanks for letting us know about the two issues above, Dongjoon.

          ----
          I've checked the release materials (signatures, tag, ...) and it looks fine, too.
          Also, I run the tests on my local Mac (java 1.8.0) with the options
          `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`
          and they passed.

          Bests,
          Takeshi



          On Sun, Aug 9, 2020 at 11:06 AM Dongjoon Hyun <
          [hidden email]> wrote:
          Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago.

          [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms
          https://github.com/apache/spark/pull/29383

          It seems that the patch is already ready in this case.
          I raised the priority of SPARK-31703 to `Blocker` for both Apache Spark 2.4.7 and 3.0.1.

          Bests,
          Dongjoon.


          On Sat, Aug 8, 2020 at 6:10 AM Holden Karau <
          [hidden email]> wrote:
          I'm going to go ahead and vote -0 then based on that then.

          On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun <
          [hidden email]> wrote:
          Hi, All.

          Unfortunately, there is an on-going discussion about the new decimal correctness.

          Although we fixed one correctness issue at master and backported it partially to 3.0/2.4, it turns out that it needs more patched to be complete.

          Please see
          https://github.com/apache/spark/pull/29125 for on-going discussion for both 3.0/2.4.

          [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with overflowed value

          I also confirmed that 2.4.7 RC1 is affected.

          Bests,
          Dongjoon.


          On Thu, Aug 6, 2020 at 2:48 PM Sean Owen <
          [hidden email]> wrote:
          +1 from me. The same as usual. Licenses and sigs look OK, builds and
          passes tests on a standard selection of profiles.

          On Thu, Aug 6, 2020 at 7:07 AM Prashant Sharma <
          [hidden email]> wrote:

          >
          > Please vote on releasing the following candidate as Apache Spark version 2.4.7.
          >
          > The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
          >
          > [ ] +1 Release this package as Apache Spark 2.4.7
          > [ ] -1 Do not release this package because ...
          >
          > To learn more about Apache Spark, please see
          http://spark.apache.org/
          >
          > There are currently no issues targeting 2.4.7 (try project = SPARK AND "Target Version/s" = "2.4.7" AND status in (Open, Reopened, "In Progress"))
          >
          > The tag to be voted on is v2.4.7-rc1 (commit dc04bf53fe821b7a07f817966c6c173f3b3788c6):
          >
          https://github.com/apache/spark/tree/v2.4.7-rc1
          >
          > The release files, including signatures, digests, etc. can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-bin/
          >
          > Signatures used for Spark RCs can be found in this file:
          >
          https://dist.apache.org/repos/dist/dev/spark/KEYS
          >
          > The staging repository for this release can be found at:
          >
          https://repository.apache.org/content/repositories/orgapachespark-1352/
          >
          > The documentation corresponding to this release can be found at:
          >
          https://dist.apache.org/repos/dist/dev/spark/v2.4.7-rc1-docs/
          >
          > The list of bug fixes going into 2.4.7 can be found at the following URL:
          >
          https://s.apache.org/spark-v2.4.7-rc1
          >
          > This release is using the release script of the tag v2.4.7-rc1.
          >
          > FAQ
          >
          >
          > =========================
          > How can I help test this release?
          > =========================
          >
          > If you are a Spark user, you can help us test this release by taking
          > an existing Spark workload and running on this release candidate, then
          > reporting any regressions.
          >
          > If you're working in PySpark you can set up a virtual env and install
          > the current RC and see if anything important breaks, in the Java/Scala
          > you can add the staging repository to your projects resolvers and test
          > with the RC (make sure to clean up the artifact cache before/after so
          > you don't end up building with an out of date RC going forward).
          >
          > ===========================================
          > What should happen to JIRA tickets still targeting 2.4.7?
          > ===========================================
          >
          > The current list of open tickets targeted at 2.4.7 can be found at:
          >
          https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.7
          >
          > Committers should look at those and triage. Extremely important bug
          > fixes, documentation, and API tweaks that impact compatibility should
          > be worked on immediately. Everything else please retarget to an
          > appropriate release.
          >
          > ==================
          > But my bug isn't fixed?
          > ==================
          >
          > In order to make timely releases, we will typically not hold the
          > release unless the bug in question is a regression from the previous
          > release. That being said, if there is something which is a regression
          > that has not been correctly targeted please ping me or a committer to
          > help target the issue.

          ---------------------------------------------------------------------
          To unsubscribe e-mail:
          [hidden email]


          --
          Twitter:
          https://twitter.com/holdenkarau
          Books (Learning Spark, High Performance Spark, etc.):
          https://amzn.to/2MaRAG9
          YouTube Live Streams:
          https://www.youtube.com/user/holdenkarau

          --
          ---
          Takeshi Yamamuro

          --




          ---------------------------------------------------------------------
          To unsubscribe e-mail: [hidden email]