[VOTE] SPARK 2.4.0 (RC3)

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] SPARK 2.4.0 (RC3)

cloud0fan
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

cloud0fan
I'm adding my own +1, since there are no known blocker issues. The correctness issue has been fixed, the streaming Java API problem has been resolved, and we have upgraded to Scala 2.12.7.

On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

RE: [VOTE] SPARK 2.4.0 (RC3)

Garlapati, Suryanarayana (Nokia - IN/Bangalore)

Might be you need to change the date(Oct 1 has already passed).

 

>> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with

>> a minimum of 3 +1 votes.

 

Regards

Surya

 

From: Wenchen Fan <[hidden email]>
Sent: Wednesday, October 10, 2018 10:20 PM
To: Spark dev list <[hidden email]>
Subject: Re: [VOTE] SPARK 2.4.0 (RC3)

 

I'm adding my own +1, since there are no known blocker issues. The correctness issue has been fixed, the streaming Java API problem has been resolved, and we have upgraded to Scala 2.12.7.

 

On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan <[hidden email]> wrote:

Please vote on releasing the following candidate as Apache Spark version 2.4.0.

 

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with

a minimum of 3 +1 votes.

 

[ ] +1 Release this package as Apache Spark 2.4.0

[ ] -1 Do not release this package because ...

 

To learn more about Apache Spark, please see http://spark.apache.org/

 

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

 

The release files, including signatures, digests, etc. can be found at:

 

Signatures used for Spark RCs can be found in this file:

 

The staging repository for this release can be found at:

 

The documentation corresponding to this release can be found at:

 

The list of bug fixes going into 2.4.0 can be found at the following URL:

 

FAQ

 

=========================

How can I help test this release?

=========================

 

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.

 

If you're working in PySpark you can set up a virtual env and install

the current RC and see if anything important breaks, in the Java/Scala

you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with a out of date RC going forward).

 

===========================================

What should happen to JIRA tickets still targeting 2.4.0?

===========================================

 

The current list of open tickets targeted at 2.4.0 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

 

Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.

 

==================

But my bug isn't fixed?

==================

 

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Imran Rashid-4
In reply to this post by cloud0fan
Sorry I had messed up my testing earlier, so I only just discovered https://issues.apache.org/jira/browse/SPARK-25704

I dont' think this is a release blocker, because its not a regression and there is a workaround, just fyi.

On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Jean Georges Perrin
Hi,

Sorry if it's stupid question, but where can I find the release notes of 2.4.0?

jg

On Oct 10, 2018, at 2:00 PM, Imran Rashid <[hidden email]> wrote:

Sorry I had messed up my testing earlier, so I only just discovered https://issues.apache.org/jira/browse/SPARK-25704

I dont' think this is a release blocker, because its not a regression and there is a workaround, just fyi.

On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Dongjoon Hyun-2
For now, you can see generated release notes. Official one will be posted on the website when the official 2.4.0 is out.


Bests,
Dongjoon.


On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin <[hidden email]> wrote:
Hi,

Sorry if it's stupid question, but where can I find the release notes of 2.4.0?

jg

On Oct 10, 2018, at 2:00 PM, Imran Rashid <[hidden email]> wrote:

Sorry I had messed up my testing earlier, so I only just discovered https://issues.apache.org/jira/browse/SPARK-25704

I dont' think this is a release blocker, because its not a regression and there is a workaround, just fyi.

On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Jean Georges Perrin
Awesome - thanks Dongjoon!

On Oct 10, 2018, at 2:36 PM, Dongjoon Hyun <[hidden email]> wrote:

For now, you can see generated release notes. Official one will be posted on the website when the official 2.4.0 is out.


Bests,
Dongjoon.


On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin <[hidden email]> wrote:
Hi,

Sorry if it's stupid question, but where can I find the release notes of 2.4.0?

jg

On Oct 10, 2018, at 2:00 PM, Imran Rashid <[hidden email]> wrote:

Sorry I had messed up my testing earlier, so I only just discovered https://issues.apache.org/jira/browse/SPARK-25704

I dont' think this is a release blocker, because its not a regression and there is a workaround, just fyi.

On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Nicholas Chammas
FYI I believe we have an open correctness issue here:

https://issues.apache.org/jira/browse/SPARK-25150

However, it needs review by another person to confirm whether it is indeed a correctness issue (and whether it still impacts this latest RC).

Nick

2018년 10월 10일 (수) 오후 3:14, Jean Georges Perrin <[hidden email]>님이 작성:
Awesome - thanks Dongjoon!


On Oct 10, 2018, at 2:36 PM, Dongjoon Hyun <[hidden email]> wrote:

For now, you can see generated release notes. Official one will be posted on the website when the official 2.4.0 is out.


Bests,
Dongjoon.


On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin <[hidden email]> wrote:
Hi,

Sorry if it's stupid question, but where can I find the release notes of 2.4.0?

jg

On Oct 10, 2018, at 2:00 PM, Imran Rashid <[hidden email]> wrote:

Sorry I had messed up my testing earlier, so I only just discovered https://issues.apache.org/jira/browse/SPARK-25704

I dont' think this is a release blocker, because its not a regression and there is a workaround, just fyi.

On Wed, Oct 10, 2018 at 11:47 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Sean Owen-3
In reply to this post by cloud0fan
+1. I tested the source build against Scala 2.12 and common build
profiles. License and sigs look OK.

No blockers; one critical:

SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4

I think this one is "won't fix" though? not trying to restore the behavior?

Other items open for 2.4.0:

SPARK-25347 Document image data source in doc site
SPARK-25584 Document libsvm data source in doc site
SPARK-25179 Document the features that require Pyarrow 0.10
SPARK-25507 Update documents for the new features in 2.4 release
SPARK-25346 Document Spark builtin data sources
SPARK-24464 Unit tests for MLlib's Instrumentation
SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
SPARK-22809 pyspark is sensitive to imports with dots
SPARK-21030 extend hint syntax to support any expression for Python and R

Anyone know enough to close or retarget them? they don't look critical
for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
features that require Pyarrow 0.10" however sounds like it could have
been important for 2.4? if not a blocker.

PS I don't think that SPARK-25150 is an issue; see JIRA. At least
there is some ongoing discussion there.

I am evaluating
https://github.com/apache/spark/pull/22259#discussion_r224252642 right
now.


On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:

>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> https://github.com/apache/spark/tree/v2.4.0-rc3
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1289
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12342385
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Hyukjin Kwon
In reply to this post by Garlapati, Suryanarayana (Nokia - IN/Bangalore)
So, which date is it?

2018년 10월 11일 (목) 오전 1:48, Garlapati, Suryanarayana (Nokia - IN/Bangalore) <[hidden email]>님이 작성:

Might be you need to change the date(Oct 1 has already passed).

 

>> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with

>> a minimum of 3 +1 votes.

 

Regards

Surya

 

From: Wenchen Fan <[hidden email]>
Sent: Wednesday, October 10, 2018 10:20 PM
To: Spark dev list <[hidden email]>
Subject: Re: [VOTE] SPARK 2.4.0 (RC3)

 

I'm adding my own +1, since there are no known blocker issues. The correctness issue has been fixed, the streaming Java API problem has been resolved, and we have upgraded to Scala 2.12.7.

 

On Thu, Oct 11, 2018 at 12:46 AM Wenchen Fan <[hidden email]> wrote:

Please vote on releasing the following candidate as Apache Spark version 2.4.0.

 

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with

a minimum of 3 +1 votes.

 

[ ] +1 Release this package as Apache Spark 2.4.0

[ ] -1 Do not release this package because ...

 

To learn more about Apache Spark, please see http://spark.apache.org/

 

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):

 

The release files, including signatures, digests, etc. can be found at:

 

Signatures used for Spark RCs can be found in this file:

 

The staging repository for this release can be found at:

 

The documentation corresponding to this release can be found at:

 

The list of bug fixes going into 2.4.0 can be found at the following URL:

 

FAQ

 

=========================

How can I help test this release?

=========================

 

If you are a Spark user, you can help us test this release by taking

an existing Spark workload and running on this release candidate, then

reporting any regressions.

 

If you're working in PySpark you can set up a virtual env and install

the current RC and see if anything important breaks, in the Java/Scala

you can add the staging repository to your projects resolvers and test

with the RC (make sure to clean up the artifact cache before/after so

you don't end up building with a out of date RC going forward).

 

===========================================

What should happen to JIRA tickets still targeting 2.4.0?

===========================================

 

The current list of open tickets targeted at 2.4.0 can be found at:

https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

 

Committers should look at those and triage. Extremely important bug

fixes, documentation, and API tweaks that impact compatibility should

be worked on immediately. Everything else please retarget to an

appropriate release.

 

==================

But my bug isn't fixed?

==================

 

In order to make timely releases, we will typically not hold the

release unless the bug in question is a regression from the previous

release. That being said, if there is something which is a regression

that has not been correctly targeted please ping me or a committer to

help target the issue.

Reply | Threaded
Open this post in threaded view
|

Fwd: [VOTE] SPARK 2.4.0 (RC3)

cloud0fan
In reply to this post by Sean Owen-3
Forgot to cc dev-list

---------- Forwarded message ---------
From: Wenchen Fan <[hidden email]>
Date: Thu, Oct 11, 2018 at 10:14 AM
Subject: Re: [VOTE] SPARK 2.4.0 (RC3)
To: <[hidden email]>
Cc: Sean Owen <[hidden email]>


Ah sorry guys, I just copy-paste the voting email from the last RC and forgot to update the date :P

The voting should be open until October 13 PST.

According to the discussion in the previous RC, I'm resolving SPARK-25378 as won't fix. It's OK to wait one or 2 weeks for the tensorflow release.

SPARK-25150 is a long-standing and known issue I believe, DataFrame join API may have confusing behavior for indirect self-join, and is relatively hard to fix, if breaking change is not allowed. I've seen many tickets complaining about it and we should definitely fix it in 3.0, which accepts necessary breaking changes.

SPARK-25588 does look like a potential issue, but there is not much we can do if this problem is not reproducible.



On Thu, Oct 11, 2018 at 7:28 AM Michael Heuer <[hidden email]> wrote:
Hello Sean, Wenchen

I could use triage on


I’ve struggled reporting Parquet+Avro dependency issues against Spark in the past, can’t seem to get any notice.

   michael


On Oct 10, 2018, at 5:00 PM, Sean Owen <[hidden email]> wrote:

+1. I tested the source build against Scala 2.12 and common build
profiles. License and sigs look OK.

No blockers; one critical:

SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4

I think this one is "won't fix" though? not trying to restore the behavior?

Other items open for 2.4.0:

SPARK-25347 Document image data source in doc site
SPARK-25584 Document libsvm data source in doc site
SPARK-25179 Document the features that require Pyarrow 0.10
SPARK-25507 Update documents for the new features in 2.4 release
SPARK-25346 Document Spark builtin data sources
SPARK-24464 Unit tests for MLlib's Instrumentation
SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
SPARK-22809 pyspark is sensitive to imports with dots
SPARK-21030 extend hint syntax to support any expression for Python and R

Anyone know enough to close or retarget them? they don't look critical
for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
features that require Pyarrow 0.10" however sounds like it could have
been important for 2.4? if not a blocker.

PS I don't think that SPARK-25150 is an issue; see JIRA. At least
there is some ongoing discussion there.

I am evaluating
https://github.com/apache/spark/pull/22259#discussion_r224252642 right
now.


On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:

Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
https://github.com/apache/spark/tree/v2.4.0-rc3

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1289

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/

The list of bug fixes going into 2.4.0 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12342385

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Sean Owen-3
In reply to this post by Sean Owen-3
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <[hidden email]> wrote:

>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Xiao Li

Let us fix all the three issues in ScalaUDF, as mentioned by Sean. 

Xiao 


Sean Owen <[hidden email]> 于2018年10月11日周四 上午9:04写道:
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <[hidden email]> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Holden Karau

On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <[hidden email]> wrote:

Let us fix all the three issues in ScalaUDF, as mentioned by Sean. 

Xiao 


Sean Owen <[hidden email]> 于2018年10月11日周四 上午9:04写道:
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <[hidden email]> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Dongjoon Hyun-2
Hi, Holden.

Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.

Bests,
Dongjoon.


On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <[hidden email]> wrote:

On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <[hidden email]> wrote:

Let us fix all the three issues in ScalaUDF, as mentioned by Sean. 

Xiao 


Sean Owen <[hidden email]> 于2018年10月11日周四 上午9:04写道:
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <[hidden email]> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Holden Karau
So if it's a blocker would you think this should be a -1?

On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, Holden.

Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.

Bests,
Dongjoon.


On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <[hidden email]> wrote:

On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <[hidden email]> wrote:

Let us fix all the three issues in ScalaUDF, as mentioned by Sean. 

Xiao 


Sean Owen <[hidden email]> 于2018年10月11日周四 上午9:04写道:
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <[hidden email]> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Dongjoon Hyun-2
Yes. From my side, it's -1 for RC3.

Bests,
Dongjoon.

On Sat, Oct 13, 2018 at 1:24 PM Holden Karau <[hidden email]> wrote:
So if it's a blocker would you think this should be a -1?

On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, Holden.

Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.

Bests,
Dongjoon.


On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <[hidden email]> wrote:

On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <[hidden email]> wrote:

Let us fix all the three issues in ScalaUDF, as mentioned by Sean. 

Xiao 


Sean Owen <[hidden email]> 于2018年10月11日周四 上午9:04写道:
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <[hidden email]> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC3)

Imran Rashid-4
I just discovered https://issues.apache.org/jira/browse/SPARK-25738 with some more testing.  I only marked it as critical, but seems pretty bad -- I'll defer to others opinion

On Sat, Oct 13, 2018 at 4:15 PM Dongjoon Hyun <[hidden email]> wrote:
Yes. From my side, it's -1 for RC3.

Bests,
Dongjoon.

On Sat, Oct 13, 2018 at 1:24 PM Holden Karau <[hidden email]> wrote:
So if it's a blocker would you think this should be a -1?

On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, Holden.

Since that's a performance at 2.4.0, I marked as `Blocker` four days ago.

Bests,
Dongjoon.


On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <[hidden email]> wrote:

On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <[hidden email]> wrote:

Let us fix all the three issues in ScalaUDF, as mentioned by Sean. 

Xiao 


Sean Owen <[hidden email]> 于2018年10月11日周四 上午9:04写道:
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for changing
ScalaUDF again before the release. Have a look, anyone familiar with
catalyst.
On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <[hidden email]> wrote:
>
> +1. I tested the source build against Scala 2.12 and common build
> profiles. License and sigs look OK.
>
> No blockers; one critical:
>
> SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
>
> I think this one is "won't fix" though? not trying to restore the behavior?
>
> Other items open for 2.4.0:
>
> SPARK-25347 Document image data source in doc site
> SPARK-25584 Document libsvm data source in doc site
> SPARK-25179 Document the features that require Pyarrow 0.10
> SPARK-25507 Update documents for the new features in 2.4 release
> SPARK-25346 Document Spark builtin data sources
> SPARK-24464 Unit tests for MLlib's Instrumentation
> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
> SPARK-22809 pyspark is sensitive to imports with dots
> SPARK-21030 extend hint syntax to support any expression for Python and R
>
> Anyone know enough to close or retarget them? they don't look critical
> for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document the
> features that require Pyarrow 0.10" however sounds like it could have
> been important for 2.4? if not a blocker.
>
> PS I don't think that SPARK-25150 is an issue; see JIRA. At least
> there is some ongoing discussion there.
>
> I am evaluating
> https://github.com/apache/spark/pull/22259#discussion_r224252642 right
> now.
>
>
> On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <[hidden email]> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version 2.4.0.
> >
> > The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 2.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v2.4.0-rc3 (commit 8e4a99bd201b9204fec52580f19ae70a229ed94e):
> > https://github.com/apache/spark/tree/v2.4.0-rc3
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1289
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/
> >
> > The list of bug fixes going into 2.4.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12342385
> >
> > FAQ
> >
> > =========================
> > How can I help test this release?
> > =========================
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===========================================
> > What should happen to JIRA tickets still targeting 2.4.0?
> > ===========================================
> >
> > The current list of open tickets targeted at 2.4.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==================
> > But my bug isn't fixed?
> > ==================
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9