[VOTE] SPARK 2.4.0 (RC2)

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[VOTE] SPARK 2.4.0 (RC2)

cloud0fan
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

cloud0fan
I'm adding my own +1, since all the problems mentioned in the RC1 voting email are all resolved. And there is no blocker issue for 2.4.0 AFAIK.

On Fri, Sep 28, 2018 at 10:59 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Saisai Shao
Only "without-hadoop" profile has 2.12 binary, is it expected?

Thanks
Saisai

Wenchen Fan <[hidden email]> 于2018年9月28日周五 上午11:08写道:
I'm adding my own +1, since all the problems mentioned in the RC1 voting email are all resolved. And there is no blocker issue for 2.4.0 AFAIK.

On Fri, Sep 28, 2018 at 10:59 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

cloud0fan
Yes, that's proposed by Sean. This time we should publish a Scala 2.12 build, both in maven and the download page.

On Fri, Sep 28, 2018 at 11:34 AM Saisai Shao <[hidden email]> wrote:
Only "without-hadoop" profile has 2.12 binary, is it expected?

Thanks
Saisai

Wenchen Fan <[hidden email]> 于2018年9月28日周五 上午11:08写道:
I'm adding my own +1, since all the problems mentioned in the RC1 voting email are all resolved. And there is no blocker issue for 2.4.0 AFAIK.

On Fri, Sep 28, 2018 at 10:59 AM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

cloud0fan
In reply to this post by cloud0fan
I don't think this bug is so serious to fail an RC, it's only about metrics IIUC, and it's not a regression in 2.4.

I agree we should backport this fix to 2.3 and 2.4, and we should update our scala 2.12 jenkins build to use scala 2.12.7. cc Shane do you know how to change it?

BTW end users can still use scala 2.12.7 with the Spark package built with scala 2.12.6, right?

On Fri, Sep 28, 2018 at 4:22 PM Darcy Shen <[hidden email]> wrote:
-1

see:


We should make sure that Spark works with Scala 2.12.7 .


This resolved bug of Scala 2.12.6 is severe and related to correctness.

We should warn our aggressive users about the Scala version.
Latest Scala (2.12.7) is preferred and should pass the unit tests at least.


---- On Fri, 28 Sep 2018 10:59:41 +0800 Wenchen Fan <[hidden email]> wrote ----

Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Sean Owen-2
I don't even know how or if this manifests as a bug. The code is
indeed incorrect and the 2.12 compiler flags it. We fixed a number of
these in SPARK-25398. While I want to get this into 2.4 if we have
another RC, I don't see evidence this is a blocker. It is not specific
to Scala 2.12.

Using Scala 2.12.7 is a not an infra change, but change to the build,
but again it's not even specific to 2.12.7. We should use the latest
if we can though.

On Fri, Sep 28, 2018 at 9:38 AM Wenchen Fan <[hidden email]> wrote:

>
> I don't think this bug is so serious to fail an RC, it's only about metrics IIUC, and it's not a regression in 2.4.
>
> I agree we should backport this fix to 2.3 and 2.4, and we should update our scala 2.12 jenkins build to use scala 2.12.7. cc Shane do you know how to change it?
>
> BTW end users can still use scala 2.12.7 with the Spark package built with scala 2.12.6, right?
>
> On Fri, Sep 28, 2018 at 4:22 PM Darcy Shen <[hidden email]> wrote:
>>
>> -1
>>
>> see:
>>
>> https://github.com/apache/spark/pull/22577
>>
>> We should make sure that Spark works with Scala 2.12.7 .
>>
>> https://github.com/scala/bug/issues/11123
>>
>> This resolved bug of Scala 2.12.6 is severe and related to correctness.
>>
>> We should warn our aggressive users about the Scala version.
>> Latest Scala (2.12.7) is preferred and should pass the unit tests at least.
>>
>>
>> ---- On Fri, 28 Sep 2018 10:59:41 +0800 Wenchen Fan <[hidden email]> wrote ----
>>
>> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>>
>> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
>> a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 2.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
>> https://github.com/apache/spark/tree/v2.4.0-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1287
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>>
>> The list of bug fixes going into 2.4.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>
>> FAQ
>>
>> =========================
>> How can I help test this release?
>> =========================
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===========================================
>> What should happen to JIRA tickets still targeting 2.4.0?
>> ===========================================
>>
>> The current list of open tickets targeted at 2.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==================
>> But my bug isn't fixed?
>> ==================
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>>
>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Dongjoon Hyun-2
In reply to this post by cloud0fan
Hi, Wenchen.

The current issue link seems to be out of order for me.

The list of bug fixes going into 2.4.0 can be found at the following URL:

Could you send out with the following issue link for next RCs?


Bests,
Dongjoon.

On Thu, Sep 27, 2018 at 8:00 PM Wenchen Fan <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Darcy Shen
In reply to this post by cloud0fan
I agree it is a non-important Spark bug. I mean the Option and String comparison. The bug is easy to fix and obvious to confirm. If the desc of PR may not be accurate,feel free to edit the title or content. I am on a vocation <a dir="ltr" href="x-apple-data-detectors://0" x-apple-data-detectors="true" x-apple-data-detectors-type="calendar-event" x-apple-data-detectors-result="0">from 9.29   :)

But the scala bug of WrappedArray is severe. We should not provide a pre-built spark packages with Scala 2.12.6 . The bug is not about compiler. But about scala-library.

If the prebuilt packages of Spark use scala-library 2.12.6 , the bug exists whatever scala version our application developer use.

For Spark we should be serious about the minor Scala version. A preferred Scala minor version should be officially stated. 


(hi wenchen sorry for the duplication of email ,just forgot to cc the list)

---- On Fri, 28 Sep 2018 22:38:05 +0800 Wenchen Fan<[hidden email]> wrote ----

I don't think this bug is so serious to fail an RC, it's only about metrics IIUC, and it's not a regression in 2.4.

I agree we should backport this fix to 2.3 and 2.4, and we should update our scala 2.12 jenkins build to use scala 2.12.7. cc Shane do you know how to change it?

BTW end users can still use scala 2.12.7 with the Spark package built with scala 2.12.6, right?

On Fri, Sep 28, 2018 at 4:22 PM Darcy Shen <[hidden email]> wrote:
-1

see:


We should make sure that Spark works with Scala 2.12.7 .


This resolved bug of Scala 2.12.6 is severe and related to correctness.

We should warn our aggressive users about the Scala version.
Latest Scala (2.12.7) is preferred and should pass the unit tests at least.


---- On Fri, 28 Sep 2018 10:59:41 +0800 Wenchen Fan <[hidden email]> wrote ----

Please vote on releasing the following candidate as Apache Spark version 2.4.0.

The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):

The release files, including signatures, digests, etc. can be found at:

Signatures used for Spark RCs can be found in this file:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:

The list of bug fixes going into 2.4.0 can be found at the following URL:

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.0?
===========================================

The current list of open tickets targeted at 2.4.0 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.



Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Sean Owen-2
Go ahead and file a JIRA to update to 2.12.7 with these details. We'll
assess whether it is a blocker.

On Fri, Sep 28, 2018 at 12:09 PM Darcy Shen <[hidden email]> wrote:

>
> I agree it is a non-important Spark bug. I mean the Option and String comparison. The bug is easy to fix and obvious to confirm. If the desc of PR may not be accurate,feel free to edit the title or content. I am on a vocation from 9.29   :)
>
> But the scala bug of WrappedArray is severe. We should not provide a pre-built spark packages with Scala 2.12.6 . The bug is not about compiler. But about scala-library.
>
> If the prebuilt packages of Spark use scala-library 2.12.6 , the bug exists whatever scala version our application developer use.
>
> For Spark we should be serious about the minor Scala version. A preferred Scala minor version should be officially stated.
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Sean Owen-2
In reply to this post by cloud0fan
+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan <[hidden email]> wrote:

>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Stavros Kontopoulos-3
+1

Stavros

On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen <[hidden email]> wrote:
+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Denny Lee
+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <[hidden email]> wrote:
+1

Stavros

On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen <[hidden email]> wrote:
+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Ted Yu
+1

-------- Original message --------
From: Denny Lee <[hidden email]>
Date: 9/30/18 10:30 PM (GMT-08:00)
To: Stavros Kontopoulos <[hidden email]>
Cc: Sean Owen <[hidden email]>, Wenchen Fan <[hidden email]>, dev <[hidden email]>
Subject: Re: [VOTE] SPARK 2.4.0 (RC2)

+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <[hidden email]> wrote:
+1

Stavros

On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen <[hidden email]> wrote:
+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Marco Gaido
-1, I was able to reproduce SPARK-25538 with the provided data.

Il giorno lun 1 ott 2018 alle ore 09:11 Ted Yu <[hidden email]> ha scritto:
+1

-------- Original message --------
From: Denny Lee <[hidden email]>
Date: 9/30/18 10:30 PM (GMT-08:00)
To: Stavros Kontopoulos <[hidden email]>
Cc: Sean Owen <[hidden email]>, Wenchen Fan <[hidden email]>, dev <[hidden email]>
Subject: Re: [VOTE] SPARK 2.4.0 (RC2)

+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <[hidden email]> wrote:
+1

Stavros

On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen <[hidden email]> wrote:
+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Holden Karau-2
Oh that does look like an important correctness issue.
-1

On Mon, Oct 1, 2018, 9:57 AM Marco Gaido <[hidden email]> wrote:
-1, I was able to reproduce SPARK-25538 with the provided data.

Il giorno lun 1 ott 2018 alle ore 09:11 Ted Yu <[hidden email]> ha scritto:
+1

-------- Original message --------
From: Denny Lee <[hidden email]>
Date: 9/30/18 10:30 PM (GMT-08:00)
To: Stavros Kontopoulos <[hidden email]>
Cc: Sean Owen <[hidden email]>, Wenchen Fan <[hidden email]>, dev <[hidden email]>
Subject: Re: [VOTE] SPARK 2.4.0 (RC2)

+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <[hidden email]> wrote:
+1

Stavros

On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen <[hidden email]> wrote:
+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Xiangrui Meng-2


On Mon, Oct 1, 2018 at 9:52 AM Holden Karau <[hidden email]> wrote:
Oh that does look like an important correctness issue.
-1

On Mon, Oct 1, 2018, 9:57 AM Marco Gaido <[hidden email]> wrote:
-1, I was able to reproduce SPARK-25538 with the provided data.

Il giorno lun 1 ott 2018 alle ore 09:11 Ted Yu <[hidden email]> ha scritto:
+1

-------- Original message --------
From: Denny Lee <[hidden email]>
Date: 9/30/18 10:30 PM (GMT-08:00)
To: Stavros Kontopoulos <[hidden email]>
Cc: Sean Owen <[hidden email]>, Wenchen Fan <[hidden email]>, dev <[hidden email]>
Subject: Re: [VOTE] SPARK 2.4.0 (RC2)

+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos <[hidden email]> wrote:
+1

Stavros

On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen <[hidden email]> wrote:
+1, with comments:

There are 5 critical issues for 2.4, and no blockers:
SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4
SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs
SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella
SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide
SPARK-25323 ML 2.4 QA: API: Python API coverage

Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?

IMHO, the use case (spark-tensorflow-connector) is very important. But whether we need to fix it in 2.4 branch depends on the release timeline. See my comment in the JIRA: https://issues.apache.org/jira/browse/SPARK-25378
 

I found two issues resolved for 2.4.1 that got into this RC, so marked
them as resolved in 2.4.0.

I checked the licenses and notice and they look correct now in source
and binary builds.

The 2.12 artifacts are as I'd expect.

I ran all tests for 2.11 and 2.12 and they pass with -Pyarn
-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.




On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan <[hidden email]> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 2.4.0.
>
> The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v2.4.0-rc2 (commit 42f25f309e91c8cde1814e3720099ac1e64783da):
> https://github.com/apache/spark/tree/v2.4.0-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1287
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/
>
> The list of bug fixes going into 2.4.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>
> FAQ
>
> =========================
> How can I help test this release?
> =========================
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===========================================
> What should happen to JIRA tickets still targeting 2.4.0?
> ===========================================
>
> The current list of open tickets targeted at 2.4.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==================
> But my bug isn't fixed?
> ==================
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


--

Xiangrui Meng

Software Engineer

Databricks Inc. http://databricks.com

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Sean Owen-3
Given that this release is probably still 2 weeks from landing, I don't think that waiting on a spark-tensorflow-connector release with TF 1.12 in mid-October is a big deal. Users can use the library with Spark 2.3.x for a week or two before upgrading, if that's the case. I think this kind of bug fix is appropriate for a minor release, while I could see trying to work around to keep the buggy behavior in a maintenance release.
On Mon, Oct 1, 2018 at 12:11 PM Xiangrui Meng <[hidden email]> wrote:

IMHO, the use case (spark-tensorflow-connector) is very important. But whether we need to fix it in 2.4 branch depends on the release timeline. See my comment in the JIRA: https://issues.apache.org/jira/browse/SPARK-25378

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

cloud0fan
This RC fails because of the correctness bug: SPARK-25538

I'll start a new RC once the fix(https://github.com/apache/spark/pull/22602) is merged.

Thanks,
Wenchen

On Tue, Oct 2, 2018 at 1:21 AM Sean Owen <[hidden email]> wrote:
Given that this release is probably still 2 weeks from landing, I don't think that waiting on a spark-tensorflow-connector release with TF 1.12 in mid-October is a big deal. Users can use the library with Spark 2.3.x for a week or two before upgrading, if that's the case. I think this kind of bug fix is appropriate for a minor release, while I could see trying to work around to keep the buggy behavior in a maintenance release.
On Mon, Oct 1, 2018 at 12:11 PM Xiangrui Meng <[hidden email]> wrote:

IMHO, the use case (spark-tensorflow-connector) is very important. But whether we need to fix it in 2.4 branch depends on the release timeline. See my comment in the JIRA: https://issues.apache.org/jira/browse/SPARK-25378

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Michael Heuer
FYI I’ve open two new issues against 2.4.0 rc2


that are regressions against 2.3.1, and may also be present in 2.3.2.  They could use triage or review.

   michael


On Oct 1, 2018, at 9:18 PM, Wenchen Fan <[hidden email]> wrote:

This RC fails because of the correctness bug: SPARK-25538

I'll start a new RC once the fix(https://github.com/apache/spark/pull/22602) is merged.

Thanks,
Wenchen

On Tue, Oct 2, 2018 at 1:21 AM Sean Owen <[hidden email]> wrote:
Given that this release is probably still 2 weeks from landing, I don't think that waiting on a spark-tensorflow-connector release with TF 1.12 in mid-October is a big deal. Users can use the library with Spark 2.3.x for a week or two before upgrading, if that's the case. I think this kind of bug fix is appropriate for a minor release, while I could see trying to work around to keep the buggy behavior in a maintenance release.
On Mon, Oct 1, 2018 at 12:11 PM Xiangrui Meng <[hidden email]> wrote:

IMHO, the use case (spark-tensorflow-connector) is very important. But whether we need to fix it in 2.4 branch depends on the release timeline. See my comment in the JIRA: https://issues.apache.org/jira/browse/SPARK-25378


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPARK 2.4.0 (RC2)

Shixiong(Ryan) Zhu
-1. Found an issue in a new 2.4 Java API: https://issues.apache.org/jira/browse/SPARK-25644 We should fix it in 2.4.0 to avoid future breaking changes.

Best Regards,

Ryan


On Mon, Oct 1, 2018 at 7:22 PM Michael Heuer <[hidden email]> wrote:
FYI I’ve open two new issues against 2.4.0 rc2


that are regressions against 2.3.1, and may also be present in 2.3.2.  They could use triage or review.

   michael


On Oct 1, 2018, at 9:18 PM, Wenchen Fan <[hidden email]> wrote:

This RC fails because of the correctness bug: SPARK-25538

I'll start a new RC once the fix(https://github.com/apache/spark/pull/22602) is merged.

Thanks,
Wenchen

On Tue, Oct 2, 2018 at 1:21 AM Sean Owen <[hidden email]> wrote:
Given that this release is probably still 2 weeks from landing, I don't think that waiting on a spark-tensorflow-connector release with TF 1.12 in mid-October is a big deal. Users can use the library with Spark 2.3.x for a week or two before upgrading, if that's the case. I think this kind of bug fix is appropriate for a minor release, while I could see trying to work around to keep the buggy behavior in a maintenance release.
On Mon, Oct 1, 2018 at 12:11 PM Xiangrui Meng <[hidden email]> wrote:

IMHO, the use case (spark-tensorflow-connector) is very important. But whether we need to fix it in 2.4 branch depends on the release timeline. See my comment in the JIRA: https://issues.apache.org/jira/browse/SPARK-25378


12