[VOTE] Spark 2.1.2 (RC1)

classic Classic list List threaded Threaded
45 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[VOTE] Spark 2.1.2 (RC1)

Holden Karau
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.

--
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Dongjoon Hyun-2
Hi, Holden.

It's not a problem, but the link of `List of JIRA ... with this filter` seems to be wrong.

Bests,
Dongjoon.


On Thu, Sep 14, 2017 at 10:47 AM, Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.

--

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Sean Owen
I think the search filter is OK, but for whatever reason the filter link includes what JIRA you're currently browsing, and that one is not actually included in the filter. It opens on a JIRA that's not included, but the search results look correct.   project = SPARK AND fixVersion = 2.1.2

On Thu, Sep 14, 2017 at 9:15 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, Holden.

It's not a problem, but the link of `List of JIRA ... with this filter` seems to be wrong.

Bests,
Dongjoon.


On Thu, Sep 14, 2017 at 10:47 AM, Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.

--

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Dongjoon Hyun-2
Yea. I think I found the root cause.

The correct one is the following as Sean said.


The current RC vote email has the following.

      List of JIRA tickets resolved in this release can be found with this filter.
      <https://issues.apache.org/jira/browse/SPARK-20134?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.1.2>

You can see the linke from the Apache archive.


SPARK-20134 is 2.1.1, so I was confused.

Thanks,
Dongjoon.


On Thu, Sep 14, 2017 at 12:18 PM, Sean Owen <[hidden email]> wrote:
I think the search filter is OK, but for whatever reason the filter link includes what JIRA you're currently browsing, and that one is not actually included in the filter. It opens on a JIRA that's not included, but the search results look correct.   project = SPARK AND fixVersion = 2.1.2

On Thu, Sep 14, 2017 at 9:15 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, Holden.

It's not a problem, but the link of `List of JIRA ... with this filter` seems to be wrong.

Bests,
Dongjoon.


On Thu, Sep 14, 2017 at 10:47 AM, Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.

--


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Sean Owen
In reply to this post by Holden Karau
+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Felix Cheung
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>


+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Denny Lee
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Ryan Blue
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Holden Karau
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix



--
Reply | Threaded
Open this post in threaded view
|

Signing releases with pwendell or release manager's key?

Sean Owen
Yeah I had meant to ask about that in the past. While I presume Patrick consents to this and all that, it does mean that anyone with access to said Jenkins scripts can create a signed Spark release, regardless of who they are.

I haven't thought through whether that's a theoretical issue we can ignore or something we need to fix up. For example you can't get a release on the ASF mirrors without more authentication.

How hard would it be to make the script take in a key? it sort of looks like the script already takes GPG_KEY, but don't know how to modify the jobs. I suppose it would be ideal, in any event, for the actual release manager to sign.

On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <[hidden email]> wrote:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Xiao Li
In reply to this post by Holden Karau
-1 


Xiao



2017-09-15 12:28 GMT-07:00 Holden Karau <[hidden email]>:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix



--

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Xiao Li
Sorry, this release candidate is 2.1.2. The issue is in 2.2.1. 

2017-09-15 14:21 GMT-07:00 Xiao Li <[hidden email]>:
-1 


Xiao



2017-09-15 12:28 GMT-07:00 Holden Karau <[hidden email]>:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix



--


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Felix Cheung
Yes ;)


From: Xiao Li <[hidden email]>
Sent: Friday, September 15, 2017 2:22:03 PM
To: Holden Karau
Cc: Ryan Blue; Denny Lee; Felix Cheung; Sean Owen; [hidden email]
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
 
Sorry, this release candidate is 2.1.2. The issue is in 2.2.1. 

2017-09-15 14:21 GMT-07:00 Xiao Li <[hidden email]>:
-1 


Xiao



2017-09-15 12:28 GMT-07:00 Holden Karau <[hidden email]>:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix



--


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Holden Karau
Xiao If it doesn't apply/you've changed your mind if you can re-vote that would be rad.

On Fri, Sep 15, 2017 at 2:22 PM, Felix Cheung <[hidden email]> wrote:
Yes ;)


From: Xiao Li <[hidden email]>
Sent: Friday, September 15, 2017 2:22:03 PM
To: Holden Karau
Cc: Ryan Blue; Denny Lee; Felix Cheung; Sean Owen; [hidden email]

Subject: Re: [VOTE] Spark 2.1.2 (RC1)
 
Sorry, this release candidate is 2.1.2. The issue is in 2.2.1. 

2017-09-15 14:21 GMT-07:00 Xiao Li <[hidden email]>:
-1 


Xiao



2017-09-15 12:28 GMT-07:00 Holden Karau <[hidden email]>:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix



--





--
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Ryan Blue
In reply to this post by Holden Karau
I'm not familiar with the release procedure, can you send a link to this Jenkins job? Can anyone run this job, or is it limited to committers?

rb

On Fri, Sep 15, 2017 at 12:28 PM, Holden Karau <[hidden email]> wrote:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix



--



--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

Re: Signing releases with pwendell or release manager's key?

Ryan Blue
In reply to this post by Sean Owen
I think this needs to be fixed. It's true that there are barriers to publication, but the signature is what we use to authenticate Apache releases.

If Patrick's key is available on Jenkins for any Spark committer to use, then the chance of a compromise are much higher than for a normal RM key.

rb

On Fri, Sep 15, 2017 at 12:34 PM, Sean Owen <[hidden email]> wrote:
Yeah I had meant to ask about that in the past. While I presume Patrick consents to this and all that, it does mean that anyone with access to said Jenkins scripts can create a signed Spark release, regardless of who they are.

I haven't thought through whether that's a theoretical issue we can ignore or something we need to fix up. For example you can't get a release on the ASF mirrors without more authentication.

How hard would it be to make the script take in a key? it sort of looks like the script already takes GPG_KEY, but don't know how to modify the jobs. I suppose it would be ideal, in any event, for the actual release manager to sign.

On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <[hidden email]> wrote:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?



--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

Re: Signing releases with pwendell or release manager's key?

Holden Karau
Changing the release jobs, beyond the available parameters, right now depends on Josh arisen as there are some scripts which generate the jobs which aren't public. I've done temporary fixes in the past with the Python packaging but my understanding is that in the medium term it requires access to the scripts.

So +CC Josh.

On Fri, Sep 15, 2017 at 4:38 PM Ryan Blue <[hidden email]> wrote:
I think this needs to be fixed. It's true that there are barriers to publication, but the signature is what we use to authenticate Apache releases.

If Patrick's key is available on Jenkins for any Spark committer to use, then the chance of a compromise are much higher than for a normal RM key.

rb

On Fri, Sep 15, 2017 at 12:34 PM, Sean Owen <[hidden email]> wrote:
Yeah I had meant to ask about that in the past. While I presume Patrick consents to this and all that, it does mean that anyone with access to said Jenkins scripts can create a signed Spark release, regardless of who they are.

I haven't thought through whether that's a theoretical issue we can ignore or something we need to fix up. For example you can't get a release on the ASF mirrors without more authentication.

How hard would it be to make the script take in a key? it sort of looks like the script already takes GPG_KEY, but don't know how to modify the jobs. I suppose it would be ideal, in any event, for the actual release manager to sign.

On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <[hidden email]> wrote:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?



--
Ryan Blue
Software Engineer
Netflix
--
Reply | Threaded
Open this post in threaded view
|

Re: Signing releases with pwendell or release manager's key?

Holden Karau
Also continuing the discussion from the vote threads, Shane probably has the best idea on the ACLs for Jenkins so I've CC'd him as well.


On Fri, Sep 15, 2017 at 5:09 PM Holden Karau <[hidden email]> wrote:
Changing the release jobs, beyond the available parameters, right now depends on Josh arisen as there are some scripts which generate the jobs which aren't public. I've done temporary fixes in the past with the Python packaging but my understanding is that in the medium term it requires access to the scripts.

So +CC Josh.

On Fri, Sep 15, 2017 at 4:38 PM Ryan Blue <[hidden email]> wrote:
I think this needs to be fixed. It's true that there are barriers to publication, but the signature is what we use to authenticate Apache releases.

If Patrick's key is available on Jenkins for any Spark committer to use, then the chance of a compromise are much higher than for a normal RM key.

rb

On Fri, Sep 15, 2017 at 12:34 PM, Sean Owen <[hidden email]> wrote:
Yeah I had meant to ask about that in the past. While I presume Patrick consents to this and all that, it does mean that anyone with access to said Jenkins scripts can create a signed Spark release, regardless of who they are.

I haven't thought through whether that's a theoretical issue we can ignore or something we need to fix up. For example you can't get a release on the ASF mirrors without more authentication.

How hard would it be to make the script take in a key? it sort of looks like the script already takes GPG_KEY, but don't know how to modify the jobs. I suppose it would be ideal, in any event, for the actual release manager to sign.

On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <[hidden email]> wrote:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?



--
Ryan Blue
Software Engineer
Netflix
--
--
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Spark 2.1.2 (RC1)

Holden Karau
In reply to this post by Ryan Blue
Indeed it's limited to a people with login permissions on the Jenkins host (and perhaps further limited, I'm not certain). Shane probably knows more about the ACLs, so I'll ask him in the other thread for specifics.

This is maybe branching a bit from the question of the current RC though, so I'd suggest we continue this discussion on the thread Sean Owen made.

On Fri, Sep 15, 2017 at 4:04 PM Ryan Blue <[hidden email]> wrote:
I'm not familiar with the release procedure, can you send a link to this Jenkins job? Can anyone run this job, or is it limited to committers?

rb

On Fri, Sep 15, 2017 at 12:28 PM, Holden Karau <[hidden email]> wrote:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?

rb

On Fri, Sep 15, 2017 at 6:36 AM, Denny Lee <[hidden email]> wrote:
+1 (non-binding)

On Thu, Sep 14, 2017 at 10:57 PM Felix Cheung <[hidden email]> wrote:
+1 tested SparkR package on Windows, r-hub, Ubuntu.

_____________________________
From: Sean Owen <[hidden email]>
Sent: Thursday, September 14, 2017 3:12 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC1)
To: Holden Karau <[hidden email]>, <[hidden email]>



+1
Very nice. The sigs and hashes look fine, it builds fine for me on Debian Stretch with Java 8, yarn/hive/hadoop-2.7 profiles, and passes tests. 

Yes as you say, no outstanding issues except for this which doesn't look critical, as it's not a regression.

SPARK-21985 PySpark PairDeserializer is broken for double-zipped RDDs


On Thu, Sep 14, 2017 at 7:47 PM Holden Karau <[hidden email]> wrote:
Please vote on releasing the following candidate as Apache Spark version 2.1.2. The vote is open until Friday September 22nd at 18:00 PST and passes if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.2
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see https://spark.apache.org/

The tag to be voted on is v2.1.2-rc1 (6f470323a0363656999dd36cb33f528afe627c12)

List of JIRA tickets resolved in this release can be found with this filter.

The release files, including signatures, digests, etc. can be found at:

Release artifacts are signed with the following key:

The staging repository for this release can be found at:

The documentation corresponding to this release can be found at:


FAQ

How can I help test this release?

If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.

If you're working in PySpark you can set up a virtual env and install the current RC and see if anything important breaks, in the Java/Scala you can add the staging repository to your projects resolvers and test with the RC (make sure to clean up the artifact cache before/after so you don't end up building with a out of date RC going forward).

What should happen to JIRA tickets still targeting 2.1.2?

Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.3.

But my bug isn't fixed!??!

In order to make timely releases, we will typically not hold the release unless the bug in question is a regression from 2.1.1. That being said if there is something which is a regression form 2.1.1 that has not been correctly targeted please ping a committer to help target the issue (you can see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

What are the unresolved issues targeted for 2.1.2?

At the time of the writing, there is one in progress major issue SPARK-21985, I believe Andrew Ray & HyukjinKwon are looking into this one.





--
Ryan Blue
Software Engineer
Netflix



--



--
Ryan Blue
Software Engineer
Netflix
--
Reply | Threaded
Open this post in threaded view
|

Re: Signing releases with pwendell or release manager's key?

Holden Karau
In reply to this post by Holden Karau
Oh yes and to keep people more informed I've been updating a PR for the release documentation as I go to write down some of this unwritten knowledge -- https://github.com/apache/spark-website/pull/66


On Fri, Sep 15, 2017 at 5:12 PM Holden Karau <[hidden email]> wrote:
Also continuing the discussion from the vote threads, Shane probably has the best idea on the ACLs for Jenkins so I've CC'd him as well.


On Fri, Sep 15, 2017 at 5:09 PM Holden Karau <[hidden email]> wrote:
Changing the release jobs, beyond the available parameters, right now depends on Josh arisen as there are some scripts which generate the jobs which aren't public. I've done temporary fixes in the past with the Python packaging but my understanding is that in the medium term it requires access to the scripts.

So +CC Josh.

On Fri, Sep 15, 2017 at 4:38 PM Ryan Blue <[hidden email]> wrote:
I think this needs to be fixed. It's true that there are barriers to publication, but the signature is what we use to authenticate Apache releases.

If Patrick's key is available on Jenkins for any Spark committer to use, then the chance of a compromise are much higher than for a normal RM key.

rb

On Fri, Sep 15, 2017 at 12:34 PM, Sean Owen <[hidden email]> wrote:
Yeah I had meant to ask about that in the past. While I presume Patrick consents to this and all that, it does mean that anyone with access to said Jenkins scripts can create a signed Spark release, regardless of who they are.

I haven't thought through whether that's a theoretical issue we can ignore or something we need to fix up. For example you can't get a release on the ASF mirrors without more authentication.

How hard would it be to make the script take in a key? it sort of looks like the script already takes GPG_KEY, but don't know how to modify the jobs. I suppose it would be ideal, in any event, for the actual release manager to sign.

On Fri, Sep 15, 2017 at 8:28 PM Holden Karau <[hidden email]> wrote:
That's a good question, I built the release candidate however the Jenkins scripts don't take a parameter for configuring who signs them rather it always signs them with Patrick's key. You can see this from previous releases which were managed by other folks but still signed by Patrick.

On Fri, Sep 15, 2017 at 12:16 PM, Ryan Blue <[hidden email]> wrote:
The signature is valid, but why was the release signed with Patrick Wendell's private key? Did Patrick build the release candidate?



--
Ryan Blue
Software Engineer
Netflix
--
--
--
123