[VOTE] Release Apache Spark 2.4.1 (RC6)

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-7
Please vote on releasing the following candidate as Apache Spark version 2.4.1.

The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.1
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
https://github.com/apache/spark/tree/v2.4.1-rc6

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1308/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/

The list of bug fixes going into 2.4.1 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/2.4.1

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.1?
===========================================

The current list of open tickets targeted at 2.4.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Mihály Tóth
Hi,

I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.

I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.

Best Regards,

  Misi

DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
Please vote on releasing the following candidate as Apache Spark version 2.4.1.

The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.1
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
https://github.com/apache/spark/tree/v2.4.1-rc6

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1308/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/

The list of bug fixes going into 2.4.1 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/2.4.1

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.1?
===========================================

The current list of open tickets targeted at 2.4.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Sean Owen-2
That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch. 

It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.

On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
Hi,

I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.

I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.

Best Regards,

  Misi

DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
Please vote on releasing the following candidate as Apache Spark version 2.4.1.

The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 2.4.1
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
https://github.com/apache/spark/tree/v2.4.1-rc6

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1308/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/

The list of bug fixes going into 2.4.1 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/2.4.1

FAQ

=========================
How can I help test this release?
=========================

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===========================================
What should happen to JIRA tickets still targeting 2.4.1?
===========================================

The current list of open tickets targeted at 2.4.1 can be found at:
https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1

Committers should look at those and triage. Extremely important bug
fixes, documentation, and API tweaks that impact compatibility should
be worked on immediately. Everything else please retarget to an
appropriate release.

==================
But my bug isn't fixed?
==================

In order to make timely releases, we will typically not hold the
release unless the bug in question is a regression from the previous
release. That being said, if there is something which is a regression
that has not been correctly targeted please ping me or a committer to
help target the issue.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Marcelo Vanzin-2
I personally find it a little weird to not have the commit in branch-2.4.

Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
(e.g. accidentally) then you lose the reference to that commit, and
then the exact commit from which the rc was generated is lost.

On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:

>
> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>
> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>
> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>
>> Hi,
>>
>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>
>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>
>> Best Regards,
>>
>>   Misi
>>
>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>
>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>
>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>> a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>
>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>
>>> FAQ
>>>
>>> =========================
>>> How can I help test this release?
>>> =========================
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===========================================
>>> What should happen to JIRA tickets still targeting 2.4.1?
>>> ===========================================
>>>
>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==================
>>> But my bug isn't fixed?
>>> ==================
>>>
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>


--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-7
I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.

Should we create a new rc7?

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>
> I personally find it a little weird to not have the commit in branch-2.4.
>
> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
> (e.g. accidentally) then you lose the reference to that commit, and
> then the exact commit from which the rc was generated is lost.
>
> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>
>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>
>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>
>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>
>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>
>>> Best Regards,
>>>
>>> Misi
>>>
>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>
>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>
>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>> a minimum of 3 +1 votes.
>>>>
>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>> [ ] -1 Do not release this package because ...
>>>>
>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>
>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>
>>>> The release files, including signatures, digests, etc. can be found at:
>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>
>>>> Signatures used for Spark RCs can be found in this file:
>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>
>>>> The staging repository for this release can be found at:
>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>
>>>> The documentation corresponding to this release can be found at:
>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>
>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>
>>>> FAQ
>>>>
>>>> =========================
>>>> How can I help test this release?
>>>> =========================
>>>>
>>>> If you are a Spark user, you can help us test this release by taking
>>>> an existing Spark workload and running on this release candidate, then
>>>> reporting any regressions.
>>>>
>>>> If you're working in PySpark you can set up a virtual env and install
>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>> you can add the staging repository to your projects resolvers and test
>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>> you don't end up building with a out of date RC going forward).
>>>>
>>>> ===========================================
>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>> ===========================================
>>>>
>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>
>>>> Committers should look at those and triage. Extremely important bug
>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>> be worked on immediately. Everything else please retarget to an
>>>> appropriate release.
>>>>
>>>> ==================
>>>> But my bug isn't fixed?
>>>> ==================
>>>>
>>>> In order to make timely releases, we will typically not hold the
>>>> release unless the bug in question is a regression from the previous
>>>> release. That being said, if there is something which is a regression
>>>> that has not been correctly targeted please ping me or a committer to
>>>> help target the issue.
>>>>
>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: [hidden email]
>>>>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Marcelo Vanzin-2
I'd be more comfortable with an rc7. Either that or manually fix the
branch with a force push, but that's a bit risky, it's easy to mess up
force pushes (if we can even do that?).

It's very possible that there is a bug in the script; IIRC it should
create the commits in the right branch when you generate the rc the
first time. Perhaps you missed some error in the command line in that
invocation (tag was created but commits not pushed to the branch, for
example).

On Fri, Mar 8, 2019 at 11:39 AM DB Tsai <[hidden email]> wrote:

>
> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>
> Should we create a new rc7?
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
> > On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
> >
> > I personally find it a little weird to not have the commit in branch-2.4.
> >
> > Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
> > (e.g. accidentally) then you lose the reference to that commit, and
> > then the exact commit from which the rc was generated is lost.
> >
> > On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
> >>
> >> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
> >>
> >> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
> >>
> >> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
> >>>
> >>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
> >>>
> >>> Best Regards,
> >>>
> >>> Misi
> >>>
> >>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
> >>>>
> >>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
> >>>>
> >>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
> >>>> a minimum of 3 +1 votes.
> >>>>
> >>>> [ ] +1 Release this package as Apache Spark 2.4.1
> >>>> [ ] -1 Do not release this package because ...
> >>>>
> >>>> To learn more about Apache Spark, please see http://spark.apache.org/
> >>>>
> >>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
> >>>> https://github.com/apache/spark/tree/v2.4.1-rc6
> >>>>
> >>>> The release files, including signatures, digests, etc. can be found at:
> >>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
> >>>>
> >>>> Signatures used for Spark RCs can be found in this file:
> >>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
> >>>>
> >>>> The staging repository for this release can be found at:
> >>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
> >>>>
> >>>> The documentation corresponding to this release can be found at:
> >>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
> >>>>
> >>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
> >>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
> >>>>
> >>>> FAQ
> >>>>
> >>>> =========================
> >>>> How can I help test this release?
> >>>> =========================
> >>>>
> >>>> If you are a Spark user, you can help us test this release by taking
> >>>> an existing Spark workload and running on this release candidate, then
> >>>> reporting any regressions.
> >>>>
> >>>> If you're working in PySpark you can set up a virtual env and install
> >>>> the current RC and see if anything important breaks, in the Java/Scala
> >>>> you can add the staging repository to your projects resolvers and test
> >>>> with the RC (make sure to clean up the artifact cache before/after so
> >>>> you don't end up building with a out of date RC going forward).
> >>>>
> >>>> ===========================================
> >>>> What should happen to JIRA tickets still targeting 2.4.1?
> >>>> ===========================================
> >>>>
> >>>> The current list of open tickets targeted at 2.4.1 can be found at:
> >>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
> >>>>
> >>>> Committers should look at those and triage. Extremely important bug
> >>>> fixes, documentation, and API tweaks that impact compatibility should
> >>>> be worked on immediately. Everything else please retarget to an
> >>>> appropriate release.
> >>>>
> >>>> ==================
> >>>> But my bug isn't fixed?
> >>>> ==================
> >>>>
> >>>> In order to make timely releases, we will typically not hold the
> >>>> release unless the bug in question is a regression from the previous
> >>>> release. That being said, if there is something which is a regression
> >>>> that has not been correctly targeted please ping me or a committer to
> >>>> help target the issue.
> >>>>
> >>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe e-mail: [hidden email]
> >>>>
> >
> >
> > --
> > Marcelo
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: [hidden email]
> >
>


--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-7
In reply to this post by DB Tsai-7
Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>
> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>
> Should we create a new rc7?
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>
>> I personally find it a little weird to not have the commit in branch-2.4.
>>
>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>> (e.g. accidentally) then you lose the reference to that commit, and
>> then the exact commit from which the rc was generated is lost.
>>
>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>
>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>
>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>
>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>
>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>
>>>> Best Regards,
>>>>
>>>> Misi
>>>>
>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>
>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>
>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>> a minimum of 3 +1 votes.
>>>>>
>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>> [ ] -1 Do not release this package because ...
>>>>>
>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>
>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>
>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>
>>>>> Signatures used for Spark RCs can be found in this file:
>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>
>>>>> The staging repository for this release can be found at:
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>
>>>>> The documentation corresponding to this release can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>
>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>
>>>>> FAQ
>>>>>
>>>>> =========================
>>>>> How can I help test this release?
>>>>> =========================
>>>>>
>>>>> If you are a Spark user, you can help us test this release by taking
>>>>> an existing Spark workload and running on this release candidate, then
>>>>> reporting any regressions.
>>>>>
>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>> you can add the staging repository to your projects resolvers and test
>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>> you don't end up building with a out of date RC going forward).
>>>>>
>>>>> ===========================================
>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>> ===========================================
>>>>>
>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>
>>>>> Committers should look at those and triage. Extremely important bug
>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>> be worked on immediately. Everything else please retarget to an
>>>>> appropriate release.
>>>>>
>>>>> ==================
>>>>> But my bug isn't fixed?
>>>>> ==================
>>>>>
>>>>> In order to make timely releases, we will typically not hold the
>>>>> release unless the bug in question is a regression from the previous
>>>>> release. That being said, if there is something which is a regression
>>>>> that has not been correctly targeted please ping me or a committer to
>>>>> help target the issue.
>>>>>
>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: [hidden email]
>>>>>
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-7
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Xiao Li-2
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-6
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
--
- DB Sent from my iPhone
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-6
BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <[hidden email]> wrote:
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
--
- DB Sent from my iPhone
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Xiao Li-2
It is common to hit this issue when driver and executors are different object layout, but Spark might not return a wrong answer. It is very hard to find out the root cause. Thus, I would suggest to include it in Spark 2.4.1. 

On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <[hidden email]> wrote:
BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <[hidden email]> wrote:
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
--
- DB Sent from my iPhone


--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-6
Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully, this is the final rc. Thanks.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <[hidden email]> wrote:
It is common to hit this issue when driver and executors are different object layout, but Spark might not return a wrong answer. It is very hard to find out the root cause. Thus, I would suggest to include it in Spark 2.4.1. 

On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <[hidden email]> wrote:
BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <[hidden email]> wrote:
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
--
- DB Sent from my iPhone


--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Sean Owen-2
FWIW RC6 looked fine to me. Passed all tests, etc.

On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <[hidden email]> wrote:
Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully, this is the final rc. Thanks.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <[hidden email]> wrote:
It is common to hit this issue when driver and executors are different object layout, but Spark might not return a wrong answer. It is very hard to find out the root cause. Thus, I would suggest to include it in Spark 2.4.1. 

On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <[hidden email]> wrote:
BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <[hidden email]> wrote:
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
--
- DB Sent from my iPhone


--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Michael Heuer
Any chance we could get some movement on this for 2.4.1?


It would require a new Parquet release, which would then need to be picked up by Spark.  We're dead in the water on 2.4.0 without a large refactoring (remove all the RDD code paths for reading Avro stored in Parquet).

   michael


On Mar 8, 2019, at 6:22 PM, Sean Owen <[hidden email]> wrote:

FWIW RC6 looked fine to me. Passed all tests, etc.

On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <[hidden email]> wrote:
Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully, this is the final rc. Thanks.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <[hidden email]> wrote:
It is common to hit this issue when driver and executors are different object layout, but Spark might not return a wrong answer. It is very hard to find out the root cause. Thus, I would suggest to include it in Spark 2.4.1. 

On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <[hidden email]> wrote:
BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <[hidden email]> wrote:
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
--
- DB Sent from my iPhone


--

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

DB Tsai-6
As we have many important fixes in 2.4 branch which we want to release asap, and this is is not a regression from Spark 2.4; as a result, 2.4.1 will be not blocked by this.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0359BC9965359766


On Sun, Mar 10, 2019 at 3:08 PM Michael Heuer <[hidden email]> wrote:
Any chance we could get some movement on this for 2.4.1?


It would require a new Parquet release, which would then need to be picked up by Spark.  We're dead in the water on 2.4.0 without a large refactoring (remove all the RDD code paths for reading Avro stored in Parquet).

   michael


On Mar 8, 2019, at 6:22 PM, Sean Owen <[hidden email]> wrote:

FWIW RC6 looked fine to me. Passed all tests, etc.

On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <[hidden email]> wrote:
Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully, this is the final rc. Thanks.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <[hidden email]> wrote:
It is common to hit this issue when driver and executors are different object layout, but Spark might not return a wrong answer. It is very hard to find out the root cause. Thus, I would suggest to include it in Spark 2.4.1. 

On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <[hidden email]> wrote:
BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <[hidden email]> wrote:
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
--
- DB Sent from my iPhone


--

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Mark Hamstra
Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.  

On Sun, Mar 10, 2019 at 3:42 PM DB Tsai <[hidden email]> wrote:
As we have many important fixes in 2.4 branch which we want to release asap, and this is is not a regression from Spark 2.4; as a result, 2.4.1 will be not blocked by this.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0359BC9965359766


On Sun, Mar 10, 2019 at 3:08 PM Michael Heuer <[hidden email]> wrote:
Any chance we could get some movement on this for 2.4.1?


It would require a new Parquet release, which would then need to be picked up by Spark.  We're dead in the water on 2.4.0 without a large refactoring (remove all the RDD code paths for reading Avro stored in Parquet).

   michael


On Mar 8, 2019, at 6:22 PM, Sean Owen <[hidden email]> wrote:

FWIW RC6 looked fine to me. Passed all tests, etc.

On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <[hidden email]> wrote:
Sounds fair to me. I'll cut another rc7 when the PR is merged. Hopefully, this is the final rc. Thanks.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <[hidden email]> wrote:
It is common to hit this issue when driver and executors are different object layout, but Spark might not return a wrong answer. It is very hard to find out the root cause. Thus, I would suggest to include it in Spark 2.4.1. 

On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <[hidden email]> wrote:
BTW, practically, is it common for users running into this bug when the driver and executors have different object layout?

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <[hidden email]> wrote:
Hi Xiao,

I already cut rc7 and start the build process. If we definitely need this fix, I can cut rc8. Let me know what you think.

Thanks,

On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <[hidden email]> wrote:
Hi, DB, 

Since this RC will fail, could you hold it until we fix https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I will submit a PR today. The PR is small and the risk is low. This is a correctness bug. It would be good to have it. 

Thanks,

Xiao


 

On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <[hidden email]> wrote:
Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc

> On Mar 8, 2019, at 11:45 AM, DB Tsai <[hidden email]> wrote:
>
> Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>> On Mar 8, 2019, at 11:39 AM, DB Tsai <[hidden email]> wrote:
>>
>> I was using `./do-release-docker.sh` to create a release. But since the gpg validation fails couple times when the script tried to publish the jars into Nexus, I re-ran the scripts multiple times without creating a new rc. I was wondering if the script will overwrite the v.2.4.1-rc6 tag instead of using the same commit causing this issue.
>>
>> Should we create a new rc7?
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin <[hidden email]> wrote:
>>>
>>> I personally find it a little weird to not have the commit in branch-2.4.
>>>
>>> Not that this would happen, but if the v2.4.1-rc6 tag is overwritten
>>> (e.g. accidentally) then you lose the reference to that commit, and
>>> then the exact commit from which the rc was generated is lost.
>>>
>>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> That's weird. I see the commit but can't find it in the branch. Was it pushed, or lost in a force push of 2.4 along the way? The change is there, just under a different commit in the 2.4 branch.
>>>>
>>>> It doesn't necessarily invalidate the RC as it is a valid public tagged commit and all that. I just want to be sure we do have the code from that commit in these tatballs. It looks like it.
>>>>
>>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on branch-2.4. Release related commits I have seen so far were also part of the branch.
>>>>>
>>>>> I guess the "Preparing Spark release v2.4.1-rc6" and "Preparing development version 2.4.2-SNAPSHOT" commits were simply not pushed to spark-2.4 just the tag itself was pushed. I dont know what is the practice in such cases but one solution is to rebase branch-2.4 changes after 3336a21 onto these commits and do a (sorry) force push. In this case there is no impact on this RC.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Misi
>>>>>
>>>>> DB Tsai <[hidden email]> ezt írta (időpont: 2019. márc. 8., P, 1:15):
>>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark version 2.4.1.
>>>>>>
>>>>>> The vote is open until March 11 PST and passes if a majority +1 PMC votes are cast, with
>>>>>> a minimum of 3 +1 votes.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 2.4.1
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>>
>>>>>> The tag to be voted on is v2.4.1-rc6 (commit 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc):
>>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/
>>>>>>
>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/
>>>>>>
>>>>>> The list of bug fixes going into 2.4.1 can be found at the following URL:
>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1
>>>>>>
>>>>>> FAQ
>>>>>>
>>>>>> =========================
>>>>>> How can I help test this release?
>>>>>> =========================
>>>>>>
>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>> an existing Spark workload and running on this release candidate, then
>>>>>> reporting any regressions.
>>>>>>
>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>> the current RC and see if anything important breaks, in the Java/Scala
>>>>>> you can add the staging repository to your projects resolvers and test
>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>
>>>>>> ===========================================
>>>>>> What should happen to JIRA tickets still targeting 2.4.1?
>>>>>> ===========================================
>>>>>>
>>>>>> The current list of open tickets targeted at 2.4.1 can be found at:
>>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target Version/s" = 2.4.1
>>>>>>
>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>> appropriate release.
>>>>>>
>>>>>> ==================
>>>>>> But my bug isn't fixed?
>>>>>> ==================
>>>>>>
>>>>>> In order to make timely releases, we will typically not hold the
>>>>>> release unless the bug in question is a regression from the previous
>>>>>> release. That being said, if there is something which is a regression
>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>> help target the issue.
>>>>>>
>>>>>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: [hidden email]
>>>>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
--
- DB Sent from my iPhone


--

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Sean Owen-2
From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:

- this is a Parquet-Avro version conflict thing
- a downstream app wants different versions of Parquet and Avro than
Spark uses, which triggers it
- it doesn't work in 2.4.0

It's not a regression from 2.4.0, which is the immediate question.
There isn't even a Parquet fix available.
But I'm not even seeing why this is excuse-making?

On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <[hidden email]> wrote:
>
> Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Mark Hamstra
It worked in 2.3. We broke it with 2.4.0 and were informed of that regression late in the 2.4.0 release process. Since we didn't fix it before the 2.4.0 release, it should have been noted as a known issue. To now claim that there is no regression from 2.4.0 is a circular argument denying the existence of a known regression from 2.3.  

On Sun, Mar 10, 2019 at 6:53 PM Sean Owen <[hidden email]> wrote:
From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:

- this is a Parquet-Avro version conflict thing
- a downstream app wants different versions of Parquet and Avro than
Spark uses, which triggers it
- it doesn't work in 2.4.0

It's not a regression from 2.4.0, which is the immediate question.
There isn't even a Parquet fix available.
But I'm not even seeing why this is excuse-making?

On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <[hidden email]> wrote:
>
> Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Michael Heuer
In reply to this post by Sean Owen-2
I'm not saying that this issue should be a blocker for 2.4.1, rather I'm looking for help moving things along.  I'm not a committer in any of the Spark, Parquet, or Avro projects.


On Mar 10, 2019, at 8:53 PM, Sean Owen <[hidden email]> wrote:

From https://issues.apache.org/jira/browse/SPARK-25588, I'm reading that:

- this is a Parquet-Avro version conflict thing
- a downstream app wants different versions of Parquet and Avro than
Spark uses, which triggers it

Prior to 2.4.0, Spark depended on versions of Parquet and Avro that did not work with each other.  In fact, a different version of Avro had to be used in Spark's test scope to prevent runtime errors.

As a workaround, we had to override Parquet to 1.8.2 (later 1.8.3) but pin parquet-avro to 1.8.1.


- it doesn't work in 2.4.0

In 2.4.0 we're no longer able to pin parquet-avro to 1.8.1, so our workaround is broken.  Using the Spark 2.4.0 versions of Parquet and Avro uncovers this new error.



It's not a regression from 2.4.0, which is the immediate question.
There isn't even a Parquet fix available.

I believe https://github.com/apache/parquet-mr/pull/560/files is a fix, but I haven't made all the necessary snapshot builds to test.  I was waiting for a valid Spark 2.4.1 RC to try.


But I'm not even seeing why this is excuse-making?

On Sun, Mar 10, 2019 at 8:44 PM Mark Hamstra <[hidden email]> wrote:

Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.

12