Apache Spark 2.2.3 ?

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache Spark 2.2.3 ?

Dongjoon Hyun-2
Hi, All.

Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.

After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).


If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).

Please let me know your opinions about this.

Bests,
Dongjoon.
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Sean Owen-2
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:

>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
>         http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
>         https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Felix Cheung
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018)

And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing?

 

From: Sean Owen <[hidden email]>
Sent: Tuesday, January 1, 2019 8:30 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Apache Spark 2.2.3 ?
 
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:
>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
> http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Xiao Li-2
Based on the commit history, https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 contains more critical fixes. Maybe the priority is higher?

On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung <[hidden email]> wrote:
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018)

And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing?

 

From: Sean Owen <[hidden email]>
Sent: Tuesday, January 1, 2019 8:30 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Apache Spark 2.2.3 ?
 
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:
>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
> http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Saisai Shao
Agreed to have a new branch-2.3 release, as we already accumulated several fixes.

Thanks
Saisai

Xiao Li <[hidden email]> 于2019年1月2日周三 下午1:32写道:
Based on the commit history, https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 contains more critical fixes. Maybe the priority is higher?

On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung <[hidden email]> wrote:
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018)

And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing?

 

From: Sean Owen <[hidden email]>
Sent: Tuesday, January 1, 2019 8:30 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Apache Spark 2.2.3 ?
 
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:
>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
> http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Dongjoon Hyun-2
Thank you for swift feedbacks and Happy New Year. :)
For 2.2.3 release on next week, I see two positive opinions (including mine)
and don't see any direct objections.

Apache Spark has a mature, resourceful, and fast-growing community.
One of the important characteristic of the mature community is
the expectable behavior where the users are able to depend on.
For instance, we have a nice tradition to cut the branch as a sign of feature freeze.
The *final* release of a branch is not only good for the end users, but also a good sign of the EOL of the branch for all.

As a junior committer of the community, I want to contribute to deliver the final 2.2.3 release to the community and to finalize `branch-2.2`.

* For Apache Spark JIRA, I checked that there is no on-going issues targeting on `2.2.3`.
* For commits, I reviewed the newly landed commits after `2.2.2` tag and updated a few missing JIRA issues accordingly.
* Apparently, we can release 2.2.3 next week. 

BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases before Spark+AI Summit (April) because we did like that usually.
Please send another email to `dev` mailing list because it's worth to receive more attentions and requests.

Bests,
Dongjoon.


On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao <[hidden email]> wrote:
Agreed to have a new branch-2.3 release, as we already accumulated several fixes.

Thanks
Saisai

Xiao Li <[hidden email]> 于2019年1月2日周三 下午1:32写道:
Based on the commit history, https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 contains more critical fixes. Maybe the priority is higher?

On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung <[hidden email]> wrote:
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018)

And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing?

 

From: Sean Owen <[hidden email]>
Sent: Tuesday, January 1, 2019 8:30 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Apache Spark 2.2.3 ?
 
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:
>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
> http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Felix Cheung
+1 on 2.2.3 of course

 

From: Dongjoon Hyun <[hidden email]>
Sent: Wednesday, January 2, 2019 12:21 PM
To: Saisai Shao
Cc: Xiao Li; Felix Cheung; Sean Owen; dev
Subject: Re: Apache Spark 2.2.3 ?
 
Thank you for swift feedbacks and Happy New Year. :)
For 2.2.3 release on next week, I see two positive opinions (including mine)
and don't see any direct objections.

Apache Spark has a mature, resourceful, and fast-growing community.
One of the important characteristic of the mature community is
the expectable behavior where the users are able to depend on.
For instance, we have a nice tradition to cut the branch as a sign of feature freeze.
The *final* release of a branch is not only good for the end users, but also a good sign of the EOL of the branch for all.

As a junior committer of the community, I want to contribute to deliver the final 2.2.3 release to the community and to finalize `branch-2.2`.

* For Apache Spark JIRA, I checked that there is no on-going issues targeting on `2.2.3`.
* For commits, I reviewed the newly landed commits after `2.2.2` tag and updated a few missing JIRA issues accordingly.
* Apparently, we can release 2.2.3 next week. 

BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases before Spark+AI Summit (April) because we did like that usually.
Please send another email to `dev` mailing list because it's worth to receive more attentions and requests.

Bests,
Dongjoon.


On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao <[hidden email]> wrote:
Agreed to have a new branch-2.3 release, as we already accumulated several fixes.

Thanks
Saisai

Xiao Li <[hidden email]> 于2019年1月2日周三 下午1:32写道:
Based on the commit history, https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 contains more critical fixes. Maybe the priority is higher?

On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung <[hidden email]> wrote:
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018)

And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing?

 

From: Sean Owen <[hidden email]>
Sent: Tuesday, January 1, 2019 8:30 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Apache Spark 2.2.3 ?
 
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:
>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
> http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Takeshi Yamamuro
Hi, all, happy new year!

+1 on the release of 2.2.3/2.3.3.
I checked there is no ongoing issue targeting on 2.3.3, too.

On Thu, Jan 3, 2019 at 8:50 AM Felix Cheung <[hidden email]> wrote:
+1 on 2.2.3 of course

 

From: Dongjoon Hyun <[hidden email]>
Sent: Wednesday, January 2, 2019 12:21 PM
To: Saisai Shao
Cc: Xiao Li; Felix Cheung; Sean Owen; dev
Subject: Re: Apache Spark 2.2.3 ?
 
Thank you for swift feedbacks and Happy New Year. :)
For 2.2.3 release on next week, I see two positive opinions (including mine)
and don't see any direct objections.

Apache Spark has a mature, resourceful, and fast-growing community.
One of the important characteristic of the mature community is
the expectable behavior where the users are able to depend on.
For instance, we have a nice tradition to cut the branch as a sign of feature freeze.
The *final* release of a branch is not only good for the end users, but also a good sign of the EOL of the branch for all.

As a junior committer of the community, I want to contribute to deliver the final 2.2.3 release to the community and to finalize `branch-2.2`.

* For Apache Spark JIRA, I checked that there is no on-going issues targeting on `2.2.3`.
* For commits, I reviewed the newly landed commits after `2.2.2` tag and updated a few missing JIRA issues accordingly.
* Apparently, we can release 2.2.3 next week. 

BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases before Spark+AI Summit (April) because we did like that usually.
Please send another email to `dev` mailing list because it's worth to receive more attentions and requests.

Bests,
Dongjoon.


On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao <[hidden email]> wrote:
Agreed to have a new branch-2.3 release, as we already accumulated several fixes.

Thanks
Saisai

Xiao Li <[hidden email]> 于2019年1月2日周三 下午1:32写道:
Based on the commit history, https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 contains more critical fixes. Maybe the priority is higher?

On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung <[hidden email]> wrote:
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018)

And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing?

 

From: Sean Owen <[hidden email]>
Sent: Tuesday, January 1, 2019 8:30 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Apache Spark 2.2.3 ?
 
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:
>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
> http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Dongjoon Hyun-2
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 12:02 AM Takeshi Yamamuro <[hidden email]> wrote:
Hi, all, happy new year!

+1 on the release of 2.2.3/2.3.3.
I checked there is no ongoing issue targeting on 2.3.3, too.

On Thu, Jan 3, 2019 at 8:50 AM Felix Cheung <[hidden email]> wrote:
+1 on 2.2.3 of course

 

From: Dongjoon Hyun <[hidden email]>
Sent: Wednesday, January 2, 2019 12:21 PM
To: Saisai Shao
Cc: Xiao Li; Felix Cheung; Sean Owen; dev
Subject: Re: Apache Spark 2.2.3 ?
 
Thank you for swift feedbacks and Happy New Year. :)
For 2.2.3 release on next week, I see two positive opinions (including mine)
and don't see any direct objections.

Apache Spark has a mature, resourceful, and fast-growing community.
One of the important characteristic of the mature community is
the expectable behavior where the users are able to depend on.
For instance, we have a nice tradition to cut the branch as a sign of feature freeze.
The *final* release of a branch is not only good for the end users, but also a good sign of the EOL of the branch for all.

As a junior committer of the community, I want to contribute to deliver the final 2.2.3 release to the community and to finalize `branch-2.2`.

* For Apache Spark JIRA, I checked that there is no on-going issues targeting on `2.2.3`.
* For commits, I reviewed the newly landed commits after `2.2.2` tag and updated a few missing JIRA issues accordingly.
* Apparently, we can release 2.2.3 next week. 

BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases before Spark+AI Summit (April) because we did like that usually.
Please send another email to `dev` mailing list because it's worth to receive more attentions and requests.

Bests,
Dongjoon.


On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao <[hidden email]> wrote:
Agreed to have a new branch-2.3 release, as we already accumulated several fixes.

Thanks
Saisai

Xiao Li <[hidden email]> 于2019年1月2日周三 下午1:32写道:
Based on the commit history, https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 contains more critical fixes. Maybe the priority is higher?

On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung <[hidden email]> wrote:
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018)

And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing?

 

From: Sean Owen <[hidden email]>
Sent: Tuesday, January 1, 2019 8:30 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Apache Spark 2.2.3 ?
 
I agree with that logic, and if you're volunteering to do the legwork,
I don't see a reason not to cut a final 2.2 release.

On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun <[hidden email]> wrote:
>
> Hi, All.
>
> Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017.
>
> http://spark.apache.org/versioning-policy.html
>
> After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches).
>
> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2
>
> If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?).
>
> Please let me know your opinions about this.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Sean Owen-2
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Dongjoon Hyun-2
Thank you, Sean!

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 2:50 PM Sean Owen <[hidden email]> wrote:
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Takeshi Yamamuro
If there is no other volunteer for the release of 2.3.3, I'd like to.

best,
takeshi

On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean!

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 2:50 PM Sean Owen <[hidden email]> wrote:
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Dongjoon Hyun-2
Great! Thank you, Takeshi! :D

Bests,
Dongjoon.

On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro <[hidden email]> wrote:
If there is no other volunteer for the release of 2.3.3, I'd like to.

best,
takeshi

On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean!

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 2:50 PM Sean Owen <[hidden email]> wrote:
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Xiao Li
Thank you, Takeshi! 

Dongjoon Hyun <[hidden email]> 于2019年1月8日周二 下午10:13写道:
Great! Thank you, Takeshi! :D

Bests,
Dongjoon.

On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro <[hidden email]> wrote:
If there is no other volunteer for the release of 2.3.3, I'd like to.

best,
takeshi

On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean!

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 2:50 PM Sean Owen <[hidden email]> wrote:
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Takeshi Yamamuro
Hi, all

I'm planning to start the release vote for v2.3.3 in the start of the next week.
# I've already checked that all the tests passed in branch-2.3 and
# there is no problem by the release scripts with dry-run.

If there is any problem, please ping me.

Best,
Takeshi


On Wed, Jan 9, 2019 at 3:16 PM Xiao Li <[hidden email]> wrote:
Thank you, Takeshi! 

Dongjoon Hyun <[hidden email]> 于2019年1月8日周二 下午10:13写道:
Great! Thank you, Takeshi! :D

Bests,
Dongjoon.

On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro <[hidden email]> wrote:
If there is no other volunteer for the release of 2.3.3, I'd like to.

best,
takeshi

On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean!

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 2:50 PM Sean Owen <[hidden email]> wrote:
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.



--
---
Takeshi Yamamuro


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Dongjoon Hyun-2
Hi, Takeshi.

Please start a new thread by using a new title containing `2.3.3`.

Bests,
Dongjoon.

On Sat, Jan 12, 2019 at 3:39 PM Takeshi Yamamuro <[hidden email]> wrote:
Hi, all

I'm planning to start the release vote for v2.3.3 in the start of the next week.
# I've already checked that all the tests passed in branch-2.3 and
# there is no problem by the release scripts with dry-run.

If there is any problem, please ping me.

Best,
Takeshi


On Wed, Jan 9, 2019 at 3:16 PM Xiao Li <[hidden email]> wrote:
Thank you, Takeshi! 

Dongjoon Hyun <[hidden email]> 于2019年1月8日周二 下午10:13写道:
Great! Thank you, Takeshi! :D

Bests,
Dongjoon.

On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro <[hidden email]> wrote:
If there is no other volunteer for the release of 2.3.3, I'd like to.

best,
takeshi

On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean!

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 2:50 PM Sean Owen <[hidden email]> wrote:
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.



--
---
Takeshi Yamamuro


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.2.3 ?

Takeshi Yamamuro
Thanks, Dongjoon!
I did it.

Best,
Takeshi

On Sun, Jan 13, 2019 at 9:33 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, Takeshi.

Please start a new thread by using a new title containing `2.3.3`.

Bests,
Dongjoon.

On Sat, Jan 12, 2019 at 3:39 PM Takeshi Yamamuro <[hidden email]> wrote:
Hi, all

I'm planning to start the release vote for v2.3.3 in the start of the next week.
# I've already checked that all the tests passed in branch-2.3 and
# there is no problem by the release scripts with dry-run.

If there is any problem, please ping me.

Best,
Takeshi


On Wed, Jan 9, 2019 at 3:16 PM Xiao Li <[hidden email]> wrote:
Thank you, Takeshi! 

Dongjoon Hyun <[hidden email]> 于2019年1月8日周二 下午10:13写道:
Great! Thank you, Takeshi! :D

Bests,
Dongjoon.

On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro <[hidden email]> wrote:
If there is no other volunteer for the release of 2.3.3, I'd like to.

best,
takeshi

On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean!

Bests,
Dongjoon.


On Thu, Jan 3, 2019 at 2:50 PM Sean Owen <[hidden email]> wrote:
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done.
Note that Spark 2.3 would be EOL around September of this year.

On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for additional support for 2.2.3, Felix and Takeshi!


The following is the update for Apache Spark 2.2.3 release.

For correctness issues, two more patches landed on `branch-2.2`.

      SPARK-22951 fix aggregation after dropDuplicates on empty dataframes
      SPARK-25591 Avoid overwriting deserialized accumulator

Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206.

      Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness)


      Affected Version: 2.2.2, 2.3.1
      Target Versions: 2.3.2, 2.4.0
      Fixed Version: 2.4.0

Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too.

I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master.

So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206.

How do you think about that?

Bests,
Dongjoon.



--
---
Takeshi Yamamuro


--
---
Takeshi Yamamuro


--
---
Takeshi Yamamuro