Apache Spark 2.4.8 (and EOL of 2.4)

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Apache Spark 2.4.8 (and EOL of 2.4)

Dongjoon Hyun-2
Hi, All.

We successfully completed Apache Spark 3.1.1 and 3.0.2 releases and started 3.2.0 discussion already.

Let's talk about branch-2.4 because there exists some discussions on JIRA and GitHub about skipping backporting to 2.4.

Since `branch-2.4` has been maintained well as LTS, I'd like to suggest having Apache Spark 2.4.8 release as the official EOL release of 2.4 line in order to focus on 3.x more from now. Please note that `branch-2.4` will be frozen officially like `branch-2.3` after EOL release.

- Apache Spark 2.4.0 was released on November 2, 2018.
- Apache Spark 2.4.7 was released on September 12, 2020.
- Since v2.4.7 tag, `branch-2.4` has 134 commits including the following 12 correctness issues.

## CORRECTNESS ISSUE
SPARK-30201 HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT
SPARK-30228 Update zstd-jni to 1.4.4-3
SPARK-30894 The nullability of Size function should not depend on SQLConf.get
SPARK-32635 When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result
SPARK-32908 percentile_approx() returns incorrect results
SPARK-33183 Bug in optimizer rule EliminateSorts
SPARK-33290 REFRESH TABLE should invalidate cache even though the table itself may not be cached
SPARK-33593 Vector reader got incorrect data with binary partition value
SPARK-33726 Duplicate field names causes wrong answers during aggregation
SPARK-34187 Use available offset range obtained during polling when checking offset validation
SPARK-34212 For parquet table, after changing the precision and scale of decimal type in hive, spark reads incorrect value
SPARK-34229 Avro should read decimal values with the file schema

## SECURITY ISSUE
SPARK-33333 Upgrade Jetty to 9.4.28.v20200408
SPARK-33831 Update to jetty 9.4.34
SPARK-34449 Upgrade Jetty to fix CVE-2020-27218

What do you think about this?

Bests,
Dongjoon.
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh

Thanks Dongjoon!

+1 and I volunteer to do the release of 2.4.8 if it passes.


Liang-Chi




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Dongjoon Hyun-2
Thank you for volunteering as Apache Spark 2.4.8 release manager, Liang-chi!

On Wed, Mar 3, 2021 at 10:13 AM Liang-Chi Hsieh <[hidden email]> wrote:

Thanks Dongjoon!

+1 and I volunteer to do the release of 2.4.8 if it passes.


Liang-Chi




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Sean Owen-2
In reply to this post by Dongjoon Hyun-2
For reference, 2.3.x was maintained from February 2018 (2.3.0) to Sep 2019 (2.3.4), or about 19 months. The 2.4 branch should probably be maintained longer than that, as the final 2.x branch. 2.4.0 was released in Nov 2018. A final release in, say, April 2021 would be about 30 months. That feels about right timing-wise.

We should in any event release 2.4.8, yes. We can of course choose to release a 2.4.9 if some critical issue is found, later.

But yeah based on the velocity of back-ports to 2.4.x, it seems about time to call it EOL.

Sean


On Wed, Mar 3, 2021 at 12:05 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

We successfully completed Apache Spark 3.1.1 and 3.0.2 releases and started 3.2.0 discussion already.

Let's talk about branch-2.4 because there exists some discussions on JIRA and GitHub about skipping backporting to 2.4.

Since `branch-2.4` has been maintained well as LTS, I'd like to suggest having Apache Spark 2.4.8 release as the official EOL release of 2.4 line in order to focus on 3.x more from now. Please note that `branch-2.4` will be frozen officially like `branch-2.3` after EOL release.

- Apache Spark 2.4.0 was released on November 2, 2018.
- Apache Spark 2.4.7 was released on September 12, 2020.
- Since v2.4.7 tag, `branch-2.4` has 134 commits including the following 12 correctness issues.

## CORRECTNESS ISSUE
SPARK-30201 HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT
SPARK-30228 Update zstd-jni to 1.4.4-3
SPARK-30894 The nullability of Size function should not depend on SQLConf.get
SPARK-32635 When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result
SPARK-32908 percentile_approx() returns incorrect results
SPARK-33183 Bug in optimizer rule EliminateSorts
SPARK-33290 REFRESH TABLE should invalidate cache even though the table itself may not be cached
SPARK-33593 Vector reader got incorrect data with binary partition value
SPARK-33726 Duplicate field names causes wrong answers during aggregation
SPARK-34187 Use available offset range obtained during polling when checking offset validation
SPARK-34212 For parquet table, after changing the precision and scale of decimal type in hive, spark reads incorrect value
SPARK-34229 Avro should read decimal values with the file schema

## SECURITY ISSUE
SPARK-33333 Upgrade Jetty to 9.4.28.v20200408
SPARK-33831 Update to jetty 9.4.34
SPARK-34449 Upgrade Jetty to fix CVE-2020-27218

What do you think about this?

Bests,
Dongjoon.
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Dongjoon Hyun-2
Thank you, Sean.

Ya, exactly, we can release 2.4.8 as a normal release first and use 2.4.9 as the EOL release.

Since 2.4.7 was released almost 6 months ago, 2.4.8 is a little late in terms of the cadence.

Bests,
Dongjoon.


On Wed, Mar 3, 2021 at 10:55 AM Sean Owen <[hidden email]> wrote:
For reference, 2.3.x was maintained from February 2018 (2.3.0) to Sep 2019 (2.3.4), or about 19 months. The 2.4 branch should probably be maintained longer than that, as the final 2.x branch. 2.4.0 was released in Nov 2018. A final release in, say, April 2021 would be about 30 months. That feels about right timing-wise.

We should in any event release 2.4.8, yes. We can of course choose to release a 2.4.9 if some critical issue is found, later.

But yeah based on the velocity of back-ports to 2.4.x, it seems about time to call it EOL.

Sean


On Wed, Mar 3, 2021 at 12:05 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

We successfully completed Apache Spark 3.1.1 and 3.0.2 releases and started 3.2.0 discussion already.

Let's talk about branch-2.4 because there exists some discussions on JIRA and GitHub about skipping backporting to 2.4.

Since `branch-2.4` has been maintained well as LTS, I'd like to suggest having Apache Spark 2.4.8 release as the official EOL release of 2.4 line in order to focus on 3.x more from now. Please note that `branch-2.4` will be frozen officially like `branch-2.3` after EOL release.

- Apache Spark 2.4.0 was released on November 2, 2018.
- Apache Spark 2.4.7 was released on September 12, 2020.
- Since v2.4.7 tag, `branch-2.4` has 134 commits including the following 12 correctness issues.

## CORRECTNESS ISSUE
SPARK-30201 HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT
SPARK-30228 Update zstd-jni to 1.4.4-3
SPARK-30894 The nullability of Size function should not depend on SQLConf.get
SPARK-32635 When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result
SPARK-32908 percentile_approx() returns incorrect results
SPARK-33183 Bug in optimizer rule EliminateSorts
SPARK-33290 REFRESH TABLE should invalidate cache even though the table itself may not be cached
SPARK-33593 Vector reader got incorrect data with binary partition value
SPARK-33726 Duplicate field names causes wrong answers during aggregation
SPARK-34187 Use available offset range obtained during polling when checking offset validation
SPARK-34212 For parquet table, after changing the precision and scale of decimal type in hive, spark reads incorrect value
SPARK-34229 Avro should read decimal values with the file schema

## SECURITY ISSUE
SPARK-33333 Upgrade Jetty to 9.4.28.v20200408
SPARK-33831 Update to jetty 9.4.34
SPARK-34449 Upgrade Jetty to fix CVE-2020-27218

What do you think about this?

Bests,
Dongjoon.
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Sean Owen-2
Sure, I'm even arguing that 2.4.8 could possibly be the final release. No objection of course to continuing to backport to 2.4.x where appropriate and cutting 2.4.9 later in the year as a final EOL release, either.

On Wed, Mar 3, 2021 at 12:59 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean.

Ya, exactly, we can release 2.4.8 as a normal release first and use 2.4.9 as the EOL release.

Since 2.4.7 was released almost 6 months ago, 2.4.8 is a little late in terms of the cadence.

Bests,
Dongjoon.


On Wed, Mar 3, 2021 at 10:55 AM Sean Owen <[hidden email]> wrote:
For reference, 2.3.x was maintained from February 2018 (2.3.0) to Sep 2019 (2.3.4), or about 19 months. The 2.4 branch should probably be maintained longer than that, as the final 2.x branch. 2.4.0 was released in Nov 2018. A final release in, say, April 2021 would be about 30 months. That feels about right timing-wise.

We should in any event release 2.4.8, yes. We can of course choose to release a 2.4.9 if some critical issue is found, later.

But yeah based on the velocity of back-ports to 2.4.x, it seems about time to call it EOL.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Hyukjin Kwon
Yeah, I would prefer to have a 2.4.8 release as an EOL too. I don't mind having 2.4.9 as EOL too if that's preferred from more people.

2021년 3월 4일 (목) 오전 4:01, Sean Owen <[hidden email]>님이 작성:
Sure, I'm even arguing that 2.4.8 could possibly be the final release. No objection of course to continuing to backport to 2.4.x where appropriate and cutting 2.4.9 later in the year as a final EOL release, either.

On Wed, Mar 3, 2021 at 12:59 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean.

Ya, exactly, we can release 2.4.8 as a normal release first and use 2.4.9 as the EOL release.

Since 2.4.7 was released almost 6 months ago, 2.4.8 is a little late in terms of the cadence.

Bests,
Dongjoon.


On Wed, Mar 3, 2021 at 10:55 AM Sean Owen <[hidden email]> wrote:
For reference, 2.3.x was maintained from February 2018 (2.3.0) to Sep 2019 (2.3.4), or about 19 months. The 2.4 branch should probably be maintained longer than that, as the final 2.x branch. 2.4.0 was released in Nov 2018. A final release in, say, April 2021 would be about 30 months. That feels about right timing-wise.

We should in any event release 2.4.8, yes. We can of course choose to release a 2.4.9 if some critical issue is found, later.

But yeah based on the velocity of back-ports to 2.4.x, it seems about time to call it EOL.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Takeshi Yamamuro
+1 for releasing 2.4.8 and thanks, Liang-chi, for volunteering.
Btw, anyone roughly know how many v2.4 users still are based on some stats (e.g., # of v2.4.7 downloads from the official repos)?
Most users have started using v3.x?

On Thu, Mar 4, 2021 at 8:34 AM Hyukjin Kwon <[hidden email]> wrote:
Yeah, I would prefer to have a 2.4.8 release as an EOL too. I don't mind having 2.4.9 as EOL too if that's preferred from more people.

2021년 3월 4일 (목) 오전 4:01, Sean Owen <[hidden email]>님이 작성:
Sure, I'm even arguing that 2.4.8 could possibly be the final release. No objection of course to continuing to backport to 2.4.x where appropriate and cutting 2.4.9 later in the year as a final EOL release, either.

On Wed, Mar 3, 2021 at 12:59 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Sean.

Ya, exactly, we can release 2.4.8 as a normal release first and use 2.4.9 as the EOL release.

Since 2.4.7 was released almost 6 months ago, 2.4.8 is a little late in terms of the cadence.

Bests,
Dongjoon.


On Wed, Mar 3, 2021 at 10:55 AM Sean Owen <[hidden email]> wrote:
For reference, 2.3.x was maintained from February 2018 (2.3.0) to Sep 2019 (2.3.4), or about 19 months. The 2.4 branch should probably be maintained longer than that, as the final 2.x branch. 2.4.0 was released in Nov 2018. A final release in, say, April 2021 would be about 30 months. That feels about right timing-wise.

We should in any event release 2.4.8, yes. We can of course choose to release a 2.4.9 if some critical issue is found, later.

But yeah based on the velocity of back-ports to 2.4.x, it seems about time to call it EOL.

Sean


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh

Thanks all for the input.

If there is no objection, I am going to cut the branch next Monday.

Thanks.
Liang-Chi


Takeshi Yamamuro wrote
> +1 for releasing 2.4.8 and thanks, Liang-chi, for volunteering.
> Btw, anyone roughly know how many v2.4 users still are based on some stats
> (e.g., # of v2.4.7 downloads from the official repos)?
> Most users have started using v3.x?
>
> On Thu, Mar 4, 2021 at 8:34 AM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>
>> Yeah, I would prefer to have a 2.4.8 release as an EOL too. I don't mind
>> having 2.4.9 as EOL too if that's preferred from more people.
>>
> Takeshi Yamamuro





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Hyukjin Kwon
Thanks [hidden email] for driving this.

2021년 3월 5일 (금) 오전 5:21, Liang-Chi Hsieh <[hidden email]>님이 작성:

Thanks all for the input.

If there is no objection, I am going to cut the branch next Monday.

Thanks.
Liang-Chi


Takeshi Yamamuro wrote
> +1 for releasing 2.4.8 and thanks, Liang-chi, for volunteering.
> Btw, anyone roughly know how many v2.4 users still are based on some stats
> (e.g., # of v2.4.7 downloads from the official repos)?
> Most users have started using v3.x?
>
> On Thu, Mar 4, 2021 at 8:34 AM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>
>> Yeah, I would prefer to have a 2.4.8 release as an EOL too. I don't mind
>> having 2.4.9 as EOL too if that's preferred from more people.
>>
> Takeshi Yamamuro





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Xiao Li-2
Thank you, Liang-Chi!

Xiao

On Thu, Mar 4, 2021 at 6:25 PM Hyukjin Kwon <[hidden email]> wrote:
Thanks [hidden email] for driving this.

2021년 3월 5일 (금) 오전 5:21, Liang-Chi Hsieh <[hidden email]>님이 작성:

Thanks all for the input.

If there is no objection, I am going to cut the branch next Monday.

Thanks.
Liang-Chi


Takeshi Yamamuro wrote
> +1 for releasing 2.4.8 and thanks, Liang-chi, for volunteering.
> Btw, anyone roughly know how many v2.4 users still are based on some stats
> (e.g., # of v2.4.7 downloads from the official repos)?
> Most users have started using v3.x?
>
> On Thu, Mar 4, 2021 at 8:34 AM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>
>> Yeah, I would prefer to have a 2.4.8 release as an EOL too. I don't mind
>> having 2.4.9 as EOL too if that's preferred from more people.
>>
> Takeshi Yamamuro





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Dongjoon Hyun-2
Thank you, Liang-Chi! Next Monday sounds good.

To All. Please ping Liang-Chi if you have a missed backport.

Bests,
Dongjoon.



On Thu, Mar 4, 2021 at 7:00 PM Xiao Li <[hidden email]> wrote:
Thank you, Liang-Chi!

Xiao

On Thu, Mar 4, 2021 at 6:25 PM Hyukjin Kwon <[hidden email]> wrote:
Thanks [hidden email] for driving this.

2021년 3월 5일 (금) 오전 5:21, Liang-Chi Hsieh <[hidden email]>님이 작성:

Thanks all for the input.

If there is no objection, I am going to cut the branch next Monday.

Thanks.
Liang-Chi


Takeshi Yamamuro wrote
> +1 for releasing 2.4.8 and thanks, Liang-chi, for volunteering.
> Btw, anyone roughly know how many v2.4 users still are based on some stats
> (e.g., # of v2.4.7 downloads from the official repos)?
> Most users have started using v3.x?
>
> On Thu, Mar 4, 2021 at 8:34 AM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>
>> Yeah, I would prefer to have a 2.4.8 release as an EOL too. I don't mind
>> having 2.4.9 as EOL too if that's preferred from more people.
>>
> Takeshi Yamamuro





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh

Thank you Dongjoon.

I'm going to cut the branch now. Hopefully I can make it soon (need to get
familiar with the process as first time :) )

Liang-Chi


Dongjoon Hyun-2 wrote
> Thank you, Liang-Chi! Next Monday sounds good.
>
> To All. Please ping Liang-Chi if you have a missed backport.
>
> Bests,
> Dongjoon.





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh
Hi devs,

I'm going to cut the branch yesterday. I'd like to share current progress. I
hit a problem during dry run of the release script. Fixed it in SPARK-34672.

The latest dry run looks okay as build, docs, publish all success. But the
last step (push the tag) has a fatal error, I'm not sure if it is due to dry
run mode?

========================
= Creating release tag v2.4.8-rc1...
Command: /opt/spark-rm/release-tag.sh
Log file: tag.log
========================
= Building Spark...
Command: /opt/spark-rm/release-build.sh package
Log file: build.log
========================
= Building documentation...
Command: /opt/spark-rm/release-build.sh docs
Log file: docs.log
========================
= Publishing release
Command: /opt/spark-rm/release-build.sh publish-release
Log file: publish.log
fatal: Not a git repository (or any parent up to mount point /opt/spark-rm)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).


Currently the 2.4 related Jenkins QA test aren't green. Some are not test
failure but

 > git fetch --tags --progress https://github.com/apache/spark.git
+refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Timeout after 10 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/spark.git
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:996)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1237)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297)
        at hudson.scm.SCM.checkout(SCM.java:505)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1206)
        at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
        at hudson.model.Run.execute(Run.java:1894)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:428)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags
--progress https://github.com/apache/spark.git
+refs/heads/*:refs/remotes/origin/*" returned status code 128:


But there is also one test failure:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.4-test-sbt-hadoop-2.7/1221/console

I will take a look.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh
I just contacted Shane and seems there is ongoing github fetches timing out
issue on Jenkins.

That being said, currently the QA test is unavailable. I guess it is unsafe
to make a release cut due to lack of reliable QA test result.

I may defer the cut until QA test comes back if no objection.

WDYT?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Dongjoon Hyun-2
Thank you for the update.

+1 for your plan.

Bests,
Dongjoon.

On Tue, Mar 9, 2021 at 12:46 PM Liang-Chi Hsieh <[hidden email]> wrote:
I just contacted Shane and seems there is ongoing github fetches timing out
issue on Jenkins.

That being said, currently the QA test is unavailable. I guess it is unsafe
to make a release cut due to lack of reliable QA test result.

I may defer the cut until QA test comes back if no objection.

WDYT?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh
To update with current status.

There are three tickets targeting 2.4 that are still ongoing.

SPARK-34719: Correctly resolve the view query with duplicated column names
SPARK-34607: Add `Utils.isMemberClass` to fix a malformed class name error
on jdk8u
SPARK-34726: Fix collectToPython timeouts

SPARK-34719 doesn't have PR for 2.4 yet.

SPARK-34607 and SPARK-34726 are under review. SPARK-34726 is a bit arguable
as it involves a behavior change even it is very rare case. Welcome any
suggestion on the PR if any. Thanks.



Dongjoon Hyun-2 wrote
> Thank you for the update.
>
> +1 for your plan.
>
> Bests,
> Dongjoon.





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Takeshi Yamamuro
Hi, viirya

I'm looking now into "SPARK-34607: Add `Utils.isMemberClass` to fix a malformed class name error
on jdk8u" .

Bests,
Takeshi

On Tue, Mar 16, 2021 at 4:45 AM Liang-Chi Hsieh <[hidden email]> wrote:
To update with current status.

There are three tickets targeting 2.4 that are still ongoing.

SPARK-34719: Correctly resolve the view query with duplicated column names
SPARK-34607: Add `Utils.isMemberClass` to fix a malformed class name error
on jdk8u
SPARK-34726: Fix collectToPython timeouts

SPARK-34719 doesn't have PR for 2.4 yet.

SPARK-34607 and SPARK-34726 are under review. SPARK-34726 is a bit arguable
as it involves a behavior change even it is very rare case. Welcome any
suggestion on the PR if any. Thanks.



Dongjoon Hyun-2 wrote
> Thank you for the update.
>
> +1 for your plan.
>
> Bests,
> Dongjoon.





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh
Thank you so much, Takeshi!


Takeshi Yamamuro wrote

> Hi, viirya
>
> I'm looking now into "SPARK-34607: Add `Utils.isMemberClass` to fix a
> malformed class name error
> on jdk8u" .
>
> Bests,
> Takeshi
>
>
> Takeshi Yamamuro





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Liang-Chi Hsieh
To update current status.

The only one remaining issue for 2.4 is:

[SPARK-34855][CORE]spark context - avoid using local lazy val for callSite

We are waiting the author to submit a PR for 2.4 branch.




Liang-Chi Hsieh wrote

> Thank you so much, Takeshi!
>
>
> Takeshi Yamamuro wrote
>> Hi, viirya
>>
>> I'm looking now into "SPARK-34607: Add `Utils.isMemberClass` to fix a
>> malformed class name error
>> on jdk8u" .
>>
>> Bests,
>> Takeshi
>>
>>
>> Takeshi Yamamuro
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

> dev-unsubscribe@.apache





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

12