[DISCUSSION]JDK11 for Apache 2.x?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSSION]JDK11 for Apache 2.x?

DB Tsai-7
Hello everyone,

Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a community.

Java 8 is already end of life for commercial users, and many companies are moving to Java 11.
The release date for Apache Spark 3.0 is still not there yet, and there are many API
incompatibility issues when upgrading from Spark 2.x. As a result, asking users to move to
Spark 3.0 to use JDK 11 is not realistic.

Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?

Should we cut a new Apache Spark 2.5 since the patches involve some of the dependencies changes
which is not desired in minor release?

Thanks.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION]JDK11 for Apache 2.x?

Holden Karau


On Tue, Aug 27, 2019 at 12:01 PM DB Tsai <[hidden email]> wrote:
Hello everyone,

Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a community.

Java 8 is already end of life for commercial users, and many companies are moving to Java 11.
The release date for Apache Spark 3.0 is still not there yet, and there are many API
incompatibility issues when upgrading from Spark 2.x. As a result, asking users to move to
Spark 3.0 to use JDK 11 is not realistic.

Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?

Should we cut a new Apache Spark 2.5 since the patches involve some of the dependencies changes
which is not desired in minor release?
I think a LTS 2.5 branch with JDK 11 would make sense given the state of Java 8.
I think backporting to 2.4 would be not ideal.


Thanks.

DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION]JDK11 for Apache 2.x?

Sean Owen-2
In reply to this post by DB Tsai-7
I think one of the key problems here are the required dependency
upgrades. It would mean many minor breaking changes and a few bigger
ones, notably around Hive, and forces a scala 2.12-only update. I
think my question is whether that even makes sense as a minor release?
it wouldn't be backwards compatible with 2.4 enough to call it a
low-risk update. It would be a smaller step than moving all the way to
3.0, sure. I am not super against it, but we have to keep in mind how
much work it would then be to maintain two LTS 2.x releases, 2.4 and
the sort-of-compatible 2.5, while proceeding with 3.x.

On Tue, Aug 27, 2019 at 2:01 PM DB Tsai <[hidden email]> wrote:

>
> Hello everyone,
>
> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a community.
>
> Java 8 is already end of life for commercial users, and many companies are moving to Java 11.
> The release date for Apache Spark 3.0 is still not there yet, and there are many API
> incompatibility issues when upgrading from Spark 2.x. As a result, asking users to move to
> Spark 3.0 to use JDK 11 is not realistic.
>
> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
>
> Should we cut a new Apache Spark 2.5 since the patches involve some of the dependencies changes
> which is not desired in minor release?
>
> Thanks.
>
> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION]JDK11 for Apache 2.x?

Jean Georges Perrin
Not a contributor, but a user perspective…

As Spark 3.x will be an evolution, I am not completely shocked that it would imply a Java 11 requirement as well. Would be great to have both Java 8 and Java 11, but one needs to be able to say goodbye. Java 8 is great, still using it actively in production, but we know its time is limited, so, by the time we evolve to Spark 3, we could combine it with Java 11.

On the other hand, not everybody may think this way and it may slow down the adoption of Spark 3…

However, I concur with Sean, I don’t think another 2.x is needed for Java 11.

> On Aug 27, 2019, at 3:09 PM, Sean Owen <[hidden email]> wrote:
>
> I think one of the key problems here are the required dependency
> upgrades. It would mean many minor breaking changes and a few bigger
> ones, notably around Hive, and forces a scala 2.12-only update. I
> think my question is whether that even makes sense as a minor release?
> it wouldn't be backwards compatible with 2.4 enough to call it a
> low-risk update. It would be a smaller step than moving all the way to
> 3.0, sure. I am not super against it, but we have to keep in mind how
> much work it would then be to maintain two LTS 2.x releases, 2.4 and
> the sort-of-compatible 2.5, while proceeding with 3.x.
>
> On Tue, Aug 27, 2019 at 2:01 PM DB Tsai <[hidden email]> wrote:
>>
>> Hello everyone,
>>
>> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a community.
>>
>> Java 8 is already end of life for commercial users, and many companies are moving to Java 11.
>> The release date for Apache Spark 3.0 is still not there yet, and there are many API
>> incompatibility issues when upgrading from Spark 2.x. As a result, asking users to move to
>> Spark 3.0 to use JDK 11 is not realistic.
>>
>> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
>>
>> Should we cut a new Apache Spark 2.5 since the patches involve some of the dependencies changes
>> which is not desired in minor release?
>>
>> Thanks.
>>
>> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION]JDK11 for Apache 2.x?

Sean Owen-2
Spark 3 will not require Java 11; it will work with Java 8 too. I
think the question is whether someone who _wants_ Java 11 should have
a 2.x release instead of 3.0.

In practice... how much are people moving off Java 8 right now? It's
still my default, because most but not all things work with 11. (Here,
I'm using 11 as shorthand for 9+; it's the LTS release in the 9 to 11
series).
I get that the issue is support, but it seems that Oracle is still
providing public updates through Dec 2020. I think it's that
commercial support is now for-pay? And OpenJDK 8 still gets updates?
that I'm unclear on.

Not that those aren't legitimate concerns, I'm just trying to figure
out how much need vs want there is and why out there.


On Tue, Aug 27, 2019 at 4:15 PM Jean Georges Perrin <[hidden email]> wrote:

>
> Not a contributor, but a user perspective…
>
> As Spark 3.x will be an evolution, I am not completely shocked that it would imply a Java 11 requirement as well. Would be great to have both Java 8 and Java 11, but one needs to be able to say goodbye. Java 8 is great, still using it actively in production, but we know its time is limited, so, by the time we evolve to Spark 3, we could combine it with Java 11.
>
> On the other hand, not everybody may think this way and it may slow down the adoption of Spark 3…
>
> However, I concur with Sean, I don’t think another 2.x is needed for Java 11.
>
> > On Aug 27, 2019, at 3:09 PM, Sean Owen <[hidden email]> wrote:
> >
> > I think one of the key problems here are the required dependency
> > upgrades. It would mean many minor breaking changes and a few bigger
> > ones, notably around Hive, and forces a scala 2.12-only update. I
> > think my question is whether that even makes sense as a minor release?
> > it wouldn't be backwards compatible with 2.4 enough to call it a
> > low-risk update. It would be a smaller step than moving all the way to
> > 3.0, sure. I am not super against it, but we have to keep in mind how
> > much work it would then be to maintain two LTS 2.x releases, 2.4 and
> > the sort-of-compatible 2.5, while proceeding with 3.x.
> >
> > On Tue, Aug 27, 2019 at 2:01 PM DB Tsai <[hidden email]> wrote:
> >>
> >> Hello everyone,
> >>
> >> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a community.
> >>
> >> Java 8 is already end of life for commercial users, and many companies are moving to Java 11.
> >> The release date for Apache Spark 3.0 is still not there yet, and there are many API
> >> incompatibility issues when upgrading from Spark 2.x. As a result, asking users to move to
> >> Spark 3.0 to use JDK 11 is not realistic.
> >>
> >> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
> >>
> >> Should we cut a new Apache Spark 2.5 since the patches involve some of the dependencies changes
> >> which is not desired in minor release?
> >>
> >> Thanks.
> >>
> >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: [hidden email]
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION]JDK11 for Apache 2.x?

Holden Karau
So I'm not a lawyer, but looking at the JDK8 License FAQ ( https://www.oracle.com/technetwork/java/javase/overview/oracle-jdk-faqs.html ) makes me feel like we should support folks who are on the 2.x branch and want to migrate away from JDK8. Although the Redhat post about OpenJDK8 is reassuring ( https://developers.redhat.com/blog/2019/05/22/openjdk-8-and-11-still-in-safe-hands/ ).

On Tue, Aug 27, 2019 at 2:22 PM Sean Owen <[hidden email]> wrote:
Spark 3 will not require Java 11; it will work with Java 8 too. I
think the question is whether someone who _wants_ Java 11 should have
a 2.x release instead of 3.0.

In practice... how much are people moving off Java 8 right now? It's
still my default, because most but not all things work with 11. (Here,
I'm using 11 as shorthand for 9+; it's the LTS release in the 9 to 11
series).
I get that the issue is support, but it seems that Oracle is still
providing public updates through Dec 2020. I think it's that
commercial support is now for-pay? And OpenJDK 8 still gets updates?
that I'm unclear on.

Not that those aren't legitimate concerns, I'm just trying to figure
out how much need vs want there is and why out there.


On Tue, Aug 27, 2019 at 4:15 PM Jean Georges Perrin <[hidden email]> wrote:
>
> Not a contributor, but a user perspective…
>
> As Spark 3.x will be an evolution, I am not completely shocked that it would imply a Java 11 requirement as well. Would be great to have both Java 8 and Java 11, but one needs to be able to say goodbye. Java 8 is great, still using it actively in production, but we know its time is limited, so, by the time we evolve to Spark 3, we could combine it with Java 11.
>
> On the other hand, not everybody may think this way and it may slow down the adoption of Spark 3…
>
> However, I concur with Sean, I don’t think another 2.x is needed for Java 11.
>
> > On Aug 27, 2019, at 3:09 PM, Sean Owen <[hidden email]> wrote:
> >
> > I think one of the key problems here are the required dependency
> > upgrades. It would mean many minor breaking changes and a few bigger
> > ones, notably around Hive, and forces a scala 2.12-only update. I
> > think my question is whether that even makes sense as a minor release?
> > it wouldn't be backwards compatible with 2.4 enough to call it a
> > low-risk update. It would be a smaller step than moving all the way to
> > 3.0, sure. I am not super against it, but we have to keep in mind how
> > much work it would then be to maintain two LTS 2.x releases, 2.4 and
> > the sort-of-compatible 2.5, while proceeding with 3.x.
> >
> > On Tue, Aug 27, 2019 at 2:01 PM DB Tsai <[hidden email]> wrote:
> >>
> >> Hello everyone,
> >>
> >> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a community.
> >>
> >> Java 8 is already end of life for commercial users, and many companies are moving to Java 11.
> >> The release date for Apache Spark 3.0 is still not there yet, and there are many API
> >> incompatibility issues when upgrading from Spark 2.x. As a result, asking users to move to
> >> Spark 3.0 to use JDK 11 is not realistic.
> >>
> >> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
> >>
> >> Should we cut a new Apache Spark 2.5 since the patches involve some of the dependencies changes
> >> which is not desired in minor release?
> >>
> >> Thanks.
> >>
> >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: [hidden email]
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION]JDK11 for Apache 2.x?

Sean Owen-2
That's just the Oracle 'commercial' JDK though. OpenJDK is still OSS
and 8 is updated there too. Oracle JDK is still free too, just not for
commercial use and commercial support for 8 (IIUC). I think that's the
'audience' here, and I don't know how large or small that is.

That said a key first question is simply, how much breaking change is
necessary to get 2.x to work on JDK 11? If it's large I think that
might resolve the question right there. SPARK-24417 has most but not
necessarily all necessary changes under it.

Heck it could be maintained as a separate community project too, if
there is both interest from some parties to make it happen, but not
enough to commit to maintain all the backports and updated from 3.x.

On Sun, Sep 1, 2019 at 9:04 PM Holden Karau <[hidden email]> wrote:

>
> So I'm not a lawyer, but looking at the JDK8 License FAQ ( https://www.oracle.com/technetwork/java/javase/overview/oracle-jdk-faqs.html ) makes me feel like we should support folks who are on the 2.x branch and want to migrate away from JDK8. Although the Redhat post about OpenJDK8 is reassuring ( https://developers.redhat.com/blog/2019/05/22/openjdk-8-and-11-still-in-safe-hands/ ).
>
> On Tue, Aug 27, 2019 at 2:22 PM Sean Owen <[hidden email]> wrote:
>>
>> Spark 3 will not require Java 11; it will work with Java 8 too. I
>> think the question is whether someone who _wants_ Java 11 should have
>> a 2.x release instead of 3.0.
>>
>> In practice... how much are people moving off Java 8 right now? It's
>> still my default, because most but not all things work with 11. (Here,
>> I'm using 11 as shorthand for 9+; it's the LTS release in the 9 to 11
>> series).
>> I get that the issue is support, but it seems that Oracle is still
>> providing public updates through Dec 2020. I think it's that
>> commercial support is now for-pay? And OpenJDK 8 still gets updates?
>> that I'm unclear on.
>>
>> Not that those aren't legitimate concerns, I'm just trying to figure
>> out how much need vs want there is and why out there.
>>
>>
>> On Tue, Aug 27, 2019 at 4:15 PM Jean Georges Perrin <[hidden email]> wrote:
>> >
>> > Not a contributor, but a user perspective…
>> >
>> > As Spark 3.x will be an evolution, I am not completely shocked that it would imply a Java 11 requirement as well. Would be great to have both Java 8 and Java 11, but one needs to be able to say goodbye. Java 8 is great, still using it actively in production, but we know its time is limited, so, by the time we evolve to Spark 3, we could combine it with Java 11.
>> >
>> > On the other hand, not everybody may think this way and it may slow down the adoption of Spark 3…
>> >
>> > However, I concur with Sean, I don’t think another 2.x is needed for Java 11.
>> >
>> > > On Aug 27, 2019, at 3:09 PM, Sean Owen <[hidden email]> wrote:
>> > >
>> > > I think one of the key problems here are the required dependency
>> > > upgrades. It would mean many minor breaking changes and a few bigger
>> > > ones, notably around Hive, and forces a scala 2.12-only update. I
>> > > think my question is whether that even makes sense as a minor release?
>> > > it wouldn't be backwards compatible with 2.4 enough to call it a
>> > > low-risk update. It would be a smaller step than moving all the way to
>> > > 3.0, sure. I am not super against it, but we have to keep in mind how
>> > > much work it would then be to maintain two LTS 2.x releases, 2.4 and
>> > > the sort-of-compatible 2.5, while proceeding with 3.x.
>> > >
>> > > On Tue, Aug 27, 2019 at 2:01 PM DB Tsai <[hidden email]> wrote:
>> > >>
>> > >> Hello everyone,
>> > >>
>> > >> Thank you all for working on supporting JDK11 in Apache Spark 3.0 as a community.
>> > >>
>> > >> Java 8 is already end of life for commercial users, and many companies are moving to Java 11.
>> > >> The release date for Apache Spark 3.0 is still not there yet, and there are many API
>> > >> incompatibility issues when upgrading from Spark 2.x. As a result, asking users to move to
>> > >> Spark 3.0 to use JDK 11 is not realistic.
>> > >>
>> > >> Should we backport PRs for JDK11 and cut a release in 2.x to support JDK11?
>> > >>
>> > >> Should we cut a new Apache Spark 2.5 since the patches involve some of the dependencies changes
>> > >> which is not desired in minor release?
>> > >>
>> > >> Thanks.
>> > >>
>> > >> DB Tsai  |  Siri Open Source Technologies [not a contribution]  |   Apple, Inc
>> > >>
>> > >>
>> > >> ---------------------------------------------------------------------
>> > >> To unsubscribe e-mail: [hidden email]
>> > >>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe e-mail: [hidden email]
>> > >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: [hidden email]
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]