drop java 7 support for spark 2.1.x or spark 2.2.x

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

drop java 7 support for spark 2.1.x or spark 2.2.x

Koert Kuipers

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Mark Hamstra
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Koert Kuipers

i care about signalling it in advance mostly. and given the performance differences we do have some interest in pushing towards java 8


On Jul 23, 2016 6:10 PM, "Mark Hamstra" <[hidden email]> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Mark Hamstra
Sure, signalling well ahead of time is good, as is getting better performance from Java 8; but do either of those interests really require dropping Java 7 support sooner rather than later?

Now, to retroactively copy edit myself, when I previously wrote "after all or nearly all relevant clusters are actually no longer running on Java 6", I meant "...no longer running on Java 7".  We should be at a point now where there aren't many Java 6 clusters left, but my sense is that there are still quite a number of Java 7 clusters around, and that there will be for a good while still.

On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <[hidden email]> wrote:

i care about signalling it in advance mostly. and given the performance differences we do have some interest in pushing towards java 8


On Jul 23, 2016 6:10 PM, "Mark Hamstra" <[hidden email]> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Koert Kuipers

They don't require dropping it sooner rather than later. But signalling in some way that java 8 is (strongly) recommend would be good.


On Jul 23, 2016 6:59 PM, "Mark Hamstra" <[hidden email]> wrote:
Sure, signalling well ahead of time is good, as is getting better performance from Java 8; but do either of those interests really require dropping Java 7 support sooner rather than later?

Now, to retroactively copy edit myself, when I previously wrote "after all or nearly all relevant clusters are actually no longer running on Java 6", I meant "...no longer running on Java 7".  We should be at a point now where there aren't many Java 6 clusters left, but my sense is that there are still quite a number of Java 7 clusters around, and that there will be for a good while still.

On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <[hidden email]> wrote:

i care about signalling it in advance mostly. and given the performance differences we do have some interest in pushing towards java 8


On Jul 23, 2016 6:10 PM, "Mark Hamstra" <[hidden email]> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Dongjoon Hyun
Hi, All.

What about providing a official benchmark result between `Apache Spark on JDK7` and `Apache Spark on JDK8`?
I think that is enough for this issue since we cannot drive users.
We had better let users choose one of JDK7/JDK8 for their own benefits.

Bests,
Dongjoon.



On Sat, Jul 23, 2016 at 4:35 PM, Koert Kuipers <[hidden email]> wrote:

They don't require dropping it sooner rather than later. But signalling in some way that java 8 is (strongly) recommend would be good.


On Jul 23, 2016 6:59 PM, "Mark Hamstra" <[hidden email]> wrote:
Sure, signalling well ahead of time is good, as is getting better performance from Java 8; but do either of those interests really require dropping Java 7 support sooner rather than later?

Now, to retroactively copy edit myself, when I previously wrote "after all or nearly all relevant clusters are actually no longer running on Java 6", I meant "...no longer running on Java 7".  We should be at a point now where there aren't many Java 6 clusters left, but my sense is that there are still quite a number of Java 7 clusters around, and that there will be for a good while still.

On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <[hidden email]> wrote:

i care about signalling it in advance mostly. and given the performance differences we do have some interest in pushing towards java 8


On Jul 23, 2016 6:10 PM, "Mark Hamstra" <[hidden email]> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Sean Owen
In reply to this post by Koert Kuipers
I had favored this for 2.0 even, so favor it sooner than later.
The general shape of the argument is:

- supporting Java 7 is starting to pinch a little because of extra
builds and the inevitable gap between what the PR builder (7) tests
and what the later Java 8 tests runs show
- requiring Java 8 would allow a few minor cleanups in the code
- Java 7 is already EOL (no, I don't count paying Oracle for support)

but mostly importantly,
- anyone can keep using 2.0 if they want to use Java 7. This does
*not* require people to use Java 8. It requires them to use Java 8 to
use the very latest version.

On Sat, Jul 23, 2016 at 10:54 PM, Koert Kuipers <[hidden email]> wrote:

> dropping java 7 support was considered for spark 2.0.x but we decided
> against it.
>
> ideally dropping support for a java version should be communicated far in
> advance to facilitate the transition.
>
> is this the right time to make that decision and start communicating it
> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>
> my general sense is that most cluster admins have plans to migrate to java 8
> before end of year. so that could line up nicely with spark 2.2
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Ofir Manor
In reply to this post by Koert Kuipers
BTW - "signalling ahead of time" is called deprecating, not dropping support...
(personally I only use JDK 8 / Scala 2.11 so I'm for it)


Ofir Manor

Co-Founder & CTO | Equalum

Mobile: <a href="tel:%2B972-54-7801286" value="+972507470820" target="_blank">+972-54-7801286 | Email: [hidden email]


On Sun, Jul 24, 2016 at 1:50 AM, Koert Kuipers <[hidden email]> wrote:

i care about signalling it in advance mostly. and given the performance differences we do have some interest in pushing towards java 8


On Jul 23, 2016 6:10 PM, "Mark Hamstra" <[hidden email]> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

Steve Loughran
In reply to this post by Mark Hamstra

On 23 Jul 2016, at 23:59, Mark Hamstra <[hidden email]> wrote:

Sure, signalling well ahead of time is good, as is getting better performance from Java 8; but do either of those interests really require dropping Java 7 support sooner rather than later?

Now, to retroactively copy edit myself, when I previously wrote "after all or nearly all relevant clusters are actually no longer running on Java 6", I meant "...no longer running on Java 7".  We should be at a point now where there aren't many Java 6 clusters left, but my sense is that there are still quite a number of Java 7 clusters around, and that there will be for a good while still.



  1. There are a lot of Hadoop clusters on Java 7; the migration to Java 8 there is pretty slow due to inertia of existing apps.
  2. But...you can run Java 8 code in a Hadoop cluster, if you set up the JVM properly
  3. And orgs playing with spark tend not to be the ones running older Hadoop versions

Ignoring runtime issues, mixing versions of: Java & Scala complicate the dev and test matrix; getting a Java 7 JDK to build and test on is getting harder and harder, so most developers are using Java 8. Even if using java 7 language, there's the differences in the JRE libraries. Scala code has it easier language wise, but there are enough diffs from 2.10 and 2.11 that again, life is complex. There's invariably a conflict between developers "I want to play with the latest toys!" and ops "It worked in 2012, let's keep using it"

Meanwhile, Java 9 is coming, which is going to break a lot of things, log4j 1.x included.

This is going to be fairly traumatic all round, as log4j 2, while a better design, doesn't parse the 1.x properties files (https://blogs.apache.org/logging/entry/moving_on_to_log4j_2). 

While Java 9 is something nobody will use in production this year, it's coming, and it's something which has a lot of consequences up the stack. Being able to stop worrying about java 7 backwards compatibility would save time which can now be directed to worrying about java 9. Indeed, the ASF builds team are being offered Java 9 EARs, along with Java 8u112"

---------------
From: Rory O'Donnell <[hidden email]>
Subject: Early Access builds of JDK 8u112 b03, JDK 9 b128 are available on java.net
Date: 22 July 2016 at 10:30:19 BST
Cc: <[hidden email]>, Dalibor Topic <[hidden email]>, Balchandra Vaidya <[hidden email]>, Muneer Kolarkunnu <[hidden email]>, <[hidden email]>, <[hidden email]>
Reply-To: <[hidden email]>


Hi Mark & Gavin,

Early Access b128 <https://jdk9.java.net/download/> for JDK 9 is available on java.net, summary of  changes are listed here <http://www.java.net/download/java/jdk9/changes/jdk-9+128.html>.

Early Access b127 <https://jdk9.java.net/jigsaw/> (#5304) for JDK 9 with Project Jigsaw is available on java.net, summary of changes are listed here <http://download.java.net/java/jigsaw/archive/127/binaries/jdk-9+127.html>

Early Access b03 <https://jdk8.java.net/download.html> for JDK 8u112 is available on java.net, summary of  changes are listed here <http://www.java.net/download/java/jdk8u112/changes/jdk8u112-b03.html>

Alan Bateman posted new EA builds contain initial implementation of current proposals , more info [0]

  The jigsaw/jake forest has been updated with an initial
  implementation of the proposals that Mark brought to the
  jpms-spec-experts mailing list last week. For those that don't build
  from source then the EA build/downloads [1] has also been refreshed.


Rgds,Rory

[0] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008467.html
[1] https://jdk9.java.net/jigsaw/

-- 
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

rxin
In reply to this post by Mark Hamstra
Bumping this.

Given we see the occassional build breaks with Java 8, we should reconsider this and do it for 2.2 or 2.3. By the time 2.2 is released, it will almost be an year since this thread started.


On Sun, Jul 24, 2016 at 12:59 AM, Mark Hamstra <[hidden email]> wrote:
Sure, signalling well ahead of time is good, as is getting better performance from Java 8; but do either of those interests really require dropping Java 7 support sooner rather than later?

Now, to retroactively copy edit myself, when I previously wrote "after all or nearly all relevant clusters are actually no longer running on Java 6", I meant "...no longer running on Java 7".  We should be at a point now where there aren't many Java 6 clusters left, but my sense is that there are still quite a number of Java 7 clusters around, and that there will be for a good while still.

On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <[hidden email]> wrote:

i care about signalling it in advance mostly. and given the performance differences we do have some interest in pushing towards java 8


On Jul 23, 2016 6:10 PM, "Mark Hamstra" <[hidden email]> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: drop java 7 support for spark 2.1.x or spark 2.2.x

rxin
BTW I created a JIRA ticket for tracking: https://issues.apache.org/jira/browse/SPARK-19493

We of course shouldn't do anything until we achieve consensus.


On Tue, Feb 7, 2017 at 3:47 PM, Reynold Xin <[hidden email]> wrote:
Bumping this.

Given we see the occassional build breaks with Java 8, we should reconsider this and do it for 2.2 or 2.3. By the time 2.2 is released, it will almost be an year since this thread started.


On Sun, Jul 24, 2016 at 12:59 AM, Mark Hamstra <[hidden email]> wrote:
Sure, signalling well ahead of time is good, as is getting better performance from Java 8; but do either of those interests really require dropping Java 7 support sooner rather than later?

Now, to retroactively copy edit myself, when I previously wrote "after all or nearly all relevant clusters are actually no longer running on Java 6", I meant "...no longer running on Java 7".  We should be at a point now where there aren't many Java 6 clusters left, but my sense is that there are still quite a number of Java 7 clusters around, and that there will be for a good while still.

On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <[hidden email]> wrote:

i care about signalling it in advance mostly. and given the performance differences we do have some interest in pushing towards java 8


On Jul 23, 2016 6:10 PM, "Mark Hamstra" <[hidden email]> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I read your "cluster admins plan to migrate by date X, so Spark should end Java 7 support then, too")?  First, I don't think we should be removing Java 7 support until some time after all or nearly all relevant clusters are actually no longer running on Java 6, and that targeting removal of support at our best guess about when admins are just *planning* to migrate isn't a very good idea.  Second, I don't see the significant difficulty or harm in continuing to support Java 7 for a while longer.

On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <[hidden email]> wrote:

dropping java 7 support was considered for spark 2.0.x but we decided against it.

ideally dropping support for a java version should be communicated far in advance to facilitate the transition.

is this the right time to make that decision and start communicating it (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?

my general sense is that most cluster admins have plans to migrate to java 8 before end of year. so that could line up nicely with spark 2.2






Loading...