Spark on JDK 14

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark on JDK 14

Sean Owen-3
For kicks, I tried Spark on JDK 14. 11 -> 14 doesn't change much, not as much as 8 -> 9 (-> 11), and indeed, virtually all tests pass. For the interested, these two seem to fail:

- ZooKeeperPersistenceEngine *** FAILED ***
  org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /spark/master_status

- parsing hour with various patterns *** FAILED ***
  java.time.format.DateTimeParseException: Text '2009-12-12 12 am' could not be parsed: Invalid value for HourOfAmPm (valid values 0 - 11): 12

I'd expect that most applications would just work now on Spark 3 + Java 14. I'd guess the same is true for Java 16 even, but, we're probably focused on the LTS releases.

Kris Mok pointed out that Project Panama (in Java 17 maybe?) might have implications as it changes off-heap memory access.
Reply | Threaded
Open this post in threaded view
|

Re: Spark on JDK 14

Dongjoon Hyun-2
Thank you for the sharing, Sean.

Although Java 14 is already EOL (Sep. 2020), that is important information because we are tracking the Java upstream.

Bests,
Dongjoon.

On Wed, Oct 28, 2020 at 1:44 PM Sean Owen <[hidden email]> wrote:
For kicks, I tried Spark on JDK 14. 11 -> 14 doesn't change much, not as much as 8 -> 9 (-> 11), and indeed, virtually all tests pass. For the interested, these two seem to fail:

- ZooKeeperPersistenceEngine *** FAILED ***
  org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /spark/master_status

- parsing hour with various patterns *** FAILED ***
  java.time.format.DateTimeParseException: Text '2009-12-12 12 am' could not be parsed: Invalid value for HourOfAmPm (valid values 0 - 11): 12

I'd expect that most applications would just work now on Spark 3 + Java 14. I'd guess the same is true for Java 16 even, but, we're probably focused on the LTS releases.

Kris Mok pointed out that Project Panama (in Java 17 maybe?) might have implications as it changes off-heap memory access.
Reply | Threaded
Open this post in threaded view
|

Re: Spark on JDK 14

Kazuaki Ishizaki
Java 16 will also includes Vector API (incubator), which is a part of Project Panama, as shown in https://mail.openjdk.java.net/pipermail/panama-dev/2020-October/011149.html

When the next LTS will be available, we could exploit it in Spark.

Kazuaki Ishizaki



From:        Dongjoon Hyun <[hidden email]>
To:        Sean Owen <[hidden email]>
Cc:        dev <[hidden email]>
Date:        2020/10/29 11:34
Subject:        Re: Spark on JDK 14



Thank you for the sharing, Sean.

Although Java 14 is already EOL (Sep. 2020), that is important information because we are tracking the Java upstream.

Bests,
Dongjoon.

On Wed, Oct 28, 2020 at 1:44 PM Sean Owen <srowen@...> wrote:
For kicks, I tried Spark on JDK 14. 11 -> 14 doesn't change much, not as much as 8 -> 9 (-> 11), and indeed, virtually all tests pass. For the interested, these two seem to fail:

- ZooKeeperPersistenceEngine *** FAILED ***
  org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /spark/master_status

- parsing hour with various patterns *** FAILED ***
  java.time.format.DateTimeParseException: Text '2009-12-12 12 am' could not be parsed: Invalid value for HourOfAmPm (valid values 0 - 11): 12


I'd expect that most applications would just work now on Spark 3 + Java 14. I'd guess the same is true for Java 16 even, but, we're probably focused on the LTS releases.

Kris Mok pointed out that Project Panama (in Java 17 maybe?) might have implications as it changes off-heap memory access.