Status of Scala 2.13 support

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Status of Scala 2.13 support

Sean Owen-3
As you can see, I've been working on Scala 2.13 support. The umbrella
is https://issues.apache.org/jira/browse/SPARK-25075 I wanted to lay
out status and strategy.

This will not be done for 3.0. At the least, there are a few key
dependencies (Chill, Kafka) that aren't published for 2.13, and at
least one change that will need removing an API deprecated as of 3.0.
Realistically: maybe Spark 3.1. I don't yet think it's pressing.


Making the change is difficult as it's hard to understand the extent
of the necessary changes until the whole thing minimally compiles for
2.13. I have gotten essentially that far in a local clone. The good
news is I don't see any obvious hard blockers, but the changes add up
to thousands of line in 200+ files.


What do we need to do for 3.0? any changes that entail breaking a
public API, ideally. The biggest issue there comes from extensive
changes to the Scala collection hierarchy mean that the types of many
public APIs that return a Seq, Map, TraversableOnce, etc _will_
actually change types in 2.13 (become immutable). See:
https://issues.apache.org/jira/browse/SPARK-27683 and
https://issues.apache.org/jira/browse/SPARK-29292 as the main
examples.

In both cases, keeping the exact same public type would require much
bigger changes. These are the type of changes that all applications
face when migrating to 2.13 though. 2.12 and 2.13 apps were never
meant to be binary-compatible. So, in both cases we're not changing
these, to avoid a lot of change and parallel source trees.

I _think_ we're done with any other must-do changes for 3.0, therefore.


What _can_ we do for 3.0? small changes that don't affect the 2.12
build are OK, and that's what you see in pull requests going in at the
moment. The big question is whether we want to do the large change for
https://issues.apache.org/jira/browse/SPARK-29292 before 3.0. It will
mean adding a ton of ".toSeq" and ".toMap" calls to make mutable
collections immutable when passed to methods. In theory, it won't
affect behavior. We'll have to see if it does in practice.

The rest will have to wait until after 3.0, I believe, including even
testing the 2.13 build, which will probably turn up some more issues.


Thoughts on approach?

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Status of Scala 2.13 support

Dongjoon Hyun-2
Thank you for sharing the status, Sean.

Given the current circumstance, our status and approach sounds realistic to me.

+1 for continuing after cutting `branch-3.0`.

Bests,
Dongjoon.


On Sun, Dec 1, 2019 at 10:50 AM Sean Owen <[hidden email]> wrote:
As you can see, I've been working on Scala 2.13 support. The umbrella
is https://issues.apache.org/jira/browse/SPARK-25075 I wanted to lay
out status and strategy.

This will not be done for 3.0. At the least, there are a few key
dependencies (Chill, Kafka) that aren't published for 2.13, and at
least one change that will need removing an API deprecated as of 3.0.
Realistically: maybe Spark 3.1. I don't yet think it's pressing.


Making the change is difficult as it's hard to understand the extent
of the necessary changes until the whole thing minimally compiles for
2.13. I have gotten essentially that far in a local clone. The good
news is I don't see any obvious hard blockers, but the changes add up
to thousands of line in 200+ files.


What do we need to do for 3.0? any changes that entail breaking a
public API, ideally. The biggest issue there comes from extensive
changes to the Scala collection hierarchy mean that the types of many
public APIs that return a Seq, Map, TraversableOnce, etc _will_
actually change types in 2.13 (become immutable). See:
https://issues.apache.org/jira/browse/SPARK-27683 and
https://issues.apache.org/jira/browse/SPARK-29292 as the main
examples.

In both cases, keeping the exact same public type would require much
bigger changes. These are the type of changes that all applications
face when migrating to 2.13 though. 2.12 and 2.13 apps were never
meant to be binary-compatible. So, in both cases we're not changing
these, to avoid a lot of change and parallel source trees.

I _think_ we're done with any other must-do changes for 3.0, therefore.


What _can_ we do for 3.0? small changes that don't affect the 2.12
build are OK, and that's what you see in pull requests going in at the
moment. The big question is whether we want to do the large change for
https://issues.apache.org/jira/browse/SPARK-29292 before 3.0. It will
mean adding a ton of ".toSeq" and ".toMap" calls to make mutable
collections immutable when passed to methods. In theory, it won't
affect behavior. We'll have to see if it does in practice.

The rest will have to wait until after 3.0, I believe, including even
testing the 2.13 build, which will probably turn up some more issues.


Thoughts on approach?

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]