Binary compatibility issues in 3.1.1?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Binary compatibility issues in 3.1.1?

Alex Ott
Hi all


I'm thinking that it's caused by following change: https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150, but not 100% sure.  If it's, then maybe we could add the backward compatible constructor for this case class?

P.S. It's not the first time when 3.1 breaks compatibility with existing connectors, for example, Spark Cassandra Connector on 3.1 doesn't work without changes: https://github.com/datastax/spark-cassandra-connector/pull/1280


--
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Reply | Threaded
Open this post in threaded view
|

Re: Binary compatibility issues in 3.1.1?

Alex Ott
although no, additional constructor won't work...

On Mon, Feb 8, 2021 at 7:01 PM Alex Ott <[hidden email]> wrote:
Hi all


I'm thinking that it's caused by following change: https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150, but not 100% sure.  If it's, then maybe we could add the backward compatible constructor for this case class?

P.S. It's not the first time when 3.1 breaks compatibility with existing connectors, for example, Spark Cassandra Connector on 3.1 doesn't work without changes: https://github.com/datastax/spark-cassandra-connector/pull/1280


--
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)


--
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Reply | Threaded
Open this post in threaded view
|

Re: Binary compatibility issues in 3.1.1?

cloud0fan
This is the cost of relying on Spark internal APIs, and the external connectors need to take care of it.

BTW, the Alias change is source-compatible, and it shouldn't break the external connectors if they are compiled with Spark 3.1.

On Tue, Feb 9, 2021 at 2:26 AM Alex Ott <[hidden email]> wrote:
although no, additional constructor won't work...

On Mon, Feb 8, 2021 at 7:01 PM Alex Ott <[hidden email]> wrote:
Hi all


I'm thinking that it's caused by following change: https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150, but not 100% sure.  If it's, then maybe we could add the backward compatible constructor for this case class?

P.S. It's not the first time when 3.1 breaks compatibility with existing connectors, for example, Spark Cassandra Connector on 3.1 doesn't work without changes: https://github.com/datastax/spark-cassandra-connector/pull/1280


--
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)


--
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)