Requesting a Plan for Avro-typed Datasets

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Requesting a Plan for Avro-typed Datasets

Aleksander Eskilson
Hi all,

There's been longstanding demand for statically typed Datasets of Avro. Functionality from the now-deprecated Databricks Spark-Avro project was folded into Spark, but can still only provide DataFrames over Avro data. As is discussed in the PR below, there are still drawbacks from not having fully, statically typed Datasets of Avro.

There's an open PR adding a first-class Encoder for statically typed Datasets of Avro:


We've tested the content of this PR widely over complex, deeply nested, Avro structures. It seems ready for a last review and nearly ready for merger. 

Alek Eskilson
github : bdrillard
Reply | Threaded
Open this post in threaded view
|

Re: Requesting a Plan for Avro-typed Datasets

Taoufik Dachraoui
Hi

Please also consider the other 2 alternatives for statically typed Datasets of Avro objects


kind regards

-Taoufik


On Thu, May 16, 2019 at 9:59 PM Aleksander Eskilson <[hidden email]> wrote:
Hi all,

There's been longstanding demand for statically typed Datasets of Avro. Functionality from the now-deprecated Databricks Spark-Avro project was folded into Spark, but can still only provide DataFrames over Avro data. As is discussed in the PR below, there are still drawbacks from not having fully, statically typed Datasets of Avro.

There's an open PR adding a first-class Encoder for statically typed Datasets of Avro:


We've tested the content of this PR widely over complex, deeply nested, Avro structures. It seems ready for a last review and nearly ready for merger. 

Alek Eskilson
github : bdrillard


--
Taoufik Dachraoui