[SPARK-30296][SQL] Add Dataset diffing feature

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[SPARK-30296][SQL] Add Dataset diffing feature

Enrico Minack
Hi Devs,

I'd like to get your thoughts on this Dataset feature proposal.
Comparing datasets is a central operation when regression testing your
code changes.

It would be super useful if Spark's Datasets provide this transformation
natively.

https://github.com/apache/spark/pull/26936

Regards,
Enrico


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [SPARK-30296][SQL] Add Dataset diffing feature

rxin
Can this perhaps exist as an utility function outside Spark?


On Tue, Jan 07, 2020 at 12:18 AM, Enrico Minack <[hidden email]> wrote:

Hi Devs,

I'd like to get your thoughts on this Dataset feature proposal. Comparing datasets is a central operation when regression testing your code changes.

It would be super useful if Spark's Datasets provide this transformation natively.

https://github.com/apache/spark/pull/26936

Regards,
Enrico

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]