SPARK-26415: Mark StreamSinkProvider and StreamSourceProvider as stable

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

SPARK-26415: Mark StreamSinkProvider and StreamSourceProvider as stable

Grant Henke
Hello Spark Developers,

Dongjoon Hyun suggested that I send an email to the dev list pointing to my suggested change. 

For convenience I will post the commit message here:

This change marks the StreamSinkProvider and StreamSourceProvider
interfaces as stable so that it can be relied on for compatibility for all of
Spark 2.x.

These interfaces have been available since Spark 2.0.0 and unchanged
since Spark 2.1.0. Additionally the Kafka integration has been using it
since Spark 2.1.0.

Because structured streaming general availability was announced in
Spark 2.2.0, I suspect there are other third-party integrations using it
already as well.

For additional context, I would like to use the  StreamSinkProvider interface in
Apache Kudu but do not feel comfortable committing the change without first 
knowing the interface is stable. 

I think leaving these interfaces as experimental was likely an oversight 
given all structured streaming was marked GA, but still wanted the annotation
change to be sure. 

I appreciate your feedback and review.

Thank you,

Grant Henke