[SS] Custom Sinks

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[SS] Custom Sinks

Anton Okolnychyi
Hi all,

I have a question about the future of custom data sinks in Structured Streaming. In particular, I want to know how continuous processing and the Datasource API V2 will impact them. 

Right now, it is possible to have custom data sinks via the current Datasource API (V1) by implementing StreamSinkProvider and Sink traits. I am wondering if this approach is planned to be the right way or it is better to wait for the Datasource API V2 and the final implementation of continuous processing.

As far as I understand SPARK-20928, there will be changes in the Source API in SS. But what about the Sink API? Is it safe to implement it now?

Best regards,
Anton
Reply | Threaded
Open this post in threaded view
|

Re: [SS] Custom Sinks

rxin
They will probably both change, but I wouldn't block on the change if you have an immediate need.


On Wed, Nov 1, 2017 at 10:41 AM, Anton Okolnychyi <[hidden email]> wrote:
Hi all,

I have a question about the future of custom data sinks in Structured Streaming. In particular, I want to know how continuous processing and the Datasource API V2 will impact them. 

Right now, it is possible to have custom data sinks via the current Datasource API (V1) by implementing StreamSinkProvider and Sink traits. I am wondering if this approach is planned to be the right way or it is better to wait for the Datasource API V2 and the final implementation of continuous processing.

As far as I understand SPARK-20928, there will be changes in the Source API in SS. But what about the Sink API? Is it safe to implement it now?

Best regards,
Anton