disable checkpointing in structured streaming

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

disable checkpointing in structured streaming

Charles vinodh
Hi ,

is it possible to disable checkpointing in structured streaming and have it replaced by our own checkpointing implementation where the offsets are saved in an external database?.. I looked up the docs and it seems this is supported on spark DStream streaming but not in structured streaming for some reason. Is that the case and if so why is this not possible with structured streaming? is it possible to have this feature in the future? 

Thanks,
Charles 
Reply | Threaded
Open this post in threaded view
|

Re: disable checkpointing in structured streaming

Charles vinodh
resurfacing this thread again.. 

On Mon, Jul 8, 2019 at 8:55 PM Charles vinodh <[hidden email]> wrote:
Hi ,

is it possible to disable checkpointing in structured streaming and have it replaced by our own checkpointing implementation where the offsets are saved in an external database?.. I looked up the docs and it seems this is supported on spark DStream streaming but not in structured streaming for some reason. Is that the case and if so why is this not possible with structured streaming? is it possible to have this feature in the future? 

Thanks,
Charles