[Spark][Kinesis] Could I get some committer review on the pull request

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[Spark][Kinesis] Could I get some committer review on the pull request

Yash Sharma
Hi All,
I've been working on a pull request [1] to allow Spark read from a specific timestamp from Kinesis. I have iterated the patch with the help of other contributors and we think that its in a good state now.

This patch would save hours of crash recovery time for Spark while reading off Kinesis. Kinesis suffers from Throttling issues unlike Kafka and hence this patch would essentially reduce the amount of data requested from Kinesis.

I would love to hear some thoughts from the committers and see if I can work on any improvements.


Best Regards,
Yash