custom FileStreamSource which reads from one partition onwards

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

custom FileStreamSource which reads from one partition onwards


to my best knowledge, the existing FileStreamSource reads all the files in a directory (hive table).
However, I need to be able to specify an initial partition it should start from (i.e. like a Kafka offset/initial warmed-up state) and then only read data which is semantically (i.e. using a file path lexicographically) greater than the minimum committed initial state?

After playing around with the internals of the file format I have come to the conclusion that manually modifying it and setting some values (i.e. the last processed & committed partition) is not  feasible as spark regardless will pick up all the files (even the older partitions)

Is this correct?

This leads me to the conclusion I need a custom StatefulFileStreamSource. I tried to create one (, but so far fail to instantiate it (even though it is just a copy of the original one as the constructor without any arguments does not seem to be defined:

NoSuchMethodException: org.apache.spark.sql.execution.streaming.StatefulFileStreamSource.<init>()

Why is the default constructor not found? Even for a simple copy of an existing (and presumably working class?

Note, I am currently working on spark 2.2.3