Read hdfs files in spark streaming

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Read hdfs files in spark streaming

Deepak Sharma

I am using spark streaming application to read fromĀ  kafka.
The value coming from kafka message is path to hdfs file.
I am using spark 2.x , spark.read.stream.
What is the best way to read this path in spark streaming and then read the json stored at the hdfs path , may be using spark.read.json , into a df inside the spark streaming app.
Thanks a lot in advance

--
Thanks
Deepak