[SS] ContinuousExecution.commit and excessive JSON serialization?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[SS] ContinuousExecution.commit and excessive JSON serialization?

Jacek Laskowski
Hi,

Why does ContinuousExecution.commit serialize an offset to JSON format just before deserializing it back (from JSON to an offset)? [1]

val offset =
          sources(0).deserializeOffset(offsetLog.get(epoch).get.offsets(0).get.json)
 

Pozdrawiam,
Jacek Laskowski
----
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals