[SS] New numSavedStates metric for StateStoreRestoreExec for saved state?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[SS] New numSavedStates metric for StateStoreRestoreExec for saved state?

Jacek Laskowski

Just reviewing StateStoreRestoreExec [1] and been wondering how to
know whether a state was available for a key. It has numOutputRows
metric [2], but that gives the number of aggregations from the child
operator only and seems to say nothing about whether state was
available for an aggregation.

What do you think about adding numSavedStates metric to
StateStoreRestoreExec? Or is there a way to find it out already
(perhaps in web UI)?

[1] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L186

[2] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L206

Jacek Laskowski
Spark Structured Streaming (Apache Spark 2.2+)
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

To unsubscribe e-mail: [hidden email]