[SQL] tracking jobs, stages and tasks per QueryExecution

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[SQL] tracking jobs, stages and tasks per QueryExecution

Oleksandr Vayda
Hey guys, 
I need to capture runtime stats and metrics for every executed SQL query. For that I'm using two listeners: the QueryExecutionListener, that gives me information about executed QueryExecution, and the SparkListener, that provides me the info about the jobs, stages and tasks. 
The problem I have is that the stats provided by the SparkListener refer the execution ID, that is generated by SQLExecution class, but is not stored in the QueryExecution object itself. Subsequently, when I catch events from the QueryExecutionListener I have an execution object, but know it's ID that it has on the time of execution. And I cannot tell what captured jobs and tasks stats on the other layer relates to it.

Any suggestions on how could I link those pieces of information?
 
Is there any reason why execution ID is not stored in the QueryExecution class? 

Thank you. 
Loading...