[SQL] tracking jobs, stages and tasks per QueryExecution
I need to capture runtime stats and metrics for every executed SQL query. For that I'm using two listeners: the QueryExecutionListener, that gives me information about executed QueryExecution, and the SparkListener, that provides me the info about the jobs, stages and tasks.
The problem I have is that the stats provided by the SparkListener refer the execution ID, that is generated by SQLExecution class, but is not stored in the QueryExecution object itself. Subsequently, when I catch events from the QueryExecutionListener I have an execution object, but know it's ID that it has on the time of execution. And I cannot tell what captured jobs and tasks stats on the other layer relates to it.
Any suggestions on how could I link those pieces of information?
Is there any reason why execution ID is not stored in the QueryExecution class?