Task partition ID in Spark event logs

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Task partition ID in Spark event logs

Michael Mior
I see there's a comment in the TaskInfo class that the index may not be the same as the ID of the RDD partition the task is computing. Under what circumstances *will* the ID by the same? If there are zero guarantees, any suggestions on how to grab this info from the scheduler to populate a new field inside TaskInfo?

Cheers,
--
Michael Mior
Loading...