Driverside in Spark while an application is running. The
Driverneeds to know the tasks progress before making any decision. I know that tasks progress can be accessed within each
RecordReaderclass by calling
The question is, how can I let the
Driver call or have an access to
getProgress() method of each task? I thought about using
accumulator variables, but I don't know how the
Driver would distinguish between different tasks.
Note that I am not looking for results displayed in Spark UI.
Any help is appreciated!