Why can per task‘s max memory only reach 1 / numTasks at most , not greater than 1 / numTasks in ExecutionMemoryPool ?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Why can per task‘s max memory only reach 1 / numTasks at most , not greater than 1 / numTasks in ExecutionMemoryPool ?

John Fang-2
1.png

In fact not all tasks belong to the same stage. Thus, per task may be is deferent for the dependence of memory. For example, the executor
 are running two tasks(A and B), and the ExecutionMemoryPool own 1000M. We can hope the task-A occupy 900M, and task-B occupy 100M due to the task-A need much more memory when it is running. But per task's max memory is limited, so the task-A can't occupy 900M memory. 



--

Regards

John Fang