Resolve _temporary directory uncleaned

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Resolve _temporary directory uncleaned

JackyLee
hi devs,

In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use
deleteExternalTmpPath to clean temporary directories after Job committed and
cancel deleteOnExit if succeeded.But sometimes (e.g., when speculative task
is enabled), temporary directories may be left uncleaned. This is happened
if there are still some tasks running after we called deleteExternalTmpPath.
Thus I tried to add JobCleaned Status to clean temporary directories. The
JobCleaned Status is happened with all stages has been cleaned in a job,
thus it is a good way to do some job cleanings.

JIRA: https://issues.apache.org/jira/browse/SPARK-31438
PR: https://github.com/apache/spark/pull/28280

There have been some discuss in https://github.com/apache/spark/pull/28129.

This PR has been submitted for about 3 months. Any one could review this?

Thanks,
Jacky Lee



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]