spark lacks fault tolerance with dynamic partition overwrite

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

spark lacks fault tolerance with dynamic partition overwrite

Koert Kuipers
i wanted to highlight here the issue we are facing with dynamic partition overwrite.

it seems that any tasks that writes to disk using this feature and that need to be retried fails upon retry, leading to a failure for the entire job.

we have seen this issue show up with preemption (task gets killed by pre-emption, and when it gets rescheduled it fails consistently). it can also show up if a hardware issue causes your task to fail, or if you have speculative execution enabled.

relevant jiras are SPARK-30320 and SPARK-29302

this affects spark 2.4.x and spark 3.0.0-SNAPSHOT
writing to hive does not seem to be impacted.

best,
koert
Reply | Threaded
Open this post in threaded view
|

Re: spark lacks fault tolerance with dynamic partition overwrite

wuyi
Hi,  Koert,

The community is back to this issue to recently and there's already a fix
https://github.com/apache/spark/pull/26339 for it.

You can track and review it there.

Best,

Yi Wu



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spark lacks fault tolerance with dynamic partition overwrite

Koert Kuipers
ah ok i was not aware of that jira issue. i will follow the progress there.
thanks for letting me known

On Tue, Apr 7, 2020 at 11:20 AM wuyi <[hidden email]> wrote:
Hi,  Koert,

The community is back to this issue to recently and there's already a fix
https://github.com/apache/spark/pull/26339 for it.

You can track and review it there.

Best,

Yi Wu



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]