Commit algorithms: what's required, what's delivered, and how they are managed
There's been discussion going on in various PRs about what committers do, are expected to do, and how they get coordinated; a general conclusion to these is "this should be covered in the developer list"
Those docs are incomplete, and I don't know of anything equivalent covering the Spark driver's commit algorithm, so it's mostly been a matter of tracing back through the IDE and having a modified committer set to do things like fail in task or
Having spent time integrating Hadoop's forthcoming S3A committers with things, I suspect that there may be some mismatch of expectations of committers & what they deliver, but I'll need to add a bit more fault injection there to be sure. I'll
have a draft of a paper up in a week or so for anyone interested in this area