Specifically, the part that says "TaskContext.get.partitionId()".
I don't know how much of that is part of Spark's public API, but there it is.
It would be useful if Spark offered a way to get a manifest of output files for any given write operation, similar to Redshift's MANIFEST option. This would help when, for example, you need to pass a list of files output by Spark to some other system (like Redshift) and don't want to have to worry about the consistency guarantees of your object store's list operations.
I should add that I tried using a waiter on the _SUCCESS file but it did not
prove successful as due to its small size compared to the part-0 file it
seems to be appearing before the part-0 file in s3, even though it was