Re: java.lang.ClassNotFoundException for s3a comitter
you are going to need hadoop-3.1 on your classpath, with hadoop-aws and the same aws-sdk it was built with (1.11.something). Mixing hadoop JARs is doomed. using a different aws sdk jar is a bit risky, though more recent upgrades have all be fairly low stress
On Fri, 19 Jun 2020 at 05:39, murat migdisoglu <[hidden email]> wrote:
I've upgraded my test cluster to spark 3 and change my comitter to directory and I still get this error.. The documentations are somehow obscure on that.
Do I need to add a third party jar to support new comitters?