Spark-submit --files option help

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark-submit --files option help

rahul c
Hi all,

I am trying to pass some configuration files via spark-submit command in cluster mode.
From logs I can see the files are transferred to each executors.
But how to build the absolute path of the file in the code?

Can anyone plz guide on it with some references.

Appreciate your help on this.

Thanks and regards
Rahul



Reply | Threaded
Open this post in threaded view
|

Re: Spark-submit --files option help

RussS
You can use SparkFiles.get(path)


Also this is probably a better question for the user list than the dev one

On Sat, Aug 1, 2020, 8:34 AM rahul c <[hidden email]> wrote:
Hi all,

I am trying to pass some configuration files via spark-submit command in cluster mode.
From logs I can see the files are transferred to each executors.
But how to build the absolute path of the file in the code?

Can anyone plz guide on it with some references.

Appreciate your help on this.

Thanks and regards
Rahul