[K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

Roland Johann
Hi all,

K8S resource manager dumps the config map to /opt/spark/conf/spark-defaults.conf and passes it to spark submit twice:
• via env var SPARK_CONF_DIR=/opt/spark/conf/
• argument --properties-file /opt/spark/conf/spark-defaults.conf
This prevents definition of user defined properties, such as from secrets - currently it seems to be the only possibility to define spark config properties from k8s secrets.

Is the current implementation intent? If yes, what are the reasons behind that decision?

Thanks and kind regards

Roland Johann
Software Developer/Data Engineer

phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: [hidden email]
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann



Reply | Threaded
Open this post in threaded view
|

Re: [K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

Roland Johann
I would like to discuss if we can introduce a change here - maybe it’s also possible to get that into 2.4.4?

I’m keen to contribute that change. Please let me know your thoughts on this.

King Regards

Roland Johann
Software Developer/Data Engineer

phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: [hidden email]
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann



Am 13.08.2019 um 21:15 schrieb Roland Johann <[hidden email]>:

Hi all,

K8S resource manager dumps the config map to /opt/spark/conf/spark-defaults.conf and passes it to spark submit twice:
• via env var SPARK_CONF_DIR=/opt/spark/conf/
• argument --properties-file /opt/spark/conf/spark-defaults.conf
This prevents definition of user defined properties, such as from secrets - currently it seems to be the only possibility to define spark config properties from k8s secrets.

Is the current implementation intent? If yes, what are the reasons behind that decision?

Thanks and kind regards

Roland Johann
Software Developer/Data Engineer

phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: [hidden email]
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann