pip/conda distribution headless mode

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

pip/conda distribution headless mode

geoHeil
Hi,

I want to use pyspark as distributed via conda in headless mode.
It looks like the hadoop binaries are bundles (= pip distributes a default version) https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn.

I want to ask if it would be possible to A) distribute the headless version (=without hadoop) instead or B) distribute the headless version additionally for pip & conda-forge distribution channels.

Best,
Georg
Reply | Threaded
Open this post in threaded view
|

Re: pip/conda distribution headless mode

Xiao Li-2
Hi, Georg,

This is being tracked by https://issues.apache.org/jira/browse/SPARK-32017 You can leave comments in the JIRA. 

Thanks,

Xiao

On Sun, Aug 30, 2020 at 3:06 PM Georg Heiler <[hidden email]> wrote:
Hi,

I want to use pyspark as distributed via conda in headless mode.
It looks like the hadoop binaries are bundles (= pip distributes a default version) https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn.

I want to ask if it would be possible to A) distribute the headless version (=without hadoop) instead or B) distribute the headless version additionally for pip & conda-forge distribution channels.

Best,
Georg


--
Reply | Threaded
Open this post in threaded view
|

Re: pip/conda distribution headless mode

geoHeil
Many thanks.

Best,
Georg

Am Mo., 31. Aug. 2020 um 01:12 Uhr schrieb Xiao Li <[hidden email]>:
Hi, Georg,

This is being tracked by https://issues.apache.org/jira/browse/SPARK-32017 You can leave comments in the JIRA. 

Thanks,

Xiao

On Sun, Aug 30, 2020 at 3:06 PM Georg Heiler <[hidden email]> wrote:
Hi,

I want to use pyspark as distributed via conda in headless mode.
It looks like the hadoop binaries are bundles (= pip distributes a default version) https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn.

I want to ask if it would be possible to A) distribute the headless version (=without hadoop) instead or B) distribute the headless version additionally for pip & conda-forge distribution channels.

Best,
Georg


--
Reply | Threaded
Open this post in threaded view
|

Re: pip/conda distribution headless mode

Hyukjin Kwon
I am going to take a look if nobody is interested in it.

2020년 8월 31일 (월) 오후 1:48, Georg Heiler <[hidden email]>님이 작성:
Many thanks.

Best,
Georg

Am Mo., 31. Aug. 2020 um 01:12 Uhr schrieb Xiao Li <[hidden email]>:
Hi, Georg,

This is being tracked by https://issues.apache.org/jira/browse/SPARK-32017 You can leave comments in the JIRA. 

Thanks,

Xiao

On Sun, Aug 30, 2020 at 3:06 PM Georg Heiler <[hidden email]> wrote:
Hi,

I want to use pyspark as distributed via conda in headless mode.
It looks like the hadoop binaries are bundles (= pip distributes a default version) https://stackoverflow.com/questions/63661404/bootstrap-spark-itself-on-yarn.

I want to ask if it would be possible to A) distribute the headless version (=without hadoop) instead or B) distribute the headless version additionally for pip & conda-forge distribution channels.

Best,
Georg


--