Debug "Java gateway process exited before sending the driver its port number"

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Debug "Java gateway process exited before sending the driver its port number"

Li Jin
Dear Spark devs,

I am debugging a weird "Java gateway process exited before sending the driver its port number" when creating SparkSession with pyspark. I am running the following simple code with pytest:

"

from pyspark.sql import SparkSession


def test_spark():

    spark = SparkSession.builder.getOrCreate()

    spark.range(10).show()

"

(I've setup an conda environment and conda installed pyspark version 2.4.4)

I am wondering if there is a way for me to find the stdout/stderr/log for the Java driver process to see what's going on? 

Help much appreciated!
Thanks,
Li