[ML][PYTHON] Collecting data in a class extending SparkSessionTestCase causes AttributeError:

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[ML][PYTHON] Collecting data in a class extending SparkSessionTestCase causes AttributeError:

zero323
Hi everyone,

It is a either to late or to early for me to think straight so please
forgive me if it is something trivial. I am trying to add a test case
extending SparkSessionTestCase to pyspark.ml.tests (example patch
attached). If test collects data, and there is another TestCase
extending extending SparkSessionTestCase executed before it, I get
AttributeError due to _jsc being None:

======================================================================

ERROR: test_foo (pyspark.ml.tests.FooTest)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/spark/python/pyspark/ml/tests.py", line 1258, in test_foo

  File "/home/spark/python/pyspark/sql/dataframe.py", line 389, in collect

    with SCCallSiteSync(self._sc) as css:

  File "/home/spark/python/pyspark/traceback_utils.py", line 72, in __enter__

    self._context._jsc.setCallSite(self._call_site)

AttributeError: 'NoneType' object has no attribute 'setCallSite'

----------------------------------------------------------------------

If TestCase is executed alone it seems to work just fine.


Can anyone reproduce this? Is there something obvious I miss here?

--
Best,
Maciej


patch (1K) Download Attachment
signature.asc (849 bytes) Download Attachment