New PySpark test style

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

New PySpark test style

Hyukjin Kwon
Hi all,

Lately, https://github.com/apache/spark/pull/23021 is merged, which tries to a big single file that contains all the tests into smaller files.

I picked up one example and follow, NumPy. Because the current style looks closer to NumPy structure and looks easier to follow. Please see https://github.com/numpy/numpy/tree/master/numpy.

I would like to say:

1. If you were working on PySpark changes, I am terribly sorry that I caused some conflicts. Please take a look for the new styles and get used it.

2. Probably, I rushed a bit. I was scared if it's going to get conflicts again and again but please take a look and see if there's outstanding structural issues. I am willing to fix or revert at worst case.

3. Now, PySpark's SQL test were newly split but other tests will be split too in the same manner.

Thank you so much.