Quantcast

[PYTHON] PySpark typing hints

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PYTHON] PySpark typing hints

zero323

Hi everyone,

For the last few months I've been working on static type annotations for PySpark. For those of you, who are not familiar with the idea, typing hints have been introduced by PEP 484 (https://www.python.org/dev/peps/pep-0484/) and further extended with PEP 526 (https://www.python.org/dev/peps/pep-0526/) with the main goal of providing information required for static analysis. Right now there a few tools which support typing hints, including Mypy (https://github.com/python/mypy) and PyCharm (https://www.jetbrains.com/help/pycharm/2017.1/type-hinting-in-pycharm.html).  Type hints can be added using function annotations (https://www.python.org/dev/peps/pep-3107/, Python 3 only), docstrings, or source independent stub files (https://www.python.org/dev/peps/pep-0484/#stub-files). Typing is optional, gradual and has no runtime impact.

At this moment I've annotated majority of the API, including majority of pyspark.sql and pyspark.ml. At this moment project is still rough around the edges, and may result in both false positive and false negatives, but I think it become mature enough to be useful in practice.

The current version is compatible only with Python 3, but it is possible, with some limitations, to backport it to Python 2 (though it is not on my todo list).

There is a number of possible benefits for PySpark users and developers:
  • Static analysis can detect a number of common mistakes to prevent runtime failures. Generic self is still fairly limited, so it is more useful with DataFrames, SS and ML than RDD, DStreams or RDD.
  • Annotations can be used for documenting complex signatures (https://git.io/v95JN) including dependencies on arguments and value (https://git.io/v95JA).
  • Detecting possible bugs in Spark (SPARK-20631) .
  • Showing API inconsistencies.

Roadmap

  • Update the project to reflect Spark 2.2.
  • Refine existing annotations.

If there will be enough interest I am happy to contribute this back to Spark or submit to Typeshed (https://github.com/python/typeshed -  this would require a formal ASF approval, and since Typeshed doesn't provide versioning, is probably not the best option in our case).

Further inforamtion:

-- 
Best,
Maciej

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PYTHON] PySpark typing hints

rxin
Seems useful to do. Is there a way to do this so it doesn't break Python 2.x?


On Sun, May 14, 2017 at 11:44 PM, Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

For the last few months I've been working on static type annotations for PySpark. For those of you, who are not familiar with the idea, typing hints have been introduced by PEP 484 (https://www.python.org/dev/peps/pep-0484/) and further extended with PEP 526 (https://www.python.org/dev/peps/pep-0526/) with the main goal of providing information required for static analysis. Right now there a few tools which support typing hints, including Mypy (https://github.com/python/mypy) and PyCharm (https://www.jetbrains.com/help/pycharm/2017.1/type-hinting-in-pycharm.html).  Type hints can be added using function annotations (https://www.python.org/dev/peps/pep-3107/, Python 3 only), docstrings, or source independent stub files (https://www.python.org/dev/peps/pep-0484/#stub-files). Typing is optional, gradual and has no runtime impact.

At this moment I've annotated majority of the API, including majority of pyspark.sql and pyspark.ml. At this moment project is still rough around the edges, and may result in both false positive and false negatives, but I think it become mature enough to be useful in practice.

The current version is compatible only with Python 3, but it is possible, with some limitations, to backport it to Python 2 (though it is not on my todo list).

There is a number of possible benefits for PySpark users and developers:
  • Static analysis can detect a number of common mistakes to prevent runtime failures. Generic self is still fairly limited, so it is more useful with DataFrames, SS and ML than RDD, DStreams or RDD.
  • Annotations can be used for documenting complex signatures (https://git.io/v95JN) including dependencies on arguments and value (https://git.io/v95JA).
  • Detecting possible bugs in Spark (SPARK-20631) .
  • Showing API inconsistencies.

Roadmap

  • Update the project to reflect Spark 2.2.
  • Refine existing annotations.

If there will be enough interest I am happy to contribute this back to Spark or submit to Typeshed (https://github.com/python/typeshed -  this would require a formal ASF approval, and since Typeshed doesn't provide versioning, is probably not the best option in our case).

Further inforamtion:

-- 
Best,
Maciej

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [PYTHON] PySpark typing hints

assaf.mendelson

Actually there is, at least for pycharm. I actually opened a jira on it (https://issues.apache.org/jira/browse/SPARK-17333). It describes two way of doing it (I also made a github stub at: https://github.com/assafmendelson/ExamplePysparkAnnotation). Unfortunately, I never found the time to follow through.

That said, If we make a decision on the way to handle it then I believe it would be a good idea to start even with the bare minimum and continue to add to it (and therefore make it so many people can contribute). The code I added in github were basically the things I needed.

 

To summarize, there are two main ways of doing it (at least in pycharm):

1.       Give the hints as part of the docstring for the function

2.       Create files with the signatures only and mark it for pycharm to use

 

The advantage of the first is that it is part of the code which means it is easier to make it updated. The main issue with this is that supporting auto generated code (as is the case in most functions) can be a little awkward and actually is a relate to a separate issue as it means pycharm marks most of the functions as an error (i.e. pyspark.sql.functions.XXX is marked as not there…)

 

The advantage of the second is that it is completely separate so messing around with it cannot harm the main code. The disadvantages are that we would need to maintain it manually and that to use it in pycharm, one needs to add them to the path (in pycharm this means mark them as source, I am not sure how other IDEs support this).

 

Lastly, I only tested these two solutions for pycharm. I am not sure of their support in other IDEs.

 

 

Thanks,

              Assaf.

 

From: rxin [via Apache Spark Developers List] [mailto:ml+[hidden email]]
Sent: Tuesday, May 23, 2017 1:10 PM
To: Mendelson, Assaf
Subject: Re: [PYTHON] PySpark typing hints

 

Seems useful to do. Is there a way to do this so it doesn't break Python 2.x?

 

 

On Sun, May 14, 2017 at 11:44 PM, Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

For the last few months I've been working on static type annotations for PySpark. For those of you, who are not familiar with the idea, typing hints have been introduced by PEP 484 (https://www.python.org/dev/peps/pep-0484/) and further extended with PEP 526 (https://www.python.org/dev/peps/pep-0526/) with the main goal of providing information required for static analysis. Right now there a few tools which support typing hints, including Mypy (https://github.com/python/mypy) and PyCharm (https://www.jetbrains.com/help/pycharm/2017.1/type-hinting-in-pycharm.html).  Type hints can be added using function annotations (https://www.python.org/dev/peps/pep-3107/, Python 3 only), docstrings, or source independent stub files (https://www.python.org/dev/peps/pep-0484/#stub-files). Typing is optional, gradual and has no runtime impact.

At this moment I've annotated majority of the API, including majority of pyspark.sql and pyspark.ml. At this moment project is still rough around the edges, and may result in both false positive and false negatives, but I think it become mature enough to be useful in practice.

The current version is compatible only with Python 3, but it is possible, with some limitations, to backport it to Python 2 (though it is not on my todo list).

There is a number of possible benefits for PySpark users and developers:

  • Static analysis can detect a number of common mistakes to prevent runtime failures. Generic self is still fairly limited, so it is more useful with DataFrames, SS and ML than RDD, DStreams or RDD.
  • Annotations can be used for documenting complex signatures (https://git.io/v95JN) including dependencies on arguments and value (https://git.io/v95JA).
  • Detecting possible bugs in Spark (SPARK-20631) .
  • Showing API inconsistencies.

Roadmap

  • Update the project to reflect Spark 2.2.
  • Refine existing annotations.

If there will be enough interest I am happy to contribute this back to Spark or submit to Typeshed (https://github.com/python/typeshed -  this would require a formal ASF approval, and since Typeshed doesn't provide versioning, is probably not the best option in our case).

Further inforamtion:

-- 
Best,
Maciej

 

 


If you reply to this email, your message will be added to the discussion below:

http://apache-spark-developers-list.1001551.n3.nabble.com/PYTHON-PySpark-typing-hints-tp21560p21611.html

To start a new topic under Apache Spark Developers List, email [hidden email]
To unsubscribe from Apache Spark Developers List, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PYTHON] PySpark typing hints

zero323
In reply to this post by rxin

It doesn't break anything at all. You can take stub files as-is, put these into PySpark root, and as long as users are not interested in type checking, it won't have any runtime impact.

Surprisingly the current MyPy build (mypy==0.511) reports only one incompatibility with Python 2 (dynamic metaclasses), which is could be resolved without significant loss of function.

On 05/23/2017 12:08 PM, Reynold Xin wrote:
Seems useful to do. Is there a way to do this so it doesn't break Python 2.x?


On Sun, May 14, 2017 at 11:44 PM, Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

For the last few months I've been working on static type annotations for PySpark. For those of you, who are not familiar with the idea, typing hints have been introduced by PEP 484 (https://www.python.org/dev/peps/pep-0484/) and further extended with PEP 526 (https://www.python.org/dev/peps/pep-0526/) with the main goal of providing information required for static analysis. Right now there a few tools which support typing hints, including Mypy (https://github.com/python/mypy) and PyCharm (https://www.jetbrains.com/help/pycharm/2017.1/type-hinting-in-pycharm.html).  Type hints can be added using function annotations (https://www.python.org/dev/peps/pep-3107/, Python 3 only), docstrings, or source independent stub files (https://www.python.org/dev/peps/pep-0484/#stub-files). Typing is optional, gradual and has no runtime impact.

At this moment I've annotated majority of the API, including majority of pyspark.sql and pyspark.ml. At this moment project is still rough around the edges, and may result in both false positive and false negatives, but I think it become mature enough to be useful in practice.

The current version is compatible only with Python 3, but it is possible, with some limitations, to backport it to Python 2 (though it is not on my todo list).

There is a number of possible benefits for PySpark users and developers:
  • Static analysis can detect a number of common mistakes to prevent runtime failures. Generic self is still fairly limited, so it is more useful with DataFrames, SS and ML than RDD, DStreams or RDD.
  • Annotations can be used for documenting complex signatures (https://git.io/v95JN) including dependencies on arguments and value (https://git.io/v95JA).
  • Detecting possible bugs in Spark (SPARK-20631) .
  • Showing API inconsistencies.

Roadmap

  • Update the project to reflect Spark 2.2.
  • Refine existing annotations.

If there will be enough interest I am happy to contribute this back to Spark or submit to Typeshed (https://github.com/python/typeshed -  this would require a formal ASF approval, and since Typeshed doesn't provide versioning, is probably not the best option in our case).

Further inforamtion:

-- 
Best,
Maciej


-- 
Maciej Szymkiewicz

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PYTHON] PySpark typing hints

zero323
In reply to this post by assaf.mendelson



On 05/23/2017 02:45 PM, Mendelson, Assaf wrote:

You are correct,

I actually did not look too deeply into it until now as I noticed you mentioned it is compatible with python 3 only and I saw in the github that mypy or pytype is required.

 

Because of that I made my suggestions with the thought of python 2.

 

Looking into it more deeply, I am wondering what is not supported? Are you talking about limitation for testing?


Since type checkers (unlike annotations) are not standardized, this varies between projects and versions. For MyPy quite a lot changed since I started annotating Spark.

Few months ago I wouldn't even bother looking at the list of issues, today (as mentioned in the other message) we could remove metaclasses, and pass both Python 2 and Python 3 checks.

The other part is typing module itself, as well as function annotations (outside docstrings). But this is not a problem with stub files.

 

If I understand correctly then one can use this without any issues for pycharm (and other IDEs supporting the type hinting) even when developing for python 2.


This strictly depends on type checker. I didn't follow the development, but I got this impression that a lot changed for example between PyCharm 2016.3 and 2017.1. I think that the important point is that lack of support, doesn't break anything.

In addition, the tests can test the existing pyspark, they just have to be run with a compatible packaging (e.g. mypy).

Meaning that porting for python 2 would provide a very small advantage over the immediate advantages (IDE usage and testing for most cases).

 

Am I missing something?

 

Thanks,

              Assaf.

 

From: Maciej Szymkiewicz [[hidden email]]
Sent: Tuesday, May 23, 2017 3:27 PM
To: Mendelson, Assaf
Subject: Re: [PYTHON] PySpark typing hints

 

 

 

On 05/23/2017 01:12 PM, assaf.mendelson wrote:

That said, If we make a decision on the way to handle it then I believe it would be a good idea to start even with the bare minimum and continue to add to it (and therefore make it so many people can contribute). The code I added in github were basically the things I needed.

I already have almost full coverage of the API, excluding some exotic part of the legacy streaming, so starting with bare minimum is not really required.

The advantage of the first is that it is part of the code which means it is easier to make it updated. The main issue with this is that supporting auto generated code (as is the case in most functions) can be a little awkward and actually is a relate to a separate issue as it means pycharm marks most of the functions as an error (i.e. pyspark.sql.functions.XXX is marked as not there…)


Comment based annotations are not suitable for complex signatures with multliversion support.

Also there is no support for overloading, therefore it is not possible to capture relationship between arguments, and arguments and return type.


-- 
Maciej Szymkiewicz

signature.asc (849 bytes) Download Attachment
Loading...