[DISCUSS][SPIP] Standardize Spark Exception Messages

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS][SPIP] Standardize Spark Exception Messages

Xinyi Yu
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing>  

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Nicholas Chammas
Just want to call out that this SPIP should probably account somehow for PySpark and the work being done in SPARK-32082 to improve PySpark exceptions. 

On Sun, Oct 25, 2020 at 8:05 PM Xinyi Yu <[hidden email]> wrote:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Hyukjin Kwon

Thanks for pointing this out, Nicholas.
This SPIP seems focused on the Scala side, grouping the exception handling and providing some guidance about error messages.
Yes, I think we can refer to it on the PySpark side. Probably I will follow up and file some JIRAs based on how this SPIP gose, and run it separately.
BTW, it’s slightly different from SPARK-32194 because this JIRA targets to have a set of proper classes instead of, for example, just plain Exception.



2020년 10월 26일 (월) 오전 10:36, Nicholas Chammas <[hidden email]>님이 작성:
Just want to call out that this SPIP should probably account somehow for PySpark and the work being done in SPARK-32082 to improve PySpark exceptions. 

On Sun, Oct 25, 2020 at 8:05 PM Xinyi Yu <[hidden email]> wrote:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Chang Chen
In reply to this post by Xinyi Yu
hi Xinyi

Just curious, which tool did you use to generate this 

Xinyi Yu <[hidden email]> 于2020年10月26日周一 上午8:05写道:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Xinyi Yu
Hi Chang,

It is a script that directly analyzes the source code searching for raw "throw new” exception. : ) Hope that give an intuitive overview of current exceptions in Spark. 
On Oct 27, 2020, 8:21 PM -0700, Chang Chen <[hidden email]>, wrote:
hi Xinyi

Just curious, which tool did you use to generate this 

Xinyi Yu <[hidden email]> 于2020年10月26日周一 上午8:05写道:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Xiao Li
+1 

This is a great proposal to improve the usability of Spark. Make Spark simple to use!

Xiao

Xinyi Yu <[hidden email]> 于2020年10月27日周二 下午8:25写道:
Hi Chang,

It is a script that directly analyzes the source code searching for raw "throw new” exception. : ) Hope that give an intuitive overview of current exceptions in Spark. 
On Oct 27, 2020, 8:21 PM -0700, Chang Chen <[hidden email]>, wrote:
hi Xinyi

Just curious, which tool did you use to generate this 

Xinyi Yu <[hidden email]> 于2020年10月26日周一 上午8:05写道:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Takeshi Yamamuro
In reply to this post by Xinyi Yu
I've already left comments about this idea in https://github.com/apache/spark/pull/29497 though,
it sounds plausible. +1.

On Mon, Oct 26, 2020 at 9:04 AM Xinyi Yu <[hidden email]> wrote:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Xinyi Yu
Hi Takeshi, Xiao and all devs,

Thank you for your kind feedback! We are going to initiate a vote for this SPIP recently.

On Oct 29, 2020, 6:41 PM -0700, Takeshi Yamamuro <[hidden email]>, wrote:
I've already left comments about this idea in https://github.com/apache/spark/pull/29497 though,
it sounds plausible. +1.

On Mon, Oct 26, 2020 at 9:04 AM Xinyi Yu <[hidden email]> wrote:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Mridul Muralidharan
In reply to this post by Xinyi Yu

I like the idea of consistent messages; it makes understanding errors easier.
Having said that, Exception messages themselves are not part of the exposed contract to users; and are subject to change.
We should leave that flexibility open to spark developers ... I am currently viewing this proposal as a internal standardization exercise within Spark codebase, and not as a public contract with users.
Is that aligned with the objectives ? Or are we looking at this as a public contract to users ?
I am not in favour of the latter.

Regards,
Mridul


On Sun, Oct 25, 2020 at 7:05 PM Xinyi Yu <[hidden email]> wrote:
Hi all,

We like to post a SPIP of Standardize Exception Messages in Spark. Here is
the document link: 
https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing
<https://docs.google.com/document/d/1XGj1o3xAFh8BA7RCn3DtwIPC6--hIFOaNUNSlpaOIZs/edit?usp=sharing

This SPIP aims to standardize the exception messages in Spark. It has three
major focuses:
1. Group exception messages in dedicated files for easy maintenance and
auditing.
2. Establish an error message guideline for developers.
3. Improve error message quality.

Thanks for your time and patience. Looking forward to your feedback!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Allison Wang
Hi Mridul,

Thanks for reviewing the SPIP. As you correctly pointed out, this proposal
is an internal, Spark engine developer-facing improvement to make error
messages more consistent and user-friendly across the Spark codebase, as
opposed to establishing a public contract with end-users.

Thanks,
Allison



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]