Chnage Restful API's default quanlites same as WebUI

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Chnage Restful API's default quanlites same as WebUI

angers.zhu
Hi devs,

These days I found that about taskSummary, the Restful API(/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary)'s default quantiles is `0.05,0.25,0.5,0.75,0.95`
But in Spark Web UI’s stage page, the quantiles is `0.0,0.25,0.5,0.75,1.0`. And the restful api’s default value
 is not shown in Spark website’s  [monitoring page](https://spark.apache.org/docs/3.0.0/monitoring.html). Detail can be seen in https://github.com/apache/spark/pull/31048#issuecomment-756773764

I have create a PR to keep them consistence, and hope you dev’s  visibility and comments.

Best regards
Angers

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Chnage Restful API's default quanlites same as WebUI

Sean Owen-2
The change makes sense of course; I think the real question for everyone is: would you change the semantics of this info from the API in a minor release, or would it have to wait? is it more bug fix than breaking change?

On Sun, Jan 10, 2021 at 7:38 PM angers.zhu <[hidden email]> wrote:
Hi devs,

These days I found that about taskSummary, the Restful API(/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary)'s default quantiles is `0.05,0.25,0.5,0.75,0.95`
But in Spark Web UI’s stage page, the quantiles is `0.0,0.25,0.5,0.75,1.0`. And the restful api’s default value
 is not shown in Spark website’s  [monitoring page](https://spark.apache.org/docs/3.0.0/monitoring.html). Detail can be seen in https://github.com/apache/spark/pull/31048#issuecomment-756773764

I have create a PR to keep them consistence, and hope you dev’s  visibility and comments.

Best regards
Angers

Reply | Threaded
Open this post in threaded view
|

Re: Chnage Restful API's default quanlites same as WebUI

ron8hu
To me, this is a bug.  In a parallel system, the duration of a stage is often
determined by the slowest task/executor. To monitor/debug a skew issue, the
maximal value (or 100% percentile value) is more useful than the 95%
percentile value.  I am happy to see that we do it right with Web UI.  For
REST API, it should be consistent with Web UI.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]