Systematically performance diagnose

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Systematically performance diagnose

Hao Lin
Hi folks,

I am trying to test the performance on a couple of my Spark applications.
For benchmarking purpose, I am wondering if there is a good performance
analysis practice. The best way I can think of is to instrument log prints
and analyze the timestamps in logs on each node.

The major metrics I am interested in are computation ratios (computation
time, data transferring time, basically a timeline of detailed events),
memory usage, disk throughput. Could I have some suggestions on how Spark
is benchmarked.

Thanks,

Max
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Systematically performance diagnose

rxin
The application web ui is pretty useful. We have been adding more and more
information to the web ui for easier performance analysis.

Look at Patrick Wendell's two talks at the Spark Summit for more
information: http://spark-summit.org/summit-2013/


On Sat, Dec 28, 2013 at 8:12 PM, Hao Lin <[hidden email]> wrote:

> Hi folks,
>
> I am trying to test the performance on a couple of my Spark applications.
> For benchmarking purpose, I am wondering if there is a good performance
> analysis practice. The best way I can think of is to instrument log prints
> and analyze the timestamps in logs on each node.
>
> The major metrics I am interested in are computation ratios (computation
> time, data transferring time, basically a timeline of detailed events),
> memory usage, disk throughput. Could I have some suggestions on how Spark
> is benchmarked.
>
> Thanks,
>
> Max
>
Loading...