Heap Settings for History Server

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

Heap Settings for History Server

Neelesh Salian
Hi folks,

I couldn't find much literature on this so I figured I could ask here.

Does anyone have experience in tuning the memory settings and interval times of the Spark History Server?
Let's say I have 500 applications at 0.5 G each with a spark.history.fs.update.interval  of 400s.
Is there a direct memory correlation that can help me set an optimum value?

Looking for some advice if anyone has tuned the History Server to render large amounts of applications.

Neelesh S. Salian