FEATURE - Spark dynamic memory allocation?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

FEATURE - Spark dynamic memory allocation?

toop
This post has NOT been accepted by the mailing list yet.
I have got spark.dynamicAllocation enabled to allow dynamic executor allocation for a scala-based Spark 2.1.1 application that imports text files, casts the data types and writes into .orc partitions for many hive tables (1 table per different incoming text file structure). Problem is: small files load great with default memory settings but for files that are over 5million rows the load fails with out of memory errors unless specific parameters like memoryOverhead/executor-memory/maxResultSize are passed in with 8GB+. Is there any feature planned to dynamically allocate memory settings? ie. similar feature to how executors are dynamically requested as needed.
Loading...