This post has NOT been accepted by the mailing list yet.
I have got spark.dynamicAllocation enabled to allow dynamic executor allocation for a scala-based Spark 2.1.1 application that imports text files, casts the data types and writes into .orc partitions for many hive tables (1 table per different incoming text file structure). Problem is: small files load great with default memory settings but for files that are over 5million rows the load fails with out of memory errors unless specific parameters like memoryOverhead/executor-memory/maxResultSize are passed in with 8GB+. Is there any feature planned to dynamically allocate memory settings? ie. similar feature to how executors are dynamically requested as needed.