Detect executor core count

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Detect executor core count

Andrew Melo
Hello,

Is there a way to detect the number of cores allocated for an executor within a java-based InputPartitionReader?

Thanks!
Andrew
Reply | Threaded
Open this post in threaded view
|

RE: Detect executor core count

Ilya Matiach-2

Hi Andrew,

I tried to do something similar to that in the LightGBM classifier/regressor/ranker in mmlspark package, I try to use the spark conf and if not configured I get the processors from the JVM directly:

https://github.com/Azure/mmlspark/blob/master/src/lightgbm/src/main/scala/LightGBMUtils.scala#L172

 

If you know of a better way, please let me know!

 

    val spark = dataset.sparkSession

    try {

      val confCores = spark.sparkContext.getConf

        .get("spark.executor.cores").toInt

      val confTaskCpus = spark.sparkContext.getConf

        .get("spark.task.cpus", "1").toInt

      confCores / confTaskCpus

    } catch {

      case _: NoSuchElementException =>

        // If spark.executor.cores is not defined, get the cores per JVM

        import spark.implicits._

        val numMachineCores = spark.range(0, 1)

          .map(_ => java.lang.Runtime.getRuntime.availableProcessors).collect.head

        numMachineCores

    }

 

Thank you, Ilya

 

From: Andrew Melo <[hidden email]>
Sent: Tuesday, June 18, 2019 11:32 AM
To: dev <[hidden email]>
Subject: Detect executor core count

 

Hello,

 

Is there a way to detect the number of cores allocated for an executor within a java-based InputPartitionReader?

 

Thanks!

Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Detect executor core count

Steve Loughran-2
be aware that older java 8 versions count the #of cores in the host, not those allocated for the container they run in

On Tue, Jun 18, 2019 at 8:13 PM Ilya Matiach <[hidden email]> wrote:

Hi Andrew,

I tried to do something similar to that in the LightGBM classifier/regressor/ranker in mmlspark package, I try to use the spark conf and if not configured I get the processors from the JVM directly:

https://github.com/Azure/mmlspark/blob/master/src/lightgbm/src/main/scala/LightGBMUtils.scala#L172

 

If you know of a better way, please let me know!

 

    val spark = dataset.sparkSession

    try {

      val confCores = spark.sparkContext.getConf

        .get("spark.executor.cores").toInt

      val confTaskCpus = spark.sparkContext.getConf

        .get("spark.task.cpus", "1").toInt

      confCores / confTaskCpus

    } catch {

      case _: NoSuchElementException =>

        // If spark.executor.cores is not defined, get the cores per JVM

        import spark.implicits._

        val numMachineCores = spark.range(0, 1)

          .map(_ => java.lang.Runtime.getRuntime.availableProcessors).collect.head

        numMachineCores

    }

 

Thank you, Ilya

 

From: Andrew Melo <[hidden email]>
Sent: Tuesday, June 18, 2019 11:32 AM
To: dev <[hidden email]>
Subject: Detect executor core count

 

Hello,

 

Is there a way to detect the number of cores allocated for an executor within a java-based InputPartitionReader?

 

Thanks!

Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Detect executor core count

Andrew Melo


On Tue, Jun 18, 2019 at 5:40 PM Steve Loughran <[hidden email]> wrote:
be aware that older java 8 versions count the #of cores in the host, not those allocated for the container they run in


Ergh, that's good to know. I suppose, though, that in any case, there should be a SparkSession available if I'm in the executor context, so I can fallback to something sensible just in case.

Thanks for the help, everyone!
 
On Tue, Jun 18, 2019 at 8:13 PM Ilya Matiach <[hidden email]> wrote:

Hi Andrew,

I tried to do something similar to that in the LightGBM classifier/regressor/ranker in mmlspark package, I try to use the spark conf and if not configured I get the processors from the JVM directly:

https://github.com/Azure/mmlspark/blob/master/src/lightgbm/src/main/scala/LightGBMUtils.scala#L172

 

If you know of a better way, please let me know!

 

    val spark = dataset.sparkSession

    try {

      val confCores = spark.sparkContext.getConf

        .get("spark.executor.cores").toInt

      val confTaskCpus = spark.sparkContext.getConf

        .get("spark.task.cpus", "1").toInt

      confCores / confTaskCpus

    } catch {

      case _: NoSuchElementException =>

        // If spark.executor.cores is not defined, get the cores per JVM

        import spark.implicits._

        val numMachineCores = spark.range(0, 1)

          .map(_ => java.lang.Runtime.getRuntime.availableProcessors).collect.head

        numMachineCores

    }

 

Thank you, Ilya

 

From: Andrew Melo <[hidden email]>
Sent: Tuesday, June 18, 2019 11:32 AM
To: dev <[hidden email]>
Subject: Detect executor core count

 

Hello,

 

Is there a way to detect the number of cores allocated for an executor within a java-based InputPartitionReader?

 

Thanks!

Andrew