adding shutdownmanagerhook to spark.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

adding shutdownmanagerhook to spark.

Nasrulla Khan Haris

HI All,

 

I am trying to add shutdown hook, but looks like shutdown manager object requires the package to be spark only, is there any other API that can help me to do this ?

https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala

 

I can see that hiveserver2 in https://github.com/apache/spark/commit/515708d5f33d5acdb4206c626192d1838f8e691f uses ShutdownHookManager in different package. But It expects me to have the package name “org.apache.spark”

 

Thanks,

Nasrulla

 

Reply | Threaded
Open this post in threaded view
|

Re: adding shutdownmanagerhook to spark.

Sean Owen-2
Spark just adds a hook to the mechanism that Hadoop exposes. You can
do the same. You shouldn't use Spark's.

On Mon, May 13, 2019 at 6:11 PM Nasrulla Khan Haris
<[hidden email]> wrote:

>
> HI All,
>
>
>
> I am trying to add shutdown hook, but looks like shutdown manager object requires the package to be spark only, is there any other API that can help me to do this ?
>
> https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala
>
>
>
> I can see that hiveserver2 in https://github.com/apache/spark/commit/515708d5f33d5acdb4206c626192d1838f8e691f uses ShutdownHookManager in different package. But It expects me to have the package name “org.apache.spark”
>
>
>
> Thanks,
>
> Nasrulla
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: adding shutdownmanagerhook to spark.

Nasrulla Khan Haris
Thanks Sean

Nasrulla

-----Original Message-----
From: Sean Owen <[hidden email]>
Sent: Monday, May 13, 2019 4:16 PM
To: Nasrulla Khan Haris <[hidden email]>
Cc: [hidden email]
Subject: Re: adding shutdownmanagerhook to spark.

Spark just adds a hook to the mechanism that Hadoop exposes. You can do the same. You shouldn't use Spark's.

On Mon, May 13, 2019 at 6:11 PM Nasrulla Khan Haris <[hidden email]> wrote:

>
> HI All,
>
>
>
> I am trying to add shutdown hook, but looks like shutdown manager object requires the package to be spark only, is there any other API that can help me to do this ?
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fapache%2Fspark%2Fblob%2Fv2.4.0%2Fcore%2Fsrc%2Fmain%2Fscala%2F
> org%2Fapache%2Fspark%2Futil%2FShutdownHookManager.scala&amp;data=01%7C
> 01%7CNasrulla.Khan%40microsoft.com%7C8aadef9c8cf844139eca08d6d7f90058%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1&amp;sdata=Si4C7iF8PN5%2FJDDQy8F
> BzzuFh2XsYD%2BfjMtT5wLxbr4%3D&amp;reserved=0
>
>
>
> I can see that hiveserver2 in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fcommit%2F515708d5f33d5acdb4206c626192d1838f8e691f&amp;data=01%7C01%7CNasrulla.Khan%40microsoft.com%7C8aadef9c8cf844139eca08d6d7f90058%7C72f988bf86f141af91ab2d7cd011db47%7C1&amp;sdata=AGNpXhauHp4Wkbe%2BVesAbO5H%2BJFhwApMCRcAi2oROl0%3D&amp;reserved=0 uses ShutdownHookManager in different package. But It expects me to have the package name “org.apache.spark”
>
>
>
> Thanks,
>
> Nasrulla
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]