Adding Custom finalize method to RDDs.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding Custom finalize method to RDDs.

Nasrulla Khan Haris

Hi All,

 

Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ?

 

Thanks,

Nasrulla

 

Reply | Threaded
Open this post in threaded view
|

RE: Adding Custom finalize method to RDDs.

Nasrulla Khan Haris

Hello Everyone,

Is there a way  to do it from user-code ?

 

Thanks,

Nasrulla

 

From: Nasrulla Khan Haris <[hidden email]>
Sent: Sunday, June 9, 2019 5:30 PM
To: [hidden email]
Subject: Adding Custom finalize method to RDDs.

 

Hi All,

 

Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ?

 

Thanks,

Nasrulla

 

Reply | Threaded
Open this post in threaded view
|

Re: Adding Custom finalize method to RDDs.

Vinoo Ganesh

Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11  https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here?

 

From: Nasrulla Khan Haris <[hidden email]>
Date: Monday, June 10, 2019 at 15:44
To: "[hidden email]" <[hidden email]>
Subject: RE: Adding Custom finalize method to RDDs.

 

Hello Everyone,

Is there a way  to do it from user-code ?

 

Thanks,

Nasrulla

 

From: Nasrulla Khan Haris <[hidden email]>
Sent: Sunday, June 9, 2019 5:30 PM
To: [hidden email]
Subject: Adding Custom finalize method to RDDs.

 

Hi All,

 

Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ?

 

Thanks,

Nasrulla

 

Reply | Threaded
Open this post in threaded view
|

RE: Adding Custom finalize method to RDDs.

Nasrulla Khan Haris

I want to delete some files which I created In my datasource api,  as soon as the RDD is cleaned up.

 

Thanks,

Nasrulla

 

From: Vinoo Ganesh <[hidden email]>
Sent: Monday, June 10, 2019 1:32 PM
To: Nasrulla Khan Haris <[hidden email]>; [hidden email]
Subject: Re: Adding Custom finalize method to RDDs.

 

Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11  https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here?

 

From: Nasrulla Khan Haris <[hidden email]>
Date: Monday, June 10, 2019 at 15:44
To: "[hidden email]" <[hidden email]>
Subject: RE: Adding Custom finalize method to RDDs.

 

Hello Everyone,

Is there a way  to do it from user-code ?

 

Thanks,

Nasrulla

 

From: Nasrulla Khan Haris <[hidden email]>
Sent: Sunday, June 9, 2019 5:30 PM
To: [hidden email]
Subject: Adding Custom finalize method to RDDs.

 

Hi All,

 

Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ?

 

Thanks,

Nasrulla

 

Reply | Threaded
Open this post in threaded view
|

Re: Adding Custom finalize method to RDDs.

Phillip Henry
That's not the kind of thing a finalize method was ever supposed to do.

Use a try/finally block instead.

Phillip


On Wed, 12 Jun 2019, 00:01 Nasrulla Khan Haris, <[hidden email]> wrote:

I want to delete some files which I created In my datasource api,  as soon as the RDD is cleaned up.

 

Thanks,

Nasrulla

 

From: Vinoo Ganesh <[hidden email]>
Sent: Monday, June 10, 2019 1:32 PM
To: Nasrulla Khan Haris <[hidden email]>; [hidden email]
Subject: Re: Adding Custom finalize method to RDDs.

 

Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11  https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here?

 

From: Nasrulla Khan Haris <[hidden email]>
Date: Monday, June 10, 2019 at 15:44
To: "[hidden email]" <[hidden email]>
Subject: RE: Adding Custom finalize method to RDDs.

 

Hello Everyone,

Is there a way  to do it from user-code ?

 

Thanks,

Nasrulla

 

From: Nasrulla Khan Haris <[hidden email]>
Sent: Sunday, June 9, 2019 5:30 PM
To: [hidden email]
Subject: Adding Custom finalize method to RDDs.

 

Hi All,

 

Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ?

 

Thanks,

Nasrulla

 

Reply | Threaded
Open this post in threaded view
|

RE: Adding Custom finalize method to RDDs.

Nasrulla Khan Haris

We cannot have control over RDD going out of scope from memory as it was handled by JVM. Thus I am not sure try and finalize will help.

 Thus I wanted to use some mechanism to cleanup of some temporary data which is created by RDD immediately as soon as it goes out of scope.

 

Any ideas ?

 

Thanks,

Nasrulla

 

From: Phillip Henry <[hidden email]>
Sent: Tuesday, June 11, 2019 11:28 PM
To: Nasrulla Khan Haris <[hidden email]>
Cc: Vinoo Ganesh <[hidden email]>; [hidden email]
Subject: Re: Adding Custom finalize method to RDDs.

 

That's not the kind of thing a finalize method was ever supposed to do.

 

Use a try/finally block instead.

 

Phillip

 

 

On Wed, 12 Jun 2019, 00:01 Nasrulla Khan Haris, <[hidden email]> wrote:

I want to delete some files which I created In my datasource api,  as soon as the RDD is cleaned up.

 

Thanks,

Nasrulla

 

From: Vinoo Ganesh <[hidden email]>
Sent: Monday, June 10, 2019 1:32 PM
To: Nasrulla Khan Haris <[hidden email]>; [hidden email]
Subject: Re: Adding Custom finalize method to RDDs.

 

Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11  https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here?

 

From: Nasrulla Khan Haris <[hidden email]>
Date: Monday, June 10, 2019 at 15:44
To: "[hidden email]" <[hidden email]>
Subject: RE: Adding Custom finalize method to RDDs.

 

Hello Everyone,

Is there a way  to do it from user-code ?

 

Thanks,

Nasrulla

 

From: Nasrulla Khan Haris <[hidden email]>
Sent: Sunday, June 9, 2019 5:30 PM
To: [hidden email]
Subject: Adding Custom finalize method to RDDs.

 

Hi All,

 

Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ?

 

Thanks,

Nasrulla

 

Reply | Threaded
Open this post in threaded view
|

Re: Adding Custom finalize method to RDDs.

Phillip Henry
If you control the codebase, you control when an RDD goes out of scope. Or am I missing something?

(Note that finalize will not necessarily executed when an object goes out of scope but when the GC runs at some indeterminate point in the future. Please avoid using finalize for the kind of task you're trying to do. It's not what it was designed for. Better to pay more attention to  house-keeping in your own code.)



On Wed, Jun 12, 2019 at 9:11 PM Nasrulla Khan Haris <[hidden email]> wrote:

We cannot have control over RDD going out of scope from memory as it was handled by JVM. Thus I am not sure try and finalize will help.

 Thus I wanted to use some mechanism to cleanup of some temporary data which is created by RDD immediately as soon as it goes out of scope.

 

Any ideas ?

 

Thanks,

Nasrulla

 

From: Phillip Henry <[hidden email]>
Sent: Tuesday, June 11, 2019 11:28 PM
To: Nasrulla Khan Haris <[hidden email]>
Cc: Vinoo Ganesh <[hidden email]>; [hidden email]
Subject: Re: Adding Custom finalize method to RDDs.

 

That's not the kind of thing a finalize method was ever supposed to do.

 

Use a try/finally block instead.

 

Phillip

 

 

On Wed, 12 Jun 2019, 00:01 Nasrulla Khan Haris, <[hidden email]> wrote:

I want to delete some files which I created In my datasource api,  as soon as the RDD is cleaned up.

 

Thanks,

Nasrulla

 

From: Vinoo Ganesh <[hidden email]>
Sent: Monday, June 10, 2019 1:32 PM
To: Nasrulla Khan Haris <[hidden email]>; [hidden email]
Subject: Re: Adding Custom finalize method to RDDs.

 

Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11  https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here?

 

From: Nasrulla Khan Haris <[hidden email]>
Date: Monday, June 10, 2019 at 15:44
To: "[hidden email]" <[hidden email]>
Subject: RE: Adding Custom finalize method to RDDs.

 

Hello Everyone,

Is there a way  to do it from user-code ?

 

Thanks,

Nasrulla

 

From: Nasrulla Khan Haris <[hidden email]>
Sent: Sunday, June 9, 2019 5:30 PM
To: [hidden email]
Subject: Adding Custom finalize method to RDDs.

 

Hi All,

 

Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ?

 

Thanks,

Nasrulla