spark messing up handling of native dependency code?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

spark messing up handling of native dependency code?

geoHeil
Hi,

There is a weird problem with spark when handling native dependency code:
I want to use a library (JAI) with spark to parse some spatial raster files. Unfortunately, there are some strange issues. JAI only works when running via the build tool i.e. `sbt run` when executed in spark.

When executed via spark-submit the error is:

    java.lang.IllegalArgumentException: The input argument(s) may not be null.
    at javax.media.jai.ParameterBlockJAI.getDefaultMode(ParameterBlockJAI.java:136)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:157)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:178)
    at org.geotools.process.raster.PolygonExtractionProcess.execute(PolygonExtractionProcess.java:171)

Which looks like some native dependency (I think GEOS is running in the background) is not there correctly.

Assuming something is wrong with the class path I tried to run a plain java/scala function. but this one works just fine.

Is spark messing with the class paths?

I created a minimal example here:


Hope someone can shed some light on this problem,
Regards,
Georg 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: spark messing up handling of native dependency code?

zero323

Maybe not related, but in general geotools are not thread safe,so using from workers is most likely a gamble.

On 06/03/2017 01:26 AM, Georg Heiler wrote:
Hi,

There is a weird problem with spark when handling native dependency code:
I want to use a library (JAI) with spark to parse some spatial raster files. Unfortunately, there are some strange issues. JAI only works when running via the build tool i.e. `sbt run` when executed in spark.

When executed via spark-submit the error is:

    java.lang.IllegalArgumentException: The input argument(s) may not be null.
    at javax.media.jai.ParameterBlockJAI.getDefaultMode(ParameterBlockJAI.java:136)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:157)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:178)
    at org.geotools.process.raster.PolygonExtractionProcess.execute(PolygonExtractionProcess.java:171)

Which looks like some native dependency (I think GEOS is running in the background) is not there correctly.

Assuming something is wrong with the class path I tried to run a plain java/scala function. but this one works just fine.

Is spark messing with the class paths?

I created a minimal example here:


Hope someone can shed some light on this problem,
Regards,
Georg 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: spark messing up handling of native dependency code?

geoHeil
When tested without any parallelism the same problem persists. Actually, NiFi shows the same issues. So probably it is not related to spark.

Maciej Szymkiewicz <[hidden email]> schrieb am Sa., 3. Juni 2017 um 01:37 Uhr:

Maybe not related, but in general geotools are not thread safe,so using from workers is most likely a gamble.

On 06/03/2017 01:26 AM, Georg Heiler wrote:
Hi,

There is a weird problem with spark when handling native dependency code:
I want to use a library (JAI) with spark to parse some spatial raster files. Unfortunately, there are some strange issues. JAI only works when running via the build tool i.e. `sbt run` when executed in spark.

When executed via spark-submit the error is:

    java.lang.IllegalArgumentException: The input argument(s) may not be null.
    at javax.media.jai.ParameterBlockJAI.getDefaultMode(ParameterBlockJAI.java:136)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:157)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:178)
    at org.geotools.process.raster.PolygonExtractionProcess.execute(PolygonExtractionProcess.java:171)

Which looks like some native dependency (I think GEOS is running in the background) is not there correctly.

Assuming something is wrong with the class path I tried to run a plain java/scala function. but this one works just fine.

Is spark messing with the class paths?

I created a minimal example here:


Hope someone can shed some light on this problem,
Regards,
Georg 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: spark messing up handling of native dependency code?

geoHeil
I read http://techblog.applift.com/upgrading-spark  an conducted further research. I think there is some problem with the class loader. Unfortunately, so far, I did not get it to work.

Georg Heiler <[hidden email]> schrieb am Sa., 3. Juni 2017 um 08:27 Uhr:
When tested without any parallelism the same problem persists. Actually, NiFi shows the same issues. So probably it is not related to spark.

Maciej Szymkiewicz <[hidden email]> schrieb am Sa., 3. Juni 2017 um 01:37 Uhr:

Maybe not related, but in general geotools are not thread safe,so using from workers is most likely a gamble.

On 06/03/2017 01:26 AM, Georg Heiler wrote:
Hi,

There is a weird problem with spark when handling native dependency code:
I want to use a library (JAI) with spark to parse some spatial raster files. Unfortunately, there are some strange issues. JAI only works when running via the build tool i.e. `sbt run` when executed in spark.

When executed via spark-submit the error is:

    java.lang.IllegalArgumentException: The input argument(s) may not be null.
    at javax.media.jai.ParameterBlockJAI.getDefaultMode(ParameterBlockJAI.java:136)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:157)
    at javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:178)
    at org.geotools.process.raster.PolygonExtractionProcess.execute(PolygonExtractionProcess.java:171)

Which looks like some native dependency (I think GEOS is running in the background) is not there correctly.

Assuming something is wrong with the class path I tried to run a plain java/scala function. but this one works just fine.

Is spark messing with the class paths?

I created a minimal example here:


Hope someone can shed some light on this problem,
Regards,
Georg 

Loading...