There is a weird problem with spark when handling native dependency code:
I want to use a library (JAI) with spark to parse some spatial raster files. Unfortunately, there are some strange issues. JAI only works when running via the build tool i.e. `sbt run` when executed in spark.
When executed via spark-submit the error is:
java.lang.IllegalArgumentException: The input argument(s) may not be null.
Which looks like some native dependency (I think GEOS is running in the background) is not there correctly.
Assuming something is wrong with the class path I tried to run a plain java/scala function. but this one works just fine.
Is spark messing with the class paths?
I created a minimal example here:
Hope someone can shed some light on this problem,
Maybe not related, but in general geotools are not thread safe,so
using from workers is most likely a gamble.
On 06/03/2017 01:26 AM, Georg Heiler wrote:
When tested without any parallelism the same problem persists. Actually, NiFi shows the same issues. So probably it is not related to spark.
Maciej Szymkiewicz <[hidden email]> schrieb am Sa., 3. Juni 2017 um 01:37 Uhr:
I read http://techblog.applift.com/upgrading-spark an conducted further research. I think there is some problem with the class loader. Unfortunately, so far, I did not get it to work.
Georg Heiler <[hidden email]> schrieb am Sa., 3. Juni 2017 um 08:27 Uhr:
|Free forum by Nabble||Edit this page|