leveldbjni dependency

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

leveldbjni dependency

mundaym
Hi all,

I am currently building Spark from source and also have to build leveldbjni
from source because the binary release (which is platform dependent) in
mvnrepository does not support my target platform (s390x). People have run
into similar problems when building for other platforms too (notably Spark
builds on arm64 pull in an alternative binary release).

The last binary release of leveldbjni was in 2013 and it does not appear to
be actively maintained. I suspect the lack of new binary releases will cause
more issues as time goes on. I am therefore curious whether anyone had any
thoughts over substituting it with an alternative library. In particular I
am interested in whether there are any constraints an alternative would need
to adhere to:

1. Would an alternative need to be leveldb compatible?
2. How performance sensitive are the uses of leveldbjni in Spark? Could a
pure Java library be used instead?

Specifically I am thinking possible alternatives that are actively
maintained might be something like https://github.com/dain/leveldb or even
RocksDB. Or perhaps there is another key-value store used in the Apache
ecosystem that could be adopted.

Thanks,
Michael



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: leveldbjni dependency

Sean Owen-2
I think you could also fork and try to compile it for s390x - I think
that's what happened for ARM.
Perf probably isn't a big deal there, but that might be easier than
rewriting or replacing it.

On Thu, Sep 10, 2020 at 1:29 AM mundaym <[hidden email]> wrote:

>
> Hi all,
>
> I am currently building Spark from source and also have to build leveldbjni
> from source because the binary release (which is platform dependent) in
> mvnrepository does not support my target platform (s390x). People have run
> into similar problems when building for other platforms too (notably Spark
> builds on arm64 pull in an alternative binary release).
>
> The last binary release of leveldbjni was in 2013 and it does not appear to
> be actively maintained. I suspect the lack of new binary releases will cause
> more issues as time goes on. I am therefore curious whether anyone had any
> thoughts over substituting it with an alternative library. In particular I
> am interested in whether there are any constraints an alternative would need
> to adhere to:
>
> 1. Would an alternative need to be leveldb compatible?
> 2. How performance sensitive are the uses of leveldbjni in Spark? Could a
> pure Java library be used instead?
>
> Specifically I am thinking possible alternatives that are actively
> maintained might be something like https://github.com/dain/leveldb or even
> RocksDB. Or perhaps there is another key-value store used in the Apache
> ecosystem that could be adopted.
>
> Thanks,
> Michael
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]