Spark 2.4.x and 3.x datasourcev2 api documentation & references

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark 2.4.x and 3.x datasourcev2 api documentation & references

J-YUE.CHUN
Hi,
i have been following closely the new proposed api, and seems until now some
lack of documentation... specially for people trying to implement them in
JAVA.

Anyway to give out some reference as starting point for java developpers as
tests classes  ?

(anyway new api looks promising) !





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.4.x and 3.x datasourcev2 api documentation & references

cloud0fan
For now you can take a look at `DataSourceV2Suite`, which contains both Java/Scala implementations. There is also an ongoing PR to implement catalog APIs for JDBC: https://github.com/apache/spark/pull/27345

We are still working on the user guide.

On Mon, Mar 16, 2020 at 4:59 AM MadDoxX <[hidden email]> wrote:
Hi,
i have been following closely the new proposed api, and seems until now some
lack of documentation... specially for people trying to implement them in
JAVA.

Anyway to give out some reference as starting point for java developpers as
tests classes  ?

(anyway new api looks promising) !





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.4.x and 3.x datasourcev2 api documentation & references

J-YUE.CHUN
This post was updated on .
Thanks for the link !

i was able to code my first datasources in microbatch and continuous by
using the v2 api...

i was wondering if there are some equivalent to SparkListener for
microbatching & continuous stream (where one could use the
spark.extra.listener) ~ monitoring :)

btw i saw some major classes name refactoring from 2.4.x to 3.x-rc, is it still going to happen a lot ? or its more or less stabilized ?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org