Spark2 sql jdbc issue

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark2 sql jdbc issue

busotir
This post has NOT been accepted by the mailing list yet.
Hello,

Could you please take a look at my issue and help me if possible. Here is code:

Dataset<Row> ds1 = spark.readStream().format("kafka").option("kafka.bootstrap.servers", brokers)
                                .option("subscribe", topics).load();
ds1.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)");
DataFrameWriter<Row> dfw = new DataFrameWriter<Row>(ds1);
dfw.mode(SaveMode.Overwrite).format("jdbc").jdbc("jdbc:mysql://localhost:3306/tab2", table, prop);


 I get an exception in method .jdbc("jdbc:mysql://localhost:3306/tab2", table, prop):

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: jdbc. Please find packages at http://spark.apache.org/third-party-projects.html
        at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:594)
        at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
        at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
        at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:516)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)
        at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:446)
        at com.ooooooo.bigdata.jobs.KafkaToRdbms.runIt(KafkaToRdbms.java:62)
        at com.ooooooo.bigdata.jobs.KafkaToRdbms.main(KafkaToRdbms.java:29)
Caused by: java.lang.ClassNotFoundException: jdbc.DefaultSource
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$25$$anonfun$apply$13.apply(DataSource.scala:579)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$25$$anonfun$apply$13.apply(DataSource.scala:579)
        at scala.util.Try$.apply(Try.scala:192)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$25.apply(DataSource.scala:579)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$25.apply(DataSource.scala:579)
        at scala.util.Try.orElse(Try.scala:84)
        at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:579)


I'm using Spark 2.1.1 version, here is part of build.gradle:

    compile group: 'mysql', name: 'mysql-connector-java', version: '5.1.37'
    compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.1.1'
    compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.1.1'
    compile group: 'org.apache.spark', name: 'spark-sql-kafka-0-10_2.11', version: '2.1.1'

BR, Ilya
Loading...