Structured Streaming Source error

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Structured Streaming Source error

Sam Elamin
Hi Folks


I am getting a weird error when trying to write a BigQuery Structured Streaming source


Error:
java.lang.AbstractMethodError: com.samelamin.spark.bigquery.streaming.BigQuerySource.commit(Lorg/apache/spark/sql/execution/streaming/Offset;)V
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$1$$anonfun$apply$mcV$sp$5.apply(StreamExecution.scala:359)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$1$$anonfun$apply$mcV$sp$5.apply(StreamExecution.scala:358)


FYI if you are interested in Spark and BigQuery feel free to give my connector a go! Still trying to get structured streaming to it as a source hence this email. but you can use it as a sink! 


Regards
Sam



Reply | Threaded
Open this post in threaded view
|

Re: Structured Streaming Source error

Shixiong(Ryan) Zhu
You used one Spark version to compile your codes but another newer version to run. As the Source APIs are not stable, Spark doesn't guarantee that they are binary compatibility.

On Tue, Jan 31, 2017 at 1:39 PM, Sam Elamin <[hidden email]> wrote:
Hi Folks


I am getting a weird error when trying to write a BigQuery Structured Streaming source


Error:
java.lang.AbstractMethodError: com.samelamin.spark.bigquery.streaming.BigQuerySource.commit(Lorg/apache/spark/sql/execution/streaming/Offset;)V
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$1$$anonfun$apply$mcV$sp$5.apply(StreamExecution.scala:359)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$1$$anonfun$apply$mcV$sp$5.apply(StreamExecution.scala:358)


FYI if you are interested in Spark and BigQuery feel free to give my connector a go! Still trying to get structured streaming to it as a source hence this email. but you can use it as a sink! 


Regards
Sam




Reply | Threaded
Open this post in threaded view
|

Re: Structured Streaming Source error

Sam Elamin
Ha Ryan your everywhere,JIRA and maillist. I thought multitasking was a myth! 

Thanks for your help. It was using different versions! 

Regards
Sam

On Tue, Jan 31, 2017 at 9:48 PM, Shixiong(Ryan) Zhu <[hidden email]> wrote:
You used one Spark version to compile your codes but another newer version to run. As the Source APIs are not stable, Spark doesn't guarantee that they are binary compatibility.

On Tue, Jan 31, 2017 at 1:39 PM, Sam Elamin <[hidden email]> wrote:
Hi Folks


I am getting a weird error when trying to write a BigQuery Structured Streaming source


Error:
java.lang.AbstractMethodError: com.samelamin.spark.bigquery.streaming.BigQuerySource.commit(Lorg/apache/spark/sql/execution/streaming/Offset;)V
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$1$$anonfun$apply$mcV$sp$5.apply(StreamExecution.scala:359)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$constructNextBatch$1$$anonfun$apply$mcV$sp$5.apply(StreamExecution.scala:358)


FYI if you are interested in Spark and BigQuery feel free to give my connector a go! Still trying to get structured streaming to it as a source hence this email. but you can use it as a sink! 


Regards
Sam