how can solve this error

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

how can solve this error

hagersaleh
I write code to connect kafka with spark using python and I run code on
jupyer
my code
import os
#os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars
/home/hadoop/Desktop/spark-program/kafka/spark-streaming-kafka-0-8-assembly_2.10-2.0.0-preview.jar
pyspark-shell'
os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 pyspark-shell"

os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
org.apache.spark:spark-streaming-kafka-0-8_2.11:2.3.0 pyspark-shell"

import pyspark
from pyspark.streaming.kafka import KafkaUtils
from pyspark.streaming import StreamingContext

#sc = SparkContext()
ssc = StreamingContext(sc,1)

broker = "iotmsgs"
directKafkaStream = KafkaUtils.createDirectStream(ssc, ["test1"],
                        {"metadata.broker.list": broker})
directKafkaStream.pprint()
ssc.start()

error display
Spark Streaming's Kafka libraries not found in class path. Try one of the
following.

  1. Include the Kafka library and its dependencies with in the
     spark-submit command as

     $ bin/spark-submit --packages
org.apache.spark:spark-streaming-kafka-0-8:2.3.0 ...

  2. Download the JAR of the artifact from Maven Central
http://search.maven.org/,
     Group Id = org.apache.spark, Artifact Id =
spark-streaming-kafka-0-8-assembly, Version = 2.3.0.
     Then, include the jar in the spark-submit command as

     $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: how can solve this error

cloud0fan

On Mon, Sep 17, 2018 at 4:46 AM hagersaleh <[hidden email]> wrote:
I write code to connect kafka with spark using python and I run code on
jupyer
my code
import os
#os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars
/home/hadoop/Desktop/spark-program/kafka/spark-streaming-kafka-0-8-assembly_2.10-2.0.0-preview.jar
pyspark-shell'
os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 pyspark-shell"

os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
org.apache.spark:spark-streaming-kafka-0-8_2.11:2.3.0 pyspark-shell"

import pyspark
from pyspark.streaming.kafka import KafkaUtils
from pyspark.streaming import StreamingContext

#sc = SparkContext()
ssc = StreamingContext(sc,1)

broker = "iotmsgs"
directKafkaStream = KafkaUtils.createDirectStream(ssc, ["test1"],
                        {"metadata.broker.list": broker})
directKafkaStream.pprint()
ssc.start()

error display
Spark Streaming's Kafka libraries not found in class path. Try one of the
following.

  1. Include the Kafka library and its dependencies with in the
     spark-submit command as

     $ bin/spark-submit --packages
org.apache.spark:spark-streaming-kafka-0-8:2.3.0 ...

  2. Download the JAR of the artifact from Maven Central
http://search.maven.org/,
     Group Id = org.apache.spark, Artifact Id =
spark-streaming-kafka-0-8-assembly, Version = 2.3.0.
     Then, include the jar in the spark-submit command as

     $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]