the scheme in stream reader

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

the scheme in stream reader

萝卜丝炒饭
Hi all,

L set the scheme for  DataStreamReader but when I print the scheme.It just printed:
root
|--value:string (nullable=true)

My code is

val line = ss.readStream.format("socket")
.option("ip",xxx)
.option("port",xxx)
.scheme(StructField("name",StringType)::(StructField("age", IntegerType))).load
line.printSchema

My spark version is 2.1.0.
I want the printSchema prints the schema I set in the code.How should I do please?
And my original target is the received data from socket is handled as schema directly.What should I do please?

thanks
Fei Shao






Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: the scheme in stream reader

Michael Armbrust
The socket source can't know how to parse your data.  I think the right thing would be for it to throw an exception saying that you can't set the schema here.  Would you mind opening a JIRA ticket?

If you are trying to parse data from something like JSON then you should use from_json` on the value returned.

On Sun, Jun 18, 2017 at 12:27 AM, 萝卜丝炒饭 <[hidden email]> wrote:
Hi all,

L set the scheme for  DataStreamReader but when I print the scheme.It just printed:
root
|--value:string (nullable=true)

My code is

val line = ss.readStream.format("socket")
.option("ip",xxx)
.option("port",xxx)
.scheme(StructField("name",StringType)::(StructField("age", IntegerType))).load
line.printSchema

My spark version is 2.1.0.
I want the printSchema prints the schema I set in the code.How should I do please?
And my original target is the received data from socket is handled as schema directly.What should I do please?

thanks
Fei Shao







Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: the scheme in stream reader

萝卜丝炒饭
In reply to this post by 萝卜丝炒饭
Hi ,
I have submitted a JIRA for this issue.
The link is 
https://issues.apache.org/jira/browse/SPARK-21147
thanks 
Fei Shao
---Original---
From: "Michael Armbrust"<[hidden email]>
Date: 2017/6/20 03:06:49
To: "萝卜丝炒饭"<[hidden email]>;
Cc: "user"<[hidden email]>;"dev"<[hidden email]>;
Subject: Re: the scheme in stream reader

The socket source can't know how to parse your data.  I think the right thing would be for it to throw an exception saying that you can't set the schema here.  Would you mind opening a JIRA ticket?

If you are trying to parse data from something like JSON then you should use from_json` on the value returned.

On Sun, Jun 18, 2017 at 12:27 AM, 萝卜丝炒饭 <[hidden email]> wrote:
Hi all,

L set the scheme for  DataStreamReader but when I print the scheme.It just printed:
root
|--value:string (nullable=true)

My code is

val line = ss.readStream.format("socket")
.option("ip",xxx)
.option("port",xxx)
.scheme(StructField("name",StringType)::(StructField("age", IntegerType))).load
line.printSchema

My spark version is 2.1.0.
I want the printSchema prints the schema I set in the code.How should I do please?
And my original target is the received data from socket is handled as schema directly.What should I do please?

thanks
Fei Shao







Loading...