Issues: Generate JSON with null values in Spark 2.0.x

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Issues: Generate JSON with null values in Spark 2.0.x

Chetan Khatri
Hello Dev / Users,

I am working with PySpark Code migration to scala, with Python - Iterating Spark with dictionary and generating JSON with null is possible with json.dumps() which will be converted to SparkSQL[Row] but in scala how can we generate json will null values as a Dataframe ?

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Issues: Generate JSON with null values in Spark 2.0.x

Dongjin Lee
Hello Chetan,

Could you post some code? If I understood correctly, you are trying to save JSON like:

{
  "first_name": "Dongjin",
  "last_name: null
}

not in omitted form, like:

{
  "first_name": "Dongjin"
}

right?

- Dongjin

On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <[hidden email]> wrote:
Hello Dev / Users,

I am working with PySpark Code migration to scala, with Python - Iterating Spark with dictionary and generating JSON with null is possible with json.dumps() which will be converted to SparkSQL[Row] but in scala how can we generate json will null values as a Dataframe ?

Thanks.



--
Dongjin Lee

Software developer in Line+.
So interested in massive-scale machine learning.
Reply | Threaded
Open this post in threaded view
|

Re: Issues: Generate JSON with null values in Spark 2.0.x

Chetan Khatri
Exactly.

On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee <[hidden email]> wrote:
Hello Chetan,

Could you post some code? If I understood correctly, you are trying to save JSON like:

{
  "first_name": "Dongjin",
  "last_name: null
}

not in omitted form, like:

{
  "first_name": "Dongjin"
}

right?

- Dongjin

On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <[hidden email]> wrote:
Hello Dev / Users,

I am working with PySpark Code migration to scala, with Python - Iterating Spark with dictionary and generating JSON with null is possible with json.dumps() which will be converted to SparkSQL[Row] but in scala how can we generate json will null values as a Dataframe ?

Thanks.



--
Dongjin Lee

Software developer in Line+.
So interested in massive-scale machine learning.

Reply | Threaded
Open this post in threaded view
|

Re: Issues: Generate JSON with null values in Spark 2.0.x

Dongjin Lee
Hi Chetan,

Sadly, you can not; Spark is configured to ignore the null values when writing JSON. (check JacksonMessageWriter and find JsonInclude.Include.NON_NULL from the code.) If you want that functionality, it would be much better to file the problem to JIRA.

Best,
Dongjin

On Mon, Mar 20, 2017 at 4:44 PM, Chetan Khatri <[hidden email]> wrote:
Exactly.

On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee <[hidden email]> wrote:
Hello Chetan,

Could you post some code? If I understood correctly, you are trying to save JSON like:

{
  "first_name": "Dongjin",
  "last_name: null
}

not in omitted form, like:

{
  "first_name": "Dongjin"
}

right?

- Dongjin

On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <[hidden email]> wrote:
Hello Dev / Users,

I am working with PySpark Code migration to scala, with Python - Iterating Spark with dictionary and generating JSON with null is possible with json.dumps() which will be converted to SparkSQL[Row] but in scala how can we generate json will null values as a Dataframe ?

Thanks.



--
Dongjin Lee

Software developer in Line+.
So interested in massive-scale machine learning.




--
Dongjin Lee

Software developer in Line+.
So interested in massive-scale machine learning.