Applying schema dynamically in dataframe

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Applying schema dynamically in dataframe

rahul c
Hi dev,

Currently I have a scenario where I am reading the data from Kafka using spark dataframe.

Multiple data sources ingest the data into kafka same topic.

My dataframe now has got json string from multiple sources. I have got key column from Kafka which can be used to identify which source data value it is.

In spark how can I now apply the schema dynamically using the key colum value.

Any help would be appreciated.

ThanksĀ and Regards
Rahul








Reply | Threaded
Open this post in threaded view
|

Re: Applying schema dynamically in dataframe

ZHANG Wei
May I get a sample scenario to understand the requirement?

--
Cheers,
-z

On Sat, 16 May 2020 11:45:03 +0530
rahul c <[hidden email]> wrote:

> Hi dev,
>
> Currently I have a scenario where I am reading the data from Kafka using
> spark dataframe.
>
> Multiple data sources ingest the data into kafka same topic.
>
> My dataframe now has got json string from multiple sources. I have got key
> column from Kafka which can be used to identify which source data value it
> is.
>
> In spark how can I now apply the schema dynamically using the key colum
> value.
>
> Any help would be appreciated.
>
> Thanks and Regards
> Rahul

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]