Schema inference for nested case class issue

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Schema inference for nested case class issue

El Houssain ALLAMI
Hi ,

i have nested case class that i wanted to fill with data read  from parquet but i got this error : cannot resolve '`fix`' given input columns: [sizeinternalname, sap_compo08_type ....]

case class Fix(a:String,b:String) 
case class Acc(fix:Fix, c:String)
spark.read.parquet("path").as[Acc]

it does not look inside nested case class 

any solution for this issue ? 

Many Thanks,
ELhoussain
Reply | Threaded
Open this post in threaded view
|

Re: Schema inference for nested case class issue

Sean Owen-2
user@ is the right place for these types of questions.
As the error says, you have a case class that defines a schema
including columns like 'fix' but these don't appear to be in your
DataFrame. It needs to match.

On Wed, Sep 4, 2019 at 6:44 AM El Houssain ALLAMI
<[hidden email]> wrote:

>
> Hi ,
>
> i have nested case class that i wanted to fill with data read  from parquet but i got this error : cannot resolve '`fix`' given input columns: [sizeinternalname, sap_compo08_type ....]
>
> case class Fix(a:String,b:String)
> case class Acc(fix:Fix, c:String)
> spark.read.parquet("path").as[Acc]
>
> it does not look inside nested case class
>
> any solution for this issue ?
>
> Many Thanks,
> ELhoussain

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]