Spark json data - avro schema validation

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Spark json data - avro schema validation

Barath Ramamoorthy

Hi I have a spark streaming application which receives logs that has encoded json in it. The json complies to a avro schema and part of the process I m converting the json to a data class which of course is a row in dataset. It’s a nested object indeed. 

In this scenario I m looking to validate the inbound json to see if it complies to the definition of avro schema. I m not finding any approach that already exists to perform this validation or not aware of . I am hoping to get some direction from this group to get going on the validation front.