Behavior of SaveMode.Append when table is not present

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Behavior of SaveMode.Append when table is not present

Shubham Chaurasia
Hi,


When saving a DataFrame to a data source, if data/table already exists, contents of the DataFrame are expected to be appended to existing data

However it does not specify behavior when the table does not exist. 
Does that throw exception or create the table or a NO-OP?

Thanks,
Shubham
Reply | Threaded
Open this post in threaded view
|

Re: Behavior of SaveMode.Append when table is not present

Ryan Blue
Right now, it is up to the source implementation to decide what to do. I think path-based tables (with no metastore component) treat an append as an implicit create.

If you're thinking that relying on sources to interpret SaveMode is bad for consistent behavior, I agree. That's why the community adopted a proposal to standardize logical plans and the behavior expected of data sources for the v2 API.

On Thu, Nov 8, 2018 at 11:53 PM Shubham Chaurasia <[hidden email]> wrote:
Hi,


When saving a DataFrame to a data source, if data/table already exists, contents of the DataFrame are expected to be appended to existing data

However it does not specify behavior when the table does not exist. 
Does that throw exception or create the table or a NO-OP?

Thanks,
Shubham


--
Ryan Blue
Software Engineer
Netflix