Clarification on the commit protocol

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Clarification on the commit protocol

rahul c
Hi team,

Just wanted to understand.
Is DBIO commit protocol available in open source spark version ?
Reply | Threaded
Open this post in threaded view
|

Re: Clarification on the commit protocol

Michael Armbrust
No, it is not. Although the commit protocol has mostly been superseded by Delta Lake, which is available as a separate open source project that works natively with Apache Spark. In contrast to the commit protocol, Delta can guarantee full ACID (rather than just partition level atomicity). It also has better performance in many cases, as it reduces the amount of metadata that needs to be retrieved from the storage system.

On Wed, Feb 26, 2020 at 9:36 PM rahul c <[hidden email]> wrote:
Hi team,

Just wanted to understand.
Is DBIO commit protocol available in open source spark version ?