Property spark.sql.streaming.minBatchesToRetain

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Property spark.sql.streaming.minBatchesToRetain

German Schiavon
Hello all,

I wanted to ask if this property is still active? I can't find it in the doc https://spark.apache.org/docs/latest/configuration.html or anywhere in the code(only in Tests).

If so, should we remove it? 
val MIN_BATCHES_TO_RETAIN = buildConf("spark.sql.streaming.minBatchesToRetain")
.internal()
.doc("The minimum number of batches that must be retained and made recoverable.")
.version("2.1.1")
.intConf
.createWithDefault(100)
Reply | Threaded
Open this post in threaded view
|

Re: Property spark.sql.streaming.minBatchesToRetain

Maxim Gekk

On Tue, Mar 9, 2021 at 3:27 PM German Schiavon <[hidden email]> wrote:
Hello all,

I wanted to ask if this property is still active? I can't find it in the doc https://spark.apache.org/docs/latest/configuration.html or anywhere in the code(only in Tests).

If so, should we remove it? 
val MIN_BATCHES_TO_RETAIN = buildConf("spark.sql.streaming.minBatchesToRetain")
.internal()
.doc("The minimum number of batches that must be retained and made recoverable.")
.version("2.1.1")
.intConf
.createWithDefault(100)
Reply | Threaded
Open this post in threaded view
|

Re: Property spark.sql.streaming.minBatchesToRetain

German Schiavon
Hey Maxim,

ok! I didn't see them.

Is this property documented somewhere?

Thanks! 

On Tue, 9 Mar 2021 at 13:57, Maxim Gekk <[hidden email]> wrote:

On Tue, Mar 9, 2021 at 3:27 PM German Schiavon <[hidden email]> wrote:
Hello all,

I wanted to ask if this property is still active? I can't find it in the doc https://spark.apache.org/docs/latest/configuration.html or anywhere in the code(only in Tests).

If so, should we remove it? 
val MIN_BATCHES_TO_RETAIN = buildConf("spark.sql.streaming.minBatchesToRetain")
.internal()
.doc("The minimum number of batches that must be retained and made recoverable.")
.version("2.1.1")
.intConf
.createWithDefault(100)
Reply | Threaded
Open this post in threaded view
|

Re: Property spark.sql.streaming.minBatchesToRetain

Jungtaek Lim-2
That property decides how many log files (log file is created per batch per type - types are like offsets, commits, etc.) to retain on the checkpoint.

Unless you're struggling with a small files problem on checkpoint, you wouldn't need to tune the value. I guess that's why the configuration is marked as "internal" meaning just some admins need to know about such configuration.

On Wed, Mar 10, 2021 at 3:58 AM German Schiavon <[hidden email]> wrote:
Hey Maxim,

ok! I didn't see them.

Is this property documented somewhere?

Thanks! 

On Tue, 9 Mar 2021 at 13:57, Maxim Gekk <[hidden email]> wrote:

On Tue, Mar 9, 2021 at 3:27 PM German Schiavon <[hidden email]> wrote:
Hello all,

I wanted to ask if this property is still active? I can't find it in the doc https://spark.apache.org/docs/latest/configuration.html or anywhere in the code(only in Tests).

If so, should we remove it? 
val MIN_BATCHES_TO_RETAIN = buildConf("spark.sql.streaming.minBatchesToRetain")
.internal()
.doc("The minimum number of batches that must be retained and made recoverable.")
.version("2.1.1")
.intConf
.createWithDefault(100)
Reply | Threaded
Open this post in threaded view
|

Re: Property spark.sql.streaming.minBatchesToRetain

German Schiavon
OK got it!


Thanks! 

On Tue, 9 Mar 2021 at 21:17, Jungtaek Lim <[hidden email]> wrote:
That property decides how many log files (log file is created per batch per type - types are like offsets, commits, etc.) to retain on the checkpoint.

Unless you're struggling with a small files problem on checkpoint, you wouldn't need to tune the value. I guess that's why the configuration is marked as "internal" meaning just some admins need to know about such configuration.

On Wed, Mar 10, 2021 at 3:58 AM German Schiavon <[hidden email]> wrote:
Hey Maxim,

ok! I didn't see them.

Is this property documented somewhere?

Thanks! 

On Tue, 9 Mar 2021 at 13:57, Maxim Gekk <[hidden email]> wrote:

On Tue, Mar 9, 2021 at 3:27 PM German Schiavon <[hidden email]> wrote:
Hello all,

I wanted to ask if this property is still active? I can't find it in the doc https://spark.apache.org/docs/latest/configuration.html or anywhere in the code(only in Tests).

If so, should we remove it? 
val MIN_BATCHES_TO_RETAIN = buildConf("spark.sql.streaming.minBatchesToRetain")
.internal()
.doc("The minimum number of batches that must be retained and made recoverable.")
.version("2.1.1")
.intConf
.createWithDefault(100)