Is spark.sql.codegen.factoryMode property really for tests only?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Is spark.sql.codegen.factoryMode property really for tests only?

Jacek Laskowski
Hi,

While reviewing the changes in 2.4 I stumbled upon spark.sql.codegen.factoryMode internal configuration property [1]. The doc says:

> Note that this config works only for tests.

Is that correct? I've got some doubts.

I found that it's used in UnsafeProjection.create [2] (through CodeGeneratorWithInterpretedFallback.createObject) which is used outside the tests and so made me think if "this config works only for tests" part is correct.

Are my doubts correct? If not, what am I missing? Thanks.


Reply | Threaded
Open this post in threaded view
|

Re: Is spark.sql.codegen.factoryMode property really for tests only?

Marco Gaido
Hi Jacek,

I do believe it is correct. Please check the method you mentioned (CodeGeneratorWithInterpretedFallback.createObject): the value is relevant only if Utils.isTesting.

Thanks,
Marco

Il giorno ven 16 nov 2018 alle ore 13:28 Jacek Laskowski <[hidden email]> ha scritto:
Hi,

While reviewing the changes in 2.4 I stumbled upon spark.sql.codegen.factoryMode internal configuration property [1]. The doc says:

> Note that this config works only for tests.

Is that correct? I've got some doubts.

I found that it's used in UnsafeProjection.create [2] (through CodeGeneratorWithInterpretedFallback.createObject) which is used outside the tests and so made me think if "this config works only for tests" part is correct.

Are my doubts correct? If not, what am I missing? Thanks.


Reply | Threaded
Open this post in threaded view
|

Re: Is spark.sql.codegen.factoryMode property really for tests only?

Jacek Laskowski
Hi Marco,

Many thanks for such a quick response. With that, I'll direct my curiosity into a different direction. Thanks!

On Fri, Nov 16, 2018 at 1:44 PM Marco Gaido <[hidden email]> wrote:
Hi Jacek,

I do believe it is correct. Please check the method you mentioned (CodeGeneratorWithInterpretedFallback.createObject): the value is relevant only if Utils.isTesting.

Thanks,
Marco

Il giorno ven 16 nov 2018 alle ore 13:28 Jacek Laskowski <[hidden email]> ha scritto:
Hi,

While reviewing the changes in 2.4 I stumbled upon spark.sql.codegen.factoryMode internal configuration property [1]. The doc says:

> Note that this config works only for tests.

Is that correct? I've got some doubts.

I found that it's used in UnsafeProjection.create [2] (through CodeGeneratorWithInterpretedFallback.createObject) which is used outside the tests and so made me think if "this config works only for tests" part is correct.

Are my doubts correct? If not, what am I missing? Thanks.