[DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog

Jungtaek Lim-2
Hi devs,

I got another report regarding configuring v2 session catalog, when Spark fails to instantiate the configured catalog. For now, it just simply logs error message without exception information, and silently uses the default session catalog.


IMO, as the user intentionally provides the session catalog, it shouldn't fail back and just throw the exception. Otherwise (if we still want to do the failback), we need to add the exception information in the error log message at least.

Would like to hear the voices.

Thanks,
Jungtaek Lim (HeartSaVioR)
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog

RussS
I was convinced that we should probably just fail, but if that is too much of a change, then logging the exception is also acceptable.

On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <[hidden email]> wrote:
Hi devs,

I got another report regarding configuring v2 session catalog, when Spark fails to instantiate the configured catalog. For now, it just simply logs error message without exception information, and silently uses the default session catalog.


IMO, as the user intentionally provides the session catalog, it shouldn't fail back and just throw the exception. Otherwise (if we still want to do the failback), we need to add the exception information in the error log message at least.

Would like to hear the voices.

Thanks,
Jungtaek Lim (HeartSaVioR)
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog

Ryan Blue
I agree. If the user configures an invalid catalog, it should fail and propagate the exception. Running with a catalog other than the one the user requested is incorrect.

On Fri, Oct 23, 2020 at 5:24 AM Russell Spitzer <[hidden email]> wrote:
I was convinced that we should probably just fail, but if that is too much of a change, then logging the exception is also acceptable.

On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <[hidden email]> wrote:
Hi devs,

I got another report regarding configuring v2 session catalog, when Spark fails to instantiate the configured catalog. For now, it just simply logs error message without exception information, and silently uses the default session catalog.


IMO, as the user intentionally provides the session catalog, it shouldn't fail back and just throw the exception. Otherwise (if we still want to do the failback), we need to add the exception information in the error log message at least.

Would like to hear the voices.

Thanks,
Jungtaek Lim (HeartSaVioR)


--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog

Jungtaek Lim-2
Yeah I'm in favor of fast-fail if things are not working out as end users intended. Spark should only fail back when it doesn't make any difference but only some sort of performance. (like whole stage codegen) This fail back brings behavioral differences, which should be considered as a bug.

I'll file an issue and raise a PR sooner. Thanks for providing voices!

On Sat, Oct 24, 2020 at 2:03 AM Ryan Blue <[hidden email]> wrote:
I agree. If the user configures an invalid catalog, it should fail and propagate the exception. Running with a catalog other than the one the user requested is incorrect.

On Fri, Oct 23, 2020 at 5:24 AM Russell Spitzer <[hidden email]> wrote:
I was convinced that we should probably just fail, but if that is too much of a change, then logging the exception is also acceptable.

On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <[hidden email]> wrote:
Hi devs,

I got another report regarding configuring v2 session catalog, when Spark fails to instantiate the configured catalog. For now, it just simply logs error message without exception information, and silently uses the default session catalog.


IMO, as the user intentionally provides the session catalog, it shouldn't fail back and just throw the exception. Otherwise (if we still want to do the failback), we need to add the exception information in the error log message at least.

Would like to hear the voices.

Thanks,
Jungtaek Lim (HeartSaVioR)


--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog

cloud0fan
+1 to fail fast. Thanks for reporting this, Jungtaek!

On Mon, Oct 26, 2020 at 8:36 AM Jungtaek Lim <[hidden email]> wrote:
Yeah I'm in favor of fast-fail if things are not working out as end users intended. Spark should only fail back when it doesn't make any difference but only some sort of performance. (like whole stage codegen) This fail back brings behavioral differences, which should be considered as a bug.

I'll file an issue and raise a PR sooner. Thanks for providing voices!

On Sat, Oct 24, 2020 at 2:03 AM Ryan Blue <[hidden email]> wrote:
I agree. If the user configures an invalid catalog, it should fail and propagate the exception. Running with a catalog other than the one the user requested is incorrect.

On Fri, Oct 23, 2020 at 5:24 AM Russell Spitzer <[hidden email]> wrote:
I was convinced that we should probably just fail, but if that is too much of a change, then logging the exception is also acceptable.

On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <[hidden email]> wrote:
Hi devs,

I got another report regarding configuring v2 session catalog, when Spark fails to instantiate the configured catalog. For now, it just simply logs error message without exception information, and silently uses the default session catalog.


IMO, as the user intentionally provides the session catalog, it shouldn't fail back and just throw the exception. Otherwise (if we still want to do the failback), we need to add the exception information in the error log message at least.

Would like to hear the voices.

Thanks,
Jungtaek Lim (HeartSaVioR)


--
Ryan Blue
Software Engineer
Netflix