[DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

Erik Erlandson-2
I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

Arguments in favor:
1) it improves testing coverage on a feature important for integrating with HDFS deployments
2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.
3) Spark 2.4 is still in RC, with outstanding correctness issues.

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.
Cheers,
Erik

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

Felix Cheung
I’m in favor of it. If you check the PR it’s a few isolated script changes and all test-only changes. Should have low impact on release but much better integration test coverage.

 

From: Erik Erlandson <[hidden email]>
Sent: Tuesday, October 16, 2018 8:20 AM
To: dev
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4
 
I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

Arguments in favor:
1) it improves testing coverage on a feature important for integrating with HDFS deployments
2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.
3) Spark 2.4 is still in RC, with outstanding correctness issues.

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.
Cheers,
Erik

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

Rob Vesse
In reply to this post by Erik Erlandson-2

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

rxin
We shouldn’t merge new features into release branches anymore. 

On Tue, Oct 16, 2018 at 6:32 PM Rob Vesse <[hidden email]> wrote:

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

liyinan926
In reply to this post by Rob Vesse
Yep, the Kerberos support for k8s is in the master but not in branch-2.4. I see no reason to get the integration tests into 2.4, which depend on the feature in the master.

On Tue, Oct 16, 2018 at 9:32 AM Rob Vesse <[hidden email]> wrote:

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

Erik Erlandson-2
In reply to this post by rxin

SPARK-23257 merged more recently than I realized. If that isn't on branch-2.4 then the first question is how soon on the release sequence that can be adopted


On Tue, Oct 16, 2018 at 9:33 AM Reynold Xin <[hidden email]> wrote:
We shouldn’t merge new features into release branches anymore. 

On Tue, Oct 16, 2018 at 6:32 PM Rob Vesse <[hidden email]> wrote:

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

ifilonenko
On Erik's note, would SPARK-23257 be included in, say, 2.4.1? When would the next RC be? I would like to propose the inclusion of the Kerberos feature sooner rather than later as it would increase Spark-on-K8S adoption in production workloads while bringing greater feature parity with Yarn and Mesos. I would like to note that the feature itself is isolated from Core and isolated via the step-based architecture of the Kubernetes Driver/Executor builders.

Furthermore, Spark users traditionally use HDFS for storage and in production use-cases these HDFS clusters would be kerberized. At Bloomberg, for example, all of the HDFS clusters are kerberized and for this reason, the only thing stopping our internal Data Science Platform from adopting Spark-on-K8S is this feature. 

On Tue, Oct 16, 2018 at 10:21 AM Erik Erlandson <[hidden email]> wrote:

SPARK-23257 merged more recently than I realized. If that isn't on branch-2.4 then the first question is how soon on the release sequence that can be adopted


On Tue, Oct 16, 2018 at 9:33 AM Reynold Xin <[hidden email]> wrote:
We shouldn’t merge new features into release branches anymore. 

On Tue, Oct 16, 2018 at 6:32 PM Rob Vesse <[hidden email]> wrote:

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

Bolke de Bruin
Chiming in here. We are in the same boat as Bloomberg. 

(But being a release manager often myself I understand the trade-off)

B. 

Op di 16 okt. 2018 21:24 schreef Ilan Filonenko <[hidden email]>:
On Erik's note, would SPARK-23257 be included in, say, 2.4.1? When would the next RC be? I would like to propose the inclusion of the Kerberos feature sooner rather than later as it would increase Spark-on-K8S adoption in production workloads while bringing greater feature parity with Yarn and Mesos. I would like to note that the feature itself is isolated from Core and isolated via the step-based architecture of the Kubernetes Driver/Executor builders.

Furthermore, Spark users traditionally use HDFS for storage and in production use-cases these HDFS clusters would be kerberized. At Bloomberg, for example, all of the HDFS clusters are kerberized and for this reason, the only thing stopping our internal Data Science Platform from adopting Spark-on-K8S is this feature. 

On Tue, Oct 16, 2018 at 10:21 AM Erik Erlandson <[hidden email]> wrote:

SPARK-23257 merged more recently than I realized. If that isn't on branch-2.4 then the first question is how soon on the release sequence that can be adopted


On Tue, Oct 16, 2018 at 9:33 AM Reynold Xin <[hidden email]> wrote:
We shouldn’t merge new features into release branches anymore. 

On Tue, Oct 16, 2018 at 6:32 PM Rob Vesse <[hidden email]> wrote:

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

Xiao Li
We need to strictly follow the backport and release policy. We can't merge such a new feature into a RC branch or a minor release (e.g., 2.4.1). 

Cheers,

Xiao

Bolke de Bruin <[hidden email]> 于2018年10月16日周二 下午12:48写道:
Chiming in here. We are in the same boat as Bloomberg. 

(But being a release manager often myself I understand the trade-off)

B. 

Op di 16 okt. 2018 21:24 schreef Ilan Filonenko <[hidden email]>:
On Erik's note, would SPARK-23257 be included in, say, 2.4.1? When would the next RC be? I would like to propose the inclusion of the Kerberos feature sooner rather than later as it would increase Spark-on-K8S adoption in production workloads while bringing greater feature parity with Yarn and Mesos. I would like to note that the feature itself is isolated from Core and isolated via the step-based architecture of the Kubernetes Driver/Executor builders.

Furthermore, Spark users traditionally use HDFS for storage and in production use-cases these HDFS clusters would be kerberized. At Bloomberg, for example, all of the HDFS clusters are kerberized and for this reason, the only thing stopping our internal Data Science Platform from adopting Spark-on-K8S is this feature. 

On Tue, Oct 16, 2018 at 10:21 AM Erik Erlandson <[hidden email]> wrote:

SPARK-23257 merged more recently than I realized. If that isn't on branch-2.4 then the first question is how soon on the release sequence that can be adopted


On Tue, Oct 16, 2018 at 9:33 AM Reynold Xin <[hidden email]> wrote:
We shouldn’t merge new features into release branches anymore. 

On Tue, Oct 16, 2018 at 6:32 PM Rob Vesse <[hidden email]> wrote:

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

Dongjoon Hyun-2
I also agree with Reynold and Xiao.

Although I love that new feature, Spark 2.4 branch-cut was made a long time ago.

We cannot backport new features at this stage at RC4.

In addition, could you split Apache SPARK issue IDs, Ilan? It's confusing during discussion.

     (1) [SPARK-23257][K8S] Kerberos Support for Spark on K8S (merged yesterday for Apache Spark 3.0)
     (2) [SPARK-23257][K8S][TESTS] Kerberos Support Integration Tests (a live PR with about 2000 lines. It's not a follow-up size.)

For (1), it's merged yesterday. That means more people start to try (1) from today. We need more time to stabilize it.
For (2), it's still under review.

Both (1) and (2) looks only valid for Spark 3.0.0.

Bests,
Dongjoon.



On Tue, Oct 16, 2018 at 1:32 PM Xiao Li <[hidden email]> wrote:
We need to strictly follow the backport and release policy. We can't merge such a new feature into a RC branch or a minor release (e.g., 2.4.1). 

Cheers,

Xiao

Bolke de Bruin <[hidden email]> 于2018年10月16日周二 下午12:48写道:
Chiming in here. We are in the same boat as Bloomberg. 

(But being a release manager often myself I understand the trade-off)

B. 

Op di 16 okt. 2018 21:24 schreef Ilan Filonenko <[hidden email]>:
On Erik's note, would SPARK-23257 be included in, say, 2.4.1? When would the next RC be? I would like to propose the inclusion of the Kerberos feature sooner rather than later as it would increase Spark-on-K8S adoption in production workloads while bringing greater feature parity with Yarn and Mesos. I would like to note that the feature itself is isolated from Core and isolated via the step-based architecture of the Kubernetes Driver/Executor builders.

Furthermore, Spark users traditionally use HDFS for storage and in production use-cases these HDFS clusters would be kerberized. At Bloomberg, for example, all of the HDFS clusters are kerberized and for this reason, the only thing stopping our internal Data Science Platform from adopting Spark-on-K8S is this feature. 

On Tue, Oct 16, 2018 at 10:21 AM Erik Erlandson <[hidden email]> wrote:

SPARK-23257 merged more recently than I realized. If that isn't on branch-2.4 then the first question is how soon on the release sequence that can be adopted


On Tue, Oct 16, 2018 at 9:33 AM Reynold Xin <[hidden email]> wrote:
We shouldn’t merge new features into release branches anymore. 

On Tue, Oct 16, 2018 at 6:32 PM Rob Vesse <[hidden email]> wrote:

Right now the Kerberos support for Spark on K8S is only on master AFAICT i.e. the feature is not present on branch-2.4

 

Therefore I don’t see any point in adding the tests into branch-2.4 unless the plan is to also merge the Kerberos support to branch-2.4

 

Rob

 

From: Erik Erlandson <[hidden email]>
Date: Tuesday, 16 October 2018 at 16:47
To: dev <[hidden email]>
Subject: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

 

I'd like to propose including integration testing for Kerberos on the Spark 2.4 release:

 

Arguments in favor:

1) it improves testing coverage on a feature important for integrating with HDFS deployments

2) its intersection with existing code is small - it consists primarily of new testing code, with a bit of refactoring into 'main' and 'test' sub-trees. These new tests appear stable.

3) Spark 2.4 is still in RC, with outstanding correctness issues.

 

The argument 'against' that I'm aware of would be the relatively large size of the PR. I believe this is considered above, but am soliciting community feedback before committing.

Cheers,

Erik