[DISCUSS][K8S] Supporting advanced pod customisation

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS][K8S] Supporting advanced pod customisation

Rob Vesse

Hey all

 

For those following the K8S backend you are probably aware of SPARK-24434 [1] (and PR 22416 [2]) which proposes a mechanism to allow for advanced pod customisation via pod templates.  This is motivated by the fact that introducing additional Spark configuration properties for each aspect of pod specification a user might wish to customise was becoming unwieldy.

 

However I am concerned that the current implementation doesn’t go far enough and actually limits the utility of the proposed new feature.  The problem stems from the fact that the implementation simply uses the pod template as a base and then Spark attempts to build a pod spec on top of that.  As the code that does this doesn’t do any kind of validation or inspection of the incoming template it is possible to provide a template that causes Spark to generate an invalid pod spec ultimately causing the job to be rejected by Kubernetes.

 

Now clearly Spark code cannot attempt to account for every possible customisation that a user may attempt to make via pod templates nor should it be responsible for ensuring that the user doesn’t start from an invalid template in the first place.  However it seems like we could be more intelligent in how we build our pod specs to avoid generating invalid specs in cases where we have a clear use case for advanced customisation.  For example the current implementation does not allow users to customise the volumes used to back SPARK_LOCAL_DIRS to better suit the compute environment the K8S cluster is running on and trying to do so with a pod template will result in an invalid spec due to duplicate volumes.

 

I think there are a few ways the community could address this:

 

  1. Status quo – provide the pod template feature as-is and simply tell users that certain customisations are never supported and may result in invalid pod specs
  2. Provide the ability for advanced users to explicitly skip pod spec building steps they know interfere with their pod templates via configuration properties
  3. Modify the pod spec building code to be aware of known desirable user customisation points and avoid generating  invalid specs in those cases

 

Currently committers seem to be going for Option 1.  Personally I would like to see the community adopt option 3 but have already received considerable pushback when I proposed that in one of my PRs hence the suggestion of the compromise option 2.  Yes this still has the possibility of ending up with invalid specs if users are over-zealous in the spec building steps they disable but since this is a power user feature I think this would be a risk power users would be willing to assume.  If we are going to provide features for power users we should avoid unnecessarily limiting the utility of those features.

 

What do other K8S folks think about this issue?

 

Thanks,

 

Rob

 

[1] https://issues.apache.org/jira/browse/SPARK-24434

[2] https://github.com/apache/spark/pull/22146

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S] Supporting advanced pod customisation

liyinan926
Thanks for bring this up. My opinion on this is this feature is really targeting advanced use cases that need more customization than what the basic k8s-related Spark config properties offer. So I think it's fair to assume that users who would like to use this feature know the risks and are responsible for making sure the supplied pod template is valid. Clear documentation on which fields of the pod spec are customizable and which are not is critical. With that being said, however, I agree that it's not a good experience for end users if the pod spec generated from a template plus overlay of customization from other Spark config properties and the k8s backend itself is invalid. Ideally, I think option #3 is favorable and the k8s backend code or the code for building the pod spec specifically should perform validation on the generated pod spec, e.g., json schema validation at bare minimum. I'm not sure how well this is supported by the client (farbric8 java client) we use though. Regarding option #2, I personally think the risks and complexity it brings far outweigh the benefits and it is much more prone to errors. For example, the k8s backend code sets some environment variables in the container that get used by the entrypoint script. If the backend skips the building of the pod spec, the end users are responsible for properly populating them and maybe other stuffs. So my opinion is I think initially #1 plus good documentation is the way to go, and we can improve on this in subsequent releases to move it towards #3. 

Yinan 

On Wed, Sep 19, 2018 at 9:12 AM Rob Vesse <[hidden email]> wrote:

Hey all

 

For those following the K8S backend you are probably aware of SPARK-24434 [1] (and PR 22416 [2]) which proposes a mechanism to allow for advanced pod customisation via pod templates.  This is motivated by the fact that introducing additional Spark configuration properties for each aspect of pod specification a user might wish to customise was becoming unwieldy.

 

However I am concerned that the current implementation doesn’t go far enough and actually limits the utility of the proposed new feature.  The problem stems from the fact that the implementation simply uses the pod template as a base and then Spark attempts to build a pod spec on top of that.  As the code that does this doesn’t do any kind of validation or inspection of the incoming template it is possible to provide a template that causes Spark to generate an invalid pod spec ultimately causing the job to be rejected by Kubernetes.

 

Now clearly Spark code cannot attempt to account for every possible customisation that a user may attempt to make via pod templates nor should it be responsible for ensuring that the user doesn’t start from an invalid template in the first place.  However it seems like we could be more intelligent in how we build our pod specs to avoid generating invalid specs in cases where we have a clear use case for advanced customisation.  For example the current implementation does not allow users to customise the volumes used to back SPARK_LOCAL_DIRS to better suit the compute environment the K8S cluster is running on and trying to do so with a pod template will result in an invalid spec due to duplicate volumes.

 

I think there are a few ways the community could address this:

 

  1. Status quo – provide the pod template feature as-is and simply tell users that certain customisations are never supported and may result in invalid pod specs
  2. Provide the ability for advanced users to explicitly skip pod spec building steps they know interfere with their pod templates via configuration properties
  3. Modify the pod spec building code to be aware of known desirable user customisation points and avoid generating  invalid specs in those cases

 

Currently committers seem to be going for Option 1.  Personally I would like to see the community adopt option 3 but have already received considerable pushback when I proposed that in one of my PRs hence the suggestion of the compromise option 2.  Yes this still has the possibility of ending up with invalid specs if users are over-zealous in the spec building steps they disable but since this is a power user feature I think this would be a risk power users would be willing to assume.  If we are going to provide features for power users we should avoid unnecessarily limiting the utility of those features.

 

What do other K8S folks think about this issue?

 

Thanks,

 

Rob

 

[1] https://issues.apache.org/jira/browse/SPARK-24434

[2] https://github.com/apache/spark/pull/22146

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S] Supporting advanced pod customisation

Erik Erlandson-2
In reply to this post by Rob Vesse
I can speak somewhat to the current design. Two of the goals for the design of this feature are that
(1) its behavior is easy to reason about
(2) its implementation in the back-end is light weight

Option 1 was chosen partly because it's behavior is relatively simple to describe to a user: "Your template will be taken as the starting point. Spark may override a certain small set of fields (documented in a table) that are necessary for its internal functioning."

This also keeps the actual back-end implementation relatively light weight. It can load the template (which also includes syntax validation) into a pod structure, then modify any fields it needs to (per above).


On Wed, Sep 19, 2018 at 9:11 AM, Rob Vesse <[hidden email]> wrote:

Hey all

 

For those following the K8S backend you are probably aware of SPARK-24434 [1] (and PR 22416 [2]) which proposes a mechanism to allow for advanced pod customisation via pod templates.  This is motivated by the fact that introducing additional Spark configuration properties for each aspect of pod specification a user might wish to customise was becoming unwieldy.

 

However I am concerned that the current implementation doesn’t go far enough and actually limits the utility of the proposed new feature.  The problem stems from the fact that the implementation simply uses the pod template as a base and then Spark attempts to build a pod spec on top of that.  As the code that does this doesn’t do any kind of validation or inspection of the incoming template it is possible to provide a template that causes Spark to generate an invalid pod spec ultimately causing the job to be rejected by Kubernetes.

 

Now clearly Spark code cannot attempt to account for every possible customisation that a user may attempt to make via pod templates nor should it be responsible for ensuring that the user doesn’t start from an invalid template in the first place.  However it seems like we could be more intelligent in how we build our pod specs to avoid generating invalid specs in cases where we have a clear use case for advanced customisation.  For example the current implementation does not allow users to customise the volumes used to back SPARK_LOCAL_DIRS to better suit the compute environment the K8S cluster is running on and trying to do so with a pod template will result in an invalid spec due to duplicate volumes.

 

I think there are a few ways the community could address this:

 

  1. Status quo – provide the pod template feature as-is and simply tell users that certain customisations are never supported and may result in invalid pod specs
  2. Provide the ability for advanced users to explicitly skip pod spec building steps they know interfere with their pod templates via configuration properties
  3. Modify the pod spec building code to be aware of known desirable user customisation points and avoid generating  invalid specs in those cases

 

Currently committers seem to be going for Option 1.  Personally I would like to see the community adopt option 3 but have already received considerable pushback when I proposed that in one of my PRs hence the suggestion of the compromise option 2.  Yes this still has the possibility of ending up with invalid specs if users are over-zealous in the spec building steps they disable but since this is a power user feature I think this would be a risk power users would be willing to assume.  If we are going to provide features for power users we should avoid unnecessarily limiting the utility of those features.

 

What do other K8S folks think about this issue?

 

Thanks,

 

Rob

 

[1] https://issues.apache.org/jira/browse/SPARK-24434

[2] https://github.com/apache/spark/pull/22146

 


Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS][K8S] Supporting advanced pod customisation

Stavros Kontopoulos-3
There is a design document that covers a lot of concerns: https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk, validation included.

We had a discussion about validation (validate before we hit the api server) and was considered too much. In general regarding Rob's options,l I prefer option 3 and that was the design's purpose.

For this feature to be completed, it needs to have some minimum functionality implemented. What that might be depends on what you offer.

For example the current implementation does not allow users to customise the volumes used to back SPARK_LOCAL_DIRS to better suit the compute environment the K8S
 
It takes time to go through all the properties align them with Spark, but for me that is the correct way to do this at the end of the day. Is the above documented at all for example for the pod template,? Are we 

missing anything else? Are we going to cover similar cases with more PRs like https://github.com/apache/spark/commit/da6fa3828bb824b65f50122a8a0a0d4741551257 ?

Option 3 would take quite some time and would add some code to the backend, not sure of the size of it though, shouldnt be that much.

Also I don't think using a pod template is an advanced use of Spark on K8s. Pod templates are used elsewhere by default, for example in kubeflow (for replicas). In addition there are key properties for normal use 

cases like pod affinity configuration, side car containers etc. These come up often in real deployments. So it depends on who is experiencing what for the UX part.


Stavros


On Wed, Sep 19, 2018 at 7:41 PM, Erik Erlandson <[hidden email]> wrote:
I can speak somewhat to the current design. Two of the goals for the design of this feature are that
(1) its behavior is easy to reason about
(2) its implementation in the back-end is light weight

Option 1 was chosen partly because it's behavior is relatively simple to describe to a user: "Your template will be taken as the starting point. Spark may override a certain small set of fields (documented in a table) that are necessary for its internal functioning."

This also keeps the actual back-end implementation relatively light weight. It can load the template (which also includes syntax validation) into a pod structure, then modify any fields it needs to (per above).


On Wed, Sep 19, 2018 at 9:11 AM, Rob Vesse <[hidden email]> wrote:

Hey all

 

For those following the K8S backend you are probably aware of SPARK-24434 [1] (and PR 22416 [2]) which proposes a mechanism to allow for advanced pod customisation via pod templates.  This is motivated by the fact that introducing additional Spark configuration properties for each aspect of pod specification a user might wish to customise was becoming unwieldy.

 

However I am concerned that the current implementation doesn’t go far enough and actually limits the utility of the proposed new feature.  The problem stems from the fact that the implementation simply uses the pod template as a base and then Spark attempts to build a pod spec on top of that.  As the code that does this doesn’t do any kind of validation or inspection of the incoming template it is possible to provide a template that causes Spark to generate an invalid pod spec ultimately causing the job to be rejected by Kubernetes.

 

Now clearly Spark code cannot attempt to account for every possible customisation that a user may attempt to make via pod templates nor should it be responsible for ensuring that the user doesn’t start from an invalid template in the first place.  However it seems like we could be more intelligent in how we build our pod specs to avoid generating invalid specs in cases where we have a clear use case for advanced customisation.  For example the current implementation does not allow users to customise the volumes used to back SPARK_LOCAL_DIRS to better suit the compute environment the K8S cluster is running on and trying to do so with a pod template will result in an invalid spec due to duplicate volumes.

 

I think there are a few ways the community could address this:

 

  1. Status quo – provide the pod template feature as-is and simply tell users that certain customisations are never supported and may result in invalid pod specs
  2. Provide the ability for advanced users to explicitly skip pod spec building steps they know interfere with their pod templates via configuration properties
  3. Modify the pod spec building code to be aware of known desirable user customisation points and avoid generating  invalid specs in those cases

 

Currently committers seem to be going for Option 1.  Personally I would like to see the community adopt option 3 but have already received considerable pushback when I proposed that in one of my PRs hence the suggestion of the compromise option 2.  Yes this still has the possibility of ending up with invalid specs if users are over-zealous in the spec building steps they disable but since this is a power user feature I think this would be a risk power users would be willing to assume.  If we are going to provide features for power users we should avoid unnecessarily limiting the utility of those features.

 

What do other K8S folks think about this issue?

 

Thanks,

 

Rob

 

[1] https://issues.apache.org/jira/browse/SPARK-24434

[2] https://github.com/apache/spark/pull/22146