spark-on-k8s is still experimental?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

spark-on-k8s is still experimental?

Takeshi Yamamuro
Hi, all

A Spark user I know asked me this question.
I checked the the spark-on-k8s document and it says;
```
**The Kubernetes scheduler is currently experimental.
In future versions, there may be behavioral changes around configuration,
container images and entrypoints.**
```
https://github.com/apache/spark/blame/master/docs/running-on-kubernetes.md#L26-L28

This statement was added when v2.3.0 released (https://issues.apache.org/jira/browse/SPARK-23104).
Does this mean that the v2.3.0 and v3.0.0 releases have the same development stage?
I'm a bit worried that users read the statement in the doc and give up starting this feature.
(In fact, the user who asked me seemed to think so)

I'm not familiar with that part, so does anyone know
when this statement can be removed from the doc?
Any milestone for that?

Thanks in advance,
Takeshi

--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: spark-on-k8s is still experimental?

Sean Owen-2
Likewise, I'm not super familiar with this integration, but, it's been
out for several minor and one new major version, and doubt that it is
any more 'experimental' now that it ever will be.
Unless someone who knows more suggests that it's still really a WIP,
or that some aspects are still fairly subject to change, I think we
could remove those statements.

On Mon, Aug 3, 2020 at 3:38 AM Takeshi Yamamuro <[hidden email]> wrote:

>
> Hi, all
>
> A Spark user I know asked me this question.
> I checked the the spark-on-k8s document and it says;
> ```
> **The Kubernetes scheduler is currently experimental.
> In future versions, there may be behavioral changes around configuration,
> container images and entrypoints.**
> ```
> https://github.com/apache/spark/blame/master/docs/running-on-kubernetes.md#L26-L28
>
> This statement was added when v2.3.0 released (https://issues.apache.org/jira/browse/SPARK-23104).
> Does this mean that the v2.3.0 and v3.0.0 releases have the same development stage?
> I'm a bit worried that users read the statement in the doc and give up starting this feature.
> (In fact, the user who asked me seemed to think so)
>
> I'm not familiar with that part, so does anyone know
> when this statement can be removed from the doc?
> Any milestone for that?
>
> Thanks in advance,
> Takeshi
>
> --
> ---
> Takeshi Yamamuro

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spark-on-k8s is still experimental?

JackyLee
+1. It has been worked well in our company and we has used it to support
online services since March in this year.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spark-on-k8s is still experimental?

Holden Karau
There was discussion around removing the statement and declaring it GA but I believe it was decided to leave it in until an external shuffle service is supported on K8s.

On Mon, Aug 3, 2020 at 2:45 AM JackyLee <[hidden email]> wrote:
+1. It has been worked well in our company and we has used it to support
online services since March in this year.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: spark-on-k8s is still experimental?

Takeshi Yamamuro
Thanks for the info, all. okay, I understood that we need more time to announce GA officially.
But, I'm still worried that users hesitate a bit to use this feature by referring to the statement in the doc,
so how about updating it according to the current situation? Please check my suggestion in https://github.com/apache/spark/pull/29368.

Anyway, many thanks!


On Tue, Aug 4, 2020 at 12:26 AM Holden Karau <[hidden email]> wrote:
There was discussion around removing the statement and declaring it GA but I believe it was decided to leave it in until an external shuffle service is supported on K8s.

On Mon, Aug 3, 2020 at 2:45 AM JackyLee <[hidden email]> wrote:
+1. It has been worked well in our company and we has used it to support
online services since March in this year.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: spark-on-k8s is still experimental?

Holden Karau
Sounds good. I think we can make a slightly stronger statement than that one (left a comment, but it's my own thoughts so others should chime in if they have a different opinion).

On Wed, Aug 5, 2020 at 7:32 PM Takeshi Yamamuro <[hidden email]> wrote:
Thanks for the info, all. okay, I understood that we need more time to announce GA officially.
But, I'm still worried that users hesitate a bit to use this feature by referring to the statement in the doc,
so how about updating it according to the current situation? Please check my suggestion in https://github.com/apache/spark/pull/29368.

Anyway, many thanks!


On Tue, Aug 4, 2020 at 12:26 AM Holden Karau <[hidden email]> wrote:
There was discussion around removing the statement and declaring it GA but I believe it was decided to leave it in until an external shuffle service is supported on K8s.

On Mon, Aug 3, 2020 at 2:45 AM JackyLee <[hidden email]> wrote:
+1. It has been worked well in our company and we has used it to support
online services since March in this year.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: spark-on-k8s is still experimental?

Prashant Sharma
My thoughts are, External shuffle service is not a blocker for spark on k8s to be a production ready.

Others may think otherwise, but there are other ways to enable auto scaling. External shuffle service feature will be useful for all the ways of deployments be it yarn, standalone, k8s and not just k8s.

About the GA, I have not yet seen a very large deployment working yet. Others can share, how they are using spark on k8s, that can give us more confidence on moving towards GA.

Thanks,

On Thu, Aug 6, 2020 at 9:18 AM Holden Karau <[hidden email]> wrote:
Sounds good. I think we can make a slightly stronger statement than that one (left a comment, but it's my own thoughts so others should chime in if they have a different opinion).

On Wed, Aug 5, 2020 at 7:32 PM Takeshi Yamamuro <[hidden email]> wrote:
Thanks for the info, all. okay, I understood that we need more time to announce GA officially.
But, I'm still worried that users hesitate a bit to use this feature by referring to the statement in the doc,
so how about updating it according to the current situation? Please check my suggestion in https://github.com/apache/spark/pull/29368.

Anyway, many thanks!


On Tue, Aug 4, 2020 at 12:26 AM Holden Karau <[hidden email]> wrote:
There was discussion around removing the statement and declaring it GA but I believe it was decided to leave it in until an external shuffle service is supported on K8s.

On Mon, Aug 3, 2020 at 2:45 AM JackyLee <[hidden email]> wrote:
+1. It has been worked well in our company and we has used it to support
online services since March in this year.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9