minikube and kubernetes cluster versions for integration testing

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

minikube and kubernetes cluster versions for integration testing

attilapiros
Hi All,

I am working on PR to change kubernetes integration testing and use the `minikube kubectl -- config view --minify` output to build the kubernetes client config.
This solution has the advantage of not using hardcoded values like 8443 for server port (which is wrong when the vm-driver is docker as the port in that case is 32788 by default).

But my question is bit more generic than my PR. It is about the supported Minikube versions and kubernetes cluster version this why I decided to write this mail.

To test this new solution I have created shell script to install each Minikube versions one by one, start a kubernetes cluster and view the config with the command above.
Running the test I found some issues. 

Currently for k8s testing we suggest to use minikube version v0.34.1 or greater with kubernetes version v1.15.12 (for details check "Testing K8S" section in the developer tools page).

The following three findings I have:

1) Looking the Minikube documentation I came across an advice about checking which kubernetes cluster versions are supported for a Minikube version:

"For up to date information on supported versions, see OldestKubernetesVersion and NewestKubernetesVersion in constants.go"

I think it would be a good idea to follow the official support matrix of Minikube so I have collected some relevant versions into this table (the link navigates to the relevant lines in `constants.go`):

                     |           kubernetes version               |
minikube version     |    oldest    |  newest      | default      |
====================================================================
v0.34.1              |    ???       |    ???       | v1.13.3      |
v1.1.0 (22 May 2019) | v1.10.13     | v1.14.2      | v1.14.2      |
v1.2.0               | v1.10.13     | v1.15.0      | v1.15.0      |
v1.3.0 (6 Aug 2019)  | v1.10.13     | v1.15.2      | v1.15.2      |
v1.6.0 (11 Dec 2019) | v1.11.10     | v1.17.0      | v1.17.0      |
v1.7.3 (8 Feb 2020)  | v1.11.10     | v1.17.3      | v1.17.3      |
v1.13.1              | v1.13.0      | v1.19.2      | v1.19.2      |
v1.17.1              | v1.13.0      | v1.20.2      | v1.20.3-rc.0 |

 
Looking this we can see if we intend to support v1.15.12 as kubernetes version we should drop everything under v1.3.0.

2) I would suggest to drop v1.15.12 as kubernetes version version because of this issue (I just found it by running my script).

3) On Minikube v1.7.2 there is this permission denied issue so I suggest to support Minikube version 1.7.3 and greater.

My test script is check_minikube_versions.zsh. It was executed on Mac but with a simple sed expression it can be tailored to linux too.

After all of this my questions:

A) What about to change the required versions and suggest to use kubernetes v1.17.3 and Minikube v1.7.3 and greater for integration testing?

I would chose v1.17.3 for k8s cluster as that is the newest supported k8s version for that Minikube v1.7.3 (hoping it will be good for us for a long time).
If you agree with this suggestion I go ahead and update the relevant documentation.

B) How about extending the integration test to check whether the Minikube version is sufficient?

By this we can provide a meaningful error when it is violated.

Bests,
Attila
Reply | Threaded
Open this post in threaded view
|

Re: minikube and kubernetes cluster versions for integration testing

Dongjoon Hyun-2
Thank you for sharing and suggestion, Attila.

Additionally, given the following information,

- The latest Minikube is v1.18.0 with K8s v1.20.2
- AWS EKS will add K8s v1.20 on April, 2021
- The end of support in AWS EKS are
    K8s v1.15 (May 3, 2021)
    K8s v1.16 (July, 2021)
    K8s v1.17 (September, 2021)

The minimum K8s versions (v1.17) sound reasonable and practical to me for Apache Spark 3.2.0.

For Minikube, I'd like to recommend to use the latest Minikube versions. However, if Minikube v1.7.3 support is easy enough in the script, +1 for using v1.7.3 as the minimum Minikube version checking.

Thanks,
Dongjoon.


On Tue, Mar 2, 2021 at 5:03 AM Attila Zsolt Piros <[hidden email]> wrote:
Hi All,

I am working on PR to change kubernetes integration testing and use the `minikube kubectl -- config view --minify` output to build the kubernetes client config.
This solution has the advantage of not using hardcoded values like 8443 for server port (which is wrong when the vm-driver is docker as the port in that case is 32788 by default).

But my question is bit more generic than my PR. It is about the supported Minikube versions and kubernetes cluster version this why I decided to write this mail.

To test this new solution I have created shell script to install each Minikube versions one by one, start a kubernetes cluster and view the config with the command above.
Running the test I found some issues. 

Currently for k8s testing we suggest to use minikube version v0.34.1 or greater with kubernetes version v1.15.12 (for details check "Testing K8S" section in the developer tools page).

The following three findings I have:

1) Looking the Minikube documentation I came across an advice about checking which kubernetes cluster versions are supported for a Minikube version:

"For up to date information on supported versions, see OldestKubernetesVersion and NewestKubernetesVersion in constants.go"

I think it would be a good idea to follow the official support matrix of Minikube so I have collected some relevant versions into this table (the link navigates to the relevant lines in `constants.go`):

                     |           kubernetes version               |
minikube version     |    oldest    |  newest      | default      |
====================================================================
v0.34.1              |    ???       |    ???       | v1.13.3      |
v1.1.0 (22 May 2019) | v1.10.13     | v1.14.2      | v1.14.2      |
v1.2.0               | v1.10.13     | v1.15.0      | v1.15.0      |
v1.3.0 (6 Aug 2019)  | v1.10.13     | v1.15.2      | v1.15.2      |
v1.6.0 (11 Dec 2019) | v1.11.10     | v1.17.0      | v1.17.0      |
v1.7.3 (8 Feb 2020)  | v1.11.10     | v1.17.3      | v1.17.3      |
v1.13.1              | v1.13.0      | v1.19.2      | v1.19.2      |
v1.17.1              | v1.13.0      | v1.20.2      | v1.20.3-rc.0 |

 
Looking this we can see if we intend to support v1.15.12 as kubernetes version we should drop everything under v1.3.0.

2) I would suggest to drop v1.15.12 as kubernetes version version because of this issue (I just found it by running my script).

3) On Minikube v1.7.2 there is this permission denied issue so I suggest to support Minikube version 1.7.3 and greater.

My test script is check_minikube_versions.zsh. It was executed on Mac but with a simple sed expression it can be tailored to linux too.

After all of this my questions:

A) What about to change the required versions and suggest to use kubernetes v1.17.3 and Minikube v1.7.3 and greater for integration testing?

I would chose v1.17.3 for k8s cluster as that is the newest supported k8s version for that Minikube v1.7.3 (hoping it will be good for us for a long time).
If you agree with this suggestion I go ahead and update the relevant documentation.

B) How about extending the integration test to check whether the Minikube version is sufficient?

By this we can provide a meaningful error when it is violated.

Bests,
Attila
Reply | Threaded
Open this post in threaded view
|

Re: minikube and kubernetes cluster versions for integration testing

shane knapp ☠
please open a jira for this and assign it to me...  shouldn't be too big of a deal to get this set up.

On Tue, Mar 2, 2021 at 6:06 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you for sharing and suggestion, Attila.

Additionally, given the following information,

- The latest Minikube is v1.18.0 with K8s v1.20.2
- AWS EKS will add K8s v1.20 on April, 2021
- The end of support in AWS EKS are
    K8s v1.15 (May 3, 2021)
    K8s v1.16 (July, 2021)
    K8s v1.17 (September, 2021)

The minimum K8s versions (v1.17) sound reasonable and practical to me for Apache Spark 3.2.0.

For Minikube, I'd like to recommend to use the latest Minikube versions. However, if Minikube v1.7.3 support is easy enough in the script, +1 for using v1.7.3 as the minimum Minikube version checking.

Thanks,
Dongjoon.


On Tue, Mar 2, 2021 at 5:03 AM Attila Zsolt Piros <[hidden email]> wrote:
Hi All,

I am working on PR to change kubernetes integration testing and use the `minikube kubectl -- config view --minify` output to build the kubernetes client config.
This solution has the advantage of not using hardcoded values like 8443 for server port (which is wrong when the vm-driver is docker as the port in that case is 32788 by default).

But my question is bit more generic than my PR. It is about the supported Minikube versions and kubernetes cluster version this why I decided to write this mail.

To test this new solution I have created shell script to install each Minikube versions one by one, start a kubernetes cluster and view the config with the command above.
Running the test I found some issues. 

Currently for k8s testing we suggest to use minikube version v0.34.1 or greater with kubernetes version v1.15.12 (for details check "Testing K8S" section in the developer tools page).

The following three findings I have:

1) Looking the Minikube documentation I came across an advice about checking which kubernetes cluster versions are supported for a Minikube version:

"For up to date information on supported versions, see OldestKubernetesVersion and NewestKubernetesVersion in constants.go"

I think it would be a good idea to follow the official support matrix of Minikube so I have collected some relevant versions into this table (the link navigates to the relevant lines in `constants.go`):

                     |           kubernetes version               |
minikube version     |    oldest    |  newest      | default      |
====================================================================
v0.34.1              |    ???       |    ???       | v1.13.3      |
v1.1.0 (22 May 2019) | v1.10.13     | v1.14.2      | v1.14.2      |
v1.2.0               | v1.10.13     | v1.15.0      | v1.15.0      |
v1.3.0 (6 Aug 2019)  | v1.10.13     | v1.15.2      | v1.15.2      |
v1.6.0 (11 Dec 2019) | v1.11.10     | v1.17.0      | v1.17.0      |
v1.7.3 (8 Feb 2020)  | v1.11.10     | v1.17.3      | v1.17.3      |
v1.13.1              | v1.13.0      | v1.19.2      | v1.19.2      |
v1.17.1              | v1.13.0      | v1.20.2      | v1.20.3-rc.0 |

 
Looking this we can see if we intend to support v1.15.12 as kubernetes version we should drop everything under v1.3.0.

2) I would suggest to drop v1.15.12 as kubernetes version version because of this issue (I just found it by running my script).

3) On Minikube v1.7.2 there is this permission denied issue so I suggest to support Minikube version 1.7.3 and greater.

My test script is check_minikube_versions.zsh. It was executed on Mac but with a simple sed expression it can be tailored to linux too.

After all of this my questions:

A) What about to change the required versions and suggest to use kubernetes v1.17.3 and Minikube v1.7.3 and greater for integration testing?

I would chose v1.17.3 for k8s cluster as that is the newest supported k8s version for that Minikube v1.7.3 (hoping it will be good for us for a long time).
If you agree with this suggestion I go ahead and update the relevant documentation.

B) How about extending the integration test to check whether the Minikube version is sufficient?

By this we can provide a meaningful error when it is violated.

Bests,
Attila


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Reply | Threaded
Open this post in threaded view
|

Re: minikube and kubernetes cluster versions for integration testing

attilapiros
Thanks Shane!

I can do the documentation task and the Minikube version check can be
incorporated into my PR.
When my PR is finalized (probably next week) I will create a jira for you
and you can set up the test systems and you can even test my PR before
merging it. Is this possible / fine for you?
 
 



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: minikube and kubernetes cluster versions for integration testing

shane knapp ☠
fwiw, upgrading minikube and the associated VM drivers is potentially a PITA.

your PR will absolutely be tested before merging.  :)

On Thu, Mar 4, 2021 at 10:13 AM attilapiros <[hidden email]> wrote:
Thanks Shane!

I can do the documentation task and the Minikube version check can be
incorporated into my PR.
When my PR is finalized (probably next week) I will create a jira for you
and you can set up the test systems and you can even test my PR before
merging it. Is this possible / fine for you?





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Reply | Threaded
Open this post in threaded view
|

Re: minikube and kubernetes cluster versions for integration testing

attilapiros
This post was updated on .
Thanks Shane!

As I promised:
- the PR about documenting the change
<https://github.com/apache/spark-website/pull/321>
- my Spark PR <https://github.com/apache/spark/pull/31829> with checking
Minikube versions and using a simpler way to configure kubernetes client
for integration testing
- the Jira you asked about upgrading the infrastructure
<https://issues.apache.org/jira/browse/SPARK-34738>

Best Regards,
Attila

On Thu, Mar 4, 2021 at 7:29 PM shane knapp ☠ <sknapp@berkeley.edu> wrote:

> fwiw, upgrading minikube and the associated VM drivers is potentially a
> PITA.
>
> your PR will absolutely be tested before merging.  :)
>
> On Thu, Mar 4, 2021 at 10:13 AM attilapiros <piros.attila.zsolt@gmail.com>
> wrote:
>
>> Thanks Shane!
>>
>> I can do the documentation task and the Minikube version check can be
>> incorporated into my PR.
>> When my PR is finalized (probably next week) I will create a jira for you
>> and you can set up the test systems and you can even test my PR before
>> merging it. Is this possible / fine for you?
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>