Create v2 of node distribution standard (issues/#494)

[cah-hbaum]

Adds the new label topology.scs.openstack.org/host-id to the standard and extend the standard to require providers to set the labels on their managed k8s clusters.

[matofeder]

LGTM

[mbuechse]

Let Team Container in its next meeting vote on whether this needs a v2; apart from that: good to go!

[mbuechse]

@cah-hbaum you are quick! Let's add a paragraph that explains the changes vs v1.

[cah-hbaum]

@mbuechse Sure thing. Gonna need to do something else first, but after that I'm on it.

[martinmo]

I agree, this was really quick! Very good. I have left some review comments we should discuss about, otherwise LGTM.

[martinmo]

@cah-hbaum I forgot to mention that the title and description of this PR should be updated to reflect that we create a new version v2.

[mbuechse]

@cah-hbaum @martinmo @matofeder Please be careful before merging. This document will be merged in stable state, so let's be extra certain that it is good this time... I pushed the date to April 4, so there's no need to hurry.

[cah-hbaum]

This should be done now. After some more approvals fly in, I will merge this.

[martinmo]

Just found two typos and a cosmetic issue – apart from that, it is good to go!

[cah-hbaum]

I changed the document state to Draft now like we discussed in the Standardization/Certification meeting.

I also fixed (probably) all problems that had remained, so it should be mergable now.

[cah-hbaum]

@mbuechse Please check this again, since this is in Draft now.

[cah-hbaum]

@artificial-intelligence I added you here for review, because we had the question, if this standard is possible to do with a Gardener setup and @mbuechse mentioned that you have some knowledge here. If that is not the case, you can "unrequest" yourself and maybe put in someone else who has more knowledge about this.

[cah-hbaum]

Also because we talked about this in the meeting:
This standard is kind of "blocked" right now, since neither the reference implementation nor a few other K8s examples automatically provide the labels mentioned here.
BUT the standard could be merged nonetheless, since it is only in Draft for now.

[chess-knight]

Draft approved

[chess-knight]

I experienced an interesting scenario, where the node distribution test will succeed.
SCS Compliance results in SovereignCloudStack/k8s-cluster-api-provider#742 (comment) show INFO: The nodes are distributed across 2 host-ids.
I am wondering now if 3 control plane nodes are distributed over 2 hosts, it means that 2 control plane nodes are located on the same physical host. What will happen with the k8s cluster if this host goes down? AFAIK etcd has in this case only one member failure tolerance, it will stop functioning!

[chess-knight]

These lines in the standard and corresponding test don't require hard anti-affinity.

[cah-hbaum]

I understand the concern. We would probably need to adapt the lines above the ones you mentioned from

The control plane nodes MUST be distributed over multiple physical machines.
Kubernetes provides [best-practices][k8s-zones] on this topic, which are also RECOMMENDED by SCS.

to

The control plane nodes MUST be distributed over multiple physical machines; a control plane MUST contain at least three nodes and be distributed over three machines.
Kubernetes provides [best-practices][k8s-zones] on this topic, which are also RECOMMENDED by SCS.

Would that be alright?

[chess-knight]

at least three nodes can this be considered as node distribution standard or is it more suited for some k8s HA standard?

[chess-knight]

Also, can we somehow incorporate the fact that etcd can be external? Then we are probably good with the current formulation, this comment is valid only if stacked etcd for k8s cluster is used.

[cah-hbaum]

Probably more relevant for a HA standard, but TBF that was always the problem I had with this standard in general, since node distribution or better fault tolerance also kind of (at least partly) includes (high) availability.
Personally, after working on this standard this long I would probably change it up altogether to create a (high) availability that is based on this document and make this distribution standard optional, since not every cluster needs to be distributed (IMO).

Coming back to the case presented by you above:

I am wondering now if 3 control plane nodes are distributed over 2 hosts, it means that 2 control plane nodes
are located on the same physical host. What will happen with the k8s cluster if this host goes down?
AFAIK etcd has in this case only one member failure tolerance, it will stop functioning!

I understand this scenario (and as far as I know you're right about the etcd functionality). The question would be how to fix this scenario in the standard. There shouldn't be a number of physical nodes smaller than three, if etcd in use has 3 members.
So I guess this means this is still a distribution issue for us and not a HA issue?

[chess-knight]

Also, can we somehow incorporate the fact that etcd can be external? Then we are probably good with the current formulation, this comment is valid only if stacked etcd for k8s cluster is used.

Now I am reading that also for external etcd you need 3+3 nodes. Do you know why it is so?

I understand this scenario (and as far as I know you're right about the etcd functionality). The question would be how to fix this scenario in the standard. There shouldn't be a number of physical nodes smaller than three, if etcd in use has 3 members.
So I guess this means this is still a distribution issue for us and not a HA issue?

https://kubernetes.io/docs/setup/best-practices/multiple-zones/#control-plane-behavior mentions:

If availability is an important concern, select at least three failure zones and replicate each individual control plane component (API server, scheduler, etcd, cluster controller manager) across at least three failure zones.

So maybe we can keep it and put it only later to the HA standard?

[cah-hbaum]

Interesting, I already read about this some time ago.
I guess the idea behind the external etcd is separation of concerns in your infrastructure and better control (and possible security) for a cluster. So it makes sense to separate the underlying etcd nodes and the kubernetes control plane. As the linked article mentions

This topology decouples the control plane and etcd member. It therefore provides an HA setup where losing a
control plane instance or an etcd member has less impact and does not affect the cluster redundancy
as much as the stacked HA topology.

Now I think that use-case wouldn't be implemented that often, but its an interesting thing to consider for a setup with required redundancy.

Personally, I would put it into a future High availability standard.

I will mention all this in #639 and we can keep on discussing everything there.

[cah-hbaum]

Wrote and mentioned most of the stuff in the description of #639

[chess-knight]

I am wondering if this standard is valid also for clusters with the managed control plane. E.g. Kamaji, where CP runs inside pods in the management cluster and when you do kubectl get nodes in the workload cluster you see only worker nodes there.

[cah-hbaum]

I am wondering if this standard is valid also for clusters with the managed control plane. E.g. Kamaji, where CP runs inside pods in the management cluster and when you do kubectl get nodes in the workload cluster you see only worker nodes there.

Thats also something we had some discussions about already and we decided that we want to have a discussion about this in the follow up issue that would make this standard Stable.

[cah-hbaum]

I'm gonna keep it like it is for now and will update the standard in the follow-up issue with regards to the things discussed here.