From 6a2fcb18a47b0b2f180b40e79355dc9a79e64109 Mon Sep 17 00:00:00 2001 From: Hannes Baum Date: Fri, 15 Mar 2024 11:34:43 +0100 Subject: [PATCH 01/12] Update node distribution standard (issues/#540) Adds the new label topology.scs.openstack.org/host-id to the standard and extend the standard to require providers to set the labels on their managed k8s clusters. Signed-off-by: Hannes Baum --- .../scs-0214-v1-k8s-node-distribution.md | 36 +++++++++++++++++-- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/Standards/scs-0214-v1-k8s-node-distribution.md b/Standards/scs-0214-v1-k8s-node-distribution.md index 2e237de07..4731c1ae4 100644 --- a/Standards/scs-0214-v1-k8s-node-distribution.md +++ b/Standards/scs-0214-v1-k8s-node-distribution.md @@ -80,15 +80,45 @@ If the standard is used by a provider, the following decisions are binding and v can also be scaled vertically first before scaling horizontally. - Worker node distribution MUST be indicated to the user through some kind of labeling in order to enable (anti)-affinity for workloads over "failure zones". +- To provide metadata about the node distribution, which also enables testing of this standard, + providers MUST label their K8s nodes with the labels listed below. + - "topology.kubernetes.io/zone" + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It provides a logical zone of failure on the side of the provider, e.g. a server rack + in the same electrical circuit or multiple machines bound to the internet through a + singular network structure. How this is defined exactly is up to the plans of the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + + - "topology.kubernetes.io/region" + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It describes the combination of one or more failure zones into a region or domain, therefore + showing a larger entity of logical failure zone. An example for this could be a building + containing racks that are put into such a zone, since they're all prone to failure, if e.g. + the power for the building is cut. How this is defined exactly is also up to the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + + - "topology.scs.community/host-id" + + This is an SCS-specific label, which MUST contain the hostID of the physical machine running + the hypervisor and not the hostID of a virtual machine. The hostID is an arbitrary identifier, + which doesn't need to contain things like hostname, but it should nonetheless be unique to the host. + This helps identify the distribution over underlying physical machines, + which would be masked if VM hostIDs would be used. ## Conformance Tests The script `k8s-node-distribution-check.py` checks the nodes available with a user-provided -kubeconfig file. It then determines based on the labels `kubernetes.io/hostname`, `topology.kubernetes.io/zone`, -`topology.kubernetes.io/region` and `node-role.kubernetes.io/control-plane`, if a distribution -of the available nodes is present. If this isn't the case, the script produces an error. +kubeconfig file. Based on the labels `topology.scs.community/host-id`, +`topology.kubernetes.io/zone`, `topology.kubernetes.io/region` and `node-role.kubernetes.io/control-plane`, +the script then determines whether the nodes are distributed according to this standard. +If this isn't the case, the script produces an error. If also produces warnings and informational outputs, if e.g. labels don't seem to be set. [k8s-ha]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ [k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/ [scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md +[k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone From d905ecb874fcf52c1d290835ff64ee39b76b2f58 Mon Sep 17 00:00:00 2001 From: Hannes Baum Date: Thu, 21 Mar 2024 15:18:39 +0100 Subject: [PATCH 02/12] Update standard to v2 (issues/#540) Like discussed in the SIG Standardization and Certification Meeting, the standard is updated to v2 with the changes made to it and immediately set to "Stable", since no major changes were done. This comes mainly from the rushed job we did. Signed-off-by: Hannes Baum --- .../scs-0214-v1-k8s-node-distribution.md | 36 +---- .../scs-0214-v2-k8s-node-distribution.md | 125 ++++++++++++++++++ 2 files changed, 128 insertions(+), 33 deletions(-) create mode 100644 Standards/scs-0214-v2-k8s-node-distribution.md diff --git a/Standards/scs-0214-v1-k8s-node-distribution.md b/Standards/scs-0214-v1-k8s-node-distribution.md index 4731c1ae4..2e237de07 100644 --- a/Standards/scs-0214-v1-k8s-node-distribution.md +++ b/Standards/scs-0214-v1-k8s-node-distribution.md @@ -80,45 +80,15 @@ If the standard is used by a provider, the following decisions are binding and v can also be scaled vertically first before scaling horizontally. - Worker node distribution MUST be indicated to the user through some kind of labeling in order to enable (anti)-affinity for workloads over "failure zones". -- To provide metadata about the node distribution, which also enables testing of this standard, - providers MUST label their K8s nodes with the labels listed below. - - "topology.kubernetes.io/zone" - - Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. - It provides a logical zone of failure on the side of the provider, e.g. a server rack - in the same electrical circuit or multiple machines bound to the internet through a - singular network structure. How this is defined exactly is up to the plans of the provider. - The field gets autopopulated most of the time by either the kubelet or external mechanisms - like the cloud controller. - - - "topology.kubernetes.io/region" - - Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. - It describes the combination of one or more failure zones into a region or domain, therefore - showing a larger entity of logical failure zone. An example for this could be a building - containing racks that are put into such a zone, since they're all prone to failure, if e.g. - the power for the building is cut. How this is defined exactly is also up to the provider. - The field gets autopopulated most of the time by either the kubelet or external mechanisms - like the cloud controller. - - - "topology.scs.community/host-id" - - This is an SCS-specific label, which MUST contain the hostID of the physical machine running - the hypervisor and not the hostID of a virtual machine. The hostID is an arbitrary identifier, - which doesn't need to contain things like hostname, but it should nonetheless be unique to the host. - This helps identify the distribution over underlying physical machines, - which would be masked if VM hostIDs would be used. ## Conformance Tests The script `k8s-node-distribution-check.py` checks the nodes available with a user-provided -kubeconfig file. Based on the labels `topology.scs.community/host-id`, -`topology.kubernetes.io/zone`, `topology.kubernetes.io/region` and `node-role.kubernetes.io/control-plane`, -the script then determines whether the nodes are distributed according to this standard. -If this isn't the case, the script produces an error. +kubeconfig file. It then determines based on the labels `kubernetes.io/hostname`, `topology.kubernetes.io/zone`, +`topology.kubernetes.io/region` and `node-role.kubernetes.io/control-plane`, if a distribution +of the available nodes is present. If this isn't the case, the script produces an error. If also produces warnings and informational outputs, if e.g. labels don't seem to be set. [k8s-ha]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ [k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/ [scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md -[k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md new file mode 100644 index 000000000..3aad49e08 --- /dev/null +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -0,0 +1,125 @@ +--- +title: Kubernetes Node Distribution and Availability +type: Standard +status: Stable +stabilized_at: 2024-03-21 +track: KaaS +--- + +## Introduction + +A Kubernetes instance is provided as a cluster, which consists of a set of machines, +so-called nodes. A cluster is composed of a control plane and at least one worker node. +The control plane manages the worker nodes and therefore the pods in the cluster by making +decisions about scheduling, event detection and rights management. Inside the control plane, +multiple components exist, which can be duplicated and distributed over multiple nodes +inside the cluster. Typically, no user workloads are run on these nodes in order to +separate the controller component from user workloads, which could pose a security risk. + +### Glossary + +The following terms are used throughout this document: + +| Term | Meaning | +|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Worker | Virtual or bare-metal machine, which hosts workloads of customers | +| Control Plane | Virtual or bare-metal machine, which hosts the container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. | +| Machine | Virtual or bare-metal entity with computational capabilities | + +## Motivation + +In normal day-to-day operation, it is not unusual for some operational failures, either +due to wear and tear of hardware, software misconfigurations, external problems or +user errors. Whichever was the source of such an outage, it always means down-time for +operations and users and possible even data loss. +Therefore, a Kubernetes cluster in a productive environment should be distributed over +multiple "failure zones" in order to provide fault-tolerance and high availability. +This is especially important for the control plane of the cluster, since it contains the +state of the whole cluster. A failure of this component could mean an unrecoverable failure +of the whole cluster. + +## Design Considerations + +Most design considerations of this standard follow the previously written Decision Record +[Kubernetes Nodes Anti Affinity][scs-0213-v1] as well as the Kubernetes documents about +[High Availability][k8s-ha] and [Best practices for large clusters][k8s-large-clusters]. + +SCS wishes to prefer distributed, highly-available systems due to their obvious advantages +like fault-tolerance and data redundancy. But it also understands the costs and overhead +for the providers associated with this effort, since the infrastructure needs to have +hardware which will just be used to provide fail-over safety or duplication. + +The document [Best practices for large clusters] describes the concept of a failure zone. +This term isn't defined any further, but can in this context be described as a number of +physical (computing) machines in such a vicinity to each other (either through physical +or logical interconnection in some way), that specific problems inside this zone would put +all these machines at risk of failure/shutdown. It is therefore necessary for important +data or services to not be present just on one failure zone. +How such a failure zone should be defined is dependent on the risk model of the service/data +and its owner as well as the capabilities of the provider. Zones could be set from things +like single machines or racks up to whole datacenters or even regions, which could be +coupled by things like electrical grids. They're therefore purely logical entities, which +shouldn't be defined further in this document. + +## Decision + +This standard formulates the requirement for the distribution of Kubernetes nodes in order +to provide a fault-tolerant and available Kubernetes cluster infrastructure. +Since some providers only have small environments to work with and therefore couldn't +comply with this standard, it will be treated as a RECOMMENDED standard, where providers +can OPT OUT. + +If the standard is used by a provider, the following decisions are binding and valid: + +- The control plane nodes MUST be distributed over multiple physical machines. Kubernetes + provides best-practices on this topic, which are also RECOMMENDED by SCS. +- At least one control plane instance MUST be run in each "failure zone", more are + RECOMMENDED in each "failure zone" to provide fault-tolerance for each zone. +- Worker nodes are RECOMMENDED to be distributed over multiple zones. This policy makes + it OPTIONAL to provide a worker node in each "failure zone", meaning that worker nodes + can also be scaled vertically first before scaling horizontally. +- Worker node distribution MUST be indicated to the user through some kind of labeling + in order to enable (anti)-affinity for workloads over "failure zones". +- To provide metadata about the node distribution, which also enables testing of this standard, + providers MUST annotate their K8s nodes with the labels listed below. These labels MUST be kept + up to date with the current state of the deployment. + - `topology.kubernetes.io/zone` + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It provides a logical zone of failure on the side of the provider, e.g. a server rack + in the same electrical circuit or multiple machines bound to the internet through a + singular network structure. How this is defined exactly is up to the plans of the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + + - `topology.kubernetes.io/region` + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It describes the combination of one or more failure zones into a region or domain, therefore + showing a larger entity of logical failure zone. An example for this could be a building + containing racks that are put into such a zone, since they're all prone to failure, if e.g. + the power for the building is cut. How this is defined exactly is also up to the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + + - `topology.scs.community/host-id` + + This is an SCS-specific label; it MUST contain the hostID of the physical machine running + the hypervisor (NOT: the hostID of a virtual machine). Here, the hostID is an arbitrary identifier, + which need not contain the actual hostname, but it should nonetheless be unique to the host. + This helps identify the distribution over underlying physical machines, + which would be masked if VM hostIDs were used. + +## Conformance Tests + +The script `k8s-node-distribution-check.py` checks the nodes available with a user-provided +kubeconfig file. Based on the labels `topology.scs.community/host-id`, +`topology.kubernetes.io/zone`, `topology.kubernetes.io/region` and `node-role.kubernetes.io/control-plane`, +the script then determines whether the nodes are distributed according to this standard. +If this isn't the case, the script produces an error. +If also produces warnings and informational outputs, if e.g. labels don't seem to be set. + +[k8s-ha]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ +[k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/ +[scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md +[k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone From 0a45c1dbaf175b0718e5870ce4a0512ea786062b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20B=C3=BCchse?= Date: Mon, 18 Mar 2024 17:17:50 +0100 Subject: [PATCH 03/12] Minor cosmetic change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthias Büchse --- .../scs-0214-v1-k8s-node-distribution.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/Standards/scs-0214-v1-k8s-node-distribution.md b/Standards/scs-0214-v1-k8s-node-distribution.md index 2e237de07..65357ed66 100644 --- a/Standards/scs-0214-v1-k8s-node-distribution.md +++ b/Standards/scs-0214-v1-k8s-node-distribution.md @@ -80,6 +80,34 @@ If the standard is used by a provider, the following decisions are binding and v can also be scaled vertically first before scaling horizontally. - Worker node distribution MUST be indicated to the user through some kind of labeling in order to enable (anti)-affinity for workloads over "failure zones". +- To provide metadata about the node distribution, which also enables testing of this standard, + providers MUST label their K8s nodes with the labels listed below. + - `topology.kubernetes.io/zone` + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It provides a logical zone of failure on the side of the provider, e.g. a server rack + in the same electrical circuit or multiple machines bound to the internet through a + singular network structure. How this is defined exactly is up to the plans of the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + + - `topology.kubernetes.io/region` + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It describes the combination of one or more failure zones into a region or domain, therefore + showing a larger entity of logical failure zone. An example for this could be a building + containing racks that are put into such a zone, since they're all prone to failure, if e.g. + the power for the building is cut. How this is defined exactly is also up to the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + + - `topology.scs.community/host-id` + + This is an SCS-specific label, which MUST contain the hostID of the physical machine running + the hypervisor and not the hostID of a virtual machine. Here, the hostID is an arbitrary identifier, + which need not to contain the actual hostname, but it should nonetheless be unique to the host. + This helps identify the distribution over underlying physical machines, + which would be masked if VM hostIDs were used. ## Conformance Tests From efa265335d7850c7f6148170fcfe2fdf0c8356b3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20B=C3=BCchse?= Date: Mon, 18 Mar 2024 17:19:48 +0100 Subject: [PATCH 04/12] Minor addendum to cosmetic change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthias Büchse --- Standards/scs-0214-v1-k8s-node-distribution.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Standards/scs-0214-v1-k8s-node-distribution.md b/Standards/scs-0214-v1-k8s-node-distribution.md index 65357ed66..2bfd8784d 100644 --- a/Standards/scs-0214-v1-k8s-node-distribution.md +++ b/Standards/scs-0214-v1-k8s-node-distribution.md @@ -103,9 +103,9 @@ If the standard is used by a provider, the following decisions are binding and v - `topology.scs.community/host-id` - This is an SCS-specific label, which MUST contain the hostID of the physical machine running - the hypervisor and not the hostID of a virtual machine. Here, the hostID is an arbitrary identifier, - which need not to contain the actual hostname, but it should nonetheless be unique to the host. + This is an SCS-specific label; it MUST contain the hostID of the physical machine running + the hypervisor (NOT: the hostID of a virtual machine). Here, the hostID is an arbitrary identifier, + which need not contain the actual hostname, but it should nonetheless be unique to the host. This helps identify the distribution over underlying physical machines, which would be masked if VM hostIDs were used. From 978095893e766c11de03bc77792967c60ed84891 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20B=C3=BCchse?= Date: Thu, 21 Mar 2024 20:19:32 +0100 Subject: [PATCH 05/12] add replaces field and push stabilization date a bit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthias Büchse --- Standards/scs-0214-v2-k8s-node-distribution.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index 3aad49e08..76ea2b9ee 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -2,7 +2,8 @@ title: Kubernetes Node Distribution and Availability type: Standard status: Stable -stabilized_at: 2024-03-21 +stabilized_at: 2024-04-04 +replaces: scs-0214-v1-k8s-node-distribution.md track: KaaS --- From 4d47645bb2281847845a4d7fc80a108cbe3aa7b3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20B=C3=BCchse?= Date: Thu, 21 Mar 2024 20:24:40 +0100 Subject: [PATCH 06/12] Add section on previous versions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthias Büchse --- Standards/scs-0214-v2-k8s-node-distribution.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index 76ea2b9ee..3b2dbb4d7 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -124,3 +124,8 @@ If also produces warnings and informational outputs, if e.g. labels don't seem t [k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/ [scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md [k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone + +## Previous standard versions + +This is version 2 of the standard; it extends [version 1](scs-0214-v1-k8s-node-distribution.md) by the +requirements regarding node labeling. From be771b6457a64b259b042709914fa1fa5341f8e7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20B=C3=BCchse?= Date: Thu, 21 Mar 2024 20:25:52 +0100 Subject: [PATCH 07/12] Improvements inspired by @martinmo's review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthias Büchse --- Standards/scs-0214-v2-k8s-node-distribution.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index 3b2dbb4d7..5fb5e1f9f 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -50,7 +50,7 @@ like fault-tolerance and data redundancy. But it also understands the costs and for the providers associated with this effort, since the infrastructure needs to have hardware which will just be used to provide fail-over safety or duplication. -The document [Best practices for large clusters] describes the concept of a failure zone. +The document [Best practices for large clusters][k8s-large-clusters] describes the concept of a failure zone. This term isn't defined any further, but can in this context be described as a number of physical (computing) machines in such a vicinity to each other (either through physical or logical interconnection in some way), that specific problems inside this zone would put @@ -82,8 +82,7 @@ If the standard is used by a provider, the following decisions are binding and v - Worker node distribution MUST be indicated to the user through some kind of labeling in order to enable (anti)-affinity for workloads over "failure zones". - To provide metadata about the node distribution, which also enables testing of this standard, - providers MUST annotate their K8s nodes with the labels listed below. These labels MUST be kept - up to date with the current state of the deployment. + providers MUST annotate their K8s nodes with the labels listed below. - `topology.kubernetes.io/zone` Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. From 757eef189261a4db8030aa43a52d6f2dd27cadf0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20B=C3=BCchse?= Date: Fri, 22 Mar 2024 16:13:22 +0100 Subject: [PATCH 08/12] Proposal: use should. Mention keeping labels up to date MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthias Büchse --- Standards/scs-0214-v2-k8s-node-distribution.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index 5fb5e1f9f..4ecbcd2e5 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -82,7 +82,8 @@ If the standard is used by a provider, the following decisions are binding and v - Worker node distribution MUST be indicated to the user through some kind of labeling in order to enable (anti)-affinity for workloads over "failure zones". - To provide metadata about the node distribution, which also enables testing of this standard, - providers MUST annotate their K8s nodes with the labels listed below. + providers SHOULD annotate their K8s nodes with the labels listed below, and the labels + SHOULD be kept up to date. - `topology.kubernetes.io/zone` Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. From d19234f24e6201735a3811927ab107c01e6b348e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20B=C3=BCchse?= Date: Fri, 22 Mar 2024 16:15:29 +0100 Subject: [PATCH 09/12] revert should to must MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Matthias Büchse --- Standards/scs-0214-v2-k8s-node-distribution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index 4ecbcd2e5..699b45afb 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -82,8 +82,8 @@ If the standard is used by a provider, the following decisions are binding and v - Worker node distribution MUST be indicated to the user through some kind of labeling in order to enable (anti)-affinity for workloads over "failure zones". - To provide metadata about the node distribution, which also enables testing of this standard, - providers SHOULD annotate their K8s nodes with the labels listed below, and the labels - SHOULD be kept up to date. + providers MUST annotate their K8s nodes with the labels listed below, and the labels + MUST be kept up to date. - `topology.kubernetes.io/zone` Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. From 69be8772f9b1325d22926a31975689cc6f9850aa Mon Sep 17 00:00:00 2001 From: Hannes Baum Date: Mon, 15 Apr 2024 14:21:38 +0200 Subject: [PATCH 10/12] Switch standard to draft Like it was discussed in the Standardization/Certification meeting, we will set this v2 to a Draft state for now. Signed-off-by: Hannes Baum --- Standards/scs-0214-v2-k8s-node-distribution.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index 699b45afb..cb9d0408b 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -1,8 +1,7 @@ --- title: Kubernetes Node Distribution and Availability type: Standard -status: Stable -stabilized_at: 2024-04-04 +status: Draft replaces: scs-0214-v1-k8s-node-distribution.md track: KaaS --- From 9b391296fbe4b972630e2d38e8d8f2da58766e9e Mon Sep 17 00:00:00 2001 From: Hannes Baum Date: Mon, 15 Apr 2024 14:29:55 +0200 Subject: [PATCH 11/12] Small cosmetic updates Some small cosmetic updates fixing some problems mentioned by @martinmo. Signed-off-by: Hannes Baum --- Standards/scs-0214-v2-k8s-node-distribution.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index cb9d0408b..5554c1ce1 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -117,14 +117,14 @@ kubeconfig file. Based on the labels `topology.scs.community/host-id`, `topology.kubernetes.io/zone`, `topology.kubernetes.io/region` and `node-role.kubernetes.io/control-plane`, the script then determines whether the nodes are distributed according to this standard. If this isn't the case, the script produces an error. -If also produces warnings and informational outputs, if e.g. labels don't seem to be set. +It also produces warnings and informational outputs, e.g., if labels don't seem to be set. + +## Previous standard versions + +This is version 2 of the standard; it extends [version 1](scs-0214-v1-k8s-node-distribution.md) with the +requirements regarding node labeling. [k8s-ha]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ [k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/ [scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md [k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone - -## Previous standard versions - -This is version 2 of the standard; it extends [version 1](scs-0214-v1-k8s-node-distribution.md) by the -requirements regarding node labeling. From 81b9ab015d5f8dc3fb998ebccf38170e96db8b3f Mon Sep 17 00:00:00 2001 From: Hannes Baum Date: Thu, 25 Apr 2024 11:41:40 +0200 Subject: [PATCH 12/12] Some more updates Did some more textual updates requested by @mbuechse. Signed-off-by: Hannes Baum --- .../scs-0214-v1-k8s-node-distribution.md | 1 + .../scs-0214-v2-k8s-node-distribution.md | 87 +++++++++---------- 2 files changed, 44 insertions(+), 44 deletions(-) diff --git a/Standards/scs-0214-v1-k8s-node-distribution.md b/Standards/scs-0214-v1-k8s-node-distribution.md index 2bfd8784d..ffec30efc 100644 --- a/Standards/scs-0214-v1-k8s-node-distribution.md +++ b/Standards/scs-0214-v1-k8s-node-distribution.md @@ -120,3 +120,4 @@ If also produces warnings and informational outputs, if e.g. labels don't seem t [k8s-ha]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ [k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/ [scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md +[k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone diff --git a/Standards/scs-0214-v2-k8s-node-distribution.md b/Standards/scs-0214-v2-k8s-node-distribution.md index 5554c1ce1..3b4915492 100644 --- a/Standards/scs-0214-v2-k8s-node-distribution.md +++ b/Standards/scs-0214-v2-k8s-node-distribution.md @@ -65,50 +65,48 @@ shouldn't be defined further in this document. This standard formulates the requirement for the distribution of Kubernetes nodes in order to provide a fault-tolerant and available Kubernetes cluster infrastructure. -Since some providers only have small environments to work with and therefore couldn't -comply with this standard, it will be treated as a RECOMMENDED standard, where providers -can OPT OUT. - -If the standard is used by a provider, the following decisions are binding and valid: - -- The control plane nodes MUST be distributed over multiple physical machines. Kubernetes - provides best-practices on this topic, which are also RECOMMENDED by SCS. -- At least one control plane instance MUST be run in each "failure zone", more are - RECOMMENDED in each "failure zone" to provide fault-tolerance for each zone. -- Worker nodes are RECOMMENDED to be distributed over multiple zones. This policy makes - it OPTIONAL to provide a worker node in each "failure zone", meaning that worker nodes - can also be scaled vertically first before scaling horizontally. -- Worker node distribution MUST be indicated to the user through some kind of labeling - in order to enable (anti)-affinity for workloads over "failure zones". -- To provide metadata about the node distribution, which also enables testing of this standard, - providers MUST annotate their K8s nodes with the labels listed below, and the labels - MUST be kept up to date. - - `topology.kubernetes.io/zone` - - Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. - It provides a logical zone of failure on the side of the provider, e.g. a server rack - in the same electrical circuit or multiple machines bound to the internet through a - singular network structure. How this is defined exactly is up to the plans of the provider. - The field gets autopopulated most of the time by either the kubelet or external mechanisms - like the cloud controller. - - - `topology.kubernetes.io/region` - - Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. - It describes the combination of one or more failure zones into a region or domain, therefore - showing a larger entity of logical failure zone. An example for this could be a building - containing racks that are put into such a zone, since they're all prone to failure, if e.g. - the power for the building is cut. How this is defined exactly is also up to the provider. - The field gets autopopulated most of the time by either the kubelet or external mechanisms - like the cloud controller. - - - `topology.scs.community/host-id` - - This is an SCS-specific label; it MUST contain the hostID of the physical machine running - the hypervisor (NOT: the hostID of a virtual machine). Here, the hostID is an arbitrary identifier, - which need not contain the actual hostname, but it should nonetheless be unique to the host. - This helps identify the distribution over underlying physical machines, - which would be masked if VM hostIDs were used. + +The control plane nodes MUST be distributed over multiple physical machines. +Kubernetes provides [best-practices][k8s-zones] on this topic, which are also RECOMMENDED by SCS. + +At least one control plane instance MUST be run in each "failure zone" used for the cluster, +more instances per "failure zone" are possible to provide fault-tolerance inside a zone. + +Worker nodes are RECOMMENDED to be distributed over multiple zones. This policy makes +it OPTIONAL to provide a worker node in each "failure zone", meaning that worker nodes +can also be scaled vertically first before scaling horizontally. + +To provide metadata about the node distribution and possibly provide the ability +to schedule workloads efficiently, which also enables testing of this standard, +providers MUST annotate their K8s nodes with the labels listed below. +These labels MUST be kept up to date with the current state of the deployment. + +- `topology.kubernetes.io/zone` + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It provides a logical zone of failure on the side of the provider, e.g. a server rack + in the same electrical circuit or multiple machines bound to the internet through a + singular network structure. How this is defined exactly is up to the plans of the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + +- `topology.kubernetes.io/region` + + Corresponds with the label described in [K8s labels documentation][k8s-labels-docs]. + It describes the combination of one or more failure zones into a region or domain, therefore + showing a larger entity of logical failure zone. An example for this could be a building + containing racks that are put into such a zone, since they're all prone to failure, if e.g. + the power for the building is cut. How this is defined exactly is also up to the provider. + The field gets autopopulated most of the time by either the kubelet or external mechanisms + like the cloud controller. + +- `topology.scs.community/host-id` + + This is an SCS-specific label; it MUST contain the hostID of the physical machine running + the hypervisor (NOT: the hostID of a virtual machine). Here, the hostID is an arbitrary identifier, + which need not contain the actual hostname, but it should nonetheless be unique to the host. + This helps identify the distribution over underlying physical machines, + which would be masked if VM hostIDs were used. ## Conformance Tests @@ -128,3 +126,4 @@ requirements regarding node labeling. [k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/ [scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md [k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone +[k8s-zones]: https://kubernetes.io/docs/setup/best-practices/multiple-zones/