[Other] Setup productive SCS cluster at Cloud&Heat with Yaook on bare metal as FITKO PoC

[anjastrunk]

Provide a productive SCS cluster as PoC for FITKO. The cluster MUST be set up with the open source Lifecycle Management Tool for OpenStack and K8S Yaook and must be SCS compliant.

In contrast to #414, productive SCS cluster is set up on bare metal.

[martinmo]

(Note: we initially used #414 as the sole issue to track both the provisioning of the virtual and the bare metal test clusters. For the sake of better documentation, we retroactively created this separate issue for the bare metal setup.)

Current state is that the installation is completed and we have a working bare metal Yaook cluster. Thanks to the contributions of @cah-hbaum this OpenStack cluster is already SCS compliant, i.e., it fulfills the stabilized standards of the IaaS track (details can be found in #415 and in https://github.com/SovereignCloudStack/standards/tree/do-not-merge/scs-compliant-yaook/Informational). Here's a recap of what happened until we got to this point:

Initial preparation for the Yaook bare metal installation started at the beginning of March. This involved a "rehearsal" of the installation procedure on an existing, simpler test cluster, because for me this was the first time conducting such an install and the hardware for the actual installation was not yet commisioned.

During the rehearsal we already had network setup issues we needed to work around:

• The first issues were related to VLAN tagging and caused connection issues between the Yaook Management Cluster and the install node. These were (probably) caused by some incompatibilities or misconfiguration of the network interfaces of the virtual install node and/or KVM (Proxmox). The exact cause is not known, because we gave up debugging at some point and switched to a bare metal install node, which immediately worked.
• The second set of issues were related to PXE booting for the actual nodes that are supposed to be provisioned by ironic. These turned out to be firmware issues (a subset of the servers "forgot" its PXE configuration after reboot) as well as misleading and very well hidden BIOS settings (especially: PXE boot timeouts).

The installation of the bare metal test cluster was conducted between March 11th and March 19th but we again bumped into a lot of technical difficulties. Debugging and fixing these was a bit more time consuming than usual because I am not yet 100% accustomed to the interactions of all the components.

• Because this is going to be a productive cluster, the network setup is a lot more complex than the rehearsal install (more redundancy, stricter isolation). In addition to separate VLANs and subnets for different purposes, we also use several dedicated switches (e.g., for the Ceph nodes). This took several iterations and debugging sessions to get everything right, i.e., that all components supposed to be communicating could talk to each other.
• Some trial and error was needed to get the netplan part of the cloud-init configuration right, also partly because I misunderstood the configuration. (This is very unfortunate and we will make this more robust and easier to verify in the future, e.g., by switching to interface selection by MAC address via the match keyword.)
• During provisioning with ironic, a subset of the nodes repeatedly ended up in the clean failed state. It took some time to debug, but the Yaook bare metal logs contained the hint ("certificate not yet valid") and we finally figured out this was caused by an extremely out of sync hardware clock.
• A similar, firmware setup/BIOS related error that cost us time was a still active hardware raid configuration on another subset of the nodes, which also lead to provisioning failures with ironic.
• Finally, we had to troubleshoot some software issues that worked flawlessly before: e.g., during the automated install we ran into the problem that the K8s APT repositories were moved. Additionally, the CNI plugin (calico) installation failed initially, which we just fixed by switching to a different release.

The next step will involve moving the hardware from our facility to its final location.

[anjastrunk]

@cah-hbaum Please provide YAML output for SCS compatible IaaS v4, to proof cluster is SCS compliant.