Deploy clusters on multiple-azs in vSphere.

[saji-pivotal]

Deploy clusters on multiple-azs in vSphere.
Signed-off-by: Sudarshan asudarshan@vmware.com

What this PR does / why we need it

This PR is to support deploying workload cluster on vSphere across multiple AZs. The AZs are custom defined in vSphere by using VsphereFailureDomains and VsphereDeploymentZone resources. The name of the VsphereDeploymentZone is referenced in the failureDomain field of the vSphere templates.

Which issue(s) this PR fixes

Fixes #827

Describe testing done for PR

Testing deployment of workers across different hosts within a single compute cluster

1. On a vsphere environment, tagged the compute cluster with the k8s-region label, and the hosts with k8s-zone labels. Set up association rules
2. Created a management cluster successfully
3. Created VsphereFailureDomains, and VsphereDeploymentZones on the management cluster
4. Set the environment variables VSPHERE_AZ_0, VSPHERE_AZ_1, VSPHERE_AZ_2
5. Set the environment variables VSPHERE_REGION to k8s-region and VSPHERE_ZONE to k8s-zone which are the required tagCategories
6. Ran tanzu cluster create... and verified that the machines were deployed in the desired failureDomain (in this case, the desired hosts)
7. Scaled up the cluster from 3-5 workers, and scaled it back down

Testing deployment of workers across different compute clusters in a single datacenter

1. On a vsphere environment, tagged the datacenter with the k8s-region label, and the compute clusters with k8s-zone labels. Set up association rules
2. Created a management cluster successfully
3. Created VsphereFailureDomains, and VsphereDeploymentZones on the management cluster
4. Set the environment variables VSPHERE_AZ_0, VSPHERE_AZ_1, VSPHERE_AZ_2
5. Set the environment variables VSPHERE_REGION to k8s-region and VSPHERE_ZONE to k8s-zone which are the required tagCategories
6. Ran tanzu cluster create... and verified that the machines were deployed in the desired failureDomains (in this case, the desired compute clusters)
7. Scaled up the cluster from 3-5 workers, and scaled it back down
8. Added a node-pool to the cluster specifying the desired az, and verified that it got deployed on the desired failureDomain

Release note

Supports deployment of workload clusters across multiple azs on vsphere
Introduces multiple machine deployments (md-1, md-2) for production clusters - This aligns with the machine deployment strategy used for AWS and Azure.

PR Checklist

• Squash the commits into one or a small number of logical commits
  
• Use good commit messages
• Ensure PR contains terms all contributors can understand and links all contributors can access

Additional information

Special notes for your reviewer

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/1108/20211110000755/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[tenczar]

LGTM

[prkalle]

LGTM.

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/1108/20211110004207/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[danniel1205]

LGTM