Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0

[yadvr]

With 4.16.0.0 release, if a VMware environment uses Vmxnet3 as the router nic controller then the isolated network VR deployment fails when router.extra.public.nics is not 0. However, VPC VR works in either case.

The workaround with 4.16.0.0 is to either use (a) E1000 nic controller, or (b) use Vmxnet3 controller with the router.extra.public.nics set to 0 if the IaaS env has all public IP ranges in the same VLAN. The additional router nics are used when users acquire public IPs from IP ranges with different VLANs, otherwise these additional router nics aren't used in isolated network VR.

As described in an old email by Kelvin below, the global setting and feature were introduced for legacy templates where hot-plugging wasn't supported:
http://mail-archives.apache.org/mod_mbox/cloudstack-users/201304.mbox/%3CCD81EFED.3B701%25kelven.yang@citrix.com%3E

All modern systemvmtemplates had open-vm-tools installed and support hot-plugging which is what is used in the case of VPC VR, but nic-hotpluging isn't implemented for isolated network VR.

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR, Isolated Networks

CLOUDSTACK VERSION

4.16.0.0

CONFIGURATION

OS / ENVIRONMENT

VMware 6.7u3, with Vmxnet3 used a nic controller for routers/systemvms

[weizhouapache]

I had some discussion and investigation with @harikrishna-patnala

when router.extra.public.nics is not 0 and vmware.systemvm.nic.device.type is set to "Vmxnet3" , the network VR will have more than 3 nics: 3 (guest, linklocal, source nat) + router.extra.public.nics (for extra public ip ranges). Therefore it will hit the problem which is not fixed by vmware for many years: the nic order inside VR is not same as defined in vmx file. please refer to https://communities.vmware.com/t5/ESXi-Discussions/Problem-with-the-order-of-interfaces-using-vmxnet3/td-p/2088500

I have tested cloudstack 4.14, which seem to be working. in cloudstack VR, there is a udev rule file to rename the network device name

root@r-6-VM:~# cat /etc/udev/rules.d/70-persistent-net.rules 

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:6e:89:00:02", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:52:a4:00:04", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="1e:00:37:00:00:04", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:3b:02:00:05", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:25:30:00:06", NAME="eth4"

and it works fine (please notice the device name)

root@r-6-VM:~# ip a |egrep 'BROADCAST|ether'
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:00:6e:89:00:02 brd ff:ff:ff:ff:ff:ff
3: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:00:25:30:00:06 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:00:52:a4:00:04 brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:37:00:00:04 brd ff:ff:ff:ff:ff:ff
6: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:00:3b:02:00:05 brd ff:ff:ff:ff:ff:ff

Unfortunately, it does not work in 4.16 (VR cannot be started). the /etc/udev/rules.d/70-persistent-net.rules still exists, but the device name are not changed, for example eth0, eth1, eth2, eth3, eth4, not the order above.
this might be caused by some code changes (e.g. /opt/cloud/bin/setup/common.sh, /opt/cloud/bin/setup/router.sh), or systemvm template changes (e.g. systemd-udevd is used in latest Debian releases).

@harikrishna-patnala is currently working on removing router.extra.public.nics, which I think is a good idea and should fix the issue above (there will be only 3 nics on network VR: guest, link local and public).

[weizhouapache]

Another issue we found in our testing is, cloudstack VR always use E1000 nic controller if systemvm template has deploy_as_is =1. see code below

cloudstack/plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Lines 2357 to 2364 in 44c08b5

	String adapterTypeStr = deployAsIs ?
	mapAdapterType(deployAsIsInfo.getNicAdapterMap().get(nicTo.getDeviceId())) :
	vmSpec.getDetails().get(VmDetailConstants.NIC_ADAPTER);
	nicDeviceType = VirtualEthernetCardType.valueOf(adapterTypeStr);
	if (s_logger.isDebugEnabled()) {
	s_logger.debug("VM " + vmInternalCSName + " will be started with NIC device type: " + nicDeviceType + " on NIC device " + nicTo.getDeviceId());
	}

I checked all platforms, the template SystemVM Template (vSphere) (id=8) has deploy_as_is = 1.
but on Hari's testing platform, it is set to 0.

@nvazquez do you have idea why it is set to 1 ? can we change it to 0 ?

[nvazquez]

@weizhouapache I have checked the codebase and it seems the auto-upgrade system VM mechanism sets it to true for VMware (https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java#L484, https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java#L816 and https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java#L838)

@davidjumani @Pearl1594 would it cause any potential regression if the above lines are removed?

[Pearl1594]

@nvazquez @weizhouapache yes, this was identified and has been fixed with #5863. With that PR we no longer enforce the systemVM template to be registered as deploy-as-is (by setting the flag to true in the upgrade path).