Vagrant vmware provider fails to work with Fusion Tech Preview

[fatso83]

Hi, I am running an Apple Silicon Macbook (ARM) and found out that VirtualBox does not work on that platform. I then learned that VMWare released a Tech Preview of Fusion for ARM last month, and so I was hopeful in getting that to work with Vagrant, as you provide a VMWare provider.

I followed all the steps for installing the provider and you can see from the output that it seems to work. I then proceeded to follow the Getting Started guide (skipping the install of VirtualBox). You could also see my own notes when doing this. All seemed fine up until the last steps:

$ vagrant init hashicorp/bionic64

$ vagrant up --provider vmware_desktop
Vagrant encountered an unexpected communications error with the
Vagrant VMware Utility driver. Please try to run the command
again. If this error persists, please contact support@hashicorp.com

When running with the debug output, it seems as if it could not find any image matching the supplied image. I am not that familiar with Vagrant, but I assumed it might have to do with the image needing to specifically work with VMWare? So I searched vagrant images for anything support vmware and tried both "generic/ubuntu2104" and "roboxes/ubuntu2104" which both were tagged vmware. No luck there either; they all erred with something like this in the debug output

 WARN vagrantfile: Performing lookup with initial values roboxes/ubuntu2104 with version
 INFO box_collection: Box not found: roboxes/ubuntu2104 (vmware_desktop, vmware_fusion, vmware_workstation)

The following is info following the Bug template

$ vagrant -v
Vagrant 2.2.18

carlerik at carl-eriks-air in ~/dev/dotfiles (master)
$ vagrant plugin list
vagrant-vmware-desktop (3.0.1, global)
  - Version Constraint: > 0

carlerik at carl-eriks-air in ~/dev/dotfiles (master)
$ /opt/vagrant-vmware-desktop/bin/vagrant-vmware-utility -v
1.0.20

carlerik at carl-eriks-air in ~/dev/dotfiles (master)
$ uname -a
Darwin carl-eriks-air.lan 21.1.0 Darwin Kernel Version 21.1.0: Sat Sep 11 12:27:45 PDT 2021; root:xnu-8019.40.67.171.4~1/RELEASE_ARM64_T8101 arm64

Debug output

https://gist.github.com/fatso83/7da6c48debb4b48333feab90b44964d8

Expected behavior

That it fired up a vm as described in getting started

Actual behavior

It erred with

Vagrant encountered an unexpected communications error with the
Vagrant VMware Utility driver.

Steps to reproduce

1. Use a M1 Mac (arm)
2. Install the Fusion Tech Preview
3. vagrant init hashicorp/bionic64
4. vagrant up --provider vmware_desktop

[fatso83]

I am suspecting this basically comes down to dealing with running an unsupported architecture (running x86-64 on ARM64), as mentioned in VMWare knowledge article 84273.

For instance, using VMware Fusion, an Arm Linux guest must be created and run on an Apple silicon Macintosh, not an X86_64 Macintosh.

There are three issues here from a user perspective that needs to be fixed:

1. There is no metadata for the Vagrant boxes available that tells me what the underlying architecture is. I cannot tell if a box is x86-64 or ARM64 when it is called bionic64. This should be fixed in the Vagrant Cloud interface.
2. "Some error occurred" is not really helping anyone. The vmware provider should be able to tell the user that the cpu platform is unsupported for that box, just like the Fusion Desktop product tells me. It does not help that an image is available for the vmware_desktop provider, if the underlying architecture itself is unsupported.
3. There are no images for ARM supported by the vmware provider. Parallels got here a year ago, so they have at least some arm boxes.

AFAIK, it seems the most likely route is to download the ARM image for Ubuntu and create a box based on that.

[caius]

I was looking into this a few weeks ago (2021-09-23 according to my notes) and reached the same error message as you. Digging through the provider code, I think it was erroring on trying to parse the version output from the vmware-vmx binary which is invoked internally by the provider.

$ /Applications/VMWare\ Fusion.app/Contents/Library/vmware-vmx -v

VMware Fusion Information:
VMware Fusion e.x.p build-18656771 Release

And then at

vagrant-vmware-desktop/go_src/vagrant-vmware-utility/driver/base.go

Line 22 in 7b26510

const VMWARE_VERSION_PATTERN = `(?i)VMware\s+(?P<product>[A-Za-z0-9-]+)\s+(?P<version>[\d.]+)\s*(?P<build>\S+)?\s*(?P<type>[A-Za-z0-9-]+)?`

it defines a regex to extract the version that doesn't match the above.

vagrant-vmware-desktop/go_src/vagrant-vmware-utility/driver/base_posix.go

Lines 109 to 113 in 7b26510

	matches, err := utility.MatchPattern(VMWARE_VERSION_PATTERN, out)
	if err != nil {
	b.logger.Trace("vmware version match failed", "output", out, "pattern", VMWARE_VERSION_PATTERN, "error", err)
	return nil, errors.New("Failed to extract VMware version information")
	}

is the point at which it's compared and raises an error internally.

I've failed to document how I found this, but I think I was running vagrant up --debug --provider vmware_desktop and digging through the vagrant-vmware-desktop rubygem without finding out much. At some point I realised there was a vagrant-vmware-utility launchd service running and that it logged to /Library/Application Support/vagrant-vmware-utility/service.log. Looking in there reveals

2021-10-25T10:12:20.649+0100 [INFO]  vagrant-vmware-utility.api: created: vmx=<nil>
2021-10-25T10:12:20.897+0100 [ERROR] vagrant-vmware-utility.api: failed to generate VMware installation information: error="Failed to extract VMware version information"
2021-10-25T10:12:20.898+0100 [ERROR] vagrant-vmware-utility.api: base driver setup failure: error="Failed to extract VMware version information"
Failed to setup Vagrant VMWare API service - Failed to extract VMware version information

which in turn loops me back round to the provider code I've noted above parsing the version number out of the vmx binary output.

[caius]

Ah, I see #16 fixed the version parsing mismatch and utility v1.0.21 has been released with the fix in it.

[fatso83]

Hmm ... good insights! I am really wondering how I ended up with 1.0.20 release, when the 1.0.21 fix arrived one week prior to me filing a bug. Pretty sure I was installing all of this just a day or two prior to filing the bug. But I will recheck! Did you get it working for you, using the 1.0.21 release, btw?

Just as an additional thing: I did end up creating an ARM image in the end and test that out as well, but I did not get any further. Just crashed with the very general error message. The Ubuntu Impish Indri image is now available on Vagrant Cloud, if anyone else would like to have a go. I will retry with the 1.0.21 fix when time permits, to see if this all just comes down to parsing the version number.

[fatso83]

Nope.

Vagrant experienced an error! Details:
ERROR vagrant: #<HashiCorp::VagrantVMwareDesktop::Errors::DriverAPIRequestUnexpectedError: Vagrant encountered an unexpected communications error with the
Vagrant VMware Utility driver. Please try to run the command
again. 

$ /opt/vagrant-vmware-desktop/bin/vagrant-vmware-utility -v
1.0.21

[yyounes75]

Got a different error message with VMware Fusion (release 18656771)

2021-10-25T13:11:15.496+0200 [INFO]  vagrant-vmware-utility.api: created: vmx=<nil>
2021-10-25T13:11:15.498+0200 [ERROR] vagrant-vmware-utility.api: path loading failure: error="Failed to locate VMware installation directory!"
2021-10-25T13:11:15.500+0200 [ERROR] vagrant-vmware-utility.api: base driver setup failure: error="Failed to locate VMware installation directory!"
Failed to setup Vagrant VMWare API service - Failed to locate VMware installation directory!

Same version as you

$ /opt/vagrant-vmware-desktop/bin/vagrant-vmware-utility -v
1.0.21

[caius]

With the following installed versions of vagrant utility, vagrant plugin and vmware fusion itself:

$ /opt/vagrant-vmware-desktop/bin/vagrant-vmware-utility -v
1.0.21

$ vagrant plugin list
vagrant-vbguest (0.30.0, global)
vagrant-vmware-desktop (3.0.1, global)
  - Version Constraint: > 0

$ /Applications/VMWare\ Fusion.app/Contents/Library/vmware-vmx -v

VMware Fusion Information:
VMware Fusion e.x.p build-18656771 Release

and a Vagrantfile set to use @fatso83's box (thanks!)

Vagrant.configure("2") do |config|
  config.vm.box = "fatso83/impish64-arm"

  config.vm.provider "vmware_desktop" do |v|
    v.gui = true
    v.memory = "1024"
  end
end

Running vagrant up I get the following output (box has already been imported at this point):

$ vagrant up           
Bringing machine 'default' up with 'vmware_desktop' provider...
==> default: Cloning VMware VM: 'fatso83/impish64-arm'. This can take some time...
An error occurred while executing `vmrun`, a utility for controlling
VMware machines. The command and output are below:

Command: ["clone", "/Users/caius/.vagrant.d/boxes/fatso83-VAGRANTSLASH-impish64-arm/1.0.1/vmware_desktop/Ubuntu 64-bit Arm Server 21.10.vmx", "/Users/caius/tmp/m1ubuntu/.vagrant/machines/default/vmware_desktop/67aff3a8-a1cf-407e-a2ea-6fba427b6c30/Ubuntu 64-bit Arm Server 21.10.vmx", "linked", {:notify=>[:stdout, :stderr]}]

Stdout: Error: The file is already in use

Stderr: 

I can reproduce it running the vmrun clone command directly with the arguments in the error message.

$ rm -f "/Users/caius/tmp/m1ubuntu/.vagrant/machines/default/vmware_desktop/b36f35be-c967-4582-aba6-5c2626453b48/Ubuntu 64-bit Arm Server 21.10.vmx"
$ vmrun clone "/Users/caius/.vagrant.d/boxes/fatso83-VAGRANTSLASH-impish64-arm/1.0.1/vmware_desktop/Ubuntu 64-bit Arm Server 21.10.vmx" "/Users/caius/tmp/m1ubuntu/.vagrant/machines/default/vmware_desktop/b36f35be-c967-4582-aba6-5c2626453b48/Ubuntu 64-bit Arm Server 21.10.vmx" linked
Error: The file is already in use

---

Looking at the docs, I can see https://www.vagrantup.com/docs/providers/vmware/configuration#linked-clones states you can do a full clone instead of a linked clone. Updating the Vagrantfile with this then leads my VM to booting at least. Vagrant gets stuck at "Waiting for the VM to receive an address" stage, although I can see the VM has booted with the GUI enabled.

Vagrant.configure("2") do |config|
  config.vm.box = "fatso83/impish64-arm"

  config.vm.provider "vmware_desktop" do |v|
    v.gui = true
    v.memory = "1024"
    v.linked_clone = false
  end
end

↳ caius$ v up
Bringing machine 'default' up with 'vmware_desktop' provider...
==> default: Cloning VMware VM: 'fatso83/impish64-arm'. This can take some time...
==> default: Verifying vmnet devices are healthy...
==> default: Preparing network adapters...
WARNING: The VMX file for this box contains a setting that is automatically overwritten by Vagrant
WARNING: when started. Vagrant will stop overwriting this setting in an upcoming release which may
WARNING: prevent proper networking setup. Below is the detected VMX setting:
WARNING: 
WARNING:   ethernet0.pcislotnumber = "160"
WARNING: 
WARNING: If networking fails to properly configure, it may require this VMX setting. It can be manually
WARNING: applied via the Vagrantfile:
WARNING: 
WARNING:   Vagrant.configure(2) do |config|
WARNING:     config.vm.provider :vmware_desktop do |vmware|
WARNING:       vmware.vmx["ethernet0.pcislotnumber"] = "160"
WARNING:     end
WARNING:   end
WARNING: 
WARNING: For more information: https://www.vagrantup.com/docs/vmware/boxes.html#vmx-allowlisting
==> default: Starting the VMware VM...
==> default: Waiting for the VM to receive an address...

[yyounes75]

@caius I managed to get the same error as you by making a symlink ln -s /Applications/VMWare\ Fusion\ Tech\ Preview.app /Applications/VMWare\ Fusion.app

[caius]

@yyounes75 oh interesting, mine's already at /Applications/VMWare\ Fusion.app - I wonder if I renamed it during a previous debugging session 🤔

[fatso83]

@caius This is very promising! At least that's the farthest I have seen anyone come to something working, at least 😄 Btw, this "Error: The file is already in use" thing, is this somehow caused by me not creating the box properly or is it something else? It's my second attempt at creating a box (all related to this issue), but I just followed a general guide, but I might have missed something. I think I removed all lock files I could find, at least.

With regards to this:

although I can see the VM has booted with the GUI enabled.

There is no gui. I just created a plain server image with no gui packages installed. I don't know what the v.gui = true config does, but I would assume it tries to make some kind of frame buffer available for the graphics system to connect to, and there should not be any such activity taking place, that I know of. Just booting into a terminal.

I wonder if I renamed it during a previous debugging session 🤔

I think that's pretty certain. The Tech Preview will install alongside any other "stable" release. Maybe the provider should be able to pick up some override from an environment variable or something else to try and find the VMWare install in an alternative location?

[caius]

v.gui = true configures VMWare to show a window for the VM booting, so you can see what it's doing. Useful for debugging issues with VMs in Vagrant. You're right that there's no GUI in the VM, so it comes up with a login prompt (serial console?) in the VMWare window.

[fatso83]

So it basically seems you have been able to get everything working, apart from networking (which fails due to something out of our control)?

[yyounes75]

Thanks to @caius the VM from @fatso83 can start but vagrant hangs at startup, it seems it can't fetch the ip address. Some debug logs :

DEBUG vmware_driver: No explicitly set MAC, looking or auto-generated one...
DEBUG vmware_driver:  -- MAC: 00:0c:29:b4:f3:36
 INFO vmware_driver: Reading DHCP lease for '00:0c:29:b4:f3:36' on 'vmnet8'
DEBUG vagrant_utility: request METHOD=GET PATH=/vmnet/vmnet8/dhcplease/00:0c:29:b4:f3:36 RESPONSE=400 ERROR=No entry found for MAC 00:0c:29:b4:f3:36
 INFO vmware_driver: Trying vmrun getGuestIPAddress...
 INFO subprocess: Starting process: ["/Applications/VMware Fusion.app/Contents/Library/vmrun", "getGuestIPAddress", "/Users/anon/Dev/.vagrant/machines/first/vmware_fusion/65b3191d-b064-48a1-979a-2322e76f934b/Ubuntu 64-bit Arm Server 21.10.vmx"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stdout: Error: The VMware Tools are not running in the virtual machine: /Users/anon/Dev/.vagrant/machines/first/vmware_fusion/65b3191d-b064-48a1-979a-2322e76f934b/Ubuntu 64-bit Arm Server 21.10.vmx
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 255
 INFO vmware_driver: vmrun getGuestIPAddress failed: VMRunError

EDIT:
Even if I set config.vm.network : private_network, ip: '10.10.10.10' the vmware_utility fails to set/fetch the IP address

DEBUG network: Normalized configuration: {:auto_config=>false, :type=>:dhcp}
DEBUG network: Adapter configuration: {:type=>:nat, :slot=>0}
DEBUG network: Network configuration: {:type=>:dhcp, :auto_config=>false}
 INFO network: Slot 1. Type: hostonly
DEBUG network: Normalized configuration: {:auto_config=>true, :netmask=>"255.255.255.0", :type=>:static, :ip=>"10.10.10.10", :protocol=>"tcp", :id=>"7b323912-91ec-4919-bc17-d2e5cce60bfe", :subnet_ip=>"10.10.10.0", :adapter_ip=>"10.10.10.1"}
DEBUG vagrant_utility: request METHOD=GET PATH=/vmnet RESPONSE=200 
 INFO network: Checking for hostonly network collisions...
 INFO subprocess: Starting process: ["/usr/sbin/netstat", "-nr", "-f", "inet"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stdout: Routing tables

Internet:
Destination        Gateway            Flags        Netif Expire
default            192.168.1.254      UGSc           en0       
default            link#22            UCSI       bridge1      !
default            link#24            UCSI       bridge1      !
127                127.0.0.1          UCS            lo0       
127.0.0.1          127.0.0.1          UH             lo0       
169.254            link#9             UCS            en0      !
172.16.248/24      link#24            UC         bridge1      !
192.168.1          link#9             UCS            en0      !
192.168.1.2        b0:b9:8a:61:53:6f  UHLWI          en0   1152
192.168.1.5        8e:ee:a3:c2:e:30   UHLWIi         en0   1171
192.168.1.7        0:1e:6:30:bd:c3    UHLWIi         en0   1191
192.168.1.8/32     link#9             UCS            en0      !
192.168.1.8        a0:78:17:6d:80:d4  UHLWI          lo0       
192.168.1.9        0:12:31:c:5c:71    UHLWI          en0   1197
192.168.1.10       2:f5:f9:57:5c:6d   UHLWI          en0      !
192.168.1.21       3c:22:fb:9b:d9:a5  UHLWIi         en0    271
192.168.1.26       70:fc:8f:7f:3b:1d  UHLWI          en0   1194
192.168.1.37       e8:39:35:d2:f7:b6  UHLWI          en0    263
192.168.1.47       ae:6d:be:61:cb:12  UHLWIi         en0    936
192.168.1.100      0:12:17:ea:6a:ea   UHLWI          en0   1199
192.168.1.101      0:12:17:ea:a:a5    UHLWI          en0   1198
192.168.1.254/32   link#9             UCS            en0      !
192.168.1.254      34:27:92:61:24:46  UHLWIir        en0   1168
192.168.1.255      ff:ff:ff:ff:ff:ff  UHLWbI         en0      !
192.168.26         link#22            UC         bridge1      !
224.0.0/4          link#9             UmCS           en0      !
224.0.0.251        1:0:5e:0:0:fb      UHmLWI         en0       
255.255.255.255/32 link#9             UCS            en0      !
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO network: No collisions detected, creating new vmnet device.
DEBUG vagrant_utility: request METHOD=POST PATH=/vmnet RESPONSE=400 ERROR=Failed to enable device
ERROR warden: Error occurred: Vagrant failed to create a new VMware networking device. The following
error message was generated while attempting to create a new device:

  Failed to enable device

Please resolve any problems reported in the error message above and
try again.

[fatso83]

This is a longshot, but maybe other paths differ in the tech preview as well? This is from go_src/vagrant-vmware-utility/utility/vmware_paths_darwin.go:

Might be worth checking if the paths in /Library/Preferences/ are the same. Maybe they need some manual fiddling as well? 🤷 At work, not able to check myself.