"networks failed: LXD object not found" after `sudo snap connect multipass:lxd lxd`

[ynott]

Describe the bug

I would like to set up a Multipass bridge.
However, the multipass network is not working.

To Reproduce

1. Install lxd
   sudo snap install lxd
2. Initialise lxd
   sudo lxd -d -v init
3. Change Multipass backend to lxd
   sudo multipass set local.driver=lxd
4. Connect to lxd
   sudo snap connect multipass:lxd lxd

Expected behavior

The multipass network is working.

Logs

$ multipass networks -vvvv
[2021-06-17T21:21:46.511] [trace] [lxd request] Requesting LXD: GET unix://multipass/var/snap/lxd/common/lxd/unix.socket@1.0/networks?recursion=1&project=multipass

networks failed: LXD object not found

 6 17 18:23:38 xxx-desktop systemd[1]: Started Service for snap application multipass.multipassd.
 6 17 18:23:38 xxx-desktop multipassd[10042]: propsReply "An AppArmor policy prevents this sender from sending this message to this recipient; type=\"method_call\",>
 6 17 18:23:38 xxx-desktop multipassd[10042]: nmReply "An AppArmor policy prevents this sender from sending this message to this recipient; type=\"method_call\", se>
 6 17 18:23:38 xxx-desktop multipassd[10042]: "Object path cannot be empty"
 6 17 18:23:38 xxx-desktop multipassd[10042]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on
 6 17 18:23:50 xxx-desktop multipassd[10042]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Host unreachabl>
 6 17 18:24:03 xxx-desktop multipassd[10042]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:d>
 6 17 18:24:15 xxx-desktop multipassd[10042]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/daily/streams/v1/index.json: Host unreachable. >
 6 17 18:24:16 xxx-desktop multipassd[10042]: Could not update manifest: failed to download from 'http://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-cor>
 6 17 18:24:21 xxx-desktop multipassd[10042]: Timeout getting response for GET operation on unix://multipass/var/snap/lxd/common/lxd/unix.socket@1.0/images?recursio>
 6 17 18:24:21 xxx-desktop multipass.multipassd[10042]: [error] [daemon] Caught an unhandled exception: Timeout getting response for GET operation on unix://multipa>
 6 17 18:24:21 xxx-desktop systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
 6 17 18:24:21 xxx-desktop systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
 6 17 18:24:21 xxx-desktop systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 4.
 6 17 18:24:21 xxx-desktop systemd[1]: Stopped Service for snap application multipass.multipassd.
 6 17 18:24:21 xxx-desktop systemd[1]: Started Service for snap application multipass.multipassd.
 6 17 18:24:21 xxx-desktop multipassd[10199]: propsReply "An AppArmor policy prevents this sender from sending this message to this recipient; type=\"method_call\",>
 6 17 18:24:21 xxx-desktop multipassd[10199]: nmReply "An AppArmor policy prevents this sender from sending this message to this recipient; type=\"method_call\", se>
 6 17 18:24:21 xxx-desktop multipassd[10199]: "Object path cannot be empty"
 6 17 18:24:21 xxx-desktop multipassd[10199]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on
 6 17 18:24:30 xxx-desktop multipassd[10199]: Starting Multipass 1.6.2
 6 17 18:24:30 xxx-desktop multipassd[10199]: Daemon arguments: /snap/multipass/4458/bin/multipassd --verbosity debug --logger platform
 6 17 18:32:08 xxx-desktop systemd[1]: Started snap.multipass.multipass.fe8d72e7-0286-48a3-8b93-1bda51898a39.scope.
 6 17 18:32:08 xxx-desktop multipassd[10199]: Goodbye!
 6 17 18:32:08 xxx-desktop systemd[1]: snap.multipass.multipass.fe8d72e7-0286-48a3-8b93-1bda51898a39.scope: Succeeded.
 6 17 18:32:08 xxx-desktop systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=42/n/a
 6 17 18:32:08 xxx-desktop systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
 6 17 18:32:09 xxx-desktop systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 5.
 6 17 18:32:09 xxx-desktop systemd[1]: Stopped Service for snap application multipass.multipassd.
 6 17 18:32:09 xxx-desktop systemd[1]: Started Service for snap application multipass.multipassd.
 6 17 18:32:09 xxx-desktop multipassd[11593]: propsReply "An AppArmor policy prevents this sender from sending this message to this recipient; type=\"method_call\",>
 6 17 18:32:09 xxx-desktop multipassd[11593]: nmReply "An AppArmor policy prevents this sender from sending this message to this recipient; type=\"method_call\", se>
 6 17 18:32:09 xxx-desktop multipassd[11593]: "Object path cannot be empty"
 6 17 18:32:09 xxx-desktop multipassd[11593]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on
 6 17 18:32:17 xxx-desktop multipassd[11593]: Starting Multipass 1.6.2
 6 17 18:32:17 xxx-desktop multipassd[11593]: Daemon arguments: /snap/multipass/4458/bin/multipassd --verbosity debug --logger platform

$ lxc network list
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
|  NAME  |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| enp2s0 | physical | NO      |                 |                           |             | 0       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| enp3s0 | physical | NO      |                 |                           |             | 0       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| lxdbr0 | bridge   | YES     | 10.248.147.1/24 | fd42:cc0b:f712:28c5::1/64 |             | 2       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| wlp1s0 | physical | NO      |                 |                           |             | 0       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+

$ lxc network show lxdbr0
config:
  ipv4.address: 10.248.147.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:cc0b:f712:28c5::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/bionic
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

$ lxc list
+--------+---------+------+------+-----------+-----------+
|  NAME  |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+--------+---------+------+------+-----------+-----------+
| bionic | RUNNING |      |      | CONTAINER | 0         |
+--------+---------+------+------+-----------+-----------+

Additional info

• OS: Ubuntu 20.04.2 LTS
• multipass version multipass 1.6.2 multipassd 1.6.2
• multipass info --all

Additional context
Please let us know if there is any information you need.

[ricab]

Hi @ynott, thanks for the report. The log does not seem to cover the request for networks, but something seems amiss with respect to networking, either Multipass <-> LXD or LXD <-> outside.

LXD's output also shows that Multipass doesn't get to create mpbr0. (Only created on first launch.)

A few questions:

• what output do you get from other multipass commands, like list or find?
• what does snap connections multipass say?
• could you please copy the relevant section from journalctl -f -n1000 --unit snap.multipass* (avoids chopped lines)?
• would you be able to provide the result of dmesg | grep -i multipass | grep apparmor=\"DENIED\"?

Thanks

---

Log highlights AFAICS, for reference:

1. NetworkManager access blocked by AppArmor
2. remotes can't be reached for image info
3. uncaught timeout exception when trying to list images from LXD
   • causes daemon to abort
4. service restarts
5. NM blocked again
   • however, multipassd appears to run happily for 8 minutes here
6. multipassd quits with 42 (config change, probably driver revert)
7. AppArmor blocks NM access again

[ricab]

Seems related to #1689, in that there is an exception that is not caught and causes the daemon to quit when updating images. In this instance, it's a LXDRuntimeError reporting a timeout.

However, in this case the timeout is probably caused by the lxd interface not being connected yet. So this is a lesser issue, especially since automatic restarts should take care of transient timeout causes.

[ynott]

@ricab
Thank you for your reply.

I have tried various settings on my Ubuntu machine and the settings on this machine seem to be confusing.
I would like to thank you very much for your support.

First of all I tried to set up a network bridge in multipass with NetworkManager and I could not get the LXD bridge to work. So I created a bridge using Netplan from NetworkManager, but I am not able to pull DNS. This caused the following error.

6 17 18:23:50 xxx-desktop multipassd[10042]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Host unreachabl>

I have modified it so that it can pull DNS and will send you the results of running the command you gave me.

Execution result: multipass find

$ multipass find
Image                       Aliases           Version          Description
core                        core16            20200818         Ubuntu Core 16
core18                                        20200812         Ubuntu Core 18
18.04                       bionic            20210604         Ubuntu 18.04 LTS
20.04                       focal,lts         20210610         Ubuntu 20.04 LTS
20.10                       groovy            20210611         Ubuntu 20.10
21.04                       hirsute           20210616         Ubuntu 21.04
daily:21.10                 devel,impish      20210611         Ubuntu 21.10
appliance:adguard-home                        20200812         Ubuntu AdGuard Home Appliance
appliance:mosquitto                           20200812         Ubuntu Mosquitto Appliance
appliance:nextcloud                           20200812         Ubuntu Nextcloud Appliance
appliance:openhab                             20200812         Ubuntu openHAB Home Appliance
appliance:plexmediaserver                     20200812         Ubuntu Plex Media Server Appliance

Execution result: multipass list

$ multipass list
No instances found.

Execution result: snap connections multipass

$ snap connections multipass
Interface          Plug                         Slot                Notes
firewall-control   multipass:firewall-control   :firewall-control   -
home               multipass:all-home           :home               -
home               multipass:home               :home               -
kvm                multipass:kvm                :kvm                -
libvirt            multipass:libvirt            -                   -
lxd                multipass:lxd                lxd:lxd             -
multipass-support  multipass:multipass-support  :multipass-support  -
network            multipass:network            :network            -
network-bind       multipass:network-bind       :network-bind       -
network-control    multipass:network-control    :network-control    -
removable-media    multipass:removable-media    -                   -
unity7             multipass:unity7             :unity7             -
wayland            multipass:wayland            :wayland            -
x11                multipass:x11                :x11                -

Execution result: journalctl -f -n1000 --unit snap.multipass*

Note: This log is for the period when DNS could not be pulled. There is currently no log output.

 6 22 11:55:48 xxx-desktop multipassd[813]: Could not update manifest: failed to download from 'http://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-core-16-amd64.img.xz': Host cdimage.ubuntu.com not found
 6 22 11:56:02 xxx-desktop multipassd[813]: Could not update manifest: failed to download from 'http://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-core-16-amd64.img.xz': Host cdimage.ubuntu.com not found
 6 22 11:56:12 xxx-desktop multipassd[813]: Could not update manifest: failed to download from 'https://cloud-images.ubuntu.com/buildd/releases/bionic/release/bionic-server-cloudimg-amd64-disk.img': Host cloud-images.ubuntu.com not found
 6 22 11:56:22 xxx-desktop multipassd[813]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Host cloud-images.ubuntu.com not found. Using cached data instead.
 6 22 11:56:32 xxx-desktop multipassd[813]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.json: Host cloud-images.ubuntu.com not found. Using cached data instead.
 6 22 11:56:42 xxx-desktop multipassd[813]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/daily/streams/v1/index.json: Host cloud-images.ubuntu.com not found. Using cached data instead.
 6 22 11:56:52 xxx-desktop multipassd[813]: Cannot retrieve last modified date for https://cloud-images.ubuntu.com/daily/streams/v1/com.ubuntu.cloud:daily:download.json: Host cloud-images.ubuntu.com not found. Using cached data instead.
 6 22 11:57:02 xxx-desktop multipassd[813]: Cannot retrieve last modified date for http://cdimage.ubuntu.com/ubuntu-core/appliances/streams/v1/index.json: Host cdimage.ubuntu.com not found. Using cached data instead.
 6 22 11:57:12 xxx-desktop multipassd[813]: Cannot retrieve last modified date for http://cdimage.ubuntu.com/ubuntu-core/appliances/streams/v1/com.ubuntu.appliances:released:download.json: Host cdimage.ubuntu.com not found. Using cached data instead.
 6 22 11:57:12 xxx-desktop multipassd[813]: Could not update manifest: failed to download from 'https://cloud-images.ubuntu.com/buildd/releases/bionic/release/bionic-server-cloudimg-amd64-disk.img': Host cloud-images.ubuntu.com not found

Execution result: dmesg | grep -i multipass | grep apparmor=\"DENIED\"

$ dmesg | grep -i multipass | grep apparmor=\"DENIED\"
[   38.584556] audit: type=1400 audit(1623980648.342:74): apparmor="DENIED" operation="capable" profile="snap.multipass.multipassd" pid=813 comm="multipassd" capability=4  capname="fsetid"

So far I've got the command results and logs, and I've run the multipass network list command again.

$ multipass networks
networks failed: LXD object not found

One more line of AppArmor log has been added

$ dmesg | grep -i multipass | grep apparmor=\"DENIED\"
[   38.584556] audit: type=1400 audit(1623980648.342:74): apparmor="DENIED" operation="capable" profile="snap.multipass.multipassd" pid=813 comm="multipassd" capability=4  capname="fsetid"
[351447.151716] audit: type=1400 audit(1624332058.365:111): apparmor="DENIED" operation="open" profile="snap.multipass.multipass" name="/etc/init.d/" pid=580802 comm="bash" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

As you say, AppArmor seems to be stopping the execution.

[ynott]

Multipass is a very useful tool and one of my favourite commands.
In this case, I believe the problem arises because I have tried various settings.

I would like to try the configuration again on a fresh clean installed Ubuntu, but it would be nice to have a manual on how to configure it, including LXD in that case.

[ricab]

Hi @ynott, thanks for the new info. Looking again I think the AppArmor errors are a red herring – they happen as a result of Qt's internal networking initialization, which nevertheless succeeds.

The issue here is really in the inability to contact LXD (which I don't think is caused by the fsetid denial). This manifests both when trying to list images (see beginning of the log in the OP) and in multipass networks. I am not sure if it could be related to your other networking problems. Any firewall in play? What does this say?

curl --unix-socket /var/snap/lxd/common/lxd/unix.socket -H "Content-Type: application/json" -X GET "lxd/1.0/networks"

I was puzzled by the fact that list and find work, but actually Multipass doesn't strictly need a reply from LXD to return a result in those commands (they work even with LXD disabled).

[...] it would be nice to have a manual on how to configure it, including LXD in that case

You're right, we should have better documentation for that. However, the steps you need are the ones you described in "To Reproduce", although perhaps with 4 before 3.

For creating the bridge, how did go about it? I add the procedure that works for me below. BTW, with the next release Multipass will be able to create the bridge for you :)

Creating a bridge with netplan

Add a yaml file (evaluated alphabetically) to /etc/netplan (e.g. /etc/netplan/50-custom.yaml) with the following contents (replacing ):

network:
  ethernets:
    <ifname>:
      dhcp4: false
  bridges:
    mybr0:
      dhcp4: true
      interfaces:
        - <ifname>

Then run:

sudo netplan apply

To revert, erase the file (or (comment out) contents) and rerun sudo netplan apply.

[ynott]

That makes sense. Thank you for your kind attention.

I'll try it on another clean Ubuntu machine, so I'll close this Issue.
If it doesn't work, please let me reopen the Issue.

[ricab]

Sure thing @ynott, I hope it works better for you now! Feel free to reopen otherwise, or just share what the ultimate cause was, if you find it.

[ynott]

@ricab
Now, It worked!

I installed a fresh Ubuntu and tested it.
I am using the Desktop Edition of Ubuntu 20.04 LTS.

Desktop Edition is NetworkManager by default, but when I switched to NetPlan, it seems that some settings were left behind.

results: $ dmesg | grep -i multipass | grep apparmor=\"DENIED\"

[ 7450.113683] audit: type=1107 audit(1624876356.220:143): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="DeviceRemoved" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.114711] audit: type=1107 audit(1624876356.220:144): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="PropertiesChanged" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.117280] audit: type=1107 audit(1624876356.224:145): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="PropertiesChanged" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.229109] audit: type=1107 audit(1624876356.336:146): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="DeviceRemoved" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.230975] audit: type=1107 audit(1624876356.336:147): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="PropertiesChanged" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.232583] audit: type=1107 audit(1624876356.340:148): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="PropertiesChanged" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.249589] audit: type=1107 audit(1624876356.356:149): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="DeviceRemoved" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.250811] audit: type=1107 audit(1624876356.356:150): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="PropertiesChanged" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7450.252192] audit: type=1107 audit(1624876356.360:151): pid=643 uid=103 auid=4294967295 ses=4294967295 subj=unconfined msg='apparmor="DENIED" operation="dbus_signal"  bus="system" path="/org/freedesktop/NetworkManager" interface="org.freedesktop.NetworkManager" member="PropertiesChanged" name=":1.13" mask="receive" pid=3925 label="snap.multipass.multipassd" peer_pid=645 peer_label="unconfined"
[ 7455.264969] audit: type=1400 audit(1624876361.372:158): apparmor="DENIED" operation="capable" profile="snap.multipass.multipassd" pid=7019 comm="multipassd" capability=4  capname="fsetid"

So I uninstalled NetworkManager completely.

sudo apt-get purge network-manager

After that, I had to reboot twice.
The multipass network is now visible.

$ multipass networks
Name   Type    Description
br0    bridge  Network bridge
mpbr0  bridge  Network bridge for Multipass

I believe that the problem was probably NetworkManager.

And then, I have launched a Multipass instance with br0 specified by the following command.

multipass launch 18.04 --network br0

The results are as follows.
It seems to be working properly.

$ multipass list
Name                    State             IPv4             Image
paramount-dace          Running           10.62.26.4       Ubuntu 18.04 LTS
                                          192.168.1XX.3

192.168.1XX.3 is the address range of my local network.

[ricab]

Puzzling, but glad you solved it 🙂

[ynott]

@ricab
I wrote a report on running multipass with bridge.
Can you take a look at it?

https://gist.github.com/ynott/f4bdc89b940522f2a0e4b32790ddb731

If you find any mistakes, please let me know.

[Saviq]

I'm reopening this as I believe I found a way to reproduce - just have a completely empty LXD, prior to launching any instances.

[ynott]

@Saviq
YES! You are absolutely right.
If it is not initialized by lxd, I would like to get a warning message.

I wrote a detailed manual on how to use multipass bridge, can you take a look at it?

Instructions for running multipass on a bridge network
https://gist.github.com/ynott/f4bdc89b940522f2a0e4b32790ddb731#5-2-initialize-lxd

Let me know if you have any advice.