sysmon not starting on ubuntu 24.04 due to

[astrakid]

Describe the bug
Apr 17 06:45:48 vm sysmon[2279738]: libbpf: prog 'FileOpenRawExit': failed to load: -22
Apr 17 06:45:48 vm sysmon[2279738]: libbpf: failed to load object './/sysmonEBPFkern5.6-_core.o'
Apr 17 06:45:48 vm sysmon[2279532]: Telemetry failed to start: eBPF object could not be loaded

To Reproduce

• we installed a naked ubuntu 24.04 LTS on hyper-v with the following kernel:
  6.11.0-1012-azure Command correction in SysmonForLinux/Readme #12~24.04.1-Ubuntu SMP
  installing sysmon leads to the mentioned error.
• we were even able to upgrade an existing 22.04 LTS (with working sysmon) and reproduce the same issue after upgrade

Sysmon version
1.3.5

Distro/kernel version
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

6.11.0-1012-azure #12~24.04.1-Ubuntu SMP

Sysmon configuration
will send it later

Logs
Apr 17 06:45:48 vm sysmon[2279738]: libbpf: prog 'FileOpenRawExit': failed to load: -22
Apr 17 06:45:48 vm sysmon[2279738]: libbpf: failed to load object './/sysmonEBPFkern5.6-_core.o'
Apr 17 06:45:48 vm sysmon[2279532]: Telemetry failed to start: eBPF object could not be loaded

Expected behavior
sysmon should be started

Additional context
by accident i found out that when starting sysmon with -bpf sysmonEBPFkern5.6-_core.o it is starting! without bpf set explicitley it does not!

[astrakid]

additional informatio: it seems that the sysmonEBPFkern is called relatively (./sysmonEBPF....), which may lead to this error.
maybe not a bug of sysmon itself, but of the way of installation

[MarioHewardt]

Hi - thanks for reporting this issue. It looks like it's working on a plain vanilla 24.04 and only occurs during the upgrade to a later kernel. Could you provide the following information:

• How did you do the upgrade to the new kernel and also how did you install Sysmon (sudo apt install sysmonforlinux?)
• Once you have it in the state where it doesn't work, what files are under /opt/sysmon?

[astrakid]

hi. well, the plain installed 24.04 was delivered by our data center service provider. so i don't know what they changed. at least nothing that has to do with the issue (see next lines)
the upgrade was done via do-release-upgrade. the latest sysmon was then installed by an ansible playbook, that i wrote. and it seems that this does not work on ubuntu 24.04 as it did on 22.04. although the service is installed more or less identically (copy config to /opt/sysmon/ and install via sysmon -i /opt/sysmon/config.xml) this worked under 22.04, but not in 24.04.
in 24.04 i have to explicitely call the install-parameter with -bpf /opt/sysmon/sysmonEBPF... (don't know the exact name by heart)

regards,
andre

[MarioHewardt]

After this step: sysmon -i /opt/sysmon/config.xml

Can you do an ls /opt/sysmon and let me know what is in that directory?

[albrecd]

I'm seeing the same issue, also on 24.04. The issue appeared after upgrading to 24.10 via do-release-upgrade (not surprising); however, I reinstalled 24.04 hoping to get it working again, and had the same issue. This was by downloading the latest 24.04 desktop ISO from Canonical (yesterday). Current Kernel is 6.11.0-24-generic.

journalctl output:

Apr 20 12:22:05 workstation sysmon[7394]: failed to resolve CO-RE relocation <byte_off> [561] struct inode___pre_v66.i_atime.tv_sec (0:0:0 @ offset 0)
Apr 20 12:22:05 workstation sysmon[7394]: processed 8469 insns (limit 1000000) max_states_per_insn 9 total_states 181 peak_states 142 mark_read 94
Apr 20 12:22:05 workstation sysmon[7394]: -- END PROG LOAD LOG --
Apr 20 12:22:05 workstation sysmon[7394]: libbpf: prog 'FileOpenRawExit': failed to load: -22
Apr 20 12:22:05 workstation sysmon[7394]: libbpf: failed to load object './/sysmonEBPFkern5.6-_core.o'
Apr 20 12:22:05 workstation sysmon[7109]: Telemetry failed to start: eBPF object could not be loaded

Contents of /opt/sysmon:

argc	    eventId    sysmonEBPFkern4.15_core.o  sysmonEBPFkern4.16.o		 sysmonEBPFkern5.2_core.o      sysmonEBPFkern5.3-5.5.o	  sysmonLogView
argv	    rules.bin  sysmonEBPFkern4.15.o	  sysmonEBPFkern4.17-5.1_core.o  sysmonEBPFkern5.2.o	       sysmonEBPFkern5.6-_core.o
config.xml  sysmon     sysmonEBPFkern4.16_core.o  sysmonEBPFkern4.17-5.1.o	 sysmonEBPFkern5.3-5.5_core.o  sysmonEBPFkern5.6-.o

Installed with sudo apt install sysmonforlinux

[astrakid]

for me it's comparable:

total 43900
-rw------- 1 root root        4 Apr 17 06:51 argc
-rw------- 1 root root       52 Apr 17 06:51 argv
-rw------- 1 root root    59778 Apr 14 11:42 config.xml
-rw------- 1 root root        8 Apr 19 05:18 eventId
-rw------- 1 root root   374312 Apr 19 05:18 rules.bin
-rwx------ 1 root root 27079096 Apr 19 05:18 sysmon
-rw------- 1 root root  1633200 Apr 19 05:18 sysmonEBPFkern4.15_core.o
-rw------- 1 root root  1067616 Apr 19 05:18 sysmonEBPFkern4.15.o
-rw------- 1 root root  1634000 Apr 19 05:18 sysmonEBPFkern4.16_core.o
-rw------- 1 root root  1068488 Apr 19 05:18 sysmonEBPFkern4.16.o
-rw------- 1 root root  1611488 Apr 19 05:18 sysmonEBPFkern4.17-5.1_core.o
-rw------- 1 root root  1053576 Apr 19 05:18 sysmonEBPFkern4.17-5.1.o
-rw------- 1 root root  1927152 Apr 19 05:18 sysmonEBPFkern5.2_core.o
-rw------- 1 root root  1923656 Apr 19 05:18 sysmonEBPFkern5.2.o
-rw------- 1 root root  1471936 Apr 19 05:18 sysmonEBPFkern5.3-5.5_core.o
-rw------- 1 root root   631304 Apr 19 05:18 sysmonEBPFkern5.3-5.5.o
-rw------- 1 root root  1471904 Apr 19 05:18 sysmonEBPFkern5.6-_core.o
-rw------- 1 root root   631280 Apr 19 05:18 sysmonEBPFkern5.6-.o
-rwx------ 1 root root  1266504 Apr 19 05:18 sysmonLogView```

[MarioHewardt]

Ah, the following line gives away the problem -

failed to resolve CO-RE relocation <byte_off> [561] struct inode___pre_v66.i_atime.tv_sec (0:0:0 @ offset 0)

Something in the kernel changed (member removed/renamed). I'm a bit backlogged at the moment but will dig into it once I get some time.

[albrecd]

Thank you!

[MarioHewardt]

Here is the net-net.

• struct timespec64 __i_atime;
• struct timespec64 __i_mtime;
• struct timespec64 _i_ctime; /* use inode*_ctime accessors! */

• time64_t i_atime_sec;
• time64_t i_mtime_sec;
• time64_t i_ctime_sec;
• u32 i_atime_nsec;
• u32 i_mtime_nsec;
• u32 i_ctime_nsec;

https://lore.kernel.org/lkml/20240520104347.j5fz4eem52bv2wzg@quack3/

[MarioHewardt]

The fix for this has been merged. We are planning on releasing a new package by end of next week.