Issue with local.file_match and Nomad log rotation

[thoward27]

What's wrong?

I'm trying to use Alloy to parse logs from Nomad allocations and forward them to Loki. Everything works at first, but then Alloy never picks up new log files as they are created.

Steps to reproduce

1. Have a nomad client, running allocs.
2. Deploy alloy with local.file_match to glob all of the log files
3. Wait for a new log file to be created, alloy does not pull log lines from the new file

System information

Ubuntu 22.04.2 LTS (GNU/Linux 6.2.0-1015-aws x86_64)

Software version

version v1.8.2 (branch: HEAD, revision: 48d0d88)

Configuration

local.file_match "nomad_logs" {
  path_targets = [
    {__path__ = "/var/lib/nomad/alloc/*/alloc/logs/*.stdout.*", __path_exclude__="/var/lib/nomad/alloc/**/*.fifo"},
    {__path__ = "/var/lib/nomad/alloc/*/alloc/logs/*.stderr.*", __path_exclude__="/var/lib/nomad/alloc/**/*.fifo"},
  ]
}
loki.source.file "nomad_logs" {
  targets = local.file_match.nomad_logs.targets
  forward_to = [loki.write.loki_writer.receiver]
}

Logs

journalctl -u grafana-alloy -g "service-foo.stdout.65"
-- No entries --

[kalleep]

Hey, you are most likely hitting the issue fixed by #3488. Can you try 1.8.3?

[thoward27]

I'm trying that now! One other thing related to nomad's log rotation is that it seems Alloy never forgets about a log file once it has read contents from it. Is there anyway to get it to purge logs from it's internal store after so many missed syncs? While restarting the service we are bombarded with logs like

ts=2025-05-14T15:01:55.236720156Z level=info msg="skipping update of position for a file which does not currently exist

[kalleep]

ts=2025-05-14T15:01:55.236720156Z level=info msg="skipping update of position for a file which does not currently exist

Was this on 1.8.3? That should go away when the files on disk does.

[thoward27]

Ah, yes, it seems 1.8.3 fixes both of these issues for me. After rolling that out I can confirm that I've seen new log files get parsed by Alloy. I can also confirm that I only see the skip message twice per deleted file. Thanks for your help @kalleep!