fix: Compute signatures from files so that loki.source.file can handle atomic writes#5143
fix: Compute signatures from files so that loki.source.file can handle atomic writes#5143
Conversation
1678109 to
6585c27
Compare
There was a problem hiding this comment.
Pull request overview
This pull request adds signature-based detection of atomic writes to the file tailer. The implementation computes a signature from the first N bytes of a file to detect when a file has been replaced atomically (e.g., by editors like neovim). The signature is progressively built as the file grows, with recomputation at specific thresholds (64, 128, 256, 512 bytes) until reaching the target size of 1024 bytes.
Changes:
- Added signature computation and comparison logic to handle atomic file replacements
- Modified file reopening logic to preserve read position when signatures match
- Added progressive signature recomputation as files grow
- Removed automatic offset reset on file deletion, now using signature comparison instead
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 13 comments.
| File | Description |
|---|---|
| internal/component/loki/source/file/internal/tail/signature.go | New file implementing signature struct and methods for computing and comparing file signatures |
| internal/component/loki/source/file/internal/tail/signature_test.go | Comprehensive unit tests for signature functionality |
| internal/component/loki/source/file/internal/tail/file.go | Integration of signature computation and comparison into file tailing logic, including initialization, recomputation, and reopen handling |
| internal/component/loki/source/file/internal/tail/file_test.go | Added atomic write test case and helper function |
|
@thampiotr I changed to use |
This is beacuse os.Rename does not work on windows
|
Windows test failing is unrelated to this pr |
The atomic package we use uses MoveFileEx on windows to replace the file but this operation do not seem work when the file is already opened by another proccess.
internal/component/loki/source/file/internal/tail/fileext/file_windows.go
Outdated
Show resolved
Hide resolved
internal/component/loki/source/file/internal/tail/fileext/file_windows.go
Outdated
Show resolved
Hide resolved
internal/component/loki/source/file/internal/tail/atomic_write_posix_test.go
Outdated
Show resolved
Hide resolved
Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com>
PR Description
Noticed when doing some manual testing. Our tailer do not handle atomic writes, e.g. neovim will perform these kind of writes on every save.
Not sure how big of a problem this is in "normal" log rotation environments. What happens before this pr is that if I have a log file in neovim and save
:wwe will detect a delete event, reopen the file and consume it fully.If this is something we want to handle we can compute a signature. The signature consist of currently hard coded number of bytes (1024).
This pr handle incomplete signatures and recompute it once we have read past desired size.
Which issue(s) this PR fixes
Notes to the Reviewer
PR Checklist