Skip to content

fix: Compute signatures from files so that loki.source.file can handle atomic writes#5143

Merged
kalleep merged 30 commits intomainfrom
kalleep/tailer-signature
Jan 22, 2026
Merged

fix: Compute signatures from files so that loki.source.file can handle atomic writes#5143
kalleep merged 30 commits intomainfrom
kalleep/tailer-signature

Conversation

@kalleep
Copy link
Contributor

@kalleep kalleep commented Dec 18, 2025

PR Description

Noticed when doing some manual testing. Our tailer do not handle atomic writes, e.g. neovim will perform these kind of writes on every save.

Not sure how big of a problem this is in "normal" log rotation environments. What happens before this pr is that if I have a log file in neovim and save :w we will detect a delete event, reopen the file and consume it fully.

If this is something we want to handle we can compute a signature. The signature consist of currently hard coded number of bytes (1024).

This pr handle incomplete signatures and recompute it once we have read past desired size.

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

  • Documentation added
  • Tests updated
  • Config converters updated

@kalleep kalleep requested a review from a team as a code owner December 18, 2025 13:17
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds signature-based detection of atomic writes to the file tailer. The implementation computes a signature from the first N bytes of a file to detect when a file has been replaced atomically (e.g., by editors like neovim). The signature is progressively built as the file grows, with recomputation at specific thresholds (64, 128, 256, 512 bytes) until reaching the target size of 1024 bytes.

Changes:

  • Added signature computation and comparison logic to handle atomic file replacements
  • Modified file reopening logic to preserve read position when signatures match
  • Added progressive signature recomputation as files grow
  • Removed automatic offset reset on file deletion, now using signature comparison instead

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 13 comments.

File Description
internal/component/loki/source/file/internal/tail/signature.go New file implementing signature struct and methods for computing and comparing file signatures
internal/component/loki/source/file/internal/tail/signature_test.go Comprehensive unit tests for signature functionality
internal/component/loki/source/file/internal/tail/file.go Integration of signature computation and comparison into file tailing logic, including initialization, recomputation, and reopen handling
internal/component/loki/source/file/internal/tail/file_test.go Added atomic write test case and helper function

@kalleep kalleep changed the title fix: add signature for tail.File so we can handle atomic writes fix: Add signature for tail.File so we can handle atomic writes Jan 12, 2026
@kalleep
Copy link
Contributor Author

kalleep commented Jan 15, 2026

@thampiotr I changed to use ReadAt in newSignatureFromFile so we no longer need to perform the extra seek calls we did before.

@kalleep kalleep changed the title fix: Add signature for tail.File so we can handle atomic writes fix: Compute signatures from files so that loki.source.file can handle atomic writes Jan 16, 2026
This is beacuse os.Rename does not work on windows
@kalleep
Copy link
Contributor Author

kalleep commented Jan 22, 2026

Windows test failing is unrelated to this pr TestSourceMapsStoreImpl_CleanOldCachedEntries. Will take a look at fixing that test in another pr.

The atomic package we use uses MoveFileEx on windows to replace the file
but this operation do not seem work when the file is already opened by
another proccess.
@kalleep kalleep merged commit 3090c4a into main Jan 22, 2026
45 of 48 checks passed
@kalleep kalleep deleted the kalleep/tailer-signature branch January 22, 2026 10:58
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants