Feature Request: Add file-based LogsReceiver with rotation and compression support

[korniltsev]

Summary

Currently, the logging block's write_to parameter only supports Loki-based LogsReceiver components (like loki.write, loki.relabel, etc.). There's no built-in way to write Alloy's internal logs directly to files with log rotation and compression capabilities.

Problem

Users who want to:

1. Write Alloy's internal logs to local files (not just stderr)
2. Implement log rotation to manage disk usage
3. Compress rotated logs to save space
4. Avoid the complexity of setting up a full Loki instance just for internal logging

Currently have limited options and must rely on external tools or complex setups.

Proposed Solution

Add a new component (e.g., local.file_write) that implements the LogsReceiver interface and provides:

• File-based log output with configurable path
• Built-in log rotation based on size, age, or both
• Compression of rotated files
• Configurable retention policies

This could leverage existing libraries like lumberjack which provides these features out of the box.

Example Usage

local.file_write "internal_logs" {
  filename     = "/var/log/alloy/alloy.log"
  max_size     = 100  // MB
  max_backups  = 3
  max_age      = 28   // days
  compress     = true
}

logging {
  level    = "info"
  format   = "json"
  write_to = [local.file_write.internal_logs.receiver]
}

Benefits

• Simplified log management for users who prefer file-based logging
• Built-in rotation and compression without external dependencies
• Consistent with Alloy's component-based architecture
• Reduces operational complexity for users not using Loki

Alternative Workarounds

Currently, users must:

• Use system-level log rotation tools with stderr redirection
• Set up a local Loki instance just for internal logs
• Implement custom log forwarding solutions

A built-in file-based LogsReceiver would eliminate these complexities.

[kalleep]

I like the idea but the component should be named loki.file_writer. This would be limited to loki pipelines. Will bring it up during our next proposal review meeting

[korniltsev]

I am not very familiar with otel , but worth double checking if they have anything like this, to not reinvent the wheel

[kalleep]

For otel this would be fileexporter.

It does handle rotation, compression etc and can be used for sveral telemetry types (logs, traces, metrics). The format would either be otel json or proto. So while this is useful it requires a specific receiver to be able to read it back otlpjsonfilereceiver. So it would not act like a regular log file

[Nachtfalkeaw]

I Like the Idea because it is Not Always possible for the alloy Administrator to configured OS Level Log Rotation. Fürther its different in Linux or Windows.

So alloy should be able to handle its own logs seoarately and without interfer with the OS system logs maintained by other administrators.

In addition it would be helpful to configure alloy to only send logs to the configured Location. In Windows everything is logged to Windows Event and If Logging Block enabled it is send to Loki, too.

I would Like to See the possibility

1. Send logs to Standard system logs
2. Send logs via Logging Block only to another component
3. Send No logs at all or to both/multiple destinations.

[korniltsev-grafanista]

Another useful usecase is separate debug logs.
Let's say some component produces a lot of debug logs to troubleshoot an issue (For example #4374)
It would be nice to have an option to keep the stderr logger clean (sometimes just default logger level, sometimes completely empty) and the debug logs are written to the new component (with compression and rotation, possibly on a separate hard drive).

It would be nice to have per line relabeling / filtering as well

[madhub]

For otel this would be fileexporter.

It does handle rotation, compression etc and can be used for sveral telemetry types (logs, traces, metrics). The format would either be otel json or proto. So while this is useful it requires a specific receiver to be able to read it back otlpjsonfilereceiver. So it would not act like a regular log file

Why not support fileexporter and allow it to be used alongside existing OpenTelemetry components?

Enabling this would support a valuable use case already possible with the OpenTelemetry Collector: receiving OTLP data and exporting it to both a file and a remote OTLP destination.

This dual-export pattern is particularly useful in on-premise environments, where having a local file copy of observability data (logs, metrics, traces) can significantly improve debugging and troubleshooting.

If issues occur while forwarding data to a centralized backend, the local file output acts as a reliable fallback, offering an immediate and accessible source of truth to help identify and resolve problems quickly.

graph LR
    %% On-Prem Applications
    A1[App 1]:::app -->|OTLP| B[Local OTel Collector]:::collector
    A2[App 2]:::app -->|OTLP| B
    A3[App 3]:::app -->|OTLP| B

    %% Collector Exports
    B -->|Export| C[File Exporter<br/> Local File]:::debug
    B -->|Export| D[Backend Exporter]:::backend

    %% Cloud Backend
    D -->|Network| D1[Cloud Backend<br/> CloudWatch, Grafana, etc.]:::cloud

    %% Groups
    subgraph On-Premise
        A1
        A2
        A3
        B
        C
        D
    end

    subgraph Cloud
        D1
    end

    %% Styles
    classDef app fill:#b3f7b3,stroke:#333,stroke-width:1px;
    classDef collector fill:#b3d1ff,stroke:#333,stroke-width:1px;
    classDef debug fill:#f9c2ff,stroke:#333,stroke-width:1px;
    classDef backend fill:#fff3b3,stroke:#333,stroke-width:1px;
    classDef cloud fill:#ffcc99,stroke:#333,stroke-width:1px;

Loading

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:

exporters:
  otlp:
    endpoint: localhost:5317
  file/rotation_with_custom_settings:
    path: C:\repos\xplor\observability-tools\otel-collector\logdata.json
    rotation:
      max_megabytes: 10
      max_days: 3
      max_backups: 3
      localtime: true
    format: json
#    compression: zstd
  debug:
    verbosity: detailed

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    logs:
      receivers: [otlp]
      processors: []
      exporters: [otlp,file/rotation_with_custom_settings]

[kalleep]

@madhub I am not saying that we would be opposed to adding fileexporter. I am just pointing out that it would not act like a regular logfile that could then be read back with something like loki.source.file.

If you want support for fileexporter I suggest you make a proposal to add it into alloy. If you want to contribute that we have some guides on how to add otel components here https://github.com/grafana/alloy/blob/main/docs/developer/add-otel-component.md

[madhub]

@madhub I am not saying that we would be opposed to adding fileexporter. I am just pointing out that it would not act like a regular logfile that could then be read back with something like loki.source.file.

If you want support for fileexporter I suggest you make a proposal to add it into alloy. If you want to contribute that we have some guides on how to add otel components here https://github.com/grafana/alloy/blob/main/docs/developer/add-otel-component.md

Added a proposal #4398