fileexporter/rotation does not work with group_by enabled

[bmiguel-teixeira]

Component(s)

exporter/file

What happened?

Description

The fileexporter rotation propery does not seems to work if group by resoruce attribute is enabled. The files are divided properly following the group by setting, but never rotated even after going beyond the max MB limits several times over.

Steps to Reproduce

      file/rotation_with_custom_settings:
        format: json
        group_by:
          enabled: true
          max_open_files: 1000
          resource_attribute: service.name
        path: /data/*
        rotation:
          localtime: true
          max_backups: 100
          max_days: 2

Expected Result

Invididual files are rotated.

Actual Result

Individua files just keep growing.

Collector version

opentelemetry-collector-contrib:0.136.0

Environment information

No response

OpenTelemetry Collector configuration

Log output

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

Sounds like a valid bug. I can take a look at this later, unless someone else wants to jump in.

[beshbeeshy]

This seems to be a decision according to the discussion on this issue. It seems that there were concerns about the interactions between the group by feature and the rotation feature, but it all boiled down to a memory leak in lumberjack.

[bmiguel-teixeira]

hi @beshbeeshy

Got it after reading the description. Some discussions topics.

I have a work around currently in which use to deal with ever growing files (using mv to relocate file and then SIGHUP the process to reload - this forces otel to re-create the original file). This allows me to "safely" dettach the active file so I can backup it up.
Is there an alternative workaround/recommendation for this current limitation?

Given the current memory leak issue around lumberjack, are there plans to address it upstream? Should we document/log otel warning if the user combines these settings?

Cheers.

[beshbeeshy]

Hi @bmiguel-teixeira

Note: I’m not a maintainer, so please wait for someone on the team to confirm, but I hope this helps.

There's a workaround in the same discussion that's referenced earlier. The idea is to add a date to the resource attribute used for grouping, which turns it into time based rotation.

Also while running the same configuration that you provided earlier, I noticed that the behaviour for rotation is that it's always writing to the original file and once it reaches the size limit it renames by adding a date to it and creates a new file with the original name, so I don't think you will need the SIGHUP part here as it's always writing to the current file with the original name.

[bmiguel-teixeira]

hey @beshbeeshy

  transform/logs:
    log_statements:
      - context: log
        statements:
        - set(resource.attributes["timestamp_formatted"], Concat([resource.attributes["service.name"], FormatTime(Now(), "%Y_%m_%d_%H_%M")], "-") )

Doing something like this seems to work, with OTEL rotating the file and closing access to the old one a short time later.

Technically this solves my main issue, though a bit undocumented to get to this.

Shall i submit a PR to update readme?

Thanks for the help

[segevfiner]

This would have been a really useful feature. It's a shame this is not supported...

[AlainGhawi]

I had the same issue locally with the fileexporter

  file/logs:
    path: /var/log/otel/*.json
    rotation:
      max_megabytes: 5        # Rotate when file reaches 5MB
      max_backups: 20         # Keep 20 old files (total ~100MB per workload)
    format: json
    flush_interval: 1s
    group_by:
      enabled: true
      resource_attribute: service.name

In my volume, I see the log files growing indefinitely by testing it with telemetrygen

We will need to wait for a fix in lumberjack if it is due to the memory leak, however, I see that there was no activity during the last 5 months.

I tried using logrotate with Linux sidecar but this can lead to log loss with copytruncate so it is only good for local setup.