Support routing actions (copy/move) in routing connector

[vengalraoguttha]

Component(s)

cmd/otelcontribcol

Is your feature request related to a problem? Please describe.

We have a use case where we want have incoming logs data stream and we want to route a subset of data to one pipeline and send entire data to another pipeline with as little copies as possible. We explored the suggestions mentioned in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/routingconnector/README.md#match_once but that seem to copy more data because we end up creating 2 different pipelines one which only has default pipeline and other has subset data router because routing connector is mutable.

Explored option:

routing/subset_data:
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
routing/default:
  default_pipelines: [ logs/default ]
service:
  pipelines:
    logs/in:
exporters: [routing/default, routing/subset]
    logs/p1:
receivers: [routing/default]
    logs/p2:
receivers: [routing/subset]

does 1 full copy of entire data.

Describe the solution you'd like

Proposed solution: to avoid copies we want to introduce a new config in routing connector with condition that also describe the type of action we want to take when condition matches.

routing/single_connector:
  default_pipelines: [ logs/default ]
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
       action: copy
service:
  pipelines:
    logs/in:
exporters: [routing/single_connector]
    logs/p1:
receivers: [routing/single_connector]
    logs/p2:
receivers: [routing/single_connector]

the config we are proposing to add is action: copy/move and this is optional config which if not mentioned will use move as default value.

Only copies subset of data as needed

Describe alternatives you've considered

Explored option:

routing/subset_data:
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
routing/default:
  default_pipelines: [ logs/default ]
service:
  pipelines:
    logs/in:
exporters: [routing/default, routing/subset]
    logs/p1:
receivers: [routing/default]
    logs/p2:
receivers: [routing/subset]

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners for connector/routing: @mwear @TylerHelmuth @evan-bradley @edmocosta @bogdandrutu. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triage' to set the priority and remove the needs-triage label.

[khushijain21]

we want to route a subset of data to one pipeline and send entire data to another pipeline with as little copies as possible

You mentioned this, wouldn't the following config solve your problem? The subset of your data will go via logs/prod and the rest will go via default pipeline. Can you explain how does action:move/copy solve your use case?

routing/single_connector:
  default_pipelines: [ logs/p1 ]
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
service:
  pipelines:
    logs/in:
exporters: [routing/single_connector]
    logs/p1:
receivers: [routing/single_connector]
    logs/prod:
receivers: [routing/single_connector]

[vengalraoguttha]

The subset of your data will go via logs/prod and the rest will go via default pipeline

like you said The subset of your data will go via logs/prod and the rest will go via default pipeline in this part. the rest of the data goes to default pipeline which is move behavior but we need copy behavior where all the data goes to default pipeline.