feat: Add unexposed otel engine and extension to codebase and change build structure

[blewis12]

What this PR Introduces

This PR introduces the Alloy Collector Distro, which has it's components defined in an OCB file and generated using the builder tool . The beefy change here is a re-jig of the build graph of the project, which is why I would like to introduce this as a single PR to ensure that folks are on board with it. The otel subcommand itself is not exposed yet, this will be introduced in a follow up PR which includes documentation changes.

Please bare in mind a fair bit of code here is generated. I will try to provide an overview in the following sections to make it easier to break down

OCB
The components to include in our otel distribution are all defined in our builder-manifest.yaml OCB file. The file itself references the extensions/exporters/processors and so forth that we want to include, it also defines replace directives that we want to include in our final build. This is what will generate the main go.mod file in the collector directory that will be the build root for the project. Note that this is important to keep in sync with the replace directives we have in the alloy module, for which we use the new dependency-replacements.yaml file and generation targets (relevant design doc)

General file structure (relevant design doc)

alloy/
│
├── collector/                     # OTel Collector build configuration
│   ├── builder-config.yaml        # OCB (OpenTelemetry Collector Builder) configuration
│   ├── generate.go                # Entry point for code generation
│   ├── generator/                 # Templating and custom generation code
│   │
│   └── [Generated Files]          # Everything else is generated by OCB + generator, including go.mod
│
├── extension/                     # OTel Collector extension
│   └── alloyengine/               # Alloy engine extension for OTel Collector
│       ├── extension.go           # Main logic for extension
│       ├── factory.go             # Factory to hook extension into collector
│
└── flowcmd/                       # command files moved from root to this directory so they can be imported by collector 

Changed build graph

Building locally / Makefile changes
When building locally via make alloy, the entire chain of replacement and collector generation/build will run - I found this to be the smoothest experience when developing locally so I didn't have to think about manually running commands before building in my changes. The time taken to do the generation is also relatively low. However in CI, the generation currently does not run. My reasoning for this is that we are checking in the built/generated code - what is built in CI should be a reflection of the files we are checking in. If we did generation as part of CI, then the user might check in files would not work when building locally but pass in CI or vice versa

Versioning
The version of the manifest is currently being kept in sync with the alloy version (currently v1.12.1). IMO we should keep the versioning synced, as otel is a subcommand of the alloy CLI and so our distro is embedded within alloy from a user-perspective. I feel it'd make more sense for users to see a single version when they run alloy --version and alloy otel --version

Testing
Very minimal, since command has not been exposed yet - once this is merged, integration tests and lifecycle tests will be added

Change in Binary Size
before:

-rwxr-xr-x@ 1 bejallewis staff 463M Dec 18 12:50 ./build/alloy 

after:

-rwxr-xr-x@ 1 bejallewis staff 464M Dec 18 12:49 ./build/alloy

So this increases the binary size by ~1Mb, due to the collector code being bundled in

relates to #4719

Edit
I've also left out versioning for the alloyextension here - from what I understand, this can be done with release-please - but I feel like this can be done in a follow up PR if we want to actually tag/release this extension for use in external OCB manifests

[Copilot]

Pull request overview

This PR introduces the foundation for an Alloy Collector Distribution built using the OpenTelemetry Collector Builder (OCB). The changes restructure the build graph to support running Alloy with an OpenTelemetry Collector runtime, though the otel subcommand is not yet exposed. The PR includes a new alloyengine extension that embeds the Alloy flow engine within the OTel Collector.

Key changes:

• Refactored command structure to allow exporting and reusing the Alloy CLI components
• Created a new alloyengine extension for the OTel Collector that runs Alloy's flow engine
• Added automated generation of OTel Collector distribution using OCB with a custom builder configuration

Reviewed changes

Copilot reviewed 25 out of 27 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
internal/alloycli/cmd_run.go	Exports RunCommand and modifies interruptContext to accept parent context
internal/alloycli/alloycli.go	Exports Command() function for reuse in collector
flowcmd/flowcmd.go	Refactors from main package to flowcmd library with exported functions
flowcmd/fallback_version.go	Changes from embedded file to reading manifest at runtime
extension/alloyengine/*	New OTel extension that wraps and runs Alloy's flow engine
collector/*	Generated OTel Collector distribution code and configuration
Makefile	Adds targets for collector generation and modifies alloy build path
Dockerfile*	Adds BUILD_DISTRO build arg to control distro generation
dependency-replacements.yaml	Adds extension and collector modules for dependency management

[github-actions]

🔍 Dependency Review

Below are the dependency upgrades and replace updates detected in go.mod changes, along with any code-impact analysis and supporting evidence.

Scope: Only dependency changes in go.mod files are reviewed. New submodules (collector/, extension/alloyengine/) were added; their dependencies are net-new and outside the scope unless they alter existing indirect dependencies in the root module.

---

go.opentelemetry.io/collector/confmap v1.45.0 -> v1.49.0 — ✅ Safe

Summary:

• Minor version bump within the v1 line; no API changes required for existing usage in this repository.
• The only usages in this repo are from the auto-generated OpenTelemetry Collector distribution code, which still compile with v1.49.0 — specifically via resolver/provider factories.

Evidence from code:

• The generated collector main still uses the same API surface (unchanged between 1.45 and 1.49):

  set := otelcol.CollectorSettings{
      // ...
      ConfigProviderSettings: otelcol.ConfigProviderSettings{
          ResolverSettings: confmap.ResolverSettings{
              ProviderFactories: []confmap.ProviderFactory{
                  envprovider.NewFactory(),
                  fileprovider.NewFactory(),
                  httpprovider.NewFactory(),
                  httpsprovider.NewFactory(),
                  yamlprovider.NewFactory(),
              },
          },
      },
  }

Changelog highlights (module releases between 1.46.0 and 1.49.0 across OpenTelemetry Collector):

• No migration notes indicating breaking changes for ResolverSettings or ProviderFactory APIs.
• Improvements tended to be internal/behavioral and additive; existing field names and types used here remain valid.

Code changes required: None.

---

go.opentelemetry.io/collector/featuregate v1.45.0 -> v1.49.0 — ✅ Safe

Summary:

• Although the require was updated to v1.49.0, your go.mod retains a replace pointing this module to a specific Grafana fork/commit:

  replace go.opentelemetry.io/collector/featuregate => github.com/grafana/opentelemetry-collector/featuregate v0.0.0-20240325174506-2fd1623b2ca0

• Because the replace takes precedence, the effective version in use remains the pinned forked commit, not the upstream v1.49.0 tag.

Impact:

• No code changes required; behavior stays aligned with the pinned fork.

Changelog notes:

• The featuregate API in the v1 line has been stable; typical updates add gates rather than change the API.

Code changes required: None.

---

go.uber.org/zap v1.27.0 -> v1.27.1 — ✅ Safe

Summary:

• Patch release with internal fixes; no public API changes signaled.
• Existing usage (zap.Logger creation and field usage via go.uber.org/zap) is unaffected.

Changelog highlights:

• 1.27.1 contains bug fixes and minor improvements; no breaking change entries.

Code changes required: None.

---

github.com/hashicorp/go-version v1.7.0 -> v1.8.0 — ✅ Safe

Summary:

• Minor release within v1 with small improvements to version/constraint parsing; no removals of public API.
• Typical code using go-version (e.g., NewVersion, NewConstraint, Constraint.Check) remains compatible.

Changelog highlights:

• v1.8.0 includes improvements/edge-case handling; no breaking API changes noted.

Code changes required: None.

---

replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter -> github.com/grafana/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.0.0-20260108134526-9817e6c361c9 — ⚠️ Needs Review

Summary:

• This adds a replace mapping to a Grafana fork of the PRW exporter. The comment states it aligns the exporter with prometheus v0.308.0 and includes “type name updates”.

  replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter => \
    github.com/grafana/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.0.0-20260108134526-9817e6c361c9

Impact analysis:

• The replace affects the module resolution for any usage of PRW exporter. In this repository, PRW exporter is used by the new OTel Collector distro (collector/go.mod), not by your root module directly.
• The fork is import-compatible (same import path via replace), so no import path changes are necessary.
• The “type name updates” due to Prometheus 0.308.0 are isolated inside the exporter fork. The root module already depends on prometheus modules that are compatible with 0.308.0, and there’s no direct usage of PRW exporter types in your codebase to update.

Suggested verification:

• Build and run the generated collector with configurations that exercise the PRW exporter (if you maintain such configs).
• If any code in this repo later starts directly importing PRW exporter internals or Prometheus’ renamed types, adjust types per the Prometheus v0.308.0 changes. For now, no code changes are required in this repo.

Relevant notes from the PR comment and replace rationale:

• “Point to fork that uses prometheus v0.308.0, changes here are mostly related to type name updates.”

Code changes required: None presently (configuration-only surface), but keep this under review if you begin importing exporter or Prometheus internals directly.

---

Notes

• New submodules added:

  • collector/ (an OTel Collector distribution): contains a generated go.mod with many direct and indirect dependencies; as this is a net-new module, its dependencies were not reviewed here per the rules.
  • extension/alloyengine/ (new extension module): likewise introduces a separate go.mod. Dependencies are net-new; not reviewed here.

• The repository also added a CI workflow to check that generated OTel collector files in collector/ are up to date. No dependency impact.

• Root module adds a replace for the PRW exporter to a Grafana fork (see above). This only affects resolution of that package and doesn’t require code changes in this repo as configured.

If you’d like a focused review of the new submodules’ dependency sets (collector/ and extension/alloyengine/), please ask and I can provide a separate, scoped review.

[thampiotr]

Looking good!
One thing I also wonder about: does the PR dependency review bot need some adjustments to its prompt to look into the right go.mod files?

[thampiotr]

I think we're almost there!

[Copilot]

Pull request overview

Copilot reviewed 26 out of 29 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

flowcmd/flowcmd.go:39

• Debug statement left in production code. The fmt.Println("hi") statement should be removed before merging as it serves no functional purpose and pollutes stdout.

[thampiotr]

From my POV this is ready % couple of minor comments.
Nice work!

[kgeckhart]

Non-blocking question: I'm guessing this is using the OTel stability definitions for upstreaming purposes. Since it lives in alloy for now should we use the alloy stability levels until it can be upstreamed?

[blewis12]

That's a good point actually, I was trying to make this as OTel-ey as possible but when we talk about this feature externally we should be sticking to Alloy stability levels for the time being. I will change that now

[kgeckhart]

Non-blocking general comment: I found the PR description incredibly helpful for understanding the layout of the changes. I think it could be useful to find a home for some of the content in the README.md and perhaps introducing a collector/README.md

[blewis12]

It could be useful to find a home for some of the content in the README.md and perhaps introducing a collector/README.md

True, there is going to be a follow up PR containing more documentation - but since this PR includes changes that primarily affect developers and therefore should be referenced in the contributing guide, i've added a README here - along with a brief summary and link in the contributing.md file