Skip to content

Commit 23c0acf

Browse files
blewis12thampiotrkgeckhart
authored
Introduce Shared YAML Dependency File To Manage Replace Directives (#4862)
* Introduce new dependency management tool * Move syntax replace to top of go.mod to avoid confusion * Apply replaces from tool * Introduce PR workflow to check modules are synced * Add E2E test * Address some PR comments Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Refactor to use a centralised cobra CLI, called by generate.go * Refactor to not generate .txt files and pass replace text in memory * Refactor E2E tests to use explicit test data * Update tools/generate-module-dependencies/replaces-mod.tpl Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com> * Return errors in FileHelper instead of using Fatalf * Address PR comments * go mod/mod changes * Reference actual calculated file to dependency yaml file in error logs * Group the generate-module-dependencies and Check for go.mod changes properly in workflow and use git diff instead so we can ignore whitespace changes * nit: a few improvements for clarity --------- Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
1 parent 8c0090c commit 23c0acf

26 files changed

+898
-64
lines changed

‎.github/workflows/check-sync-module-dependencies.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Check generate-module-dependencies
2+
permissions:
3+
contents: read
4+
on: pull_request
5+
6+
jobs:
7+
check:
8+
runs-on: ubuntu-latest
9+
10+
steps:
11+
- name: Checkout
12+
uses: actions/checkout@v4
13+
with:
14+
fetch-depth: 0
15+
16+
- name: Set up Go
17+
uses: actions/setup-go@v5
18+
with:
19+
go-version: '1.25.x'
20+
21+
- name: Run tests
22+
run: |
23+
cd tools/generate-module-dependencies
24+
go test -v .
25+
26+
- name: Run generate-module-dependencies
27+
run: |
28+
make generate-module-dependencies
29+
30+
- name: Check for go.mod changes
31+
run: |
32+
# List changed go.mod files (added/modified/deleted)
33+
CHANGED=$(git diff --name-only --ignore-all-space | grep -E '(^|/)go\.mod$' || true)
34+
35+
if [ -n "$CHANGED" ]; then
36+
echo "::error::go.mod files are out of sync with generate-module-dependencies."
37+
echo "The following go.mod files changed:"
38+
echo "$CHANGED"
39+
echo
40+
echo "Diff:"
41+
git --no-pager diff -- $CHANGED || true
42+
echo
43+
echo "To fix locally:"
44+
echo " Ensure that you update the dependency-replacements.yaml file and run make generate-module-dependencies"
45+
exit 1
46+
fi

‎Makefile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
## generate-versioned-files Generate versioned files.
5050
## generate-winmanifest Generate the Windows application manifest.
5151
## generate-snmp Generate SNMP modules from prometheus/snmp_exporter for prometheus.exporter.snmp and bumps SNMP version in _index.md.t.
52+
## generate-module-dependencies Generate replace directives from dependency-replacements.yaml and inject them into go.mod and builder-config.yaml.
5253
##
5354
## Other targets:
5455
##
@@ -257,6 +258,13 @@ else
257258
bash ./operations/helm/scripts/rebuild-tests.sh
258259
endif
259260

261+
generate-module-dependencies:
262+
ifeq ($(USE_CONTAINER),1)
263+
$(RERUN_IN_CONTAINER)
264+
else
265+
cd ./tools/generate-module-dependencies && $(GO_ENV) go generate
266+
endif
267+
260268
generate-ui:
261269
ifeq ($(USE_CONTAINER),1)
262270
$(RERUN_IN_CONTAINER)

‎dependency-replacements.yaml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# NOTE: replace directives below must always be *temporary*.
2+
#
3+
# Adding a replace directive to change a module to a fork of a module will
4+
# only be accepted when a PR or an issue upstream has been opened to accept the new
5+
# change.
6+
#
7+
# Contributors are expected to work with upstream to make their changes
8+
# acceptable, and remove the `replace` directive as soon as possible.
9+
#
10+
# If upstream is unresponsive, you should consider making a hard fork
11+
# (i.e., creating a new Go module with the same source) or picking a different
12+
# dependency.
13+
14+
# Modules define the target files where replaces will be applied.
15+
# Each module specifies:
16+
# - name: the name of the module
17+
# - path: the path to the module's go.mod file
18+
# - file_type: enum of what sort of generation pipeline will be used, currently the only available value is "mod"
19+
20+
# Replace directives define the replacements that will be applied to the defined modules
21+
# Each replace directive specifies:
22+
# - comment: a brief explanation as to why the replace is necessary. This must contain a link to an upstream issue or a PR that is required to complete in order to remove the replace directive
23+
# - dependency: the dependency to be replaced
24+
# - replacement: the replacement for the dependency
25+
26+
modules:
27+
- name: alloy
28+
path: go.mod
29+
file_type: mod
30+
31+
replaces:
32+
- comment: >
33+
TODO: remove this replace directive once the upstream issue is fixed:
34+
https://github.com/prometheus/prometheus/issues/13842
35+
dependency: go.opentelemetry.io/collector/featuregate
36+
replacement: github.com/grafana/opentelemetry-collector/featuregate v0.0.0-20240325174506-2fd1623b2ca0
37+
38+
- comment: Replace directives from Prometheus
39+
dependency: github.com/fsnotify/fsnotify v1.8.0
40+
replacement: github.com/fsnotify/fsnotify v1.7.0
41+
42+
- comment: >
43+
TODO: remove replace directive once there is a release of Prometheus
44+
which addresses https://github.com/prometheus/prometheus/issues/14049,
45+
for example, via this implementation: https://github.com/grafana/prometheus/pull/34
46+
dependency: github.com/prometheus/prometheus
47+
replacement: github.com/grafana/prometheus v1.8.2-0.20251030104821-c9e0b31e9aeb
48+
49+
- comment: Replace yaml.v2 with fork
50+
dependency: gopkg.in/yaml.v2
51+
replacement: github.com/rfratto/go-yaml v0.0.0-20211119180816-77389c3526dc
52+
53+
- comment: Replace directives from Loki — Azure SDK
54+
dependency: github.com/Azure/azure-sdk-for-go
55+
replacement: github.com/Azure/azure-sdk-for-go v68.0.0+incompatible
56+
57+
- comment: Replace directives from Loki — Azure storage blob fork
58+
dependency: github.com/Azure/azure-storage-blob-go
59+
replacement: github.com/MasslessParticle/azure-storage-blob-go v0.14.1-0.20240322194317-344980fda573
60+
61+
- comment: Use fork of gocql that has gokit logs and Prometheus metrics
62+
dependency: github.com/gocql/gocql
63+
replacement: github.com/grafana/gocql v0.0.0-20200605141915-ba5dc39ece85
64+
65+
- comment: Insist on the optimised version of grafana/regexp
66+
dependency: github.com/grafana/regexp
67+
replacement: github.com/grafana/regexp v0.0.0-20240518133315-a468a5bfb3bc
68+
69+
- comment: Replace memberlist with Grafana fork which includes unmerged upstream fixes
70+
dependency: github.com/hashicorp/memberlist
71+
replacement: github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe
72+
73+
- comment: Use forked syslog implementation by leodido for continued support
74+
dependency: github.com/influxdata/go-syslog/v3
75+
replacement: github.com/leodido/go-syslog/v4 v4.2.0
76+
77+
- comment: Replace thanos-io/objstore with Grafana fork
78+
dependency: github.com/thanos-io/objstore
79+
replacement: github.com/grafana/objstore v0.0.0-20250210100727-533688b5600d
80+
81+
- comment: TODO - remove forks when changes are merged upstream — non-singleton cadvisor
82+
dependency: github.com/google/cadvisor
83+
replacement: github.com/grafana/cadvisor v0.0.0-20240729082359-1f04a91701e2
84+
85+
- comment: TODO - track exporter-package-v0.18.1 branch of grafana fork; remove once merged upstream
86+
dependency: github.com/prometheus-community/postgres_exporter
87+
replacement: github.com/grafana/postgres_exporter v0.0.0-20250930111128-c8f6a9f4d363
88+
89+
- comment: TODO - remove once PR is merged upstream - https://github.com/prometheus/mysqld_exporter/pull/774
90+
dependency: github.com/prometheus/mysqld_exporter
91+
replacement: github.com/grafana/mysqld_exporter v0.17.2-0.20250226152553-be612e3fdedd
92+
93+
- comment: >
94+
TODO: replace node_exporter with custom fork for multi usage.
95+
https://github.com/prometheus/node_exporter/pull/2812
96+
dependency: github.com/prometheus/node_exporter
97+
replacement: github.com/grafana/node_exporter v0.18.1-grafana-r01.0.20251024135609-318b01780c89
98+
99+
- comment: Use Grafana fork of smimesign
100+
dependency: github.com/github/smimesign
101+
replacement: github.com/grafana/smimesign v0.2.1-0.20220408144937-2a5adf3481d3
102+
103+
- comment: Replace OpenTelemetry OBI with Grafana fork
104+
dependency: go.opentelemetry.io/obi
105+
replacement: github.com/grafana/opentelemetry-ebpf-instrumentation v1.3.13
106+
107+
- comment: Replace OpenTelemetry eBPF profiler with Grafana fork
108+
dependency: go.opentelemetry.io/ebpf-profiler
109+
replacement: github.com/grafana/opentelemetry-ebpf-profiler v0.0.202546-0.20251106085643-a00a0ef2a84c
110+
111+
- comment: Update openshift/client-go to version compatible with structured-merge-diff v6
112+
dependency: github.com/openshift/client-go
113+
replacement: github.com/openshift/client-go v0.0.0-20251015124057-db0dee36e235
114+
115+
- comment: Use Grafana's patched k8sattributesprocessor with support for k8s.io/client-go v0.34.1
116+
dependency: github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor
117+
replacement: github.com/grafana/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.0.0-20251021125353-73458b01ab23
118+
119+
- comment: >
120+
Do not remove until bug in walqueue backwards compatibility is resolved:
121+
https://github.com/deneonet/benc/issues/13
122+
dependency: github.com/deneonet/benc
123+
replacement: github.com/deneonet/benc v1.1.7
124+
125+
- comment: Pin runc to v1.2.8 for compatibility with cadvisor requiring libcontainer/cgroups packages
126+
dependency: github.com/opencontainers/runc
127+
replacement: github.com/opencontainers/runc v1.2.8
128+
129+
- comment: Replace controller-runtime with pinned version
130+
dependency: sigs.k8s.io/controller-runtime
131+
replacement: sigs.k8s.io/controller-runtime v0.20.4

‎dirwatch-test/.remote_config_hash‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
��PR���'Ȣ�ƿ�J龏�؊�ď�t)8

‎dirwatch-test/config.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
service:
2+
telemetry:
3+
logs:
4+
processors:
5+
- batch:
6+
exporter:
7+
otlp:
8+
endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/logs
9+
headers:
10+
- name: Authorization
11+
value: Basic ${env:GCLOUD_BASIC_AUTH}
12+
protocol: http/protobuf
13+
metrics:
14+
readers:
15+
- periodic:
16+
exporter:
17+
otlp:
18+
endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/metrics
19+
headers:
20+
- name: Authorization
21+
value: Basic ${env:GCLOUD_BASIC_AUTH}
22+
protocol: http/protobuf
23+
traces:
24+
processors:
25+
- batch:
26+
exporter:
27+
otlp:
28+
endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/traces
29+
headers:
30+
- name: Authorization
31+
value: Basic ${env:GCLOUD_BASIC_AUTH}
32+
protocol: http/protobuf

0 commit comments

Comments
 (0)