-
Notifications
You must be signed in to change notification settings - Fork 537
Description
Component(s)
loki.write
What's wrong?
loki.write properly compresses Loki write API payloads with Snappy but does not set the Content-Encoding header;
alloy/internal/component/common/loki/client/batch.go
Lines 139 to 149 in e6d2a12
| // encode the batch as snappy-compressed push request, and returns | |
| // the encoded bytes and the number of encoded entries | |
| func (b *batch) encode() ([]byte, int, error) { | |
| req, entriesCount := b.createPushRequest() | |
| buf, err := proto.Marshal(req) | |
| if err != nil { | |
| return nil, 0, err | |
| } | |
| buf = snappy.Encode(nil, buf) | |
| return buf, entriesCount, nil | |
| } |
alloy/internal/component/common/loki/client/shards.go
Lines 468 to 484 in ff508c0
| func (s *shards) send(ctx context.Context, tenantID string, buf []byte) (int, error) { | |
| ctx, cancel := context.WithTimeout(ctx, s.cfg.Timeout) | |
| defer cancel() | |
| req, err := http.NewRequestWithContext(ctx, "POST", s.cfg.URL.String(), bytes.NewReader(buf)) | |
| if err != nil { | |
| return -1, err | |
| } | |
| const contentType = "application/x-protobuf" | |
| req.Header.Set("Content-Type", contentType) | |
| req.Header.Set("User-Agent", userAgent) | |
| // If the tenant ID is not empty alloy is running in multi-tenant mode, so | |
| // we should send it to Loki | |
| if tenantID != "" { | |
| req.Header.Set("X-Scope-OrgID", tenantID) | |
| } |
Sending compressed data without the proper encoding header is a violation of RFC 9110 Section 8.4 Content-Encoding
Attempting to inject the header via the http_headers field of the endpoint block results in an error:
Jan 23 11:24:02 alma9 alloy[59987]: Error: remote:115:9: Failed to build component: decoding configuration: setting header "Content-Encoding" is not allowed
I've tested injecting the header upstream in a proxy and Loki ingest works fine both with and without the header set so I assume Loki detects the encoding with byte identifiers instead of header values, but any intermediate HTTP services might not be as flexible.
Steps to reproduce
- Send any log data with
loki.write - Observe request headers either via tcpdump if plain http, a mitm proxy, request debug logging on the remote server, etc
System information
No response
Software version
Grafana Alloy v1.12.1
Configuration
Sourced from the `grafana/k8s-monitoring` helm chart,
loki.write "logsservice" {
endpoint {
url = "http://loki.meta-monitor.svc.cluster.local:3100/loki/api/v1/push"
retry_on_http_429 = true
tls_config {
insecure_skip_verify = false
}
min_backoff_period = "500ms"
max_backoff_period = "5m"
max_backoff_retries = "10"
}
external_labels = {
"cluster" = "monitoring",
"k8s_cluster_name" = "monitoring",
}
}
Logs
Not traditional logs, but observing `tcpdump` byte stream at the remote endpoint shows the absence of the required header.
07:53:13.100928 eth0 In IP 172-31-108-191.envoy-eg-d8c59e83.envoy-gateway-system.svc.cluster.local.45518 > redacted-vector-0.redacted-vector-headless.vector.svc.cluster.local.3100: Flags [P.], seq 2124:3245, ack 151, win 491, options [nop,nop,TS val 4054487890 ecr 3668709224], length 1121
E....h@.}..}..l...l~....4.PY........*......
...R...hPOST / HTTP/1.1
host: loki-write.monitor.redacted.com
user-agent: Alloy/v1.12.1 (linux; helm)
content-length: 592
authorization: Basic redacted
content-type: application/x-protobuf
x-agent-id: 93a3fb7c-5b52-4b4e-9636-e361a864f9c4
x-alloy-id: 93a3fb7c-5b52-4b4e-9636-e361a864f9c4
x-scope-orgid: redacted
x-forwarded-for: redacted
x-forwarded-proto: https
x-envoy-external-address: redacted
x-request-id: 3dc11957-06cb-41b9-9dbd-86eace53c65a
...F
..
..{app="konnectivity-agent", cluster="redacted", container=V<.@job="kube-system/Rb..k8s_cl.f._name=Nk....space2Q.$", service.:Ra..#.JD..}...
..............XI0123 07:53:11.796345 ..d1 client.go:557] "remote c-@.on EOF"..HID=472876.*
.pod.#k.,9lP-7b5785b7c4-7lhkk.Y
...0instance_id.B=R...H.J$..[.n[....)y..%y.......6y..2.340637%w. .y..024.........-....j...y......>...2.687861.y..3069.y..y..y..y.:y....
.M..........2l..2.84692Nz.p449] "error dialing backend" ...="..\ tcp 10.3.64.212:19100: ... ...refused".QTID=5932207892241112149.8.io...279./$Address="1>l..".......j.........
07:53:13.101085 eth0 Out IP redacted-vector-0.redacted-vector-headless.vector.svc.cluster.local.3100 > 172-31-108-191.envoy-eg-d8c59e83.envoy-gateway-system.svc.cluster.local.45518: Flags [P.], seq 151:226, ack 3245, win 1702, options [nop,nop,TS val 3668710724 ecr 4054487890], length 75
E....K@.......l~..l.........4.T.....1......
...D...RHTTP/1.1 200 OK
content-length: 0
date: Fri, 23 Jan 2026 07:53:12 GMT
Tip
React with 👍 if this issue is important to you.