Skip to content

loki.write component should set a "Content-Encoding: snappy" header #5339

@a-abella

Description

@a-abella

Component(s)

loki.write

What's wrong?

loki.write properly compresses Loki write API payloads with Snappy but does not set the Content-Encoding header;

// encode the batch as snappy-compressed push request, and returns
// the encoded bytes and the number of encoded entries
func (b *batch) encode() ([]byte, int, error) {
req, entriesCount := b.createPushRequest()
buf, err := proto.Marshal(req)
if err != nil {
return nil, 0, err
}
buf = snappy.Encode(nil, buf)
return buf, entriesCount, nil
}

func (s *shards) send(ctx context.Context, tenantID string, buf []byte) (int, error) {
ctx, cancel := context.WithTimeout(ctx, s.cfg.Timeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, "POST", s.cfg.URL.String(), bytes.NewReader(buf))
if err != nil {
return -1, err
}
const contentType = "application/x-protobuf"
req.Header.Set("Content-Type", contentType)
req.Header.Set("User-Agent", userAgent)
// If the tenant ID is not empty alloy is running in multi-tenant mode, so
// we should send it to Loki
if tenantID != "" {
req.Header.Set("X-Scope-OrgID", tenantID)
}

Sending compressed data without the proper encoding header is a violation of RFC 9110 Section 8.4 Content-Encoding

Attempting to inject the header via the http_headers field of the endpoint block results in an error:

Jan 23 11:24:02 alma9 alloy[59987]: Error: remote:115:9: Failed to build component: decoding configuration: setting header "Content-Encoding" is not allowed

I've tested injecting the header upstream in a proxy and Loki ingest works fine both with and without the header set so I assume Loki detects the encoding with byte identifiers instead of header values, but any intermediate HTTP services might not be as flexible.

Steps to reproduce

  • Send any log data with loki.write
  • Observe request headers either via tcpdump if plain http, a mitm proxy, request debug logging on the remote server, etc

System information

No response

Software version

Grafana Alloy v1.12.1

Configuration

Sourced from the `grafana/k8s-monitoring` helm chart,


loki.write "logsservice" {
  endpoint {
    url = "http://loki.meta-monitor.svc.cluster.local:3100/loki/api/v1/push"
    retry_on_http_429 = true
    tls_config {
      insecure_skip_verify = false
    }
    min_backoff_period = "500ms"
    max_backoff_period = "5m"
    max_backoff_retries = "10"
  }
  external_labels = {
    "cluster" = "monitoring",
    "k8s_cluster_name" = "monitoring",
  }
}

Logs

Not traditional logs, but observing `tcpdump` byte stream at the remote endpoint shows the absence of the required header.



07:53:13.100928 eth0  In  IP 172-31-108-191.envoy-eg-d8c59e83.envoy-gateway-system.svc.cluster.local.45518 > redacted-vector-0.redacted-vector-headless.vector.svc.cluster.local.3100: Flags [P.], seq 2124:3245, ack 151, win 491, options [nop,nop,TS val 4054487890 ecr 3668709224], length 1121
E....h@.}..}..l...l~....4.PY........*......
...R...hPOST / HTTP/1.1
host: loki-write.monitor.redacted.com
user-agent: Alloy/v1.12.1 (linux; helm)
content-length: 592
authorization: Basic redacted
content-type: application/x-protobuf
x-agent-id: 93a3fb7c-5b52-4b4e-9636-e361a864f9c4
x-alloy-id: 93a3fb7c-5b52-4b4e-9636-e361a864f9c4
x-scope-orgid: redacted
x-forwarded-for: redacted
x-forwarded-proto: https
x-envoy-external-address: redacted
x-request-id: 3dc11957-06cb-41b9-9dbd-86eace53c65a

...F
..
..{app="konnectivity-agent", cluster="redacted", container=V<.@job="kube-system/Rb..k8s_cl.f._name=Nk....space2Q.$", service.:Ra..#.JD..}...
..............XI0123 07:53:11.796345  ..d1 client.go:557] "remote c-@.on EOF"..HID=472876.*
.pod.#k.,9lP-7b5785b7c4-7lhkk.Y
...0instance_id.B=R...H.J$..[.n[....)y..%y.......6y..2.340637%w. .y..024.........-....j...y......>...2.687861.y..3069.y..y..y..y.:y....
.M..........2l..2.84692Nz.p449] "error dialing backend" ...="..\ tcp 10.3.64.212:19100: ...	...refused".QTID=5932207892241112149.8.io...279./$Address="1>l..".......j.........
07:53:13.101085 eth0  Out IP redacted-vector-0.redacted-vector-headless.vector.svc.cluster.local.3100 > 172-31-108-191.envoy-eg-d8c59e83.envoy-gateway-system.svc.cluster.local.45518: Flags [P.], seq 151:226, ack 3245, win 1702, options [nop,nop,TS val 3668710724 ecr 4054487890], length 75
E....K@.......l~..l.........4.T.....1......
...D...RHTTP/1.1 200 OK
content-length: 0
date: Fri, 23 Jan 2026 07:53:12 GMT

Tip

React with 👍 if this issue is important to you.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions