Querier: Error while trying to remote-read "snappy: corrupt input"

[nschad]

Describe the bug
Upon upgrading from 1.18.1 to 1.19.0 we were hit with this unexpected log in the querier.
This gets logged everytime a Prometheus client (fairly new v3) tries to remote-read some data.
It was working fine v1.18.1 (nothing else was changed).

Z stderr F ts=2025-08-15T09:46:00.462412068Z caller=remote_read.go:26 level=error org_id=<org> msg="failed to parse proto" err="snappy: corrupt input"

To Reproduce
Steps to reproduce the behavior:

1. Run Cortex
2. Perform remote-Read via Rule eval
3. Error is presented in the UI of the prometheus trying to read OR visible in the querier / query-frontend logs.

Expected behavior
remote-read works

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context

• For the upgrade from 1.18.1 to v1.19 no cortex settings were changed

[yeya24]

Can you help take a look @afhassan

[nschad]

Maybe some additional info

Cortex config (only overrides from default)

alertmanager:
  data_dir: /data
  enable_api: true
  external_url: <url>
  sharding_ring:
    kvstore:
      store: memberlist
alertmanager_storage:
  backend: local
  local:
    path: /data
api:
  response_compression_enabled: true
blocks_storage:
  bucket_store:
    bucket_index:
      enabled: true
    chunks_cache:
      backend: memcached
      memcached:
        addresses: dns+cortex-memcached-blocks.cortex.svc.cluster.local:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 33554432
        timeout: 300ms
    ignore_deletion_mark_delay: 1h0m0s
    index_cache:
      backend: memcached
      memcached:
        addresses: dns+cortex-memcached-blocks-index.cortex.svc.cluster.local:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 16777216
        timeout: 300ms
    index_header_lazy_loading_enabled: true
    metadata_cache:
      backend: memcached
      memcached:
        addresses: dns+cortex-memcached-blocks-metadata.cortex.svc.cluster.local:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 16777216
        timeout: 300ms
    sync_dir: /data/tsdb-sync
  s3:
    [...]
  tsdb:
    close_idle_tsdb_timeout: 14h0m0s
    dir: /data/tsdb
    retention_period: 24h0m0s
compactor:
  compaction_concurrency: 2
  sharding_enabled: true
  sharding_ring:
    kvstore:
      store: memberlist
  sharding_strategy: shuffle-sharding
distributor:
  extend_writes: false
  remote_timeout: 5s
  ring:
    kvstore:
      store: memberlist
  shard_by_all_labels: true
frontend:
  log_queries_longer_than: 1m0s
  query_stats_enabled: true
frontend_worker:
  frontend_address: dns+cortex-query-frontend-headless:9095
ingester:
  lifecycler:
    join_after: 10s
    num_tokens: 8192
    observe_period: 10s
    ring:
      kvstore:
        store: memberlist
    tokens_file_path: /data/tokens
    unregister_on_shutdown: false
ingester_client:
  grpc_compression: "snappy-block"
  grpc_client_config:
    max_recv_msg_size: 10485760
    max_send_msg_size: 10485760
limits:
  compactor_blocks_retention_period: 1y
  compactor_tenant_shard_size: 3
  ingestion_rate: 300000
  max_label_names_per_series: 100
  max_series_per_metric: 0
  reject_old_samples: true
  reject_old_samples_max_age: 4d
memberlist:
  join_members:
  - cortex-memberlist
querier:
  active_query_tracker_dir: /data/active-query-tracker
  query_ingesters_within: 13h0m0s
  query_store_after: 12h0m0s
  timeout: 5m0s
query_range:
  align_queries_with_step: true
  cache_results: true
  results_cache:
    cache:
      memcached:
        expiration: 1h0m0s
      memcached_client:
        max_item_size: 33554432
        timeout: 1s
  split_queries_by_interval: 24h0m0s
ruler:
  enable_api: true
  enable_sharding: true
  ring:
    kvstore:
      store: memberlist
ruler_storage:
  backend: local
  local:
    directory: /data/rules
runtime_config:
  file: /etc/cortex-runtime-config/runtime_config.yaml
server:
  grpc_server_max_concurrent_streams: 10000
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  http_listen_port: 8080
store_gateway:
  sharding_enabled: true
  sharding_ring:
    kvstore:
      store: memberlist
target: compactor

The prometheus trying to read was using v3.5.0

[cles]

Also blocked on a hefty cluster upgrade from v1.18.1 to v1.19.0 due to this issue, with a configuration almost identical to the one provided above.

Logs from the query-frontend:

{"caller":"handler.go:454","component":"query-frontend","error":"snappy: corrupt input\n","fetched_chunks_bytes":0,"fetched_chunks_count":0,"fetched_data_bytes":0,"fetched_samples_count":0,"fetched_series_count":0,"level":"error","method":"POST","msg":"query stats","org_id":"tenant","path":"/prometheus/api/v1/read","query_wall_time_seconds":0.00005836,"response_series_count":0,"response_size":22,"response_time":"1.046859ms","split_queries":1,"status_code":400,"ts":"2025-09-09T11:23:47.204014345Z","user_agent":"Prometheus/3.5.0"}

and from querier:

{"caller":"scheduler_processor.go:160","level":"info","msg":"started running request","org_id":"tenant","ts":"2025-09-09T11:23:47.203423223Z"}
{"caller":"remote_read.go:26","err":"snappy: corrupt input","level":"error","msg":"failed to parse proto","org_id":"tenant","ts":"2025-09-09T11:23:47.203595481Z"}
{"caller":"scheduler_processor.go:194","level":"info","msg":"finished request","org_id":"tenant","response_size":22,"status_code":400,"ts":"2025-09-09T11:23:47.203637197Z"}

(no logs whatsoever from query-scheduler)

The client is a simple prometheus 3.5.0 remote reader UI with the following config:

---
global:
  scrape_interval: 30s
  scrape_timeout: 15s
  evaluation_interval: 1m

remote_read:
  - url: http://query-frontend.cortex.svc.cluster.local/prometheus/api/v1/read
    headers:
      X-Scope-OrgID: "tenant"

exact same config works just fine on v1.18.1. Seeing snappy mentioned in the error message, I tried to play around with the grpc_client_config.grpc_compression configuration, disabling compression for frontend, frontend_worker and even query_scheduler, all to no effect.

Something I found rather curious is that the trace spans for querier (I enabled tracing to see if anything of use showed up there) show event=util.ParseProtoRequest[decompress] size=0, so it seems the buffer that reached the decompression stage was zero-sized? No clue about the possible cause of that, though, that's as far as I got.

[yeya24]

I am able to root cause the bug was introduced in #6390 as it didn't handle remote read request and set empty body buffer for it. I will add a fix to skip logging query request string for remote read request as this requires additional proto body parsing and decompression

[yeya24]

Created fix PR: #7025

[yeya24]

I have published release 1.19.1 https://github.com/cortexproject/cortex/releases/tag/v1.19.1