What happens
Production relay pods (bb-public cell) intermittently log:
Internal error: error: database error: database error: error returned from database: invalid byte sequence for encoding "UTF8": 0x00 at line 1870
27 occurrences over the last 14 days (first sighting 2026-07-22, ~5/day since, most recent 2026-07-27 12:14Z), across many different pods/hosts, often in pairs seconds apart — consistent with a client retrying after a 500. Datadog: https://app.datadoghq.com/logs?query=kube_cluster_name%3Aproduction-bke-bb-public-infra-cell-01-us-west-2%20kube_namespace%3Abuzz%20%22invalid%20byte%20sequence%22
Likely root cause
JSON permits \u0000 escapes in strings, but Postgres text/jsonb columns cannot store NUL bytes. An event whose content (or a tag value) contains an embedded NUL passes JSON parsing and signature checks, then blows up at insert time as a Postgres encoding error, which surfaces as Internal error (500 / generic failure) to the client instead of a clean invalid: rejection.
Expected
Events containing NUL bytes in content or tag values should be rejected at validation time (invalid: content contains disallowed control characters or similar), before any DB write — not surfaced as a database internal error.
Notes
- Blast radius today is one client request per hit (no panic/restart, the transaction just fails), but each hit is a 500 to a real user and an error-stream pollution source during launch monitoring.
- Worth checking all ingest text paths (event content, tags, profile fields, workflow definitions) for the same gap, since the error doesn't identify which statement fired.
What happens
Production relay pods (bb-public cell) intermittently log:
27 occurrences over the last 14 days (first sighting 2026-07-22, ~5/day since, most recent 2026-07-27 12:14Z), across many different pods/hosts, often in pairs seconds apart — consistent with a client retrying after a 500. Datadog: https://app.datadoghq.com/logs?query=kube_cluster_name%3Aproduction-bke-bb-public-infra-cell-01-us-west-2%20kube_namespace%3Abuzz%20%22invalid%20byte%20sequence%22
Likely root cause
JSON permits
\u0000escapes in strings, but Postgrestext/jsonbcolumns cannot store NUL bytes. An event whose content (or a tag value) contains an embedded NUL passes JSON parsing and signature checks, then blows up at insert time as a Postgres encoding error, which surfaces asInternal error(500 / generic failure) to the client instead of a cleaninvalid:rejection.Expected
Events containing NUL bytes in content or tag values should be rejected at validation time (
invalid: content contains disallowed control charactersor similar), before any DB write — not surfaced as a database internal error.Notes