This is the design record for the chat-moderation work now up as draft PRs — putting the goal and the path here so we can discuss async instead of in DMs.
Goal
Let players use free-form chat without the failure modes that made preset-only the safe default: harassment, slurs, and threats reaching other players, with no moderation cost that scales per message and no third-party dependency. The bar: a blocked message never reaches another player, a clean message adds ~no latency, and an outage never lets anything through unmoderated.
Where we started
The original decision record settled on preset-only chat after rejecting blocklists (evadable), hosted ML (~$200/mo at our volume), and self-hosted ML (assumed to cost more than the server). Re-examining that: small guard models now run in-process on CPU at real-time latency, so the cost premise changed. Before building anything, we measured instead of assumed — the full 1.34M-message chat history: 99.3% completely clean; genuine harm ~0.03% (violent threats: 5 messages ever). The conclusion cuts both ways: chat is safe enough to open up, and the problem is small enough that deterministic rules do most of the work — the ML model is a backstop, not the foundation.
The path we took
A separate verdict-only moderation service; the relay stays the edge authority.
client → relay POST /chat → POST /moderate → verdict (allow / reject / rewrite)
↳ relay publishes on allow, maps rejects to HTTP
- The relay owns identity, membership, age gate, bans, publishing. It sends
{playerId (opaque UUID), displayName, lobbyCode, message} and gets a verdict. The service never sees Steam/Discord identity.
- The service owns content judgment: deterministic tiers first (curated preset allowlist ~5k entries built from our real chat, hard rules for directed threats, slur blocklist, link stripping, vocab rewrites), then a small fine-tuned guard model (~300ms, in-process, $0/message) only for what falls through. Clean messages short-circuit at 1–9ms and never touch the model.
- Fail-closed (deliberate): service unreachable ⇒ chat 503s, or degrades to allowlist-only (
MODERATION_OUTAGE_POLICY=presets) — presets are safe by construction. Free-form never proceeds unmoderated.
- No automatic retries: the sender gets instant feedback (see UX below), so the player is the retry loop; auto-retries would amplify load exactly when the service is struggling.
Player-facing UX (the part we care most about): your message echoes instantly when you hit enter. If it was blocked you immediately see "not delivered to other players"; if it was auto-cleaned you see "delivered as: …" — no silent shadow-banning, no mystery. Recipients only ever receive the delivered form.
Evidence: the reported-lobby buffer keeps the original text — a rewrite never launders the record a moderator later reviews. The service also keeps its own message+verdict log.
Privacy split: word lists / rewrite rules / eval data live in a private config repo (so slurs never appear in public code); the service repo itself is public-ready.
v1 scope — deliberately small
v1 ships blocking + the delivery notices, nothing else. Player reports, appeals, and a moderator review console are already built but parked on branches — shipping the simple thing first, then iterating on what real usage shows we need.
v2 and beyond (parked, mostly already built)
Roughly in the order real usage will probably demand them:
- Player reports — in-client report flow + relay intake endpoints with reported-message evidence capture. Built; parked on branches (relay
feat/moderation-intake-full, client feat/moderation-ux).
- Review surface — a Discord webhook for flagged/reported messages first (zero UI to build), an optional web console later.
- Sanctions ladder + appeals — the service already issues mutes/rate-limits; v2 adds the escalation policy (warn → timed mutes → permanent) driven by strikes, and an appeal path.
- Wire contract as a package — see open question 4.
- Transport evolution — if the HTTP hop ever matters, the relay can subscribe to MQTT inbox topics and republish post-verdict; the verdict service itself doesn't change.
- Reuse — the same
POST /moderate can front any other chat surface unchanged.
None of this blocks v1 — it's listed so we agree on direction, not to commit to any of it now.
The PRs
Rollout: merge → set env vars → run service-side shadow mode on real traffic (logs would-blocks, blocks nothing) → flip enforcement when the calls look right.
Open questions
- Rewrite vs block — mild profanity gets auto-rewritten ("delivered as…") rather than blocked. Is rewriting someone's words acceptable given the sender sees exactly what was delivered, or should everything just block?
- Retention — the service currently keeps chat logs indefinitely. Proposal: clean messages 30 days (rolling window — enough to tune shadow mode and investigate anything recent), blocked/flagged messages + verdicts 12 months (repeat-offender history, appeal evidence), sanction records permanent (the sanction + its single triggering message only), aggregate stats forever. Indefinite full logs are all liability and no moderation value past the window. Enforced by a daily TTL purge job on the service. Objections?
- Hosting/ownership — the service runs on separate infra right now (HTTPS + bearer, CI-deployed). Happy to hand over, replicate, or keep running it.
- Wire contract — the band vocabulary is currently documented in the gateway, and the relay is built to tolerate drift (unknown bands degrade to a generic block), so v1 is fine as-is. Longer term the shared types should live in one place — either a tiny contract package published from the service repo, or
@bmp/types. This couples with question 3: whoever ends up owning the service should own the contract package.
This is the design record for the chat-moderation work now up as draft PRs — putting the goal and the path here so we can discuss async instead of in DMs.
Goal
Let players use free-form chat without the failure modes that made preset-only the safe default: harassment, slurs, and threats reaching other players, with no moderation cost that scales per message and no third-party dependency. The bar: a blocked message never reaches another player, a clean message adds ~no latency, and an outage never lets anything through unmoderated.
Where we started
The original decision record settled on preset-only chat after rejecting blocklists (evadable), hosted ML (~$200/mo at our volume), and self-hosted ML (assumed to cost more than the server). Re-examining that: small guard models now run in-process on CPU at real-time latency, so the cost premise changed. Before building anything, we measured instead of assumed — the full 1.34M-message chat history: 99.3% completely clean; genuine harm ~0.03% (violent threats: 5 messages ever). The conclusion cuts both ways: chat is safe enough to open up, and the problem is small enough that deterministic rules do most of the work — the ML model is a backstop, not the foundation.
The path we took
A separate verdict-only moderation service; the relay stays the edge authority.
{playerId (opaque UUID), displayName, lobbyCode, message}and gets a verdict. The service never sees Steam/Discord identity.MODERATION_OUTAGE_POLICY=presets) — presets are safe by construction. Free-form never proceeds unmoderated.Player-facing UX (the part we care most about): your message echoes instantly when you hit enter. If it was blocked you immediately see "not delivered to other players"; if it was auto-cleaned you see "delivered as: …" — no silent shadow-banning, no mystery. Recipients only ever receive the delivered form.
Evidence: the reported-lobby buffer keeps the original text — a rewrite never launders the record a moderator later reviews. The service also keeps its own message+verdict log.
Privacy split: word lists / rewrite rules / eval data live in a private config repo (so slurs never appear in public code); the service repo itself is public-ready.
v1 scope — deliberately small
v1 ships blocking + the delivery notices, nothing else. Player reports, appeals, and a moderator review console are already built but parked on branches — shipping the simple thing first, then iterating on what real usage shows we need.
v2 and beyond (parked, mostly already built)
Roughly in the order real usage will probably demand them:
feat/moderation-intake-full, clientfeat/moderation-ux).POST /moderatecan front any other chat surface unchanged.None of this blocks v1 — it's listed so we agree on direction, not to commit to any of it now.
The PRs
MODERATION_SERVICE_URLis set — merging changes nothing by itself)Rollout: merge → set env vars → run service-side shadow mode on real traffic (logs would-blocks, blocks nothing) → flip enforcement when the calls look right.
Open questions
@bmp/types. This couples with question 3: whoever ends up owning the service should own the contract package.