kind:9033 auth failure logs nothing server-side when get_relay_member returns None

[marcellodesales]

🐛 Problem

When a kind:9033 (RELAY_ADMIN_SET_WORKSPACE_PROFILE) event is rejected with "actor not authorized: must be admin or owner", the relay emits no server-side log identifying which pubkey was looked up or why get_relay_member returned None. The only signal is the client-facing ["OK", <id>, false, "invalid: actor not authorized: must be admin or owner"] NIP-01 response over the WebSocket.

This makes the failure completely invisible from the operator side. The kind:9033 path goes over WebSocket (not the HTTP bridge), so it never appears in the bridge INFO logs at all.

🔍 What Happened

During a live deployment of buzz 0.2.0, a relay admin attempt to upload the workspace/community icon consistently failed with "actor not authorized: must be admin or owner" despite the relay_members table containing rows for both the operator and user pubkeys with role owner.

Root cause: a manual DB repair job had inserted pubkeys as 66-char \x-prefixed strings (e.g. \x6abac6326580...) into the relay_members.pubkey TEXT column. The relay's get_relay_member queries with 64-char plain hex (as documented in relay_members.rs: "pubkey values are 64-char lowercase hex strings"), so the lookup returned None → sender_role = "" → auth denied.

# What was in the DB (wrong — 66 chars with \x prefix):
\x6abac6326580def897eb8d73903ae742f071603870020e65ccaf71edf697213e  | owner
\x4a2efbc16eb5c5257d869412c0a60c15ec80f2a1df220917ed903759d6057514  | owner

# What the relay queries for (correct — 64 chars, no prefix):
6abac6326580def897eb8d73903ae742f071603870020e65ccaf71edf697213e    | (not found → None)

has_admin_or_owner (no pubkey predicate) still returned true because the \x-prefixed rows exist → community_has_steward = true → the may_set_workspace_profile gate required admin/owner → sender_role = "" → denied.

The mismatch would have been immediately obvious with one debug! line.

🕵️ Relevant Code Paths

• crates/buzz-relay/src/handlers/relay_admin.rs — execute_relay_admin_command, the kind:9033 branch
• crates/buzz-db/src/relay_members.rs — get_relay_member (SQL: WHERE community_id = $1 AND pubkey = $2)
• may_set_workspace_profile returns false when sender_role == "" and community_has_steward == true

✅ Suggested Fix

Add a debug! (or warn! since it results in a denial) log in execute_relay_admin_command after the get_relay_member call, before the may_set_workspace_profile gate:

let sender_member = state
    .db
    .get_relay_member(tenant.community(), &sender_hex)
    .await
    .map_err(|e| format!("database error: {e}"))?;

let sender_role = sender_member
    .as_ref()
    .map(|m| m.role.as_str())
    .unwrap_or("");

// Suggested addition:
if sender_member.is_none() {
    tracing::debug!(
        pubkey = %sender_hex,
        community = %tenant.community(),
        kind = event.kind.as_u16(),
        "relay_admin: no relay_member row found for sender — will be treated as no role"
    );
}

This would have immediately surfaced the lookup key and confirmed the format mismatch without any DB spelunking.

🌍 Environment

• buzz-relay 0.2.0 (image tag main, rolling)
• PostgreSQL 16 (Azure DB for PostgreSQL Flexible Server)
• relay_members.pubkey column type: TEXT
• require_relay_membership: false (open relay with steward detection via has_admin_or_owner)

[marcellodesales]

🔧 Workaround — Kubernetes Job to repair relay_members pubkey format

Since we operate on AKS (Azure Kubernetes Service) with DATABASE_URL stored in a Kubernetes Secret (ESO-materialized from Vault), we can't psql directly. All DB repairs are run as one-shot Kubernetes Job objects that mount the secret and use the postgres:16 (Debian) image.

🕵️ How we diagnosed it

The relay was rejecting kind:9033 with "actor not authorized: must be admin or owner" despite relay_members containing rows for our pubkeys. We ran a diagnostic Job:

# Excerpt — query that revealed the format mismatch
psql "$DATABASE_URL" -c "
  SELECT pubkey, length(pubkey) AS len, role
  FROM relay_members
  WHERE community_id = '<our-community-uuid>';"

Output:

pubkey                                                              | len | role
\x6abac6326580def897eb8d73903ae742f071603870020e65ccaf71edf697213e |  66 | owner
\x4a2efbc16eb5c5257d869412c0a60c15ec80f2a1df220917ed903759d6057514 |  66 | owner

The \x prefix and len=66 immediately flagged the mismatch — after reading crates/buzz-db/src/relay_members.rs which clearly documents "pubkey values are 64-char lowercase hex strings" and confirmed get_relay_member passes the pubkey string directly to SQL with no transformation.

✅ Fix Job

buzz-db-fix-relay-members-pubkey-format.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: buzz-db-fix-relay-members-pubkey-format
  namespace: buzz
spec:
  ttlSecondsAfterFinished: 3600
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: fix
          image: postgres:16
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: buzz-secrets
                  key: DATABASE_URL
          command:
            - bash
            - -ec
            - |
              COMMUNITY=$(psql -t -A "$DATABASE_URL" -c "SELECT id FROM communities LIMIT 1;")

              echo "[pre] Current relay_members:"
              psql "$DATABASE_URL" -c "
                SELECT pubkey, length(pubkey) AS len, role
                FROM relay_members WHERE community_id = '${COMMUNITY}'
                ORDER BY updated_at;"

              # Re-insert any \x-prefixed rows as correct 64-char hex.
              # ON CONFLICT DO NOTHING: if 64-char row already exists, skip.
              psql "$DATABASE_URL" -c "
                INSERT INTO relay_members (community_id, pubkey, role, added_by)
                SELECT community_id,
                       substring(pubkey FROM 3),   -- strip leading \x
                       role,
                       added_by
                FROM relay_members
                WHERE community_id = '${COMMUNITY}'
                  AND pubkey LIKE '\\x%'
                  AND length(pubkey) = 66
                ON CONFLICT (community_id, pubkey) DO NOTHING;"

              echo ""
              echo "[post] relay_members after fix:"
              psql "$DATABASE_URL" -c "
                SELECT pubkey, length(pubkey) AS len, role
                FROM relay_members WHERE community_id = '${COMMUNITY}'
                ORDER BY length(pubkey), pubkey;"
              echo "Done."

The fix strips the \x prefix from any 66-char malformed rows and inserts the correct 64-char versions. The relay's get_relay_member finds the new rows immediately — no restart required. The old \x-prefixed rows become harmless junk; bootstrap_owner at the next relay restart will demote them to admin (still sufficient for may_set_workspace_profile).

💡 Why the debug! log would have saved hours

Without server-side logging of the lookup key vs. what was in the DB, the only observable symptom was the client-side ["OK", <id>, false, "invalid: actor not authorized: must be admin or owner"] NIP-01 message. The kind:9033 event travels over WebSocket and is never logged at the HTTP bridge INFO level, so it was completely invisible in production logs. The entire investigation required reading the relay source, writing diagnostic DB queries as Kubernetes Jobs, and cross-referencing pod logs across three relay replicas.

🤝 Offering a PR

Happy to open a PR adding the debug! / warn! log to relay_admin.rs if that would help — the change is small and we've already identified the exact location.