feat: Guardrail prerequisites — outbound deny recording + on_error policy

[huang195]

Summary

Two framework prerequisites for shipping guardrail plugins (IBAC, PII
scrubbers, jailbreak classifiers, etc.). Independent changes, bundled
because they land together in the same PR cycle:

1. feat(listeners): Record pipeline denials as SessionDenied events — close a symmetry gap where outbound rejects (extproc, forwardproxy) and inbound rejects (reverseproxy) didn't emit session events, so blocked calls never appeared in /v1/sessions or abctl.
2. feat(pipeline): Add on_error policy (enforce | observe | off) — per-plugin knob letting operators canary a new guardrail in shadow mode before flipping to enforce. Plugin code is unchanged under all three modes; the rollout knob lives outside the plugin.

Together these close the last two framework gaps identified in the pre-IBAC research pass. A third concern — cross-session intent contamination under concurrent multi-tenant traffic — is not in scope, because the current deployment model is single-tenant per pod (documented at store.go:309). If we ever move off that model, correlation via a session header becomes the follow-up.

Commit 1 — Outbound deny recording

extproc listener already had recordInboundReject on inbound denials (emitted a SessionDenied event with Invocations / host / identity). The symmetric gaps:

• extproc outbound rejects → no session event
• forwardproxy outbound rejects → no session event
• reverseproxy inbound rejects → no session event

Fixed with a local helper in each listener, called from the reject path before rejectFromAction / writeRejection returns:

Phase:       SessionDenied
Invocations: phase-filtered from pctx.Extensions.Invocations
Host:        pctx.Host
StatusCode + Error: from action.Violation

Bucketing mirrors the accept-path recording in each listener:

• extproc outbound + forwardproxy → ActiveSession() → default
• reverseproxy inbound → A2A.SessionID → ActiveSession() → default

Skip rule: denials with no Invocations are not recorded — a content-free SessionDenied event would be noise without attribution, and non-plugin-originated denials (body-too-large, etc.) belong in /stats counters.

Response-phase rejects (RunResponse denying after the upstream returned) are out of scope for this PR — no in-tree plugin rejects in that path today, and the wire-phase semantics are subtly different (the upstream already ran).

Commit 2 — on_error policy

pipeline:
  outbound:
    plugins:
      - name: token-exchange
      - name: pii-scrubber
        on_error: observe             # log would-blocks, don't block
        config: { ... }

Three values:

Policy	Plugin dispatched?	Reject →	Body mutation →	Typical use
enforce (default)	yes	HTTP error, pipeline stops	applied to wire	Production guardrails
observe	yes	shadow Invocation, request passes	suppressed (no-op)	Canarying a new plugin
off	no	n/a	n/a	Kill-switch without redeploy

Observe mechanics

The plugin's OnRequest / OnResponse runs exactly as under enforce. If it returns pipeline.Deny(...), the framework intercepts: it marks the plugin's last Invocation with Shadow: true (or synthesizes one if the plugin didn't record), logs a WARN pipeline: plugin would have denied (shadow) line, and continues the pipeline. The request is not blocked.

Body-mutation calls (SetBody / SetResponseBody) under observe record a Shadow: true Invocation but do not alter the in-memory body or the wire bytes — downstream plugins and the upstream see the original.

What this commit does NOT include

The parent branch feat/plugin-on-error also sealed the built-in plugin registry (reservedBuiltinNames, RegisterBuiltin, IsReservedBuiltin, ChainDirection, BuildChain, built-ins switched to RegisterBuiltin, chain-placement validation). That's a different concern — preventing custom plugins from replacing jwt-validation / token-exchange — and can land separately without blocking the rollout mechanism guardrail plugins need today. We split it off so this PR is narrowly about on_error.

If you want that seal later, it's a small follow-up PR on top of this one.

Dashboard queries

count(Invocations where shadow=true)                                  # rollout volume
count(Invocations where shadow=true and action="deny") by plugin      # per-plugin shadow deny rate
count(Invocations where shadow=false and action="deny") by plugin     # enforced denials (unchanged)

What on_error does NOT do

• Not a circuit breaker — a crashing plugin in observe still crashes on every request.
• Not sampling — observe runs the plugin on 100% of traffic.
• Not a timeout — a slow plugin still blocks.
• Does not cover runtime error returns / panics. Policy applies only to intentional Reject actions; a panic in observe still surfaces as a 500.

Test plan

• go build ./... and go test -race ./... pass in both authlib and cmd/authbridge under GOWORK=off
• gofmt -l clean on all files touched by this PR
• 8 new tests in commit 1 (reject-recording + bucketing fallbacks + skip-without-invocations across 3 listeners)
• 7 new tests in commit 2 (observe conversion, enforce default, off skipping, response-phase observe, synthesized shadow record, SetBody observe-is-noop, WithPolicies overflow)
• 2 new config tests (on_error YAML parse / valid values / invalid rejection / omitted defaults to enforce)
• End-to-end smoke: deploy on weather-agent with an intentionally-rejecting plugin in observe, confirm shadow invocations land in abctl and traffic passes; flip to enforce, confirm traffic is blocked

Follow-ups (not in this PR)

• Seal built-in registry (reserved names + chain-placement validation). Drop-in on top of this branch once prioritized.
• Cross-session intent correlation (session header injected by inbound listener, read by outbound). Required before IBAC / any multi-tenant guardrail under concurrent multi-user load; deferred while single-tenant-per-pod is the operational model.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

[huang195]

Addressing review in f7bebe2. Five fixes + one acknowledgment:

#1 Reason content leaking to logs / session store — Added a "Don't put matched content in Violation.Reason" section to plugin-reference.md §on_error with a GOOD/BAD example. The same rule covers both the WARN log on shadow denies and the session store (both outside the authorization domain of the request). Pointed at Details["match_sha256"] as the right place for match fingerprints when debugging needs them.

#2 Duplicate filterInvocationsByPhase — Promoted to (*Invocations).FilteredByPhase in pipeline/. Forwardproxy and reverseproxy now call the method; local helpers gone. Took the chance to fix a subtle inconsistency: forwardproxy/reverseproxy had been accepting untagged Phase == "" entries, extproc's snapshotInvocations had been strict. Strict is correct (framework always populates Phase via Context.Record); aligned on strict. Extproc's snapshotInvocations collapses to a one-line delegation.

#3 Synthesized Invocation Reason obscures the deny code — Fixed. The synthesized Invocation.Reason is now action.Violation.Code (the plugin's machine-stable deny code); Details["synthesized"] = "true" distinguishes the synthesized-vs-recorded path when debugging. Dashboards grouping by Reason now see the actual deny across both paths. Test updated.

#4 on_error name is misleading — Led the section with a blockquote naming the caveat up front: "despite the name, this controls intentional Deny, not panics/errors." The point was previously buried under "What on_error does NOT do"; promoted to the top.

#5 abctl not updated to surface Shadow — actionCell now appends * to the action string when Invocation.Shadow is true (e.g., deny*). Width fits inside the existing 8-char column budget. Detail pane picks up Invocation.Shadow automatically via the existing JSON marshal tag. New test case added.

#6 Missing test: policy references plugin that doesn't exist — No change; documenting why. The schema can't express this case:

• PluginEntry.OnError is inline with the plugin entry in YAML — there's no separate policy map keyed by name where a "missing plugin" reference could arise.
• WithPolicies(...ErrorPolicy) is positional, indexed parallel to the plugin slice. The nearest-equivalent typo ("more policies than plugins") is already covered by TestNew_RejectsTooManyPolicies.

If a future schema adds a name-keyed policy-override section (e.g., policy_overrides: { pii-scrubber: observe }), the missing-plugin case appears then and the test goes in with it.

[pdettori]

Well-structured PR with two cleanly separable features bundled together. The deny-recording gap closure (commit 1) and the on_error policy (commit 2) are both well-implemented with thorough tests (17 new tests total) and clear documentation. The FilteredByPhase consolidation in commit 3 is a nice cleanup.

One blocking issue: a code comment claims auth plugins are validated against shadowing at startup, when that validation was intentionally deferred to the built-in registry sealing follow-up. The claim could mislead operators or auditors.

Areas reviewed: Go (pipeline, config, listeners, plugins, abctl), Docs
Commits: 3 commits, all signed-off
CI status: 16/16 passing (1 skip: Spellcheck)

[huang195]

Addressed in 7a71aa6. You're right — the sentence was leftover from the parent feat/plugin-on-error branch (which included the built-in registry seal). When I extracted the on_error piece for this PR the seal got dropped, but this godoc paragraph didn't get updated to match. Rewrote it as advisory + an explicit "not enforced at startup" note pointing at the sealing follow-up:

// Shadowing or disabling auth gates (jwt-validation, token-exchange)
// is an authentication bypass dressed as a feature — operators SHOULD
// leave those plugins on ErrorPolicyEnforce (the default).
//
// The framework does NOT enforce this at startup: nothing prevents
// on_error: observe on jwt-validation today. A built-in-registry
// sealing pass is planned as a follow-up to reject a non-enforce
// policy on reserved gates at build time.

Also verified the matching doc section in plugin-reference.md §"Applicability to auth gates" — it was already advisory ("auth gates should stay on enforce"), no false-claim there. No other occurrences in the tree.

[pdettori]

Previous blocking issue (stale godoc claiming startup enforcement of auth-gate policies) is resolved — the fixup commit correctly rewrites the paragraph as advisory with an honest "not enforced at startup" statement.

The two features are well-implemented: deny-recording helpers are symmetric across all three listeners, the observe/enforce/off dispatch logic is clean with proper bounds checking, and body mutation suppression under observe is correct (both in-memory state and wire bytes preserved). 17 new tests cover happy paths, edge cases, and both request/response phases. Documentation is accurate and includes appropriate caveats.

Areas reviewed: Go (pipeline, config, listeners, plugins, abctl), Docs
Commits: 4 commits, all signed-off
CI status: 16/16 passing (1 skip: Spellcheck)

[pdettori]

nit: recordShadowBodyMutation fires once per SetBody call under observe. If a plugin calls SetBody in a loop (unlikely but possible), this creates N shadow invocations. Not a bug — just something to be aware of if shadow invocation counts look inflated during canary analysis.