Bounded and asynchronous log emission: cap Google write latency, add a background emitter, and agent-native stdout format

[anth-volk]

Problem

The google_cloud_logging destination emits synchronously on the caller's request path via Logger.log_struct, and the underlying WriteLogEntries call carries google-cloud-logging's defaults: a 60-second retry deadline and 60-second timeout. Consequences for consumers (diagnosed in the policyengine-household-api, which promises customers ≤30s full runtimes — see PolicyEngine/policyengine-household-api#1596/#1597):

• Healthy path: every request pays 2–3 synchronous Logging API round-trips (~15–60ms).
• Degraded path: during a Cloud Logging incident, each emission can block up to 60s before it counts as a strike against the 3-strike destination breaker — so the first requests of each fresh process can stall 60–180s, violating downstream latency promises even though the fail-open design (added in Disable a log destination after repeated consecutive emit failures #21) prevents outright failure.

Standard practice separates the synchronous log call (capture context, enqueue — microseconds, cannot block or raise) from asynchronous emission (network I/O off the caller's thread, batched, bounded, best-effort). Python stdlib ships QueueHandler/QueueListener for exactly this, and google-cloud-logging's own CloudLoggingHandler defaults to BackgroundThreadTransport — though its queue is unbounded (queue.Queue(0)), which is why this issue proposes a bounded implementation instead of adopting it.

Proposal (one feature branch, gated commits)

1. Bounded Google writes (safety floor, default behavior change 60s → 2s): replace log_struct with hand-built entries written via the gapic layer with retry=None, timeout=<config, default 2.0s> (OBSERVABILITY_GOOGLE_LOG_TIMEOUT_SECONDS), keeping client.logger() for full_name/default_resource, with a fail-open fallback to plain write_entries on non-gapic transports. Also exposes emit_batch (one bounded call for N entries).
2. Agent-native structured stdout (OBSERVABILITY_STDOUT_FORMAT=google): map payloads onto the special keys Cloud Run's log agent promotes natively (severity, time, logging.googleapis.com/trace|spanId|labels), enabling full-fidelity ingestion with zero in-process network emission on platforms with an agent.
3. BackgroundEmitDestination: bounded deque (default 1000, drop-newest with counted drops), lazy daemon worker (pid-aware for fork safety), batch drain preferring emit_batch, consecutive-batch-failure trip that flows through the existing manager breaker/stdout-fallback path unchanged, flush(deadline)/close()/restart() lifecycle with lazy atexit registration.
4. Config + wiring + facade: OBSERVABILITY_LOG_EMIT_MODE=sync|async (default sync), queue/batch/flush knobs; manager wraps non-stdout destinations in async mode (stdout is the fallback and is never wrapped); runtime.shutdown() flushes log destinations; public flush_observability()/restart_observability() (the latter required by consumers whose runtimes restore from memory snapshots — Modal — since threads do not survive snapshot restore).
5. Docs: README env-var documentation, loss semantics, snapshot/fork consumer guidance, runbook kill-switches (OBSERVABILITY_LOG_DESTINATIONS=stdout, OBSERVABILITY_LOG_EMIT_MODE=sync).

Compatibility contract

Everything defaults off (sync, plain): existing consumers see no behavior change except the bounded 2s write timeout, which is strictly safer. All existing fail-open invariants from #21 (per-destination guards, DESTINATION_FAILURE_LIMIT, stdout fallback) are preserved and covered by the new tests.

Consumer adoption (platform-split defaults, Modal snapshot hook, GCP Log Router sink) happens separately in policyengine-household-api after this releases.

[anth-volk]

Redesign after the abandoned PR #23 (see its closing comment for why). The replacement, agreed with the maintainer, is deliberately smaller and strategy-shaped:

Consumer contract: one env var, OBSERVABILITY_LOG_PROFILE — gcp-agent (google-format stdout only, for platforms whose agent ingests stdout, e.g. Cloud Run), gcp-direct (stdout + queued direct Cloud Logging writes, e.g. Modal), plain-sync (plain stdout, guaranteed zero threads — dev/kill-switch), auto (default; platform detection via OBSERVABILITY_PLATFORM/K_SERVICE/Modal markers). The package owns all routing and process handling; consumers stop carrying platform logic.

Architecture: the core is backend-agnostic — a destination registry, a generic QueuedLogDestination transport built on stdlib queue.Queue + logging.handlers.QueueListener (CPython owns the thread/shutdown machinery), stdout formatter hooks, profiles, and bounded lifecycle. Cloud Logging is one registered strategy at the edge (google_cloud_logging.py: client, credentials, bounded writes, google stdout formatter). Future backends plug into the same registry and transport.

Key semantics vs today:

• Remote destinations are always queued — no remote write ever happens on a request thread, eliminating the 60s-stall failure mode structurally (today's sync log_struct path inherits a 60s retry deadline). Delivery becomes best-effort by contract; stdout is the durable sibling record.
• The queue is bounded (drop-and-count with throttled reports), writes carry an explicit budget, close/shutdown are deadline-bounded, and recovery for snapshot/fork runtimes is a documented single-threaded restart_observability() — no breakers, generations, or fork hooks in the transport.