Re-land Google Cloud Logging routing with kill-switch and startup diagnostics - #178
Merged
Conversation
…ng-consumer" This reverts commit f743750.
Set OBSERVABILITY_LOG_DESTINATIONS explicitly in every Modal secret: previews default to google_cloud_logging as the live test bed, and production stays on stdout until the repo variable flips it — recovery from any future destination failure is a secret change, not a revert. Log one startup line (platform, destinations, identity-token presence) so a misrouted destination is visible in a container's first seconds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Beta preview has been cleaned up because this PR was closed. |
…breaker Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #141
Re-lands #142 (reverted in f743750 after the production outage) now that the root cause is fixed and verified, plus the operational rails that make a recurrence recoverable in seconds instead of a revert cycle.
Root cause of the outage — fixed and proven
The WIF provider mapped
google.subject = assertion.sub; Modal'ssub(full function identity) exceeds GCP's 127-byte limit, so every STS exchange failed after ~25s of internal retries, per log write, inside the request path. The provider now mapsgoogle.subject = assertion.workspace_id + "/" + assertion.app_name, which is provably ≤ 90 bytes for any app this workspace can create (measured workspace_id: 25 bytes; Modal caps app names at 64). Verified end to end: a Modal container exchanged its real identity token through the remapped provider and wrote to Cloud Logging in 2.1s, and the entry is queryable inpolicyengine-api.What this PR adds beyond the revert-of-revert
OBSERVABILITY_LOG_DESTINATIONSset explicitly in every Modal secret: previews default togoogle_cloud_logging(the live test bed —peukchat-*is already in the provider trust condition); production stays onstdoutuntil theOBSERVABILITY_LOG_DESTINATIONS_PRODUCTIONrepo variable flips it. Any future destination failure is recoverable by changing a variable and redeploying — no revert.Merge preconditions
1.3.2; pin bumped in 4183edc. The breaker converts any remaining emit failure into lost logs + internal records instead of request-path damage.Rollout
jsonPayload.service_name="policyengine-uk-chat"inpolicyengine-api).OBSERVABILITY_LOG_DESTINATIONS_PRODUCTION=google_cloud_logging→ next deploy routes production logs; smoke test guards it.Verification
Backend suite with this branch: 241 passed, 10 skipped.
🤖 Generated with Claude Code