From dba8b562354473d97a7538dab133588cccae96dc Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Tue, 30 Jun 2026 21:03:17 +0200 Subject: [PATCH 1/2] Document fixed Google observability destination --- README.md | 3 + .../google-cloud-stage3-runbook.changed.md | 1 + .../operations/google-cloud-stage3-runbook.md | 182 ++++++++++++++++++ 3 files changed, 186 insertions(+) create mode 100644 changelog.d/google-cloud-stage3-runbook.changed.md create mode 100644 docs/operations/google-cloud-stage3-runbook.md diff --git a/README.md b/README.md index c756b24..5a53261 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,9 @@ The Google Cloud setup needs: - An IAM binding granting the workload identity principal `roles/iam.workloadIdentityUser` on that service account. +For the fixed PolicyEngine Google Cloud destination, see +[`docs/operations/google-cloud-stage3-runbook.md`](docs/operations/google-cloud-stage3-runbook.md). + ## Release workflow Changes should include a Towncrier fragment in `changelog.d/`. Pull requests diff --git a/changelog.d/google-cloud-stage3-runbook.changed.md b/changelog.d/google-cloud-stage3-runbook.changed.md new file mode 100644 index 0000000..4fb0283 --- /dev/null +++ b/changelog.d/google-cloud-stage3-runbook.changed.md @@ -0,0 +1 @@ +Document the fixed Google Cloud Stage 3 observability destination, including Cloud Run routing, Modal WIF, and smoke-test queries. diff --git a/docs/operations/google-cloud-stage3-runbook.md b/docs/operations/google-cloud-stage3-runbook.md new file mode 100644 index 0000000..7cc0998 --- /dev/null +++ b/docs/operations/google-cloud-stage3-runbook.md @@ -0,0 +1,182 @@ +# Google Cloud Stage 3 Observability Runbook + +This runbook documents the fixed Google Cloud Logging destination for +PolicyEngine observability records. + +## Central Destination + +- Project: `policyengine-observability` +- Project number: `790230211054` +- Log bucket: `policyengine-observability` +- Bucket location: `global` +- Bucket retention: 30 days +- Log Analytics: enabled +- Primary log name: `policyengine-observability` + +Required APIs: + +```bash +gcloud services enable \ + logging.googleapis.com \ + iamcredentials.googleapis.com \ + sts.googleapis.com \ + --project=policyengine-observability +``` + +## Cloud Run Logging + +Cloud Run has two log paths. + +App observability records are written directly by `policyengine-observability` +to the central project. Household API Cloud Run runtime service accounts need: + +```text +roles/logging.logWriter +``` + +on `policyengine-observability`. + +Initial household service accounts: + +```text +household-api-gateway@policyengine-household-api.iam.gserviceaccount.com +household-api-worker@policyengine-household-api.iam.gserviceaccount.com +``` + +Native Cloud Run logs are routed with a Cloud Logging sink from +`policyengine-household-api`: + +```text +sink: household-cloud-run-to-policyengine-observability +destination: logging.googleapis.com/projects/policyengine-observability/locations/global/buckets/policyengine-observability +filter: resource.type="cloud_run_revision" +writer: serviceAccount:service-120046258570@gcp-sa-logging.iam.gserviceaccount.com +``` + +The sink writer has `roles/logging.bucketWriter` on +`policyengine-observability`. + +## Modal Logging + +Modal platform logs stay in Modal for now. Modal simulation container +observability records are written directly to Google Cloud Logging by +`policyengine-observability`. + +The Modal writer service account is: + +```text +observability-writer@policyengine-observability.iam.gserviceaccount.com +``` + +It has `roles/logging.logWriter` on `policyengine-observability`. + +Modal Workload Identity Federation: + +```text +provider: projects/790230211054/locations/global/workloadIdentityPools/modal/providers/modal +issuer: https://oidc.modal.com +audience: oidc.modal.com +``` + +The writer service account grants `roles/iam.workloadIdentityUser` to: + +```text +principalSet://iam.googleapis.com/projects/790230211054/locations/global/workloadIdentityPools/modal/* +``` + +The provider condition should allow household Modal app names beginning with +`policyengine-household-api`. UK Chat patterns may be included for later reuse, +but UK Chat is not required for the household API cutover. + +## Runtime Configuration + +Use these values for deployed household API Cloud Run and Modal environments: + +```bash +OBSERVABILITY_LOG_DESTINATIONS=google_cloud_logging +OBSERVABILITY_GOOGLE_CLOUD_PROJECT=policyengine-observability +OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME=policyengine-observability +OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER=projects/790230211054/locations/global/workloadIdentityPools/modal/providers/modal +OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL=observability-writer@policyengine-observability.iam.gserviceaccount.com +``` + +Cloud Run does not need the Modal WIF variables. Modal does. + +Do not use `OBSERVABILITY_GOOGLE_LOG_NAME`; the package reads +`OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME`. + +## Smoke Tests + +Write a manual structured log: + +```bash +gcloud logging write policyengine-observability \ + '{"event":"stage3_smoke_test","service_name":"policyengine-observability","schema_version":"policyengine.observability.smoke.v1"}' \ + --payload-type=json \ + --project=policyengine-observability \ + --severity=INFO +``` + +Read it back: + +```bash +gcloud logging read \ + 'logName="projects/policyengine-observability/logs/policyengine-observability" AND jsonPayload.event="stage3_smoke_test"' \ + --project=policyengine-observability \ + --limit=1 \ + --format=json +``` + +Find household API app observability logs: + +```text +logName="projects/policyengine-observability/logs/policyengine-observability" +jsonPayload.service_name="policyengine-household-api" +``` + +Find Cloud Run app observability logs: + +```text +jsonPayload.service_name="policyengine-household-api" +jsonPayload.platform="google_cloud_run" +``` + +Find Modal simulation container observability logs: + +```text +jsonPayload.service_name="policyengine-household-api" +jsonPayload.platform="modal" +jsonPayload.service_role="modal_worker" +``` + +Find native Cloud Run logs routed through the sink: + +```text +resource.type="cloud_run_revision" +``` + +Find observability failures: + +```text +jsonPayload.event="observability_internal_error" +``` + +## Rollback + +For a failing runtime, set: + +```bash +OBSERVABILITY_LOG_DESTINATIONS=stdout +``` + +To restore the temporary bridge destination, set: + +```bash +OBSERVABILITY_GOOGLE_CLOUD_PROJECT=policyengine-api +OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME=policyengine-observability +OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER=projects/389282473430/locations/global/workloadIdentityPools/modal/providers/modal +OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL=observability-writer@policyengine-api.iam.gserviceaccount.com +``` + +Leave the central project and sink in place during rollback unless the sink +itself is the source of the problem. From 966a9f4eab683fbfaeb7cf9d15436295f207cf4e Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Wed, 1 Jul 2026 21:46:02 +0200 Subject: [PATCH 2/2] Remove live infrastructure identifiers from runbook --- .../operations/google-cloud-stage3-runbook.md | 99 ++++++++++--------- 1 file changed, 53 insertions(+), 46 deletions(-) diff --git a/docs/operations/google-cloud-stage3-runbook.md b/docs/operations/google-cloud-stage3-runbook.md index 7cc0998..6f0ea5d 100644 --- a/docs/operations/google-cloud-stage3-runbook.md +++ b/docs/operations/google-cloud-stage3-runbook.md @@ -3,15 +3,20 @@ This runbook documents the fixed Google Cloud Logging destination for PolicyEngine observability records. +The public runbook intentionally uses placeholders for live project IDs, +project numbers, service accounts, workload identity provider paths, and sink +writer identities. Keep concrete values in private infrastructure state, +repository/environment secrets, or internal operations documentation. + ## Central Destination -- Project: `policyengine-observability` -- Project number: `790230211054` -- Log bucket: `policyengine-observability` -- Bucket location: `global` +- Project: `` +- Project number: `` +- Log bucket: `` +- Bucket location: `` - Bucket retention: 30 days - Log Analytics: enabled -- Primary log name: `policyengine-observability` +- Primary log name: `` Required APIs: @@ -20,60 +25,61 @@ gcloud services enable \ logging.googleapis.com \ iamcredentials.googleapis.com \ sts.googleapis.com \ - --project=policyengine-observability + --project= ``` ## Cloud Run Logging Cloud Run has two log paths. -App observability records are written directly by `policyengine-observability` -to the central project. Household API Cloud Run runtime service accounts need: +App observability records are written directly by the observability package to +the central project. Cloud Run runtime service accounts need: ```text roles/logging.logWriter ``` -on `policyengine-observability`. +on ``. -Initial household service accounts: +Service accounts should be granted explicitly per deployed service and +environment: ```text -household-api-gateway@policyengine-household-api.iam.gserviceaccount.com -household-api-worker@policyengine-household-api.iam.gserviceaccount.com + + ``` Native Cloud Run logs are routed with a Cloud Logging sink from -`policyengine-household-api`: +the source application project: ```text -sink: household-cloud-run-to-policyengine-observability -destination: logging.googleapis.com/projects/policyengine-observability/locations/global/buckets/policyengine-observability +sink: +destination: logging.googleapis.com/projects//locations//buckets/ filter: resource.type="cloud_run_revision" -writer: serviceAccount:service-120046258570@gcp-sa-logging.iam.gserviceaccount.com +writer: ``` The sink writer has `roles/logging.bucketWriter` on -`policyengine-observability`. +``. ## Modal Logging Modal platform logs stay in Modal for now. Modal simulation container -observability records are written directly to Google Cloud Logging by -`policyengine-observability`. +observability records are written directly to Google Cloud Logging by the +observability package. -The Modal writer service account is: +The Modal writer service account is an environment-specific service account: ```text -observability-writer@policyengine-observability.iam.gserviceaccount.com + ``` -It has `roles/logging.logWriter` on `policyengine-observability`. +It has `roles/logging.logWriter` on ``. Modal Workload Identity Federation: ```text -provider: projects/790230211054/locations/global/workloadIdentityPools/modal/providers/modal +provider: projects//locations/global/workloadIdentityPools//providers/ issuer: https://oidc.modal.com audience: oidc.modal.com ``` @@ -81,23 +87,24 @@ audience: oidc.modal.com The writer service account grants `roles/iam.workloadIdentityUser` to: ```text -principalSet://iam.googleapis.com/projects/790230211054/locations/global/workloadIdentityPools/modal/* +principalSet://iam.googleapis.com/projects//locations/global/workloadIdentityPools// ``` -The provider condition should allow household Modal app names beginning with -`policyengine-household-api`. UK Chat patterns may be included for later reuse, -but UK Chat is not required for the household API cutover. +The impersonation grant must be constrained by Modal OIDC claims or by an +equivalent provider condition. Do not use an unconstrained wildcard grant. The +condition should allow only the expected Modal app names and environments for +the services being deployed. ## Runtime Configuration -Use these values for deployed household API Cloud Run and Modal environments: +Use these variables for deployed Cloud Run and Modal environments: ```bash OBSERVABILITY_LOG_DESTINATIONS=google_cloud_logging -OBSERVABILITY_GOOGLE_CLOUD_PROJECT=policyengine-observability -OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME=policyengine-observability -OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER=projects/790230211054/locations/global/workloadIdentityPools/modal/providers/modal -OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL=observability-writer@policyengine-observability.iam.gserviceaccount.com +OBSERVABILITY_GOOGLE_CLOUD_PROJECT= +OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME= +OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER=projects//locations/global/workloadIdentityPools//providers/ +OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL= ``` Cloud Run does not need the Modal WIF variables. Modal does. @@ -110,10 +117,10 @@ Do not use `OBSERVABILITY_GOOGLE_LOG_NAME`; the package reads Write a manual structured log: ```bash -gcloud logging write policyengine-observability \ - '{"event":"stage3_smoke_test","service_name":"policyengine-observability","schema_version":"policyengine.observability.smoke.v1"}' \ +gcloud logging write \ + '{"event":"stage3_smoke_test","service_name":"","schema_version":"policyengine.observability.smoke.v1"}' \ --payload-type=json \ - --project=policyengine-observability \ + --project= \ --severity=INFO ``` @@ -121,30 +128,30 @@ Read it back: ```bash gcloud logging read \ - 'logName="projects/policyengine-observability/logs/policyengine-observability" AND jsonPayload.event="stage3_smoke_test"' \ - --project=policyengine-observability \ + 'logName="projects//logs/" AND jsonPayload.event="stage3_smoke_test"' \ + --project= \ --limit=1 \ --format=json ``` -Find household API app observability logs: +Find app observability logs: ```text -logName="projects/policyengine-observability/logs/policyengine-observability" -jsonPayload.service_name="policyengine-household-api" +logName="projects//logs/" +jsonPayload.service_name="" ``` Find Cloud Run app observability logs: ```text -jsonPayload.service_name="policyengine-household-api" +jsonPayload.service_name="" jsonPayload.platform="google_cloud_run" ``` Find Modal simulation container observability logs: ```text -jsonPayload.service_name="policyengine-household-api" +jsonPayload.service_name="" jsonPayload.platform="modal" jsonPayload.service_role="modal_worker" ``` @@ -172,10 +179,10 @@ OBSERVABILITY_LOG_DESTINATIONS=stdout To restore the temporary bridge destination, set: ```bash -OBSERVABILITY_GOOGLE_CLOUD_PROJECT=policyengine-api -OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME=policyengine-observability -OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER=projects/389282473430/locations/global/workloadIdentityPools/modal/providers/modal -OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL=observability-writer@policyengine-api.iam.gserviceaccount.com +OBSERVABILITY_GOOGLE_CLOUD_PROJECT= +OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME= +OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER= +OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL= ``` Leave the central project and sink in place during rollback unless the sink