diff --git a/README.md b/README.md index c756b24..5a53261 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,9 @@ The Google Cloud setup needs: - An IAM binding granting the workload identity principal `roles/iam.workloadIdentityUser` on that service account. +For the fixed PolicyEngine Google Cloud destination, see +[`docs/operations/google-cloud-stage3-runbook.md`](docs/operations/google-cloud-stage3-runbook.md). + ## Release workflow Changes should include a Towncrier fragment in `changelog.d/`. Pull requests diff --git a/changelog.d/google-cloud-stage3-runbook.changed.md b/changelog.d/google-cloud-stage3-runbook.changed.md new file mode 100644 index 0000000..4fb0283 --- /dev/null +++ b/changelog.d/google-cloud-stage3-runbook.changed.md @@ -0,0 +1 @@ +Document the fixed Google Cloud Stage 3 observability destination, including Cloud Run routing, Modal WIF, and smoke-test queries. diff --git a/docs/operations/google-cloud-stage3-runbook.md b/docs/operations/google-cloud-stage3-runbook.md new file mode 100644 index 0000000..6f0ea5d --- /dev/null +++ b/docs/operations/google-cloud-stage3-runbook.md @@ -0,0 +1,189 @@ +# Google Cloud Stage 3 Observability Runbook + +This runbook documents the fixed Google Cloud Logging destination for +PolicyEngine observability records. + +The public runbook intentionally uses placeholders for live project IDs, +project numbers, service accounts, workload identity provider paths, and sink +writer identities. Keep concrete values in private infrastructure state, +repository/environment secrets, or internal operations documentation. + +## Central Destination + +- Project: `` +- Project number: `` +- Log bucket: `` +- Bucket location: `` +- Bucket retention: 30 days +- Log Analytics: enabled +- Primary log name: `` + +Required APIs: + +```bash +gcloud services enable \ + logging.googleapis.com \ + iamcredentials.googleapis.com \ + sts.googleapis.com \ + --project= +``` + +## Cloud Run Logging + +Cloud Run has two log paths. + +App observability records are written directly by the observability package to +the central project. Cloud Run runtime service accounts need: + +```text +roles/logging.logWriter +``` + +on ``. + +Service accounts should be granted explicitly per deployed service and +environment: + +```text + + +``` + +Native Cloud Run logs are routed with a Cloud Logging sink from +the source application project: + +```text +sink: +destination: logging.googleapis.com/projects//locations//buckets/ +filter: resource.type="cloud_run_revision" +writer: +``` + +The sink writer has `roles/logging.bucketWriter` on +``. + +## Modal Logging + +Modal platform logs stay in Modal for now. Modal simulation container +observability records are written directly to Google Cloud Logging by the +observability package. + +The Modal writer service account is an environment-specific service account: + +```text + +``` + +It has `roles/logging.logWriter` on ``. + +Modal Workload Identity Federation: + +```text +provider: projects//locations/global/workloadIdentityPools//providers/ +issuer: https://oidc.modal.com +audience: oidc.modal.com +``` + +The writer service account grants `roles/iam.workloadIdentityUser` to: + +```text +principalSet://iam.googleapis.com/projects//locations/global/workloadIdentityPools// +``` + +The impersonation grant must be constrained by Modal OIDC claims or by an +equivalent provider condition. Do not use an unconstrained wildcard grant. The +condition should allow only the expected Modal app names and environments for +the services being deployed. + +## Runtime Configuration + +Use these variables for deployed Cloud Run and Modal environments: + +```bash +OBSERVABILITY_LOG_DESTINATIONS=google_cloud_logging +OBSERVABILITY_GOOGLE_CLOUD_PROJECT= +OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME= +OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER=projects//locations/global/workloadIdentityPools//providers/ +OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL= +``` + +Cloud Run does not need the Modal WIF variables. Modal does. + +Do not use `OBSERVABILITY_GOOGLE_LOG_NAME`; the package reads +`OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME`. + +## Smoke Tests + +Write a manual structured log: + +```bash +gcloud logging write \ + '{"event":"stage3_smoke_test","service_name":"","schema_version":"policyengine.observability.smoke.v1"}' \ + --payload-type=json \ + --project= \ + --severity=INFO +``` + +Read it back: + +```bash +gcloud logging read \ + 'logName="projects//logs/" AND jsonPayload.event="stage3_smoke_test"' \ + --project= \ + --limit=1 \ + --format=json +``` + +Find app observability logs: + +```text +logName="projects//logs/" +jsonPayload.service_name="" +``` + +Find Cloud Run app observability logs: + +```text +jsonPayload.service_name="" +jsonPayload.platform="google_cloud_run" +``` + +Find Modal simulation container observability logs: + +```text +jsonPayload.service_name="" +jsonPayload.platform="modal" +jsonPayload.service_role="modal_worker" +``` + +Find native Cloud Run logs routed through the sink: + +```text +resource.type="cloud_run_revision" +``` + +Find observability failures: + +```text +jsonPayload.event="observability_internal_error" +``` + +## Rollback + +For a failing runtime, set: + +```bash +OBSERVABILITY_LOG_DESTINATIONS=stdout +``` + +To restore the temporary bridge destination, set: + +```bash +OBSERVABILITY_GOOGLE_CLOUD_PROJECT= +OBSERVABILITY_GOOGLE_CLOUD_LOG_NAME= +OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER= +OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL= +``` + +Leave the central project and sink in place during rollback unless the sink +itself is the source of the problem.