diff --git a/docs/bridge-integration/CUTOVER-RUNBOOK.md b/docs/bridge-integration/CUTOVER-RUNBOOK.md new file mode 100644 index 000000000..a608e7f1c --- /dev/null +++ b/docs/bridge-integration/CUTOVER-RUNBOOK.md @@ -0,0 +1,509 @@ +# Cash Wallet Cutover Day Runbook + +This runbook is the one-time execution script for moving Flash Cash Wallet users from legacy IBEX USD presentation to IBEX USDT-backed Cash Wallet operation. + +Use `OPERATIONS.md` as the reference manual for steady-state Bridge operations, replay, reconciliation, webhook incidents, and general deploy checks. This document is intentionally focused on cutover-day decisions and sequencing. + +## Scope + +This runbook covers: + +- T-7 days through T-0 readiness checks. +- T-1 hour go/no-go decision. +- T-0 cutover execution. +- Hard abort criteria. +- Verification after `IN_PROGRESS` and `COMPLETE`. +- T+1 hour and T+24 hour reconciliation. +- Handoff to rollback planning when abort criteria are met. + +Out of scope: + +- Bridge steady-state operations. See `OPERATIONS.md`. +- Server implementation. See ENG-345. +- Mobile implementation. See ENG-346. +- Legal/customer communications approval. See COM-44. +- Full rollback procedure. See ENG-364. + +## Cutover Principles + +- No silent ambiguity: every go/no-go decision needs a named owner and timestamp. +- Do not manually mutate user wallet state during cutover without a written repair plan and second reviewer. +- Keep old-client compatibility intact throughout the cutover. Clients without `X-Flash-Client-Capabilities: cash-wallet-usdt-v1` must continue to see legacy `USD`. +- Capable clients with `X-Flash-Client-Capabilities: cash-wallet-usdt-v1` see `USDT` only after their account has completed migration, or after global cutover is `COMPLETE`. +- Do not mark cutover `COMPLETE` until migration, wallet presentation, reconciliation, and support readiness all pass. +- If rollback is needed after irreversible wallet/default changes, treat it as a data repair project, not a normal deploy rollback. + +## Roles + +Fill this table before the staging rehearsal and again before production cutover. + +| Role | Name | Backup | Required at T-1h? | Required at T-0? | +| --- | --- | --- | --- | --- | +| Cutover commander | Dread | TBD | Yes | Yes | +| Ops executor | Olaniran | TBD | Yes | Yes | +| Backend owner | TBD | TBD | Yes | Yes | +| Mobile owner | Nick | TBD | Yes | On call | +| Support lead | TBD | TBD | Yes | On call | +| Comms/legal approver | TBD | TBD | Yes | No | +| Incident commander fallback | TBD | TBD | Yes | Yes | + +## Required Artifacts + +Create or link these before T-1 day: + +- Production release commit SHA. +- Bridge operations `OPERATIONS.md` branch/commit. +- Staging rehearsal notes. +- Backend deployment evidence. +- Mobile rollout status for ENG-346. +- COM-44 approved comms. +- Dashboard links. +- PagerDuty/Slack on-call schedule. +- Bridge dashboard/config screenshots or internal references. +- Cutover run ID. +- Config snapshot reference. +- Rollback/repair owner for ENG-364. + +Do not paste secrets, bearer tokens, Bridge API keys, webhook public keys copied from private channels, replay secrets, or Bruno local environment files into this runbook. + +## Dependencies + +Do not proceed to production cutover unless all required dependencies are deployed or explicitly waived by the cutover commander. + +| Dependency | Required state | Owner | Status | +| --- | --- | --- | --- | +| ENG-345 server cutover support | Deployed and smoke-tested | Backend | TBD | +| ENG-346 mobile cutover UX | Shipped or rollout plan approved | Mobile | TBD | +| ENG-296 / ENG-297 USDT wallet support | Deployed and verified | Backend | TBD | +| COM-44 comms/legal | Approved | Comms/legal | TBD | +| Bridge webhook server | Healthy in target env | Ops | TBD | +| Bridge reconciliation | Runs cleanly or known exceptions documented | Ops/backend | TBD | +| Monitoring dashboards | Live | Ops | TBD | +| PagerDuty/Slack on-call | Active | Ops | TBD | +| ENG-364 rollback plan | Draft reviewed or waived | Ops/backend | TBD | + +## T-7 Days Checklist + +- Confirm production cutover date and window. +- Confirm the cutover commander, ops executor, backend owner, mobile owner, support lead, and comms/legal approver. +- Confirm ENG-345 is merged and deployed to staging. +- Confirm ENG-346 status and mobile rollout plan. +- Confirm COM-44 comms are drafted and routed for approval. +- Confirm Bridge dashboard access for the ops executor and backup. +- Confirm production config owner and secret owner. +- Confirm rollback owner and decision window. +- Confirm customer support has an escalation channel and internal FAQ draft. +- Schedule staging rehearsal. + +Exit criteria: + +- All named roles have accepted ownership. +- Dependencies have owners and target dates. +- Staging rehearsal is scheduled. + +## T-3 Days Checklist + +- Deploy the release candidate to staging. +- Run Bridge GraphQL smoke checks. +- Run Bridge webhook `/health`. +- Run replay dry-run for a narrow recent window. +- Run Bridge reconciliation with a 24-hour window. +- Verify no unresolved high-severity `bridge_without_ibex` orphan. +- Verify old-client no-header wallet query returns legacy `USD`. +- Verify capable-client wallet query matches the staged cutover phase: `USD` before `IN_PROGRESS`, `USDT` only for migrated accounts during `IN_PROGRESS`, and `USDT` after `COMPLETE`. +- Verify admin `cashWalletCutover` query works. +- Confirm dashboards show webhook status, reconciliation orphan counts, Bridge API errors, and cutover state changes. +- Confirm support can identify and escalate cutover issues. + +Exit criteria: + +- Staging passes the smoke matrix. +- Any known issue has an owner and explicit production go/no-go impact. + +## T-1 Day Checklist + +- Freeze unrelated Bridge/Cash Wallet deploys unless approved by the cutover commander. +- Confirm production release SHA. +- Confirm production config snapshot. +- Confirm production Bridge webhook public keys and replay secret are present. +- Confirm mobile rollout state. +- Confirm customer/support comms are approved. +- Confirm PagerDuty/Slack rotation. +- Confirm the rollback/repair decision tree is available. +- Confirm a staging rehearsal has completed, or record the explicit waiver. +- Create the production cutover run ID. + +Exit criteria: + +- Cutover commander signs off. +- Ops executor signs off. +- Backend owner signs off. +- Support lead signs off. +- Comms/legal approver signs off if user-facing comms are planned. + +## T-1 Hour Go/No-Go + +Record the decision in the issue, deployment channel, or operator log before touching production state. + +| Gate | Go condition | Abort condition | Owner | Result | +| --- | --- | --- | --- | --- | +| Release SHA | Expected SHA deployed | Wrong or unknown SHA | Backend | TBD | +| Migrations | Applied with no failure | Pending or failed migration | Backend | TBD | +| Webhook health | `/health` returns OK | Unhealthy webhook server | Ops | TBD | +| Bridge API | Normal latency/error rate | Bridge outage or auth failures | Ops | TBD | +| Reconciliation | No high-severity unresolved orphan | New critical orphan growth | Backend/Ops | TBD | +| Old-client compatibility | No-header query returns `USD` | No-header query returns `USDT` or errors | Backend | TBD | +| Capable-client behavior | Before `IN_PROGRESS`, capability query returns legacy `USD`; during cutover, migrated canary returns `USDT`; after `COMPLETE`, capability query returns `USDT` | Result does not match current cutover phase, or query errors | Backend/Mobile | TBD | +| Mobile readiness | ENG-346 status accepted | Unknown or unacceptable rollout risk | Mobile | TBD | +| Support readiness | Support lead online | Support not staffed | Support | TBD | +| On-call | PagerDuty/Slack rotation active | No accountable responder | Ops | TBD | +| Comms | Approved or waived | Required comms not approved | Comms/legal | TBD | + +Decision: + +- `GO`: proceed to T-0 execution. +- `NO-GO`: keep state at `PRE`, announce abort, record blocker and owner. +- `GO WITH WAIVER`: only allowed if the cutover commander records the waived gate, reason, risk owner, and rollback implication. + +Hard abort criteria: + +- Production webhook server is unhealthy. +- Bridge API auth fails. +- Old-client no-header query does not return legacy `USD`. +- Capable-client query does not return `USDT`. +- Reconciliation shows unresolved high-severity deposit mismatch. +- Required decision owner is unavailable. +- Support cannot staff the cutover window. +- Unknown production commit or config. + +## T-0 Execution + +Use one shared operator log. Record timestamps, operator, command/tool used, and result for each step. + +Set these variables once and record them in the operator log: + +```bash +export CUTOVER_VERSION=345 +export CUTOVER_RUN_ID="" +export OPERATOR="" +export PROD_CONFIG_PATH="" +``` + +Before running any command, confirm `git rev-parse HEAD`, `PROD_CONFIG_PATH`, and the config snapshot's Bridge environment/base URL match the target production environment. + +### 1. Confirm Starting State + +Query current admin cutover state: + +```graphql +query CashWalletCutoverState { + cashWalletCutover { + state + scheduledAt + startedAt + completedAt + pausedAt + pauseReason + cutoverVersion + runId + updatedBy + updatedAt + } +} +``` + +Expected: + +- `state` is `PRE`. +- `runId` is either absent or matches the planned production run. +- No unexpected `pausedAt` or `pauseReason`. + +If unexpected, stop and resolve before proceeding. + +### 2. Run Final Pre-Migration Smoke Checks + +Run both wallet presentation checks against a controlled production canary. Before `IN_PROGRESS`, both no-header and capability-header requests are expected to remain on legacy `USD`; this verifies the cutover has not started early. + +No capability header expected result: + +- Cash wallet presents as `USD`. +- Legacy balance is USD-compatible. + +Capability header expected result: + +- Before `IN_PROGRESS`: cash wallet still presents as `USD`. +- During `IN_PROGRESS`: only accounts with migration status `complete` or `skipped_already_migrated` present as `USDT`; not-started accounts remain `USD`, and active migrations may block. +- After `COMPLETE`: cash wallet presents as `USDT`. +- When presented as `USDT`, balance uses USDT smallest-unit semantics. + +Example header: + +```text +X-Flash-Client-Capabilities: cash-wallet-usdt-v1 +``` + +Do not proceed if either presentation check fails. + +### 3. Set Cutover `IN_PROGRESS` + +Preview and prepare the production cohort before setting `IN_PROGRESS`: + +```bash +node lib/scripts/cash-wallet-cutover.js preview \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" + +node lib/scripts/cash-wallet-cutover.js prepare \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +Use the operator CLI for production `IN_PROGRESS`; it preserves the lifecycle path used by the cutover worker: + +```bash +node lib/scripts/cash-wallet-cutover.js start \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +The admin `cashWalletCutoverUpdate` mutation and Bruno `03-set-in-progress.bru` file directly mutate state and bypass CLI lifecycle checks. Use them for production `IN_PROGRESS` only as an emergency override with backend-owner approval, a written operator-log entry, and the reason the CLI cannot be used. Track additional start lifecycle hardening through ENG-403. + +Record: + +- Operator. +- Timestamp. +- Run ID. +- Cutover version. +- Commit SHA. +- State response. + +Expected: + +- Public and admin `cashWalletCutover.state` return `IN_PROGRESS`. +- No spike in GraphQL errors. +- Old-client/capable-client presentation still passes. + +### 4. Run Migration + +Run one batch at a time and inspect status after each batch: + +```bash +node lib/scripts/cash-wallet-cutover.js run-batch \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" \ + --worker-id "$OPERATOR-manual-1" \ + --limit 25 + +node lib/scripts/cash-wallet-cutover.js status \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +Repeat `run-batch` until `status` shows no remaining runnable migration records. Preserve each JSON output in the operator log. If a batch exits non-zero or reports `failed` or `requires_operator_review`, stop and invoke the abort/rollback handoff below. + +Record: + +- Command/tool name and exact args. +- Start timestamp. +- End timestamp. +- Cohort size. +- Success count. +- Failure count. +- Checkpoint reference. +- Log reference. + +Monitor: + +- Migration failures. +- GraphQL error rate. +- Bridge webhook failures. +- IBEX deposit/ledger anomalies. +- Cash wallet presentation errors. +- Support escalations. + +If the job fails before irreversible changes, pause and decide whether to resume or abort. If failures occur after irreversible changes, stop manual action and invoke the ENG-364 repair/rollback decision process. + +### 5. Verify During `IN_PROGRESS` + +Before completing: + +- Admin cutover state is `IN_PROGRESS`. +- Migration job reports complete or expected partial state. +- No unresolved migration failures. +- No-header wallet query returns `USD`. +- Capability wallet query returns `USDT`. +- Bridge reconciliation runs successfully. +- No new unresolved high-severity orphan. +- Support has no blocker-level customer escalation. + +Do not set `COMPLETE` if any check fails. + +### 6. Set Cutover `COMPLETE` + +Set the cutover state to `COMPLETE` only after the `IN_PROGRESS` verification passes. + +Use the operator CLI for production `COMPLETE`; it checks failed, manual-review, and runnable migration counts before completing: + +```bash +node lib/scripts/cash-wallet-cutover.js complete \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +The admin `cashWalletCutoverUpdate` mutation and Bruno `04-set-complete.bru` file directly mutate state and bypass completion checks. Use them for production `COMPLETE` only as an emergency override with backend-owner approval, a written operator-log entry, and preserved `status` output proving why the override is safe. + +Record: + +- Operator. +- Timestamp. +- State response. +- Commit SHA. +- Run ID. + +Expected: + +- Admin and public `cashWalletCutover.state` return `COMPLETE`. +- Old clients continue to see legacy `USD`. +- Capable clients see `USDT`. + +### 7. Post-Complete Smoke Checks + +Run immediately after `COMPLETE`: + +- No-header wallet query. +- Capability-header wallet query. +- Cash wallet transaction-history query. +- Known account transaction detail query. +- Bridge reconciliation with at least the cutover window. +- Webhook health check. +- Bridge GraphQL smoke query. + +Expected: + +- No-header clients: legacy `USD` compatibility. +- Capable clients: `USDT`. +- No new critical reconciliation orphan. +- No webhook signature or auth failures. +- No unexpected support escalation. + +## T+1 Hour Checks + +- Re-run Bridge reconciliation. +- Review dashboard error rates. +- Review webhook request status by route. +- Review Bridge API latency and non-2xx rate. +- Review support tickets. +- Spot-check at least three migrated accounts. +- Confirm old-client compatibility still holds. +- Confirm capable-client USDT presentation still holds. +- Record known issues and owners. + +If any high-severity issue appears, open an incident and decide whether to invoke ENG-364. + +## T+24 Hour Checks + +- Run reconciliation with a 24-hour window. +- Review all reconciliation orphans. +- Confirm no orphan needs manual repair. +- Spot-check migrated account balances and recent transaction history. +- Review failed/pending Bridge transfers. +- Review webhook replay logs. +- Review support volume and themes. +- Decide whether to continue, pause, or accelerate legacy sunset planning. +- Schedule postmortem if there was any incident, waiver, failed gate, or customer-impacting issue. + +## Abort and Rollback Handoff + +Abort before `IN_PROGRESS`: + +- Keep cutover state at `PRE`. +- Announce no-go. +- Record blocker, owner, and next review time. + +Abort during `IN_PROGRESS`: + +- Pause or stop the migration worker according to the implementation. +- Preserve logs, checkpoints, and run ID. +- Do not manually mutate wallets. +- Decide whether to resume, complete, or invoke ENG-364. + +Abort after `COMPLETE`: + +- Treat as production incident and data repair. +- Keep client-aware presentation intact unless it is the root cause. +- Invoke ENG-364 rollback/repair process. +- Require written repair plan and second reviewer before changing wallet/default state. + +Rollback handoff packet: + +- Run ID. +- Cutover version. +- Production commit SHA. +- Config snapshot. +- Migration checkpoint. +- Failed gate or incident trigger. +- Affected account count. +- Known affected account IDs, if any. +- Reconciliation output. +- Support impact summary. +- Proposed next action. + +## Staging Rehearsal Log + +Complete before production or record a signed waiver. + +| Date | Environment | Commit SHA | Operator | Result | Notes | +| --- | --- | --- | --- | --- | --- | +| TBD | Staging | TBD | TBD | TBD | TBD | + +Minimum rehearsal evidence: + +- `PRE` state verified. +- `IN_PROGRESS` state set. +- Migration/cutover flow exercised. +- Preview and prepare outputs preserved. +- At least one partial batch run exercised. +- Resume after a stopped or stale worker lock exercised. +- Failed batch handling exercised, including `failed` or `requires_operator_review` operator decision. +- Abort before irreversible changes exercised. +- Abort after partial irreversible changes tabletop completed. +- `COMPLETE` state set. +- No-header wallet check passed. +- Capability-header wallet check passed. +- Old-client compatibility checked during partial migration. +- Reconciliation ran. +- Reconciliation orphan triage path exercised. +- Abort/rollback tabletop completed. + +## Production Signoff + +| Checkpoint | Name | Timestamp | Result | Notes | +| --- | --- | --- | --- | --- | +| T-1d readiness | TBD | TBD | TBD | TBD | +| T-1h go/no-go | TBD | TBD | TBD | TBD | +| `IN_PROGRESS` set | TBD | TBD | TBD | TBD | +| Migration complete | TBD | TBD | TBD | TBD | +| `COMPLETE` set | TBD | TBD | TBD | TBD | +| T+1h check | TBD | TBD | TBD | TBD | +| T+24h check | TBD | TBD | TBD | TBD | + +## Links + +- OPS-50: Cash Wallet Cutover Day go/no-go runbook. +- ENG-345: Server-side Cash Wallet cutover orchestration. +- ENG-346: Mobile Cash Wallet cutover screens and USDT card state. +- ENG-364: Cash Wallet cutover rollback contingency plan. +- COM-44: Customer/legal communications. +- `OPERATIONS.md`: Bridge steady-state operations reference. diff --git a/docs/bridge-integration/OPERATIONS.md b/docs/bridge-integration/OPERATIONS.md new file mode 100644 index 000000000..1b64cd8c0 --- /dev/null +++ b/docs/bridge-integration/OPERATIONS.md @@ -0,0 +1,623 @@ +# Bridge Operations Runbook + +This runbook covers production operation for the Bridge.xyz integration and the related cash wallet cutover work. It is written for deploys, incident response, replay, reconciliation, and cutover execution. + +The code is the source of truth. Older Bridge docs may mention `/bridge/webhooks/*`; the current webhook server exposes `/kyc`, `/deposit`, `/transfer`, `/internal/replay`, and `/health`. + +## Scope + +Bridge enables Flash users to move between USD bank rails and USDT: + +- On-ramp: Bridge virtual account receives USD, Bridge converts to USDT, IBEX credits Flash. +- Off-ramp: Flash initiates Bridge transfer from USDT to a linked external bank account. +- Webhooks: Bridge reports KYC, deposit, and transfer events to Flash. +- Reconciliation: Flash compares Bridge deposits with IBEX deposit records and records orphans. +- Cash wallet cutover: ENG-345 makes USD/USDT wallet presentation client-aware while migration runs. + +## Services + +- GraphQL API: user and admin Bridge operations. +- Bridge webhook server: standalone Express server for Bridge webhooks. +- MongoDB: Bridge account mappings, deposit logs, replay logs, reconciliation orphans, and cutover state. +- Bridge.xyz API: KYC, virtual accounts, events, external accounts, and transfers. +- IBEX: USDT receive addresses and deposit crediting. + +Key code paths: + +- `src/services/bridge/client.ts` +- `src/services/bridge/index.ts` +- `src/services/bridge/webhook-server/index.ts` +- `src/services/bridge/reconciliation.ts` +- `src/scripts/replay-bridge-events.ts` +- `src/scripts/reconcile-bridge-ibex-deposits.ts` +- `src/graphql/admin/root/query/bridge-reconciliation-orphans.ts` +- `src/migrations/20260423000000-bridge-virtual-account-unique-accountid.ts` + +## Required Config + +Bridge config lives under `bridge` in the runtime config. + +Required fields: + +- `bridge.enabled` +- `bridge.apiKey` +- `bridge.baseUrl` +- `bridge.webhook.port` +- `bridge.webhook.publicKeys.kyc` +- `bridge.webhook.publicKeys.deposit` +- `bridge.webhook.publicKeys.transfer` +- `bridge.webhook.timestampSkewMs` + +Optional or defaulted fields: + +- `bridge.minWithdrawalAmount` +- `bridge.timeoutMs` defaults to `10000` +- `bridge.webhook.replaySecret`, or `BRIDGE_WEBHOOK_REPLAY_SECRET` + +Never commit production API keys, replay secrets, webhook public keys copied from private channels, bearer tokens, or Bruno local environment files. + +## Pre-Deploy Checklist + +1. Confirm the exact branch and commit being deployed. +2. Confirm Bridge config is present for the target environment. +3. Confirm webhook public keys match Bridge dashboard values for KYC, deposit, and transfer events. +4. Confirm the webhook replay secret is present for operators, not exposed to clients. +5. Confirm MongoDB migration status. +6. Confirm Bridge API base URL points at the intended environment. +7. Confirm mobile client rollout status for `cash-wallet-usdt-v1` capability. +8. Confirm rollback owner, deploy owner, and decision owner. + +Useful checks: + +```bash +git rev-parse HEAD +yarn build +git diff --check +``` + +Full `yarn test:unit` and `yarn tsc-check` may currently include unrelated repo debt. For Bridge deploy confidence, use focused Bridge and cutover suites plus build and lint checks. + +## Deployment Order + +Deploy in this order: + +1. Run database migrations. +2. Deploy GraphQL/backend. +3. Deploy Bridge webhook server. +4. Verify webhook health. +5. Verify GraphQL Bridge surfaces. +6. Verify replay dry run. +7. Run reconciliation. +8. Only after the above is healthy, execute cash wallet cutover phases. + +### 1. Run Migrations + +Use the repo's normal Mongo migration process for the target environment. The Bridge virtual account uniqueness migration must be applied before relying on production Bridge account mappings: + +```bash +./scripts/mongodb-migrate.sh +``` + +Confirm no migration failed before continuing. + +### 2. Deploy GraphQL + +Deploy the backend with Bridge config enabled for the environment. + +Post-deploy smoke checks: + +- authenticated `bridgeKycStatus` +- authenticated `bridgeVirtualAccount` +- authenticated `bridgeExternalAccounts` +- admin `bridgeReconciliationOrphans` +- shared `cashWalletCutover` after ENG-345 is deployed + +Example shape: + +```graphql +query BridgeSmoke { + bridgeKycStatus { + status + } + bridgeVirtualAccount { + id + status + } + bridgeExternalAccounts { + id + status + } +} +``` + +### 3. Deploy Webhook Server + +The package script for local/dev execution is: + +```bash +yarn bridge-webhook +``` + +Production should run the same server entrypoint with the production config and process supervisor. The server starts from: + +```text +src/servers/bridge-webhook-server.ts +``` + +Current routes: + +- `GET /health` +- `POST /kyc` +- `POST /deposit` +- `POST /transfer` +- `POST /internal/replay` + +Health check: + +```bash +curl -fsS "$BRIDGE_WEBHOOK_URL/health" +``` + +Expected response: + +```json +{"status":"ok","service":"bridge-webhook"} +``` + +### 4. Verify Webhook Signatures + +Bridge webhooks must include: + +- `X-Webhook-Signature` + +The expected header format is: + +```text +X-Webhook-Signature: t=,v0= +``` + +Flash is intended to verify the Bridge signature for `.` using RSA-SHA256 and the configured public key for the event family. Before production cutover, verify this behavior with a Bridge-provided fixture or captured sandbox webhook because the current middleware must match Bridge's exact signing input. Track this through ENG-402. + +Expected behavior: + +- `200`: accepted or idempotent duplicate. +- `400`: malformed payload. +- `401`: invalid or missing signature/timestamp. +- `500`: retryable server failure. + +If signature failures spike after deploy, treat it as a config or raw-body parsing incident until proven otherwise. + +## Replay Missed Bridge Events + +Replay is for missed or failed Bridge webhooks. Always dry-run first. + +Build first: + +```bash +yarn build +``` + +Production replay must use the target environment config. Do not use `dev/config/base-config.yaml` for production; that file is for local/non-production testing. + +Before any replay, record these values in the operator log: + +```bash +test -n "$PROD_CONFIG_PATH" +test -n "$BRIDGE_WEBHOOK_URL" +git rev-parse HEAD +printf 'configPath=%s\nwebhookUrl=%s\ndryRun=%s\n' \ + "$PROD_CONFIG_PATH" "$BRIDGE_WEBHOOK_URL" "true" +``` + +Also confirm the config snapshot's `bridge.baseUrl` and environment name match the target environment before continuing. + +Dry run: + +```bash +BRIDGE_WEBHOOK_REPLAY_SECRET="$BRIDGE_WEBHOOK_REPLAY_SECRET" \ +BRIDGE_WEBHOOK_URL="$BRIDGE_WEBHOOK_URL" \ +node lib/scripts/replay-bridge-events.js \ + --configPath "$PROD_CONFIG_PATH" \ + --start 2026-05-01T00:00:00Z \ + --end 2026-05-02T00:00:00Z \ + --event-type deposit \ + --dry-run \ + --operator "ops@example.com" +``` + +Live replay: + +```bash +BRIDGE_WEBHOOK_REPLAY_SECRET="$BRIDGE_WEBHOOK_REPLAY_SECRET" \ +BRIDGE_WEBHOOK_URL="$BRIDGE_WEBHOOK_URL" \ +node lib/scripts/replay-bridge-events.js \ + --configPath "$PROD_CONFIG_PATH" \ + --start 2026-05-01T00:00:00Z \ + --end 2026-05-02T00:00:00Z \ + --event-type deposit \ + --operator "ops@example.com" +``` + +Useful filters: + +- `--event-type kyc` +- `--event-type deposit` +- `--event-type transfer` +- `--transfer-id ` + +Replay writes to the replay log and posts to `/internal/replay`. If `/internal/replay` returns `503`, the replay secret is not configured on the webhook server. + +## Reconciliation + +Run reconciliation after deploy, after replay, and during incidents involving deposits. + +```bash +yarn build +node lib/scripts/reconcile-bridge-ibex-deposits.js \ + --configPath "$PROD_CONFIG_PATH" \ + --window-hours 24 +``` + +Default `--window-hours` is `0.25` or 15 minutes. Use a larger window for deploy verification or after an outage. + +Admin query: + +```graphql +query BridgeOrphans { + bridgeReconciliationOrphans(limit: 50, orphanType: null, status: null) { + id + orphanType + status + amount + createdAt + updatedAt + } +} +``` + +Primary orphan classes: + +- `bridge_without_ibex`: Bridge says a deposit completed, but IBEX credit is missing or delayed. +- `ibex_without_bridge`: IBEX has a deposit that Flash cannot match to a Bridge deposit. + +Do not manually credit or reverse funds from an orphan without a written repair plan and a second reviewer. + +## Cash Wallet Cutover + +ENG-345 introduces client-aware wallet presentation during the cash wallet migration. + +Compatibility rule: + +- `PRE`: old clients and capable clients both see legacy `USD`. +- `IN_PROGRESS`: not-started accounts stay legacy `USD`; active migrations may block; accounts with migration status `complete` or `skipped_already_migrated` show `USDT` to capable clients and legacy-compatible `USD` to old clients. +- `COMPLETE`: capable clients see `USDT`; old clients continue to see legacy-compatible `USD`. + +Operator Bruno files from ENG-345: + +- `dev/bruno/Flash GraphQL API/admin/cash-wallet-cutover/01-query-admin-state.bru` +- `dev/bruno/Flash GraphQL API/admin/cash-wallet-cutover/02-set-scheduled-pre.bru` +- `dev/bruno/Flash GraphQL API/admin/cash-wallet-cutover/03-set-in-progress.bru` +- `dev/bruno/Flash GraphQL API/admin/cash-wallet-cutover/04-set-complete.bru` +- `dev/bruno/Flash GraphQL API/notoken/queries/cash-wallet-cutover.bru` + +The no-token Bruno file is an operator smoke check. Client-facing docs and support snippets should use the minimal public query below unless wider operational field exposure has been explicitly approved. + +Cutover states: + +- `PRE`: migration not active, schedule metadata may be present. +- `IN_PROGRESS`: migration is running or being actively validated. +- `COMPLETE`: cutover completed and client-aware presentation remains active for compatibility. + +Admin state query: + +```graphql +query CashWalletCutoverAdminState { + cashWalletCutover { + state + scheduledAt + startedAt + completedAt + pausedAt + pauseReason + cutoverVersion + runId + updatedBy + updatedAt + } +} +``` + +Public/client state query: + +```graphql +query CashWalletCutoverPublicState { + cashWalletCutover { + state + scheduledAt + } +} +``` + +The schema may expose additional operational fields for shared resolver reuse. Do not rely on, publish, or paste `runId`, `updatedBy`, `updatedAt`, `pauseReason`, or migration audit fields in client-facing artifacts without explicit approval. + +### Cutover Preflight + +Before moving to `IN_PROGRESS`: + +1. Confirm Bridge GraphQL smoke tests pass. +2. Confirm webhook `/health` passes. +3. Run Bridge deposit reconciliation. +4. Confirm no unresolved high-severity `bridge_without_ibex` orphan. +5. Confirm old-client no-header wallet query returns legacy `USD`. +6. Confirm capable-client wallet query also returns legacy `USD`; `USDT` should not appear before cutover starts unless using a separately completed canary environment. +7. Confirm support and mobile teams know the scheduled window. +8. Save the exact run ID, operator, commit SHA, and config snapshot location. + +### Cutover Execution + +Set the operator variables once and reuse them: + +```bash +export CUTOVER_VERSION=345 +export CUTOVER_RUN_ID="" +export OPERATOR="" +export PROD_CONFIG_PATH="" +``` + +1. Set `PRE` with schedule metadata using the admin `cashWalletCutoverUpdate` mutation or the Bruno operator file `02-set-scheduled-pre.bru`. +2. Re-run no-header and capable-client wallet smoke checks. +3. Preview and prepare migration records: + +```bash +node lib/scripts/cash-wallet-cutover.js preview \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" + +node lib/scripts/cash-wallet-cutover.js prepare \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +4. Set `IN_PROGRESS` with the CLI: + +```bash +node lib/scripts/cash-wallet-cutover.js start \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +The admin `cashWalletCutoverUpdate` mutation and Bruno `03-set-in-progress.bru` file directly mutate state and bypass CLI lifecycle checks. Use them for production `IN_PROGRESS` only as an emergency override with backend-owner approval and a written operator-log entry. Track additional start lifecycle hardening through ENG-403. + +5. Run migration batches and inspect status after each batch: + +```bash +node lib/scripts/cash-wallet-cutover.js run-batch \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" \ + --worker-id "$OPERATOR-manual-1" \ + --limit 25 + +node lib/scripts/cash-wallet-cutover.js status \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +Repeat `run-batch` until `status` shows no remaining runnable migration records. Preserve every JSON output in the operator log. + +6. Monitor batch/checkpoint logs and GraphQL error rate. +7. Re-run wallet smoke checks during migration. +8. Run reconciliation. +9. Set `COMPLETE` only after migration and presentation checks pass, using the CLI: + +```bash +node lib/scripts/cash-wallet-cutover.js complete \ + --configPath "$PROD_CONFIG_PATH" \ + --cutover-version "$CUTOVER_VERSION" \ + --run-id "$CUTOVER_RUN_ID" \ + --operator "$OPERATOR" +``` + +The admin `cashWalletCutoverUpdate` mutation and Bruno `04-set-complete.bru` file directly mutate state and bypass completion checks. Use them for production `COMPLETE` only as an emergency override with backend-owner approval, a written operator-log entry, and preserved `status` output proving why the override is safe. + +10. Re-run old-client and capable-client wallet checks after `COMPLETE`. + +For a controlled canary account, record the expected balances before execution. Example from the review canary: + +- No capability header: legacy `USD` wallet, `balance: 1000`. +- `X-Flash-Client-Capabilities: cash-wallet-usdt-v1`: `USDT` wallet, `balance: 10000000`. + +## Rollback and Pause + +Rollback is safest before irreversible wallet updates or default-wallet flips. + +If Bridge deploy fails before cutover: + +1. Disable Bridge feature access if needed. +2. Keep webhook server online if it can safely accept retries. +3. Roll back GraphQL/backend to the previous release. +4. Re-run reconciliation after recovery. + +If webhook server fails: + +1. Keep GraphQL available if user flows are safe. +2. Restore webhook route health. +3. Replay missed events by time window. +4. Run reconciliation. + +If cutover is in `PRE`: + +1. Keep state at `PRE`. +2. Fix the blocker. +3. Do not start migration. + +If cutover is `IN_PROGRESS`: + +1. Pause or stop the worker according to the cutover implementation. +2. Preserve logs, checkpoints, and run ID. +3. Do not manually mutate wallet state. +4. Decide whether to resume, complete, or write a repair plan. + +If cutover is `COMPLETE`: + +1. Treat rollback as a data repair project, not a normal deploy rollback. +2. Keep client-aware presentation intact unless it is the root cause. +3. Require a written repair plan and second reviewer before changing wallet/default state. + +## Incident Playbooks + +### Bridge API Down + +Symptoms: + +- KYC, virtual account, external account, or transfer operations fail. +- Bridge client errors rise. + +Actions: + +1. Confirm Bridge status and API base URL. +2. Pause user-facing Bridge mutations if failure rate is high. +3. Keep read-only status endpoints available if they are safe. +4. Retry only idempotent operations. +5. Reconcile when Bridge recovers. + +### Webhook Signature Failures + +Symptoms: + +- High `401` rate on `/kyc`, `/deposit`, or `/transfer`. + +Actions: + +1. Confirm public keys for the affected event family. +2. Confirm proxy preserves raw body. +3. Confirm the timestamp in `X-Webhook-Signature` is inside `timestampSkewMs`. +4. Do not bypass signature verification in production. +5. After fixing, replay the affected event window. + +### Bridge Deposit Without IBEX Credit + +Symptoms: + +- Reconciliation creates `bridge_without_ibex`. + +Actions: + +1. Confirm Bridge deposit status and transfer hash if available. +2. Check IBEX deposit ingestion delay. +3. Re-run reconciliation with a larger window. +4. If still orphaned, prepare a written repair plan. +5. Do not manually credit without approval and audit trail. + +### IBEX Deposit Without Bridge Match + +Symptoms: + +- Reconciliation creates `ibex_without_bridge`. + +Actions: + +1. Confirm whether the IBEX deposit belongs to Bridge. +2. Check for delayed Bridge event ingestion. +3. Replay Bridge events around the deposit timestamp. +4. Investigate unknown source deposits before assigning them to a user. + +### Withdrawal Stuck or Failed + +Symptoms: + +- Bridge transfer remains pending or failed. + +Actions: + +1. Query Bridge transfer state. +2. Check `/transfer` webhook delivery. +3. Replay transfer events for the affected transfer ID. +4. Confirm user-visible status matches Bridge state. +5. Escalate to Bridge support for unclear terminal states. + +### Old Clients See Wrong Wallet + +Symptoms: + +- Clients without capability header see `USDT` unexpectedly. + +Actions: + +1. Confirm request headers at GraphQL edge. +2. Query wallet list with no capability header. +3. Query with `cash-wallet-usdt-v1`. +4. Check cash wallet cutover state. +5. Roll back presentation code if compatibility is broken before data repair is needed. + +### Capable Clients Do Not See USDT + +Symptoms: + +- Clients with `X-Flash-Client-Capabilities: cash-wallet-usdt-v1` still see legacy `USD`. + +Actions: + +1. Confirm the exact header reaches GraphQL. +2. Confirm cutover state. +3. Check account wallet inventory and migration records. +4. Re-run focused wallet presentation tests. +5. Do not mark cutover `COMPLETE` until fixed. + +## Monitoring Signals + +Track these at minimum: + +- Bridge GraphQL mutation error rate. +- Bridge API latency and non-2xx rate. +- Webhook request counts by route and status. +- Webhook signature failures. +- Webhook handler exceptions. +- Replay success, failure, and skipped counts. +- Reconciliation orphan counts by type and status. +- Bridge transfer pending age. +- Cash wallet cutover state changes. +- Cash wallet cutover batch/checkpoint failures. +- Old-client wallet presentation requests. +- Capable-client wallet presentation requests. + +Alert on: + +- webhook `5xx` sustained above baseline. +- any sustained webhook `401` after deploy. +- reconciliation orphan growth. +- transfer pending age exceeding expected Bridge SLA. +- cutover worker failure during `IN_PROGRESS`. +- old-client compatibility smoke failure. + +## Post-Deploy Verification + +After deploy: + +1. `GET /health` on webhook server returns OK. +2. Bridge GraphQL smoke queries return without GraphQL errors. +3. A no-header wallet query shows legacy `USD`. +4. A capable-client wallet query matches the active phase: `USD` during `PRE`, `USDT` only for migrated accounts during `IN_PROGRESS`, and `USDT` after `COMPLETE`. +5. Replay dry-run succeeds for a narrow recent window. +6. Reconciliation completes. +7. No new unresolved high-severity orphans appear. +8. Logs show no signature, config, or Bridge API auth errors. + +## Documentation Follow-Ups + +This runbook should be updated when: + +- Production process manager names and service names are finalized. +- Exact deploy commands for the target environment are confirmed. +- Bridge dashboard webhook URLs are finalized. +- ENG-345 cutover worker command names are finalized. +- Observability dashboards and alert links exist.