Skip to content

fix(cloud): stop the provider-sync engine dispose/create loop (active workspace reload churn) - #2530

Merged
benjaminshafii merged 2 commits into
devfrom
fix/provider-sync-reload-loop
Jul 7, 2026
Merged

fix(cloud): stop the provider-sync engine dispose/create loop (active workspace reload churn)#2530
benjaminshafii merged 2 commits into
devfrom
fix/provider-sync-reload-loop

Conversation

@benjaminshafii

@benjaminshafii benjaminshafii commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

Fixes the customer-reported engine lifecycle loop where the active workspace's OpenCode instance is disposed and re-created continuously (~1x/sec), with the status bar stuck flashing "Reloading OpenCode config" and the /event SSE cycling connected → disconnected at ~1 Hz. Matches the field report from Go Autonomous (0.17.8-alpha + 0.17.13) and the customer's hunch that it only happens when org inference providers are configured.

Root cause (verified live on a Daytona repro)

The loop is a cloud-provider sync reconcile feedback cycle:

  1. POST /workspace/:id/desktop-cloud-sync erased the provider import baseline on every call (primary): it injected cloudImports from the cloud-plugin DB — whose providers record is always empty — and wrote that whole object back into the workspace config, wiping cloudImports.providers that the provider import had just persisted. The app calls this endpoint right after every provider import, so the baseline was erased immediately after being written. Every sync pass then saw "never imported" and re-imported all org providers.
  2. PATCH /workspace/:id/config emitted a config reload event unconditionally — even for a byte-identical no-op provider patch — so every re-import forced an engine reload (dispose→create of the active workspace instance).
  3. The session provider-auth store was recreated on every reload flip: its useMemo depended on the reloadCoordinator context value, whose identity changes on each reloadPending/reloadBusy flip. Every fresh store immediately started another cloud sync pass (empty sync-context key) → goto 1. This is why the loop followed the active workspace and background workspaces stayed stable.
  4. Contributing hardening gaps: isCloudProviderOutOfSync compared trimmed baseline model ids against raw live ids (whitespace/empty ids ⇒ permanently out-of-sync), and a failed baseline read was treated as "no imports" (re-import storm) instead of aborting the pass.

Diagnostic fingerprint on the repro box: runtime_opencode_configs had the provider, while openwork_workspace_configs.cloudImports.providers stayed {} forever, and den-api logs showed continuous GET /v1/llm-providers/:id/connect reconciles.

Fixes

  • server / desktop-cloud-sync: only persist desktopCloudSync state (via the merge updater); the provider import baseline in the workspace config is preserved, and providers now diff against the real baseline.
  • server / PATCH config: writeRuntimeOpencodeConfig detects no-op writes (skips the upsert and the write listeners) and the route only emits a reload event when the runtime config actually changed.
  • app / session provider-auth: depend on the stable markReloadRequired callback instead of the whole coordinator context value — the store is no longer recreated on reload flips (settings route already did this).
  • app / cloud-provider-config: symmetric model-id comparison (trim/filter both sides).
  • app / provider sync: abort the pass when the import baseline cannot be read instead of re-importing everything.

Reproduction & proof (Daytona, seeded cloud, signed in as Alex/Acme)

Setup: seeded OpenWork Cloud (Acme Robotics demo org, alex@acme.test) on a Daytona server sandbox, one custom org LLM provider ("Acme Azure Foundry", 2 models — mirroring the customer's Azure Foundry custom provider), Electron desktop in a second Daytona sandbox connected to that Den.

metric (same Den, same org, same provider, same flow) dev (pre-fix) this branch
engine creating instance for the active workspace 2001 in ~26.5 min (~1.26/sec, unbounded) 4 total (workspace open + one legitimate import reload), then silent
den-api GET /llm-providers/:id/connect reconciles continuous 1 per import
workspace config cloudImports.providers {} forever (erased each sync) baseline persisted
status bar "Reloading OpenCode config" churn quiet — 30/30 samples over 60 s clean

The pre-fix loop only stopped when the sandbox's Den connectivity was cut (sync pass fails before reconciling) — confirming the mechanism.

fraimz (frame-by-frame proof with the full user journey — handoff sign-in, org pick, provider auto-import witnessed in the workspace config, 60 s quiet window) is posted as a comment below.

Tests

  • pnpm --filter openwork-server test -- src/runtime-opencode-config-store.test.ts src/runtime-config-patch-reload.e2e.test.ts src/desktop-cloud-sync-preserves-imports.e2e.test.ts src/desktop-cloud-sync.test.ts12 pass, 0 fail (includes new: no-op write skips listeners; identical provider PATCH emits no second reload event; desktop-cloud-sync preserves cloudImports.providers)
  • pnpm --filter @openwork/app exec bun test src/react-app/domains/connections/provider-auth/cloud-provider-config.test.ts4 pass, 0 fail (new: symmetric out-of-sync comparison incl. whitespace model ids)
  • pnpm --filter openwork-server typecheck / pnpm --filter @openwork/app typecheck — clean
  • pnpm fraimz --flow provider-sync-stable-engine --cdp-url <daytona-electron>PASSED (5/5 frames)

Follow-up

The settings-open engine reloads (cloud-MCP re-mint per settings mount, provider-sync target race, workspace re-select) are fixed in #2544.

The active workspace's opencode instance was disposed/re-created ~1x/sec
when an org had cloud LLM providers configured. Root cause chain:

- POST /workspace/:id/desktop-cloud-sync clobbered the provider import
  baseline (cloudImports.providers) in the workspace config with the
  plugin DB copy (always empty), so every cloud provider sync pass
  re-imported all org providers
- each re-import PATCHed an identical provider block, and the route
  emitted a config reload event unconditionally, forcing an engine
  reload per pass
- every reloadPending/reloadBusy flip changed the reloadCoordinator
  context identity, recreating the session provider-auth store, whose
  fresh sync-context key immediately started the next sync pass

Fixes:
- desktop-cloud-sync only updates desktopCloudSync state; the provider
  baseline in the workspace config is preserved (and providers now diff
  against the real baseline)
- PATCH /workspace/:id/config emits a reload event only when the runtime
  opencode config actually changed; no-op writes skip listeners
- the session provider-auth store depends on the stable
  markReloadRequired callback instead of the coordinator context value
- isCloudProviderOutOfSync normalizes model ids on both sides
- cloud provider sync aborts when the import baseline cannot be read
  instead of treating it as empty and re-importing everything
Voiceover-first demo proving org cloud providers import once and the
engine connection stays stable: desktop handoff sign-in as the org
owner, Acme org pick, provider auto-import witnessed in the workspace
config (one lpr_ baseline + runtime provider), then a 60s sampled
quiet window with zero 'Reloading OpenCode config' flashes.
@vercel

vercel Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 7, 2026 8:05am
openwork-den Ready Ready Preview, Comment Jul 7, 2026 8:05am
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 7, 2026 8:05am
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 7, 2026 8:05am

@benjaminshafii

Copy link
Copy Markdown
Member Author

fraimz — ✅ PASSED

1 passed · 0 failed · 0 skipped — run 2026-07-07T07-59-19-731Z

Full frame proof with validated screenshots: evals/results/2026-07-07T07-59-19-731Z/fraimz.html (re-run: pnpm fraimz --flow provider-sync-stable-engine)

✅ provider-sync-stable-engine — Org cloud providers import once and the engine connection stays stable

User-facing flow demo

  1. A clean signed-out workspace boots to a quiet session surface

    🎙 Alex starts on a clean workspace, signed out of the cloud — the composer is idle and the status bar is quiet.

    • ✅ Visible text includes "Describe your task"
    • ✅ Visible text does not include "Reloading OpenCode config"
    • 📸 provider-sync-stable-engine-01-signed-out-session.png — 5 validations passed
    provider-sync-stable-engine-01-signed-out-session.png
  2. Desktop handoff sign-in lands on the org picker and Alex picks Acme Robotics

    🎙 He signs in to OpenWork Cloud with a pasted sign-in code and lands on the organization picker — he chooses Acme Robotics, the org whose provider used to trigger the loop.

    • ✅ Visible text includes "Acme Robotics"
    • ✅ Visible text includes "Continue with organization"
    • 📸 provider-sync-stable-engine-02-org-picker-acme.png — 6 validations passed
    provider-sync-stable-engine-02-org-picker-acme.png
  3. Org resources travel with the sign-in: AI provider models are visible before entering the workspace

    🎙 Acme's resources come with him: the onboarding summary shows the org's AI provider models, and he continues into the workspace.

    • ✅ Visible text includes "AI Providers"
    • ✅ Visible text includes "Continue to workspace"
    • 📸 provider-sync-stable-engine-03-org-resources.png — 5 validations passed
    provider-sync-stable-engine-03-org-resources.png
  4. The org provider imported once and the import baseline is persisted in the workspace config

    🎙 The org provider imported exactly once — under the hood the workspace config now remembers the import baseline, so cloud sync has nothing left to re-import.

    • ✅ Workspace config has one cloud import (lpr_01kwxmywtmfagtzpnbpgn536ph) and the runtime provider includes it.
    • 📸 provider-sync-stable-engine-04-provider-imported.png — 5 validations passed
    provider-sync-stable-engine-04-provider-imported.png
  5. Sixty seconds in the session with zero 'Reloading OpenCode config' flashes — the reload loop is gone

    🎙 The proof is in the waiting: a full minute in the session and the status bar never flashes "Reloading OpenCode config" — no dispose and re-create churn, the engine connection stays stable.

    • ✅ No Reloading OpenCode config flashes across 30 samples.
    • ✅ Visible text includes "Describe your task"
    • ✅ Visible text does not include "Reloading OpenCode config"
    • 📸 provider-sync-stable-engine-05-engine-stable.png — 6 validations passed
    provider-sync-stable-engine-05-engine-stable.png
    • ✅ Script frame 1 narrated: "Alex starts on a clean workspace, signed out of the cloud — the composer is idle and the"
    • ✅ Script frame 2 narrated: "He signs in to OpenWork Cloud with a pasted sign-in code and lands on the organization p"
    • ✅ Script frame 3 narrated: "Acme's resources come with him: the onboarding summary shows the org's AI provider model"
    • ✅ Script frame 4 narrated: "The org provider imported exactly once — under the hood the workspace config now remembe"
    • ✅ Script frame 5 narrated: "The proof is in the waiting: a full minute in the session and the status bar never flash"

@benjaminshafii
benjaminshafii merged commit bbdfa47 into dev Jul 7, 2026
12 of 14 checks passed
@benjaminshafii

Copy link
Copy Markdown
Member Author

Settings-open reload fixes — live verification (commit c59b61e)

Environment: same Daytona rig as the fraimz above — seeded Acme Robotics cloud, signed in as alex@acme.test, org provider imported and in-sync, openwork-cloud MCP stuck status=failed (unreachable minted URL — deliberately kept, since a persistently-unhealthy cloud MCP is what the reporting customer has). Engine instance churn measured via the opencode log; server writes via the embedded server request log.

Before this commit (previous head, observed)

action writes engine disposes
settings open #1 POST /mcp (re-mint) 1 reload
settings open #2 PUT auth + 2× PATCH /config (spurious full provider re-import) 3 reloads
settings open #3 POST /mcp (re-mint again — guard reset by store remount) 1 reload

After this commit (observed)

action writes engine disposes
settings open #1 POST /mcp — the one allowed per-episode re-mint attempt; marker openwork.den.mcp.unhealthyRemintAttempt persisted 1 reload
settings open #2 none 0
settings open #3 none 0
click current workspace in settings menu none (no POST /workspaces/:id/activate) 0

Engine create-count stayed flat (13 → 13) across opens #2, #3 and the workspace re-select. The openwork-cloud entry still reports failed (genuinely unreachable here); the re-mint retries again only when the entry reconnects (new episode) or via the user-facing Refresh (force clears the marker).

Tests added: workspace-activate.e2e.test.ts (switch reloads once, re-activate doesn't — engine stub counts /instance/dispose), cloud-mcp-user-state.test.ts (episode marker helpers). All targeted suites + typechecks pass; commands in the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant