fix(lifecycle): re-derive git env from the DB on config-drift recreate (ent#109) - #1913
Conversation
d981989 to
d36f12e
Compare
…109) `recreate_container_with_updated_config` seeds env from the OLD container and re-derived only `GITHUB_PAT`, replaying whatever `GITHUB_REPO` / `GIT_SYNC_*` each container happened to be carrying. The git-env derivation lived only in `_apply_persisted_auth_env` (`recreate_missing_container`). That split is a pre-existing fleet-wide bug, not a cosmetic one: the recreate has exactly one production caller, `start_agent_internal`, which fires on nine config-drift predicates AND on base-image drift at cold start — so a base-image rebuild arms the replay for every agent at once. `_apply_git_env_from_db` is now the single writer. Three load-bearing details: * **The PAT gate is a parameter, never inherited.** The two paths gate differently on purpose. `per_agent_only` (config-drift recreate) preserves #211 verbatim — resolve the effective PAT only when the container already carries one or a per-agent PAT row exists — so a global-only platform PAT is never injected into a previously-tokenless container. A verbatim lift would have swapped that for the 2-tier per-agent -> GLOBAL resolver used by `effective` (the rebuild-from-nothing path, which has no old container to inherit a token from): `configure_push_remote` then clears the push blackhole and a tokenless agent can push a private KB to the shared public upstream. learnings.md ent#162 names this class exactly. * **Set-or-clear**, since the recreate writes into a carried-forward dict. A deleted `agent_git_config` row pops the whole owned set; a `source_mode` flip clears the mode/branch pair. `GITHUB_PAT` alone stays set-only while a repo is bound — clearing it would revoke a live agent`s push on an unrelated recreate. * **`GIT_SYNC_AUTO` = DB flag OR baked env**, plus a convergence backfill. crud.py`s two writers genuinely disagree (`and not config.ephemeral` sits inside a swallowing try/except on the DB side only; the column defaults to 0), so deriving from `auto_sync_enabled` alone would silently stop auto-push for that slice of the fleet. The backfill writes the column the moment the disagreement is observed, so the OR retires itself. Making the #389 toggle authoritative is a separate follow-up. ent#123 is preserved: the gate is the REPO, not the PAT, so a tokenless agent rebuilt after container loss still clones (#843/#1439 silent-empty class). One deliberate divergence from a verbatim lift, asserted by test: a container with a baked `GITHUB_PAT` and NO git binding previously had that token refreshed from the global platform PAT on every recreate; it is now popped. The per-agent PAT is a column ON `agent_git_config`, so "no row" means no per-agent credential and no repo to push to by construction. Tests: tests/unit/test_ent109_git_env_seam.py — each of the four behaviours proved to have teeth by mutation (un-gate the PAT, flip the call site to `effective`, derive GIT_SYNC_AUTO DB-only, drop the clear sweep, drop the source-mode clear, diverge the GIT_SYNC_AUTO literal, unguard the backfill: all seven go red). Plus a static call-site guard, so flipping either gate fails CI even though no behavioural test of the helper alone would catch it. Refs Abilityai/trinity-enterprise#109 (PR 1 of 3) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ld paths (ent#109) Adds the missing agent_service/lifecycle.py catalog entry and records the per-call-site PAT gate, the set-or-clear contract, and the GIT_SYNC_AUTO OR-derivation. Amends the ent#123 clause to point at the new shared seam instead of _apply_persisted_auth_env. Refs Abilityai/trinity-enterprise#109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Refs Abilityai/trinity-enterprise#109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-sync.md: retitle the rebuild-recovery section to "Container-rebuild env — lifecycle.py::_apply_git_env_from_db" and document the per-call-site PAT gate, the set-or-clear contract, the GIT_SYNC_AUTO OR-derivation, and the two vars deliberately NOT owned. git-sync-health.md: GIT_SYNC_AUTO is re-derived on every rebuild as auto_sync_enabled OR the baked env (the two creation writers disagree), with a self-retiring backfill; kill-switch row and file table corrected. agent-lifecycle.md: Revision History row. feature-flows.md: hand-added Recent Updates row (the skill drops it past ~400 lines). Note: that table is at 56 rows against its stated ~20 cap (#1360) — pre-existing drift, deliberately not trimmed here. Refs Abilityai/trinity-enterprise#109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…it_env_from_db ent#109 moved GITHUB_PAT derivation out of the inline block in recreate_container_with_updated_config (which the guard anchored on via the comment "Update GITHUB_PAT") into the shared _apply_git_env_from_db. The guard intent is unchanged and still enforced: that block resolves the effective per-agent PAT, never the platform-only get_github_pat(). Also fixes a silent-degradation flaw in the guard itself. str.find returns -1 on a miss, and src[-1:-1+300] slices to an EMPTY string — so a moved anchor made the guard assert "get_github_pat_for_agent in \x27\x27", failing with no hint about why. The anchor is now asserted first with a message naming the fix (re-point it, do not delete it), and the block is sliced to the next top-level def rather than a fixed byte window. Both failure modes proved red by mutation: swapping the helper to get_github_pat() and renaming the anchored function. Refs Abilityai/trinity-enterprise#109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ackfill (ent#109) Two defects found reviewing the ent#109 PR 1 env seam. 1. The config-drift recreate blackholed push for agents bound post-creation. The repo half of the block is repo-gated (ent#123) while the PAT half keeps #211's narrower per-agent gate, and those two disagree for one real row shape: an agent bound via POST /{agent}/git/initialize on the GLOBAL platform PAT. That path writes an agent_git_config row and pushes, but never recreates the container, never bakes git env, never persists a per-agent PAT row, and never writes the token into the workspace .env — so its only credential is the one embedded in .git/config's origin URL, and startup.sh's #1264 fallback does not cover it. Handing startup.sh GIT_SYNC_ENABLED=true with no GITHUB_PAT is exactly what it reads as "deliberately tokenless": the restart branch rewrites origin to the credential-less CLONE_URL, destroying that token, and configure_push_remote blackholes the push remote — silently, and fleet-wide on the same base-image drift this helper exists to fix. `per_agent_only` now writes the block only when the old container already carried GITHUB_REPO or a PAT resolves. It still corrects a stale repo, a flipped source_mode and a deleted row — every case the fix is about; a tokenless ent#123 agent carries GITHUB_REPO from creation, so the flagship is unaffected. `effective` is exempt: with no old container, NOT introducing the block is the #843/#1439 silently-empty-agent bug. 2. The GIT_SYNC_AUTO backfill erased an owner's explicit disable. PUT /{agent}/git/auto-sync writes the row and nothing else while the agent gates on container env, and creation sets both true for the ordinary non-source-mode PAT agent — so "baked true / DB 0" is also exactly what an owner's disable looks like. The backfill re-enabled it on the next recreate and erased the only record of the intent, so the toggle could never stick. It was a privilege boundary too: PUT .../auto-sync is OwnedAgentByName while POST .../start, which triggers the recreate, is AuthorizedAgentByName — so a shared non-owner, or an agent-scoped key resolving to its owner with the owner's role (trinity-ops-agent#232), flipped an owner-only flag arming a 15-minute background commit-and-push loop. The OR-derivation stays (crud.py's two creation writers genuinely disagree, and DB-only derivation would silently stop auto-push for that slice). The write-back is gone; the disagreement is logged. Making the #389 toggle authoritative remains the tracked follow-up that retires the OR honestly. Tests 17 -> 22: a TestIntroduceGuard class (unbaked container untouched, carried repo still corrected, resolvable PAT still introduces, effective exempt, clear sweep unaffected) and the derive-only assertion. Both fixes proved to have teeth by mutation — removing the guard and restoring the backfill each go red on exactly one test. Two learnings.md entries. Refs trinity-enterprise#109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…urce grep
The previous call-site guard sliced `lifecycle.py` by function header and
counted a single-line literal in each half. Two blind spots:
1. It pinned only the two KNOWN sites. ent#109's bug WAS that git env had
two writers and one of them was wrong; a THIRD writer added later on any
container-seeded path re-opens exactly that hole, and the grep version
stayed green through a planted `pat_gate="effective"` writer (verified by
mutation).
2. `lifecycle.py` names the helper in two comments, so a substring count
read prose as call sites — the same first blind spot the #1871 guard hit.
The AST walk maps `{enclosing function: pat_gate literal}` and asserts the set
equals exactly `{recreate_container_with_updated_config: per_agent_only,
_apply_persisted_auth_env: effective}`. It also fails loud on a non-literal or
omitted `pat_gate` and on a duplicate call in one function — each of which
would make the guard silently vacuous, which is worse than the leak it guards.
Also drops the stale "convergence backfill" wording from the module docstring
and the registry entry (d8da9d08 removed the backfill; the description still
described it) and re-states the idempotence test as "the DB row is never
mutated".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e check Follow-on to the AST guard: `architecture.md` and the `agent-lifecycle.md` change log both said the static guard "fails CI if either call site flips", which understates what it now enforces. It pins the whole writer SET, so a third writer on any container-seeded path fails CI too — the property that matters, since ent#109's bug was two writers with one of them wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d36f12e to
14172ba
Compare
|
Resolve by running |
…env-seam-fix # Conflicts: # docs/memory/learnings.md
vybe
left a comment
There was a problem hiding this comment.
/validate-pr: PASS.
- Base
dev✅ · 10 files ✅ · security scan clean ✅ - No new top-level backend module.
TRINITY_GIT_BASE_URLis pre-existing on dev (already in.env.example, already read bycrud.py/git_service.py/startup.sh) — not a new config var, so no #1056-class gap introduced ✅ - Docs: architecture + 4 feature flows ✅
- Test adequacy:
test_ent109_git_env_seam.pypins the writer SET with an AST guard rather than a call-site grep — that survives refactors, which a string match would not ✅
The core call is right: recreate_container_with_updated_config re-derived only GITHUB_PAT and replayed whatever GITHUB_REPO/GIT_SYNC_* the old container carried, and since #1809 made base-image drift trigger a cold-start recreate, that replay is fleet-wide. Consolidating both rebuild paths onto one owner is the correct shape.
Two things I checked rather than took on trust:
pat_gateis a required keyword, never defaulted. The two paths gate the PAT differently and a shared default would inject a global-only platform PAT into a previously-tokenless container —per_agent_onlyreproduces #211's guard verbatim.- The correct-never-introduce guard is load-bearing, not defensive padding.
POST /{agent}/git/initializewrites a git-config row whose only credential lives in the container's.git/configorigin URL. IntroducingGIT_SYNC_ENABLED=truewith noGITHUB_PATis exactly what startup.sh reads as deliberately-tokenless — it rewrites origin to the credential-less CLONE_URL and blackholes the push remote. Gating the config-drift path on an already-carried repo avoids that; exempting the rebuild-from-nothing path is correct because there NOT introducing the block is the #843/#1439 silently-empty-agent bug.
TRINITY_GIT_BASE_URL is in .env.example but in neither compose file's backend.environment:, and the backend service uses an explicit env list with no env_file. So the self-hosted-git override is inert on a compose deploy today. Worth a separate issue; this PR's set-or-clear behaviour is coherent either way.
status-in-dev set by hand — issue-status-on-merge.yml is same-repo only.
#1913 (this branch's parent) was squash-merged to dev, so its 9 commits are no longer ancestors and the stack surfaced as a conflict. Resolved additively: - lifecycle.py docstring only — this branch's text is the superset (it names repo_binding's bind path as the second caller of recreate_container_with_updated_config, which this PR adds) - architecture.md kept both: this branch's repo_binding.py row plus dev's three existing agent_service rows - feature-flows.md kept both changelog rows, newest-first: ent#109 (08-02) above #1932 (08-01) - learnings.md positional conflict against an adjacent dev addition; dev added nothing in the hunk, so both sets of entries stand - registry.json this branch's 5 new entries; dev's 104 all retained (109 total) Verified: zero conflict markers tree-wide, registry.json parses, lifecycle.py compiles, and `git diff origin/dev` shows no deletions in any of the four docs.
dev has since taken #1913/#1937/#1947/#1949/#1899. Seven conflicts, resolved so that no side's change is lost: SOURCE - static_checks.py the one real semantic conflict. ent#128 (#1899) flipped the per-check swallow from _skip to _fail so a crashed check is counted by _counts; ent#89 kept _skip and added logging. Taking this branch's side verbatim would have silently reverted the HARD-count fix. Merged: _fail from #1899 + logger.error(exc_info=True) from ent#89, which is strictly more diagnostic than the logger.warning it replaces. The docstring directly above already asserts "a check that could not evaluate is not a check that passed". - template_service three hunks, all adjacent additions: both import blocks kept (template_schedules got its own statement — the two sides shared a closing paren), both new functions kept, and both pre-literal computations kept at each of the two call sites. - crud.py import list, both symbols kept. DOCS - architecture.md two hunks where BOTH sides had edited the same three bullets (template_service / fork_to_own / crud). Not a pick — each line was 3-way merged at word granularity against the merge base; no edit pairs overlapped, so both sides' text survives verbatim. dev's bullet order preserved, ent#89's new template_schedules.py bullet appended. - feature-flows.md all rows kept, table stays reverse-chronological. - learnings.md both sets of entries kept. - registry.json both entry lists kept. The conflict opened after a bare '{' and closed before a bare '}', so each side was an object BODY -- a naive concatenation produced invalid JSON. Re-added the '},{' separator. 109 -> 112 entries, none dropped. Verified: zero markers tree-wide, registry.json parses (112 entries), all three touched modules compile, and every deletion vs origin/dev is one of ent#89's own intended replacements (crud docstring three->four, the cron helper replaced by the shared validator, T-018 wired into the dispatch map).
Blast radius first
This changes what every agent container recreate produces, fleet-wide — it is not a refactor.
recreate_container_with_updated_configseedsenv_varsfrom the old container and, before this PR, re-derived onlyGITHUB_PAT. Everything else in the GitHub-sync block —GITHUB_REPO,GIT_SYNC_ENABLED,GIT_SOURCE_MODE,GIT_SOURCE_BRANCH,GIT_SYNC_AUTO— was replayed from whatever that container happened to be carrying. The actual derivation lived only on the other rebuild path (_apply_persisted_auth_env→recreate_missing_container).That path has exactly one production caller:
start_agent_internal. It fires on nine config-drift predicates and on base-image drift at cold start (#1809) — so a base-image rebuild makes every cold start a recreate, and arms the replay for the entire fleet on the next release. A stale repo binding, asource_modeflip, or a deletedagent_git_configrow survived indefinitely._apply_git_env_from_db(agent_name, env_vars, *, pat_gate)is now the single owner of that block on both rebuild paths.The two regressions this deliberately prevents
The naive fix — lift the block verbatim from the rebuild-from-nothing path into the config-drift path — ships two bugs. Both are guarded here.
1. Global-PAT injection into tokenless containers (the ent#162 class).
The two paths gate the PAT differently on purpose.
_apply_persisted_auth_envuses the 2-tierget_github_pat_for_agent(per-agent → global platform PAT); the config-drift recreate deliberately keeps #211's narrower per-agent-only gate. A verbatim lift would share the broad gate and bake the platform PAT into every previously-tokenless agent on its next drift recreate.configure_push_remotethen clears the push blackhole — and a tokenless ent#123 agent (Cornelius) could push a user's private knowledge base to the shared public upstream.So
pat_gateis a required keyword parameter, never a default:per_agent_onlyfor config drift,effectivefor rebuild-from-nothing. An AST guard pins the writer set —{recreate_container_with_updated_config: per_agent_only, _apply_persisted_auth_env: effective}exactly — so CI fails on a flipped gate and on a third writer appearing on any container-seeded path. That count is load-bearing: ent#109's bug was two writers with one of them wrong. (A source-grep version of this guard stayed green through a planted thirdeffectivewriter; verified by mutation.)2.
GIT_SYNC_AUTOstripped where the DB flag and baked env disagree.crud.py's two creation writers genuinely disagree — the DB opt-in carriesand not config.ephemeralinside a swallowingtry/exceptwhile_apply_github_envdoes not, and the column defaults to0. Soenv=true / DB=0is reachable from one transient DB hiccup at creation and permanently for ghosts. Deriving from the column alone would silently stop auto-push for that slice of the fleet — no error, just a staleagent_sync_state. HenceDB auto_sync_enabled OR baked env.The OR is derive-only — the column is never written back. A "convergence backfill" was implemented and then removed, because
PUT /{agent}/git/auto-sync {enabled:false}writes the row and nothing else while the agent gates on container env — sobaked true / DB 0is also exactly what an owner's explicit disable looks like, and no backfill can tell them apart. It would erase that intent and make the toggle unable to ever stick. It was also a privilege escalation:PUT .../auto-syncisOwnedAgentByName, butPOST /api/agents/{name}/start— the recreate's trigger — isAuthorizedAgentByName. The disagreement is logged, not resolved.Also in scope
_GIT_ENV_KEYS, because the config-drift path writes into a carried-forward dict. A deletedagent_git_configrow pops the whole set (including an orphanedGITHUB_PAT— the per-agent token is a column on that row, so no row ⇒ no credential and no repo); asource_modeflip clears the mode/branch pair.GITHUB_PATstays set-only while a repo is bound, so an unrelated recreate never revokes a live agent's push.POST /{agent}/git/initializeon the global PAT, whose only credential lives in the container's.git/configorigin URL. Handing startup.shGIT_SYNC_ENABLED=truewith noGITHUB_PATis precisely the input it reads as "deliberately tokenless" — the restart branch rewrites origin to the credential-lessCLONE_URL, destroying that token, and blackholes the push remote. So this path only ever corrects env the container already carries.effectiveis exempt: with no old container, not introducing the block is the bug: GET /api/templates omits local templates from config/agent-templates/ #843/bug: GitHub-template agent deploy — race intermittently leaves a silent empty agent (clone into non-empty /home/developer, exit 128, not surfaced) #1439 silently-empty-agent bug.GIT_WORKING_BRANCH(read only by startup.sh's clone branch, which a recreate never reaches) andGIT_UPSTREAM_REPO(no DB column). Making the Sync health observability (S1) #389 toggle authoritative is a tracked follow-up.Tests
tests/unit/test_ent109_git_env_seam.py— 22 tests, registered intests/registry.json. Every behaviour was proved to have teeth by mutation: un-gate the PAT, flip the call site toeffective, plant a third writer, deriveGIT_SYNC_AUTODB-only, drop the clear sweep, drop the source-mode clear, restore the backfill, remove the introduce-guard — all go red.test_github_pat_fallback.py's #735 call-site guard is re-anchored onto_apply_git_env_from_db(itsstr.findmiss previously returned-1, silently slicing from the end of the file and asserting against an empty string).Verification: full unit suite, deterministic order — 5872 passed, 0 failed (baseline
origin/dev: 5850 passed; delta is exactly the 22 new tests). Ruff on the changed files is byte-identical to baseline.tests/lint_sys_modules.py: no new violations.Issue references
Refs Abilityai/trinity-enterprise#109 — context only. This PR is PR 1 of 3 (the env seam); it does not implement the ent#109 feature and must not close it.
No public bug issue exists for this defect — I searched the public tracker by title and body (
GITHUB_REPO,GIT_SYNC_AUTO,_apply_persisted_auth_env,recreate_container_with_updated_config) and found only adjacent base-image-drift bugs (#1809, #1816, #1860). Rather than invent a closing reference, this PR carries none; happy to file one and re-point it if you'd prefer the fix tracked publicly.Not included
No
VERSIONbump, no changelog. No schema change, no migration, no new config, no endpoint change.🤖 Generated with Claude Code