Skip to content

fix(lifecycle): re-derive git env from the DB on config-drift recreate (ent#109) - #1913

Merged
vybe merged 9 commits into
devfrom
AndriiPasternak31/git-env-seam-fix
Aug 3, 2026
Merged

fix(lifecycle): re-derive git env from the DB on config-drift recreate (ent#109)#1913
vybe merged 9 commits into
devfrom
AndriiPasternak31/git-env-seam-fix

Conversation

@AndriiPasternak31

@AndriiPasternak31 AndriiPasternak31 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Blast radius first

This changes what every agent container recreate produces, fleet-wide — it is not a refactor.

recreate_container_with_updated_config seeds env_vars from the old container and, before this PR, re-derived only GITHUB_PAT. Everything else in the GitHub-sync block — GITHUB_REPO, GIT_SYNC_ENABLED, GIT_SOURCE_MODE, GIT_SOURCE_BRANCH, GIT_SYNC_AUTO — was replayed from whatever that container happened to be carrying. The actual derivation lived only on the other rebuild path (_apply_persisted_auth_envrecreate_missing_container).

That path has exactly one production caller: start_agent_internal. It fires on nine config-drift predicates and on base-image drift at cold start (#1809) — so a base-image rebuild makes every cold start a recreate, and arms the replay for the entire fleet on the next release. A stale repo binding, a source_mode flip, or a deleted agent_git_config row survived indefinitely.

_apply_git_env_from_db(agent_name, env_vars, *, pat_gate) is now the single owner of that block on both rebuild paths.

Rebased onto #1912 (fix: fleet restart adopts rebuilt base images through the canonical lifecycle path, #1860), which landed on dev mid-review — and which widens this bug's reach. That PR adds restart_agent_internal (explicit stop → start_agent_internal) so a fleet "Restart All" reliably performs a cold start. Cold start is exactly the path that runs the #1809 image-drift predicate and therefore recreate_container_with_updated_config. So after #1912, the git-env replay this PR removes is reachable from an ordinary operator "Restart All", not just from a base-image rebuild. The two changes are additive (no shared writer), and this branch keeps #1912's helper, its registry entry, and its docs rows intact; test_1860_fleet_restart_adoption.py, test_1809_image_drift_recreate.py and both test_1816_* suites pass alongside the ent#109 suite (154 tests).

The two regressions this deliberately prevents

The naive fix — lift the block verbatim from the rebuild-from-nothing path into the config-drift path — ships two bugs. Both are guarded here.

1. Global-PAT injection into tokenless containers (the ent#162 class).
The two paths gate the PAT differently on purpose. _apply_persisted_auth_env uses the 2-tier get_github_pat_for_agent (per-agent → global platform PAT); the config-drift recreate deliberately keeps #211's narrower per-agent-only gate. A verbatim lift would share the broad gate and bake the platform PAT into every previously-tokenless agent on its next drift recreate. configure_push_remote then clears the push blackhole — and a tokenless ent#123 agent (Cornelius) could push a user's private knowledge base to the shared public upstream.

So pat_gate is a required keyword parameter, never a default: per_agent_only for config drift, effective for rebuild-from-nothing. An AST guard pins the writer set{recreate_container_with_updated_config: per_agent_only, _apply_persisted_auth_env: effective} exactly — so CI fails on a flipped gate and on a third writer appearing on any container-seeded path. That count is load-bearing: ent#109's bug was two writers with one of them wrong. (A source-grep version of this guard stayed green through a planted third effective writer; verified by mutation.)

2. GIT_SYNC_AUTO stripped where the DB flag and baked env disagree.
crud.py's two creation writers genuinely disagree — the DB opt-in carries and not config.ephemeral inside a swallowing try/except while _apply_github_env does not, and the column defaults to 0. So env=true / DB=0 is reachable from one transient DB hiccup at creation and permanently for ghosts. Deriving from the column alone would silently stop auto-push for that slice of the fleet — no error, just a stale agent_sync_state. Hence DB auto_sync_enabled OR baked env.

The OR is derive-only — the column is never written back. A "convergence backfill" was implemented and then removed, because PUT /{agent}/git/auto-sync {enabled:false} writes the row and nothing else while the agent gates on container env — so baked true / DB 0 is also exactly what an owner's explicit disable looks like, and no backfill can tell them apart. It would erase that intent and make the toggle unable to ever stick. It was also a privilege escalation: PUT .../auto-sync is OwnedAgentByName, but POST /api/agents/{name}/start — the recreate's trigger — is AuthorizedAgentByName. The disagreement is logged, not resolved.

Also in scope

  • Set-or-clear over _GIT_ENV_KEYS, because the config-drift path writes into a carried-forward dict. A deleted agent_git_config row pops the whole set (including an orphaned GITHUB_PAT — the per-agent token is a column on that row, so no row ⇒ no credential and no repo); a source_mode flip clears the mode/branch pair. GITHUB_PAT stays set-only while a repo is bound, so an unrelated recreate never revokes a live agent's push.
  • Correct, never introduce (config-drift path only). The repo half is repo-gated (ent#123) while the PAT half keeps feat: Auto-propagate GitHub PAT to all agents on update #211's per-agent gate, and those two disagree for one real row shape: an agent bound post-creation via POST /{agent}/git/initialize on the global PAT, whose only credential lives in the container's .git/config origin URL. Handing startup.sh GIT_SYNC_ENABLED=true with no GITHUB_PAT is precisely the input it reads as "deliberately tokenless" — the restart branch rewrites origin to the credential-less CLONE_URL, destroying that token, and blackholes the push remote. So this path only ever corrects env the container already carries. effective is exempt: with no old container, not introducing the block is the bug: GET /api/templates omits local templates from config/agent-templates/ #843/bug: GitHub-template agent deploy — race intermittently leaves a silent empty agent (clone into non-empty /home/developer, exit 128, not surfaced) #1439 silently-empty-agent bug.
  • Not owned here, deliberately: GIT_WORKING_BRANCH (read only by startup.sh's clone branch, which a recreate never reaches) and GIT_UPSTREAM_REPO (no DB column). Making the Sync health observability (S1) #389 toggle authoritative is a tracked follow-up.

Tests

tests/unit/test_ent109_git_env_seam.py — 22 tests, registered in tests/registry.json. Every behaviour was proved to have teeth by mutation: un-gate the PAT, flip the call site to effective, plant a third writer, derive GIT_SYNC_AUTO DB-only, drop the clear sweep, drop the source-mode clear, restore the backfill, remove the introduce-guard — all go red.

test_github_pat_fallback.py's #735 call-site guard is re-anchored onto _apply_git_env_from_db (its str.find miss previously returned -1, silently slicing from the end of the file and asserting against an empty string).

Verification: full unit suite, deterministic order — 5872 passed, 0 failed (baseline origin/dev: 5850 passed; delta is exactly the 22 new tests). Ruff on the changed files is byte-identical to baseline. tests/lint_sys_modules.py: no new violations.

Pre-existing, not from this PR: test_1081_physical_meter.pytest_telegram_webhook_backfill.py are order-dependent under pytest-randomly. Reproduced identically on pristine origin/dev; my new file provably does not pollute them.

Issue references

Refs Abilityai/trinity-enterprise#109 — context only. This PR is PR 1 of 3 (the env seam); it does not implement the ent#109 feature and must not close it.

No public bug issue exists for this defect — I searched the public tracker by title and body (GITHUB_REPO, GIT_SYNC_AUTO, _apply_persisted_auth_env, recreate_container_with_updated_config) and found only adjacent base-image-drift bugs (#1809, #1816, #1860). Rather than invent a closing reference, this PR carries none; happy to file one and re-point it if you'd prefer the fix tracked publicly.

Not included

No VERSION bump, no changelog. No schema change, no migration, no new config, no endpoint change.

🤖 Generated with Claude Code

@AndriiPasternak31
AndriiPasternak31 force-pushed the AndriiPasternak31/git-env-seam-fix branch 3 times, most recently from d981989 to d36f12e Compare July 31, 2026 14:02
AndriiPasternak31 and others added 8 commits July 31, 2026 15:40
…109)

`recreate_container_with_updated_config` seeds env from the OLD container and
re-derived only `GITHUB_PAT`, replaying whatever `GITHUB_REPO` / `GIT_SYNC_*`
each container happened to be carrying. The git-env derivation lived only in
`_apply_persisted_auth_env` (`recreate_missing_container`). That split is a
pre-existing fleet-wide bug, not a cosmetic one: the recreate has exactly one
production caller, `start_agent_internal`, which fires on nine config-drift
predicates AND on base-image drift at cold start — so a base-image rebuild
arms the replay for every agent at once.

`_apply_git_env_from_db` is now the single writer. Three load-bearing details:

* **The PAT gate is a parameter, never inherited.** The two paths gate
  differently on purpose. `per_agent_only` (config-drift recreate) preserves
  #211 verbatim — resolve the effective PAT only when the container already
  carries one or a per-agent PAT row exists — so a global-only platform PAT is
  never injected into a previously-tokenless container. A verbatim lift would
  have swapped that for the 2-tier per-agent -> GLOBAL resolver used by
  `effective` (the rebuild-from-nothing path, which has no old container to
  inherit a token from): `configure_push_remote` then clears the push
  blackhole and a tokenless agent can push a private KB to the shared public
  upstream. learnings.md ent#162 names this class exactly.
* **Set-or-clear**, since the recreate writes into a carried-forward dict. A
  deleted `agent_git_config` row pops the whole owned set; a `source_mode`
  flip clears the mode/branch pair. `GITHUB_PAT` alone stays set-only while a
  repo is bound — clearing it would revoke a live agent`s push on an unrelated
  recreate.
* **`GIT_SYNC_AUTO` = DB flag OR baked env**, plus a convergence backfill.
  crud.py`s two writers genuinely disagree (`and not config.ephemeral` sits
  inside a swallowing try/except on the DB side only; the column defaults to
  0), so deriving from `auto_sync_enabled` alone would silently stop auto-push
  for that slice of the fleet. The backfill writes the column the moment the
  disagreement is observed, so the OR retires itself. Making the #389 toggle
  authoritative is a separate follow-up.

ent#123 is preserved: the gate is the REPO, not the PAT, so a tokenless agent
rebuilt after container loss still clones (#843/#1439 silent-empty class).

One deliberate divergence from a verbatim lift, asserted by test: a container
with a baked `GITHUB_PAT` and NO git binding previously had that token
refreshed from the global platform PAT on every recreate; it is now popped.
The per-agent PAT is a column ON `agent_git_config`, so "no row" means no
per-agent credential and no repo to push to by construction.

Tests: tests/unit/test_ent109_git_env_seam.py — each of the four behaviours
proved to have teeth by mutation (un-gate the PAT, flip the call site to
`effective`, derive GIT_SYNC_AUTO DB-only, drop the clear sweep, drop the
source-mode clear, diverge the GIT_SYNC_AUTO literal, unguard the backfill:
all seven go red). Plus a static call-site guard, so flipping either gate
fails CI even though no behavioural test of the helper alone would catch it.

Refs Abilityai/trinity-enterprise#109 (PR 1 of 3)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ld paths (ent#109)

Adds the missing agent_service/lifecycle.py catalog entry and records the
per-call-site PAT gate, the set-or-clear contract, and the GIT_SYNC_AUTO
OR-derivation. Amends the ent#123 clause to point at the new shared seam
instead of _apply_persisted_auth_env.

Refs Abilityai/trinity-enterprise#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Refs Abilityai/trinity-enterprise#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-sync.md: retitle the rebuild-recovery section to
"Container-rebuild env — lifecycle.py::_apply_git_env_from_db" and document
the per-call-site PAT gate, the set-or-clear contract, the GIT_SYNC_AUTO
OR-derivation, and the two vars deliberately NOT owned.

git-sync-health.md: GIT_SYNC_AUTO is re-derived on every rebuild as
auto_sync_enabled OR the baked env (the two creation writers disagree), with a
self-retiring backfill; kill-switch row and file table corrected.

agent-lifecycle.md: Revision History row.
feature-flows.md: hand-added Recent Updates row (the skill drops it past ~400
lines). Note: that table is at 56 rows against its stated ~20 cap (#1360) —
pre-existing drift, deliberately not trimmed here.

Refs Abilityai/trinity-enterprise#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…it_env_from_db

ent#109 moved GITHUB_PAT derivation out of the inline block in
recreate_container_with_updated_config (which the guard anchored on via the
comment "Update GITHUB_PAT") into the shared _apply_git_env_from_db. The
guard intent is unchanged and still enforced: that block resolves the
effective per-agent PAT, never the platform-only get_github_pat().

Also fixes a silent-degradation flaw in the guard itself. str.find returns
-1 on a miss, and src[-1:-1+300] slices to an EMPTY string — so a moved
anchor made the guard assert "get_github_pat_for_agent in \x27\x27", failing with no
hint about why. The anchor is now asserted first with a message naming the
fix (re-point it, do not delete it), and the block is sliced to the next
top-level def rather than a fixed byte window.

Both failure modes proved red by mutation: swapping the helper to
get_github_pat() and renaming the anchored function.

Refs Abilityai/trinity-enterprise#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ackfill (ent#109)

Two defects found reviewing the ent#109 PR 1 env seam.

1. The config-drift recreate blackholed push for agents bound post-creation.
   The repo half of the block is repo-gated (ent#123) while the PAT half keeps
   #211's narrower per-agent gate, and those two disagree for one real row
   shape: an agent bound via POST /{agent}/git/initialize on the GLOBAL
   platform PAT. That path writes an agent_git_config row and pushes, but never
   recreates the container, never bakes git env, never persists a per-agent PAT
   row, and never writes the token into the workspace .env — so its only
   credential is the one embedded in .git/config's origin URL, and startup.sh's
   #1264 fallback does not cover it. Handing startup.sh GIT_SYNC_ENABLED=true
   with no GITHUB_PAT is exactly what it reads as "deliberately tokenless": the
   restart branch rewrites origin to the credential-less CLONE_URL, destroying
   that token, and configure_push_remote blackholes the push remote — silently,
   and fleet-wide on the same base-image drift this helper exists to fix.

   `per_agent_only` now writes the block only when the old container already
   carried GITHUB_REPO or a PAT resolves. It still corrects a stale repo, a
   flipped source_mode and a deleted row — every case the fix is about; a
   tokenless ent#123 agent carries GITHUB_REPO from creation, so the flagship
   is unaffected. `effective` is exempt: with no old container, NOT introducing
   the block is the #843/#1439 silently-empty-agent bug.

2. The GIT_SYNC_AUTO backfill erased an owner's explicit disable.
   PUT /{agent}/git/auto-sync writes the row and nothing else while the agent
   gates on container env, and creation sets both true for the ordinary
   non-source-mode PAT agent — so "baked true / DB 0" is also exactly what an
   owner's disable looks like. The backfill re-enabled it on the next recreate
   and erased the only record of the intent, so the toggle could never stick.
   It was a privilege boundary too: PUT .../auto-sync is OwnedAgentByName while
   POST .../start, which triggers the recreate, is AuthorizedAgentByName — so a
   shared non-owner, or an agent-scoped key resolving to its owner with the
   owner's role (trinity-ops-agent#232), flipped an owner-only flag arming a
   15-minute background commit-and-push loop.

   The OR-derivation stays (crud.py's two creation writers genuinely disagree,
   and DB-only derivation would silently stop auto-push for that slice). The
   write-back is gone; the disagreement is logged. Making the #389 toggle
   authoritative remains the tracked follow-up that retires the OR honestly.

Tests 17 -> 22: a TestIntroduceGuard class (unbaked container untouched,
carried repo still corrected, resolvable PAT still introduces, effective
exempt, clear sweep unaffected) and the derive-only assertion. Both fixes
proved to have teeth by mutation — removing the guard and restoring the
backfill each go red on exactly one test. Two learnings.md entries.

Refs trinity-enterprise#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…urce grep

The previous call-site guard sliced `lifecycle.py` by function header and
counted a single-line literal in each half. Two blind spots:

  1. It pinned only the two KNOWN sites. ent#109's bug WAS that git env had
     two writers and one of them was wrong; a THIRD writer added later on any
     container-seeded path re-opens exactly that hole, and the grep version
     stayed green through a planted `pat_gate="effective"` writer (verified by
     mutation).
  2. `lifecycle.py` names the helper in two comments, so a substring count
     read prose as call sites — the same first blind spot the #1871 guard hit.

The AST walk maps `{enclosing function: pat_gate literal}` and asserts the set
equals exactly `{recreate_container_with_updated_config: per_agent_only,
_apply_persisted_auth_env: effective}`. It also fails loud on a non-literal or
omitted `pat_gate` and on a duplicate call in one function — each of which
would make the guard silently vacuous, which is worse than the leak it guards.

Also drops the stale "convergence backfill" wording from the module docstring
and the registry entry (d8da9d08 removed the backfill; the description still
described it) and re-states the idempotence test as "the DB row is never
mutated".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e check

Follow-on to the AST guard: `architecture.md` and the `agent-lifecycle.md`
change log both said the static guard "fails CI if either call site flips",
which understates what it now enforces. It pins the whole writer SET, so a
third writer on any container-seeded path fails CI too — the property that
matters, since ent#109's bug was two writers with one of them wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AndriiPasternak31
AndriiPasternak31 marked this pull request as ready for review July 31, 2026 14:41
@AndriiPasternak31
AndriiPasternak31 force-pushed the AndriiPasternak31/git-env-seam-fix branch from d36f12e to 14172ba Compare July 31, 2026 14:41
@AndriiPasternak31 AndriiPasternak31 self-assigned this Jul 31, 2026
@AndriiPasternak31
AndriiPasternak31 requested review from dolho and vybe July 31, 2026 15:00
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

…env-seam-fix

# Conflicts:
#	docs/memory/learnings.md

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/validate-pr: PASS.

  • Base dev ✅ · 10 files ✅ · security scan clean ✅
  • No new top-level backend module. TRINITY_GIT_BASE_URL is pre-existing on dev (already in .env.example, already read by crud.py/git_service.py/startup.sh) — not a new config var, so no #1056-class gap introduced ✅
  • Docs: architecture + 4 feature flows ✅
  • Test adequacy: test_ent109_git_env_seam.py pins the writer SET with an AST guard rather than a call-site grep — that survives refactors, which a string match would not ✅

The core call is right: recreate_container_with_updated_config re-derived only GITHUB_PAT and replayed whatever GITHUB_REPO/GIT_SYNC_* the old container carried, and since #1809 made base-image drift trigger a cold-start recreate, that replay is fleet-wide. Consolidating both rebuild paths onto one owner is the correct shape.

Two things I checked rather than took on trust:

  1. pat_gate is a required keyword, never defaulted. The two paths gate the PAT differently and a shared default would inject a global-only platform PAT into a previously-tokenless container — per_agent_only reproduces #211's guard verbatim.
  2. The correct-never-introduce guard is load-bearing, not defensive padding. POST /{agent}/git/initialize writes a git-config row whose only credential lives in the container's .git/config origin URL. Introducing GIT_SYNC_ENABLED=true with no GITHUB_PAT is exactly what startup.sh reads as deliberately-tokenless — it rewrites origin to the credential-less CLONE_URL and blackholes the push remote. Gating the config-drift path on an already-carried repo avoids that; exempting the rebuild-from-nothing path is correct because there NOT introducing the block is the #843/#1439 silently-empty-agent bug.

⚠️ Non-blocking, pre-existing and NOT introduced here: TRINITY_GIT_BASE_URL is in .env.example but in neither compose file's backend.environment:, and the backend service uses an explicit env list with no env_file. So the self-hosted-git override is inert on a compose deploy today. Worth a separate issue; this PR's set-or-clear behaviour is coherent either way.

⚠️ Cross-tracker: ent#109 needs status-in-dev set by hand — issue-status-on-merge.yml is same-repo only.

@vybe
vybe merged commit c269adc into dev Aug 3, 2026
22 checks passed
vybe pushed a commit that referenced this pull request Aug 3, 2026
#1913 (this branch's parent) was squash-merged to dev, so its 9 commits are no
longer ancestors and the stack surfaced as a conflict. Resolved additively:

- lifecycle.py       docstring only — this branch's text is the superset (it
                     names repo_binding's bind path as the second caller of
                     recreate_container_with_updated_config, which this PR adds)
- architecture.md    kept both: this branch's repo_binding.py row plus dev's
                     three existing agent_service rows
- feature-flows.md   kept both changelog rows, newest-first: ent#109 (08-02)
                     above #1932 (08-01)
- learnings.md       positional conflict against an adjacent dev addition;
                     dev added nothing in the hunk, so both sets of entries stand
- registry.json      this branch's 5 new entries; dev's 104 all retained (109 total)

Verified: zero conflict markers tree-wide, registry.json parses, lifecycle.py
compiles, and `git diff origin/dev` shows no deletions in any of the four docs.
vybe pushed a commit that referenced this pull request Aug 3, 2026
dev has since taken #1913/#1937/#1947/#1949/#1899. Seven conflicts, resolved so
that no side's change is lost:

SOURCE
- static_checks.py  the one real semantic conflict. ent#128 (#1899) flipped the
                    per-check swallow from _skip to _fail so a crashed check is
                    counted by _counts; ent#89 kept _skip and added logging.
                    Taking this branch's side verbatim would have silently
                    reverted the HARD-count fix. Merged: _fail from #1899 +
                    logger.error(exc_info=True) from ent#89, which is strictly
                    more diagnostic than the logger.warning it replaces. The
                    docstring directly above already asserts "a check that could
                    not evaluate is not a check that passed".
- template_service  three hunks, all adjacent additions: both import blocks kept
                    (template_schedules got its own statement — the two sides
                    shared a closing paren), both new functions kept, and both
                    pre-literal computations kept at each of the two call sites.
- crud.py           import list, both symbols kept.

DOCS
- architecture.md   two hunks where BOTH sides had edited the same three bullets
                    (template_service / fork_to_own / crud). Not a pick — each
                    line was 3-way merged at word granularity against the merge
                    base; no edit pairs overlapped, so both sides' text survives
                    verbatim. dev's bullet order preserved, ent#89's new
                    template_schedules.py bullet appended.
- feature-flows.md  all rows kept, table stays reverse-chronological.
- learnings.md      both sets of entries kept.
- registry.json     both entry lists kept. The conflict opened after a bare '{'
                    and closed before a bare '}', so each side was an object
                    BODY -- a naive concatenation produced invalid JSON. Re-added
                    the '},{' separator. 109 -> 112 entries, none dropped.

Verified: zero markers tree-wide, registry.json parses (112 entries), all three
touched modules compile, and every deletion vs origin/dev is one of ent#89's own
intended replacements (crud docstring three->four, the cron helper replaced by
the shared validator, T-018 wired into the dispatch map).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants