Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/memory/feature-flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

| Date | ID | Feature | Flow |
|------|-----|---------|------|
| 2026-05-18 | #888 | write_user_memory MCP tool — per-user memory write with server-side email resolution, fixing PII cross-user memory leak | [write-user-memory.md](feature-flows/write-user-memory.md) |
| 2026-05-17 | #862 | fix(cleanup): execution retention sweeps were no-ops — `prune_execution_logs`/`prune_execution_rows` queried `status IN ('completed','failed','terminated')` but `TaskExecutionStatus` uses `'success'/'failed'/'cancelled'/'skipped'`; only `'failed'` rows ever pruned; fixed SQL predicates + `idx_executions_completed_terminal` partial index + migration to drop/recreate existing wrong index on live installs | [cleanup-service.md](feature-flows/cleanup-service.md) |
| 2026-05-13 | #831 | feat: platform default model — admin sets `platform_default_model` in Settings General tab; `task_execution_service.execute_task()` resolves `model=None` → platform default (TTL-cached, write-through invalidation); `GET /api/settings/feature-flags` exposes value for frontend; SchedulesPanel shows "platform default (X)" when no model set; PRESET_MODELS updated to canonical Anthropic list (Opus 4.7 / Sonnet 4.6 / Haiku 4.5) | [model-selection.md](feature-flows/model-selection.md), [platform-settings.md](feature-flows/platform-settings.md), [task-execution-service.md](feature-flows/task-execution-service.md) |
| 2026-05-12 | #808 | fix(orphan-killer): `_set_idle_priority()` (SCHED_IDLE/nice) + `_scan_deadline` 8s per-iteration budget — prevents orphan-killer daemon thread from starving uvicorn health probes on 1-CPU containers and triggering circuit breaker | [parallel-headless-execution.md](feature-flows/parallel-headless-execution.md) |
Expand Down
1 change: 1 addition & 0 deletions docs/memory/feature-flows/agent-monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -881,6 +881,7 @@ Monitoring service stopped

| Date | Changes |
|------|---------|
| 2026-05-18 | **#873 — pipe-drop responses classified as HTTP 502 on agent server** (`docker/base-image/agent_server/services/headless_executor.py:856-874`). `BrokenPipeError`/`ConnectionResetError` in `execute_headless` now raise 502 instead of 500, avoiding collision with the 503 auto-switch path in `task_execution_service.py`. From the monitoring side, 502 and 500 are equivalent: both are HTTP responses that call `circuit.record_success()` in `check_network_health()` (agent is TCP-reachable) and are then flagged `UNHEALTHY` by the `status_code >= 500` branch in `aggregate_health()`. No change to `monitoring_service.py` was required. |
| 2026-05-13 | **#474 Layer 2 — `/health` probe exception classification split** (`services/monitoring_service.py:check_network_health()`, commit d53a2d6b). Layered on top of #798 (below). Two new exception handlers inserted BEFORE the shared `TRANSIENT_TRANSPORT_EXCEPTIONS` handler so they win Python's first-match: (1) `BrokenPipeError` / `ConnectionResetError` → client-side transport drop (upstream MCP-sync cancellation cascading into a pooled keepalive socket); returns `reachable=False` with `error="Connection dropped: <ExceptionName>"` but does **NOT** call `circuit.record_failure()` — the agent's health was never observed, so it must not trip the breaker on healthy agents. (2) `httpx.ReadError` / `httpx.WriteError` / `httpx.RemoteProtocolError` → genuine agent liveness signals on a `/health` probe (partial write then socket drop = event-loop wedge, OOM mid-write, segfault); calls `circuit.record_failure()` and returns `reachable=False` with `error="HTTP transport error on /health: <ExceptionName>"`. This is the documented `/health`-specific divergence from `AgentClient._request()` — on `/api/*` paths #798's tuple-based handler keeps the same exceptions circuit-neutral because the cause space is broader. Regression test: `tests/unit/test_monitoring_health_check_classification.py`. |
| 2026-05-12 | **Circuit-breaker classification mirrored on the /health probe (#474)**: `check_network_health()` now lazy-imports `CIRCUIT_FAILURE_EXCEPTIONS` / `TRANSIENT_TRANSPORT_EXCEPTIONS` from `services/agent_client.py` and applies the same rule as inline `/api/*` requests. Any HTTP response (200..599) records circuit success so stale failure counters clear as soon as the agent answers. Only `ConnectError`/`ConnectTimeout` increment the failure counter; read-timeouts, pool exhaustion, mid-write broken-pipe/reset, and garbled framing surface as `reachable=False` but don't poison the circuit. `aggregate_health()` adds an explicit `network.status_code >= 500 → UNHEALTHY` branch (`monitoring_service.py:419`) so a wedged-but-listening agent isn't silently HEALTHY under the new rule. |
| 2026-03-03 | **SUB-002 credential monitoring removal**: Removed credential file checks from `check_business_health()`, `aggregate_health()`, and `perform_health_check()`. Removed `alert_subscription_credentials_missing()` from `monitoring_alerts.py`. Removed auto-remediation via `inject_subscription_on_start()`. `credential_status` field deprecated (always `None`). Tokens now injected as container env vars. |
Expand Down
1 change: 1 addition & 0 deletions docs/memory/feature-flows/email-authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Passwordless email-based authentication with verification codes. Users enter the
## Revision History
| Date | Changes |
|------|---------|
| 2026-05-18 | **Contextual email subjects (#890)**: `send_verification_code()` gains optional `agent_name` and `context_label` params. When `agent_name` is set, subject becomes `Your Trinity access code for "{agent_name}"`; when `context_label` is set, subject becomes `Your {context_label} verification code`. A new private `_get_verification_email_html()` generates an HTML body with inline CSS, 36px monospace code block, and contextual intro text. `auth.py` now passes `context_label="Trinity login"`; `public.py` now passes `agent_name=link["agent_name"]`. |
| 2026-03-26 | **OTP rate limiting added**: `verify_email_login_code()` now enforces both IP-based rate limiting (existing) and new per-email OTP rate limiting (5 failed attempts → 429 for 10 minutes, Redis key `otp_attempts:{email}`). `confirm_verification_code()` in public.py also gains IP-based rate limiting. New helpers: `check_otp_rate_limit(email)`, `record_otp_attempt(email, success)`. |
| 2026-03-20 | **ROLE-001**: `get_or_create_email_user()` now assigns role `"creator"` (was `"user"`) to new email-authenticated users. Existing users unaffected. See [role-model.md](role-model.md). |
| 2026-02-23 | **Related security fixes**: M-003 removed plaintext password fallback (affects admin login path). M-005 added rate limiting to admin `/token` endpoint (5 attempts per 10 minutes per IP). See [admin-login.md](admin-login.md) for details. Email auth flow unchanged but now has consistent security posture with admin login. |
Expand Down
14 changes: 14 additions & 0 deletions docs/memory/feature-flows/gemini-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,20 @@ The `tools` array in templates is informational only.

---

## Error Handling

- **Pipe-drop (`BrokenPipeError` / `ConnectionResetError`) in `execute_headless`**: raised as HTTP 502, not 500. This keeps pipe-drops out of the `agent_client.py` circuit-breaker failure counter — 4xx/5xx/502/503/504 are treated as application errors and skip the failure increment (#474/#873).

---

## Revision History

| Date | Change |
|------|--------|
| 2026-05-18 | Pipe-drop reclassification (#474/#873): `BrokenPipeError`/`ConnectionResetError` in `execute_headless` now raise HTTP 502 instead of 500, preventing false circuit-breaker trips when the Gemini child process exits early. |

---

## Related Documentation

- [Gemini Support Guide](../../GEMINI_SUPPORT.md) - User-facing setup guide
Expand Down
5 changes: 4 additions & 1 deletion docs/memory/feature-flows/parallel-headless-execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@
> **Requirement**: 12.1 - Parallel Headless Execution
> **Status**: Implemented
> **Created**: 2025-12-22
> **Updated**: 2026-05-13 (#474: agent-server classifies subprocess pipe drop (`BrokenPipeError`/`ConnectionResetError`) as HTTP 502 — not 500 — in both `headless_executor.py` and `gemini_runtime.py`; avoids SUB-003 503-auth-class collision on benign causes (auth abort, permission-mode kill, upstream cancel) and downgrades log level from ERROR to INFO; prior 2026-05-12 #808: orphan-killer SCHED_IDLE / scan deadline)
> **Updated**: 2026-05-18 (#873: pipe-drop handler ported to refactored `headless_executor.py` after #122 extraction; prior 2026-05-13 #474: original pipe-drop reclassification in both runtimes)
> **Verified**: 2026-02-05

## Revision History

| Date | Changes |
|------|---------|
| 2026-05-18 | **Issue #873 - Pipe-drop handler ported to refactored `headless_executor.py`**: After PR #122 extracted `execute_headless_task()` into its own module (`services/headless_executor.py`), the `except (BrokenPipeError, ConnectionResetError)` handler from #474 needed to be present in the new file's outer `try/except` chain. PR #873 applied the same pattern — handler at `headless_executor.py:856-874`, semantically identical to the `gemini_runtime.py:665-676` sibling — ensuring the pipe-drop → HTTP 502 reclassification and INFO-level logging survive the module split. No behaviour change; pure carry-forward of the #474 fix. `src/frontend/src/composables/useProcessWebSocket.js` was deleted as part of this cleanup (composable was no longer used after the SSE streaming surface was consolidated). |
| 2026-05-13 | **Issue #474 - Subprocess pipe-drop misclassified as HTTP 500 (and triggering SUB-003 auto-switch on the 503 sibling path)**: When the agent-server tried to write a task request to the Claude (or Gemini) subprocess and the child had already exited — typical causes: OAuth/auth abort, permission-mode validation kill, upstream cancellation by the backend's `terminate_execution_on_agent()`, or any race where the child closes stdin before the parent finishes writing — the resulting `BrokenPipeError` / `ConnectionResetError` fell through to the generic `except Exception` clause and surfaced as HTTP 500 with a misleading `[Errno 32] Broken pipe` ERROR log line. The 503 sibling shape of this failure additionally collided with SUB-003 (`services/task_execution_service.py:628`) which interprets any 503 from an agent endpoint as auth-class failure and auto-rotates the subscription — meaning a benign pipe drop could spuriously burn through subscription rotations. Fix: dedicated `except (BrokenPipeError, ConnectionResetError) as pipe_err` handler inserted *before* the generic `except Exception` clause in both runtimes — `docker/base-image/agent_server/services/headless_executor.py:814-832` (Claude path) and `docker/base-image/agent_server/services/gemini_runtime.py:665-676` (Gemini path) — both raise `HTTPException(status_code=502, detail="Agent subprocess closed before task could complete")`. 502 ("Bad Gateway to Claude subprocess") is semantically correct and collision-free: SUB-003's 503-or-auth-string predicate at `task_execution_service.py:628` ignores 502, so 502 flows through as plain FAILED without rotating the subscription. Log level downgraded to `logger.info()` (not `logger.error()`) because the pipe drop is a known benign race — ERROR-level spam misleads operators into hunting non-existent server-side faults. Sibling of #520 (502 for empty result) and #516 (504 for signal exit) — same pattern of "give the symptom a precise HTTP code so backend classification doesn't conflate it with auth failure." Regression tests: `tests/unit/test_headless_executor_pipe_drop.py` (handler behavior, status code, log level, ordering relative to the generic catch-all) and `tests/unit/test_pipe_close_no_auto_switch.py` (asserts SUB-003 does NOT trigger on 502 from the pipe-drop path). |
| 2026-05-12 | **Issue #808 - orphan-killer daemon thread spins at 100% CPU for ~15 min, starving uvicorn health probes and triggering circuit breaker**: Previous fixes (#649, #730, #747) bounded how long the *caller* waits for the orphan scan, but the daemon thread itself continued running at default (normal) scheduling priority. On 1-CPU containers a thread spinning at 100% preempts all other runnable threads — the uvicorn event loop cannot service health probe requests, probes time out repeatedly, and the circuit breaker (#631) opens and stays dormant for ~40 minutes. Two complementary fixes (`subprocess_pgroup.py`): (1) `_set_idle_priority()` — called at the start of `_run_orphan_killer()`, sets the daemon thread to `SCHED_IDLE` (Linux kernel "run only when nothing else wants CPU") via `os.sched_setscheduler(0, os.SCHED_IDLE, os.sched_param(0))`; falls back to `os.nice(19)` on non-Linux POSIX (macOS CI); no-op on other platforms. (2) `_scan_deadline = time.monotonic() + _ORPHAN_SCAN_WALL_SECONDS` (8 s) — passed as `_scan_deadline` kwarg to `_kill_orphan_pipe_writers`; per-iteration `time.monotonic() >= _scan_deadline` check in the `/proc` loop aborts scanning after budget expires, bounding total CPU time regardless of individual `readlink()` delays from D-state processes. 8 s is inside the existing 11 s `asyncio.wait_for` ceiling with 3 s margin. `_kill_orphan_pipe_writers` signature changed from `(pipe_read_fd, our_pgid)` to `(pipe_read_fd, our_pgid, _scan_deadline=None)` — keyword-only, backward-compatible. Tests: mock `_slow_orphan_killer` signature updated (`_scan_deadline=None`); new `TestKillOrphanPipeWriters.test_scan_deadline_stops_scan_early` (passes expired deadline, asserts scan aborts in < 1 s); new `TestSetIdlePriority` (does_not_raise, idempotent). |
| 2026-05-09 | **Issue #728 (follow-up) - `os.stat()` on `/proc/pid/fd/N` blocks indefinitely on D-state processes, causing orphan scan to silently miss orphan writers**: `_kill_orphan_pipe_writers` used `os.stat(f"/proc/{pid}/fd/{fd_name}")` to follow the symlink to the pipe inode and then read `/proc/{pid}/fdinfo/{fd_name}` to check write/read flags. Both operations acquire the inode lock at the kernel level; on a D-state (uninterruptible sleep) process this lock may be held indefinitely, causing the entire 10 s orphan-scan daemon thread to exit its cap without ever reaching the orphan writer. Fix (`subprocess_pgroup.py`): replace `os.stat()` + fdinfo with `os.readlink(f"/proc/{pid}/fd/{fd_name}")`. On Linux `/proc/pid/fd/N` is a symlink whose target text is `"pipe:[inode_number]"` — `readlink()` reads this string from proc's own metadata without following the symlink and without acquiring any inode lock, making it safe to call on D-state processes. The fdinfo write/read flag check is removed entirely: an explicit `our_pid = os.getpid()` self-skip is added so the read-end holder is excluded unconditionally; all remaining holders of the target pipe inode outside the killed pgid are by definition orphan writers. Regression test added: `tests/unit/test_subprocess_pgroup.py` — `TestKillOrphanPipeWriters.test_kills_orphan_even_when_stat_raises_dstate_simulation` monkeypatches `os.stat` to raise `OSError` and verifies the orphan is still found and killed via the `readlink` path (would have FAILED against the old implementation). Complements the `_drain_bounded` 90 s cap added in PR #730. |
Expand Down Expand Up @@ -389,6 +390,8 @@ After execution completes, `error_type` determines the HTTP response:
| `"execution_error"` | 503 | Falls through to non-zero return code handling |
| `null` | 200 | Normal success path |

**Pipe-drop reclassification (#474 / #873):** When the subprocess stdin write fails because the Claude child already exited (`BrokenPipeError` / `ConnectionResetError`), both `headless_executor.py` (lines 856–874) and `gemini_runtime.py` (lines 665–676) raise HTTP **502** instead of falling through to the generic `except Exception` → 500 path. 502 ("Bad Gateway to Claude subprocess") is semantically correct and avoids the SUB-003 auto-switch predicate in `task_execution_service.py:628`, which only triggers on 503. Log level for this path is `logger.info` (not `logger.error`) because the cause is a benign race (auth abort, permission-mode kill, upstream backend cancel), not a server-side fault.

The `_format_rate_limit_error()` helper (`claude_code.py:674-683`) uses `metadata.error_message` to build an actionable error detail string that suggests resolution steps (wait for reset, set API key, or reassign subscription).

## Key Differences: Chat vs Task
Expand Down
3 changes: 2 additions & 1 deletion docs/memory/feature-flows/public-agent-links.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ Public User -> GET /api/public/link/{token}

Public User -> POST /api/public/verify/request
-> Backend generates 6-digit code
-> Email service sends code
-> Email service sends code (subject/body now include agent name — #890)
-> Return {expires_in_seconds: 600}

Public User -> POST /api/public/verify/confirm
Expand Down Expand Up @@ -1978,6 +1978,7 @@ const viewingHistorySession = ref(null) // non-null = read-only history mode

| Date | Changes |
|------|---------|
| 2026-05-18 | **#890**: `request_verification_code()` now passes `agent_name` to `email_service.send_verification_code()` so the verification email subject and body name the agent. |
| 2026-04-29 | **#587 Chat History for Logged-In Users**: Two new JWT-authenticated endpoints (`GET /api/public/sessions/{token}`, `GET /api/public/sessions/{token}/{session_id}`) in `public.py`. New `ChatHistoryDropdown.vue` component. `PublicChat.vue` gains `viewingHistorySession` ref, `handleHistorySessionSelected()`, `exitHistoryView()`, amber read-only banner, and hidden `ChatInput` while in history mode. No new DB tables — reuses `chat_sessions`/`chat_messages`. |
| 2026-04-27 | **fix #539 Context duplication**: `build_public_chat_context()` was called AFTER `add_public_chat_message(role="user")`, causing the current user message to appear twice in every agent prompt (once in "Previous conversation:", once in "Current message:"). Fixed by swapping the call order — context built first from prior history, user message stored after. Added 6 unit tests in `tests/unit/test_public_chat_context.py`. Updated PUB-005 data flow and backend implementation step ordering to reflect correct call order. |
| 2026-02-19 | **CHAT-001 Shared Components Refactor**: PublicChat.vue now uses shared components from `components/chat/` (ChatMessages, ChatInput, ChatBubble, ChatLoadingIndicator). Shared with new ChatPanel.vue authenticated chat. Updated method line numbers, added Shared Chat Components section. File now 611 lines. |
Expand Down
Loading
Loading