Skip to content

fix(operator-queue): stop 5s error-loop when an agent queue item lacks created_at (#1426)#1435

Merged
vybe merged 5 commits into
devfrom
feature/1426-operator-queue-created-at
Jul 8, 2026
Merged

fix(operator-queue): stop 5s error-loop when an agent queue item lacks created_at (#1426)#1435
vybe merged 5 commits into
devfrom
feature/1426-operator-queue-created-at

Conversation

@dolho

@dolho dolho commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

On eu2 (v0.7.0) the backend logged this every ~5s, indefinitely:

ERROR services.operator_queue_service: Failed to create queue item req-...-001: 'created_at'

An agent wrote a pending item to ~/.trinity/operator-queue.json without a created_at field. OperatorQueueOperations.create_item hard-indexes item["created_at"] (also item["title"], item["question"]), so it raised KeyError. Because the item stayed pending in the agent file, the 5s sync retried and error-logged it forever — the request never reached the Operating Room and the operator never saw it.

Fix

Agents author this JSON free-form, so the sync ingest boundary must be defensive. create_item now defaults the three hard-indexed fields, mirroring how it already .get()-defaults type/status/priority:

  • created_atutc_now_iso() (ingest time, ISO-Z per Invariant #16) — the issue's preferred fix.
  • title"Agent request"; question → the title, else "(no details provided)".

The item is now created once, so the next cycle sees it via exists() and the loop stops — the request surfaces in the Operating Room instead of vanishing behind unbounded log spam.

Acceptance criteria

  • Tolerate the missing field: default created_at to now on ingest.
  • No error-loop — the item is created, so the retry stops permanently.
  • The request reaches the Operating Room (not silently invisible).

Verification

  • Unittests/unit/test_1426_operator_queue_created_at.py (3, real create_item + throwaway SQLite): missing created_at defaulted (not raised); missing title/question defaulted; present fields preserved.
  • Live — against the rebuilt backend image + real DB: a created_at-less item creates without KeyError (created_at defaulted to 2026-07-03T10:18:55Z), a fully-minimal {id,status} item defaults all three, and the running 5s sync loop logged zero "Failed to create queue item" errors.

Fixes #1426

🤖 Generated with Claude Code

…estion

Agents author ~/.trinity/operator-queue.json free-form. An item that omitted
created_at made OperatorQueueOperations.create_item raise KeyError; because the
item stayed `pending` in the agent file, the 5s operator-queue sync retried and
error-logged it EVERY cycle, indefinitely ("Failed to create queue item <id>:
'created_at'" on eu2/v0.7.0) — the request never reached the Operating Room and
the operator never saw it. title/question had the same latent KeyError.

Make the sync ingest boundary defensive: default the three hard-indexed fields
in create_item (mirroring the existing type/status/priority .get() defaults).
created_at defaults to utc_now_iso() (ingest time, ISO-Z per Invariant #16);
title/question get sensible fallbacks. The item is now created once, so the
next cycle sees it via exists() and the loop stops — the request surfaces
instead of vanishing.

Verified:
- tests/unit/test_1426_operator_queue_created_at.py (3, real code + SQLite):
  missing created_at defaulted (not raised); missing title/question defaulted;
  present fields preserved.
- Live against the rebuilt backend image + real DB: a created_at-less item
  creates without KeyError (created_at defaulted to now), and the 5s sync loop
  logged zero "Failed to create queue item" errors.

Related to #1426

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dolho
dolho requested a review from vybe July 3, 2026 10:26
@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

dolho and others added 4 commits July 7, 2026 10:49
… CI fail

The test overrode TRINITY_DB_PATH to a fresh file, but init_database() runs only
once (at first `database` import, under the conftest-pinned path), so
get_engine() then hit an uninitialized DB — every case failed with
'no such table: operator_queue', which the regression-diff job flagged as a
head-only regression. Use the conftest's already-initialized per-process DB
(drop the override). The operator_queue.py fix itself is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dolho

dolho commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the failing regression diff check. Root cause was in the test, not the fix: it overrode TRINITY_DB_PATH to a fresh temp file, but init_database() runs only once (at first database import, under the conftest-pinned path), so get_engine() then hit an uninitialized DB — all three cases failed with no such table: operator_queue, which regression-diff flagged as a head-only regression. Switched the test to the unit conftest's already-initialized per-process DB (dropped the override). The operator_queue.py fix itself is unchanged. Verified: 3/3 pass in isolation and under suite ordering (test_73 … test_1426 → 7 passed). Pushed as 952dc8eb; CI re-running.

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via /validate-pr. Root-cause fix: create_item now defaults created_at/title/question (mirrors the existing type/status/priority defaults) so the item is created once and the 5s sync retry-loop stops. Verified the remaining item["id"] hard-index is unreachable — the caller skips id-less items. Named regression test present. Body updated to Fixes #1426 so the issue auto-promotes.

@vybe
vybe merged commit ac10399 into dev Jul 8, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants