Structured highlight descriptors + routine-replay attention decay fixes + label placement hardening

[softpudding]

Summary

Three themes in this branch:

1. Highlight / element-descriptor pipeline: the LLM now sees a compact structured descriptor per highlighted element instead of raw outerHTML, with a corner-badge label overlay that is always "above" (viewport-top exception), defers on collision, and clamps its x-offset inside the element's width.
2. Routine-replay attention hardening: the replay prompt no longer duplicates the routine identity into the user turn, and the condenser now fires on small models with a 1M-advertised context by explicit token cap. SDK pinned to versions that carry matching agent-side fixes.
3. Eval harness isolation + new 35-task × 4-model benchmark: each test now spawns its own eval/server.py on an OS-assigned port, eliminating cross-conversation event commingling and mid-conversation sessionId rotation bugs that were silently suppressing scores on --parallel > 1 runs. Fresh benchmark snapshot checked in.

Headline commits

1. 3b052cf — Harden routine replay against long-context attention decay

• frontend/index.html::buildRoutinePrompt sends only the SOP markdown; routine name/goal/framing lives in the ROUTINE_REPLAY system-prompt block.
• server/agent/browser_condenser.py gains SMALL_MODEL_TOKEN_OVERRIDES; seeded with qwen3.5-flash → 100k so condensation actually fires.
• Pinned agent-sdk to 66ed257b (task_tracker plan pinning in routine_replay mode, small-model confirmation-reasoning gate, refocused condenser prompt).
• Tests: new test_browser_condenser.py covers override/substring/fallback/configure paths.

2. e445430 — Bump agent-sdk to c92a185a

Picks up the SMALL_MODEL_GUIDANCE / LARGE_MODEL_GUIDANCE → <ACTION_PROTOCOL> tag rename so the model-tier identity no longer leaks into rendered prompts.

3. 0bc4c16 — Persist highlights across commands + outline for better coverage

• Keep injected highlights until the next user-visible command so users can see what was highlighted.
• Element borders switch from inset box-shadow to outline with negative offset — opaque full-bleed children no longer hide the border.
• Labels move into document-coordinate space so they scroll with the element.

4. 728eafa — Corner-badge label overlay for unambiguous element binding

• 4-side placement replaced by a strict top-or-bottom corner badge anchored at the element's top-left, touching the bbox edge. Sideways placements are disabled.
• When a label fits neither above nor below, the element defers to a later page rather than being placed ambiguously; total_pages absorbs overflow.
• Opaque darker-shade fill + 11 px font so badges recede into the visual hierarchy.

5. f915ae4 — Labels always "above", defer on collision; tolerate shared-border bbox overlap

• getFeasiblePositions returns ['above'] (or ['below'] at the viewport top). Collisions defer to a later page instead of flipping sides.
• bboxesPartiallyOverlap requires ≥ 3 px on both axes before treating an intersection as an overlap → 1–2 px shared-border artefacts no longer force pagination.
• Label-vs-neighbor uses strict bboxesIntersect so labels touching an adjacent element's top edge at a shared row border are no longer collisions.
• On https://finviz.com/screener.ashx?v=121: 7 pages → 4 pages; all 6 filter tabs land on page 1.

6. ea8cf32 — Emit structured element descriptors instead of raw HTML

• The "Highlighted Elements" LLM observation now emits one line per element built in the page world from the live DOM:

  id(type): <tag> "text" · attr=val … flags
  

• <select> appends an indented options: block listing every <option> in full.

• Anonymous nodes harvest up to 3 filtered semantic class tokens and an icon hint (<use xlink:href>, <img alt>, [aria-label]).

• InteractiveElement.html stays internal; the descriptor is the single source of truth on the wire.

• Shared extractor (extension/src/commands/element-descriptor.injected.js) is inlined into both highlight-detection and drag-drop scripts.

• Server-side formatter (server/agent/tools/base.py::_format_highlighted_element_lines) is shared between the main list and drag-and-drop inner-elements.

• Prompts updated: big_model/highlight_tool.j2, big_model/element_interaction_tool.j2, and select_element description in server/models/commands.py.

Follow-up fixes and refinements

• 728eafa follow-ups:
  • f915ae4 / 08206ca / 43f2643 / 77f9e01 — iterative hardening of label placement: always-above rule, horizontal shift of corner badges, tiny-element filtering before the collision planner, and per-tab scope on deferred highlight cleanup (so switching tabs doesn't nuke the other tab's highlights).
• Eval / small-model stability (addressing regressions the descriptor rewrite surfaced):
  • d492019 fix(agent): reduce eval regressions from highlight/descriptor branch
  • 64c14e1 fix(extension): hash element ids on stable fingerprint, not volatile outerHTML — element IDs now hash on a fingerprint (type + tag + stable attrs + text), so focus / ephemeral-value / aria-expanded flips don't reassign IDs mid-conversation.
• Per-test mock-server isolation in the eval harness (74dcddc fix(eval): per-test mock-site server isolation):
  • Each test spawns its own eval/server.py on an OS-assigned port; localhost:16605 in start_url / instruction is rewritten to that port via _rewrite_eval_server_urls.
  • New EvalServerProcess class owns spawn / handshake / drain / SIGTERM→SIGKILL teardown with an atexit hook and per-instance process-group.
  • EvalServerClient is now port-addressed; the ?site= filter is removed.
  • ServiceManager.start_eval_server is gone; ensure_services only checks OpenBrowser (8765).
  • Fixes an entire bug class: under --parallel > 1, the old shared events_store let parallel conversations commingle, and within a single conversation a page-reload sessionId rotation could silently hide the early events from the scorer. Isolating the server per test makes the track file contain exactly one sessionId.
• New benchmark snapshot (78dd2ce eval: refresh benchmark to the 35-task × 4-model suite (2026-04-21)):
  • 140 runs across qwen3.5-{flash,plus} + qwen3.6-{flash,plus}, 111 passed (79.3 %). See eval/evaluation_report.json and the updated README eval section.
  • qwen3.5-plus 30/35 (85.7 %), qwen3.6-flash 29/35 (82.9 %), qwen3.6-plus 28/35 (80.0 %), qwen3.5-flash 24/35 (68.6 %). qwen3.6-flash is the fastest model of the four (252 s avg vs 310-338 s for the others).
• Dependency pins: c374403 / d559ea7 bump litellm + openhands-sdk to rev c92a185a (Qwen prompt-cache enablement, <ACTION_PROTOCOL> rename).
• Skill packaging: e29e65a moves Claude skills under ~/.claude/skills/ (user-scoped) instead of repo-scoped, so they survive checkout.
• CI / lint: 4102276 + 316a4a2 + 61e3868 apply pre-commit prettier/black formatting flagged by CI; 71df427 repoints two lockfile entries (ws-8.20.0, @types/ws-8.18.1) from registry.anpm.alibaba-inc.com (Alibaba internal, unreachable from GitHub Actions runners) to registry.npmjs.org. Integrity hashes preserved because the mirror tarballs are bit-identical.

Test plan

• cd extension && bun test — 195/195 pass.
• uv run pytest server/tests/unit — 411/411 pass.
• Full end-to-end eval, 4 models × 35 tasks, with per-test isolation — 111 / 140 passed (79.3 %). Report checked in at eval/evaluation_report.json.
• Per-test isolation verified live: two concurrent EvalServerProcess instances get distinct OS-assigned ports, their events_store content is independent, and both ports are unreachable after stop().
• Exercise a page with <select> (e.g. country picker) and confirm the indented options: block lists every option with group/selected/disabled flags.
• Exercise a page with icon-only controls (Finviz filter tabs, BlueBook like button) and confirm the descriptor surfaces class=... + icon=... rather than a bare <span>.
• Load the extension in Chrome, run /ob-routines execute value-stocks-monthly-drop, and confirm the replay completes without please_help_me, with task_tracker stepping through end-to-end.

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.