refactor!: extract shared agent system engine

[shadaj]

Stack

This is PR 1 of 2. Review and merge this PR first. The additive high-level API is split into #92, which is based on this branch.

Summary

Extract the daemon's agent execution machinery into a shared infinity_agent_core::system engine, migrate the daemon and Lambda runtimes onto it, and extract the protocol components those embeddings share.

This PR intentionally stops at the engine and embedding boundary. Static builder conveniences, local MCP/RAP tool-set adapters, ThreadHandle, launcher mode, per-thread launch configuration, and the new usage guides are introduced by #92.

Review guide

Most of this diff is relocation. The old daemon thread_worker.rs and agent_loop.rs loops, plus the old core batch processor, are replaced by focused core modules. The migrated daemon tests in system/tests.rs and their unchanged snapshots are the fidelity evidence for the port.

Shared engine

• system/local/driver.rs ports the worker loop: batching, cancellation, deferral, compaction, tool-result waiting, and idle exit.
• system/thread.rs owns the completion pipeline, commit ordering, observer calls, config resolution, and tool dispatch.
• system/local/router.rs owns input routing, driver spawning, wake admission, subscriptions, lifecycle notifications, and shutdown.
• system/builder.rs exposes embedding-oriented configuration, local observer startup, and batch-shaped AgentSystem::step for serverless platforms.
• stores.rs centralizes ancestor, compaction, deduplication, snapshot, and restore behavior. The daemon store wraps it with persisted extras.

Behavior worth reviewing closely

• Router-owned event admission: StateStore::should_wake_thread_for_event prevents stale callbacks, tool results, OAuth responses, and choices from creating unknown or shut-down threads. User text remains able to create threads. Errors fail open.
• Lifecycle and subscription-aware activity: RunningSystem::thread_lifecycle emits Live and Idle at the same points the router mutates its live set. The daemon keeps subscribed threads active for RAP server lifetime even though their drivers may idle.
• Idle-exit races: drivers and the router share a LocalSet; the final no-await channel checks make exit atomic with respect to routing. Input either reaches the existing driver or observes closed channels and causes a respawn.
• Observer ordering: history sync completes before CompletionFinished and tool dispatch. Failed dispatch enqueues an error tool result so the agent can recover.
• Batch-shaped step mode: one delivery is partitioned by group_id, deferral is applied sequentially per thread, and thread slices run concurrently.

Protocol foundations

• infinity-rap-bridge provides callback conversion, prepared deduplication IDs, callback serving, and view-update forwarding for daemon and Lambda embeddings.
• infinity-mcp-bridge provides lazy stdio and Streamable HTTP clients plus canonical metadata and dispatch used by the daemon proxy.
• invoke_rap_tool is the shared RAP invocation path used by core, Lambda, and daemon tools, including a uniform non-2xx error result.
• Application-facing RapToolSet and McpToolSet adapters are deferred to feat(infinity-agent-core)!: add high-level agent system API #92.

Production migrations

• Daemon: one daemon-lifetime agent system replaces per-session loops and restart-on-send behavior. RAP servers boot lazily from thread configuration and follow lifecycle/subscription activity.
• Lambda: a batch is processed through one AgentSystem::step; per-session tools resolve through ThreadConfigSource; the Rust rap-receiver uses shared callback conversion and replaces the JavaScript receiver.
• CLI: DisplayEvent moves to the CLI because rendering is no longer a core engine concern.

Diff orientation

The new core system modules include the migrated daemon fidelity tests and shared fixtures. The daemon itself shrinks substantially because agent_loop.rs and thread_worker.rs are removed. Application conveniences and the documentation rewrite are kept out of this PR so reviewers can focus on the engine and production migrations.

Validation

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace

Breaking changes

• Removes batch_processor (process_batch, process_input_item, and core DisplayEvent) in favor of AgentSystem::step and observer-based local execution.
• Thread is internal and AgentSystem is not Clone.
• ThreadObserver replaces inline daemon persistence/display hooks; it has no on_commit.
• EventCollector::take returns (thread_id, event) pairs.
• Builder tools are stored as Rc; Tool gains defaulted is_passive.
• StateStore gains provided should_wake_thread_for_event.
• ToolContext and its builder lose input_queue_arn.
• Resident runtime types such as RunningSystem, SubscribeHandle, ChannelSender, and ChannelSendError live under system::local.
• The daemon no longer exports its old worker/loop/session implementation modules, sleep/RAP tool wrappers, or boot_rap_servers.
• rap_callback::start_callback_server is replaced by infinity_daemon::launch_session_manager; callback serving accepts a RapCallbackBridge.
• SessionManager::switch_model accepts the requester's sender and returns Result<(), String>; send_input accepts user_driven: bool; SharedSessionManager is Rc.
• CLI DisplayEvent lives in infinity_agent_cli::display.

[cloudflare-workers-and-pages]

Deploying infinity with    Cloudflare Pages

Latest commit:	db1d501
Status:	✅  Deploy successful!
Preview URL:	https://6502eb17.infinity-dc7.pages.dev
Branch Preview URL:	https://sandbox-0f86f15a-4591-435a-b.infinity-dc7.pages.dev

View logs