Skip to content

ROADMAP.md

Latest commit

 

History

History
42 lines (32 loc) · 10.2 KB

ROADMAP.md

File metadata and controls

42 lines (32 loc) · 10.2 KB

Milady roadmap

High-level direction and rationale. Not exhaustive; see the published Changelog (or docs/changelog.mdx in-repo) for shipped changes with WHYs.

Principles: energy and experience (desktop)

Goal: Beat the felt cost of always-on dev shells on laptop battery while staying more visually distinctive than a flat editor surface—better UX and DX, not a spec-sheet stunt.

  • Honest comparison: Cursor (and similar) ships a large persistent surface: Chromium/Electron-style shell, editor, extensions, LSP, indexing, often multiple web contexts. Milady’s desktop UI is narrower and task-shaped: companion, chat, settings, and bridges—on Electrobun / WKWebView plus optional 3D. Total J/s is workload-dependent; apples-to-apples needs the same scenario and tools (Activity Monitor, powermetrics, Instruments). Our bar is excellent experience per watt for a local AI companion, not claiming victory in every head-to-head against a full IDE.
  • What we optimize: Wasted work—GPU and timers for hidden documents, off-screen canvases, and redundant HTTP polling. Battery-aware quality when unplugged (DPR cap, tighter Spark splats, no directional shadows, fewer background API ticks). Rich by default when the user is looking at the app and on AC.
  • Shipped levers (see changelog + desktop docs): VrmViewer visibility pause; desktop:getPowerState → VrmEngine.setLowPowerRenderMode; visibility-gated intervals (dashboard, stream, game logs, fine-tuning, cloud credits); vector 3D graph rAF pause when hidden; dev hooks opt-out so DX tooling does not accidentally burn watts (screenshot proxy, aggregated console).
  • Next UX/DX directions: User-visible Efficiency / Performance profile (single toggle), prefers-reduced-motion, optional idle frame cap for the avatar when motion fidelity matters less than battery, and clearer in-app copy when battery savings are active (so users trust the tradeoff).

Done (this cycle)

  • Port collisions (dev + embedded desktop) — dev:desktop / dev:desktop:watch pre-allocate free loopback ports for MILADY_API_PORT and MILADY_PORT (Vite) before spawning API, Vite, and Electrobun so env, proxy, and renderer URL stay aligned. Embedded agent: Electrobun picks the next free port from the preferred MILADY_PORT instead of default lsof + SIGKILL; optional MILADY_AGENT_RECLAIM_STALE_PORT=1 restores reclaim. Runtime: eliza.ts / dev-server.ts sync process.env to the API’s actual bind port where safe. UI: injectApiBase on agent status for main + all surface windows. Why: two Milady stacks or stray processes should not require manual port hunting or killing unrelated processes; dynamic binds must propagate to renderer and dev tooling. Docs: docs/apps/desktop-local-development.md, docs/apps/desktop.md (port sections). Code: scripts/lib/allocate-loopback-port.mjs, apps/app/electrobun/src/native/loopback-port.ts, agent.ts, index.ts, surface-windows.ts, vite.config.ts, dev-server.ts, eliza.ts.
  • Desktop dev observability (IDEs / agents) — GET /api/dev/stack, desktop:stack-status, default-on screenshot proxy (/api/dev/cursor-screenshot, loopback + token), default-on aggregated console (.milady/desktop-dev-console.log + /api/dev/console-log tail with basename allow-list). Why: multi-process dev is opaque to tools that cannot see the native window; explicit HTTP + file hooks avoid guessing ports and keep loopback/tokens bounded. Opt-out env vars documented. Docs: docs/apps/desktop-local-development.md (section IDE and agent observability). Rules: .cursor/rules/milady-desktop-dev-observability.mdc.
  • Electrobun Darwin → macOS mapping (WebGPU) — getMacOSMajorVersion() uses Darwin − 9 for 20–24 (macOS 11–15) and Darwin + 1 for ≥ 25 (macOS 26+ Tahoe). Why: os.release() is Darwin; Tahoe is macOS 26 on Darwin 25—the old single formula reported 16 and broke WKWebView WebGPU messaging and gating. Docs: docs/apps/electrobun-darwin-macos-webgpu-version.md. Tests: webgpu-browser-support.test.ts.
  • Desktop menu reset (main process) — Confirm + API reset + restart + status poll run in Electrobun main; renderer syncs via menu-reset-milady-applied and shared completeResetLocalStateAfterServerWipe. Why: WKWebView deferred renderer networking after native dialogs; users saw “nothing happens” after confirm. Reachable-base probe uses res.ok only. Docs: docs/apps/desktop-main-process-reset.md. Tests: menu-reset-from-main.test.ts, reset-main-process.test.ts.
  • Edge TTS disclosure — Document and surface MILADY_DISABLE_EDGE_TTS / ELIZA_DISABLE_EDGE_TTS (registry + docs/cli/environment.md + TTS doc). Why: orchestrator auto-loads Edge TTS → node-edge-tts → Microsoft; “no API key” is not “offline.”
  • Vitest app-core coverage — Root config globs packages/app-core/test/**/*.test.ts(x) and src/**/*.test.tsx; excludes app-core e2e under test/ from the default unit job. Why: new tests under test/state and test/runtime were skipped; a single hard-coded TSX path was brittle.
  • Node.js CI timeouts — Use useblacksmith/setup-node@v5 on Blacksmith for the desktop and API CI jobs; pin actions/setup-node@v3 + check-latest: false everywhere else; add Bun global cache and timeout-minutes to test, release, nightly, benchmark-tests, publish-npm. Why: v4 timeouts from nodejs.org and slow post-action; Blacksmith’s colocated cache and v3 fix it. See docs/build-and-release.md "Node.js and Bun in CI: WHYs".
  • Release workflow hardening — Strict shell (bash -euo pipefail) for fail-fast steps; retry loops for bun install with a final run so the step fails if all retries failed; crash dump uses the maintained ASAR CLI; find -print0 / while IFS= read -r -d '' for safe paths; DMG path via find+stat; node-gyp artifact removal before pack; size report includes milady-dist; single Capacitor build step; packaged DMG E2E uses 240s CDP timeout in CI and dumps stdout/stderr on timeout. Why: Reproducible builds, clear failures, and debuggable CI; see docs/build-and-release.md "Release workflow: design and WHYs".
  • Plugin resolution (NODE_PATH) — Set NODE_PATH in three places so dynamic import("@elizaos/plugin-*") resolves from CLI (run-node.mjs child), direct eliza load (eliza.ts on load), and Electrobun (dev: walk up to find node_modules; packaged: ASAR node_modules). Why: Node does not search repo root when the entry is under dist/ or cwd is a subdir; without this, "Cannot find module" broke coding-agent and others. See docs/plugin-resolution-and-node-path.md.
  • Electrobun startup resilience — Keep API server up when runtime fails to load so the UI can show an error instead of "Failed to fetch". Why: A single missing native module (e.g. onnxruntime on Intel Mac) used to make the whole window dead with no explanation.
  • Intel Mac x64 DMG — Release workflow runs install and desktop build under arch -x86_64 for the macos-x64 artifact so native .node binaries are x64. Why: CI runs on arm64; without Rosetta we shipped arm64 binaries and Intel users got "Cannot find module .../darwin/x64/...".
  • Auto-derived plugin deps — copy-electrobun-plugins-and-deps.mjs walks each @elizaos package's package.json dependencies instead of a curated list. Why: Curated lists missed new plugin deps and caused silent failures in packaged app; auto-walk stays correct as plugins change.
  • Regression tests for startup — E2E tests assert keep-server-alive and eliza.js load-failure behavior. Why: A failing test prevents removal of the exception-handling guards better than docs alone.
  • Plugin resolution fix — NODE_PATH set to repo root node_modules in eliza.ts, run-node.mjs, and agent.ts (Electrobun dev). Why: Dynamic import("@elizaos/plugin-*") from bundled eliza.js couldn't resolve packages at root; NODE_PATH tells Node where to look. No-op in packaged app (existsSync guard). See docs/plugin-resolution-and-node-path.md.
  • Bun exports patch — Postinstall in patch-deps.mjs rewrites @elizaos/plugin-coding-agent (and any similar package) so exports["."] no longer has "bun": "./src/index.ts" when that file doesn't exist. Why: The published tarball only ships dist/; Bun picks the "bun" condition first and fails. Removing the dead condition lets Bun use "import" → ./dist/index.js. See "Bun and published package exports" in docs/plugin-resolution-and-node-path.md.
  • Release size-report: SIGPIPE 141 — du | sort | head pipelines in the "Report packaged app size" step run in a subshell with || r=$? and allow exit 141; sort stderr silenced. Why: Under -euo pipefail, 141 would exit the step before we could allow it; subshell captures it. See docs/build-and-release.md.
  • NFA routes: optional plugin — /api/nfa/status and /api/nfa/learnings lazy-load @elizaos/plugin-bnb-identity and fall back when missing. Why: Core and tests work without the plugin; ambient type declaration keeps typecheck happy.

Short-term / follow-ups

  • Upstream plugin hygiene — Some plugins (e.g. @elizaos/plugin-discord) list typescript in dependencies instead of devDependencies; we skip it via DEP_SKIP to avoid bundle bloat. Why: Fixing upstream would reduce our skip list and keep plugin package.json correct.
  • Optional: filter bundled deps — We intentionally copy all transitive deps (including ones tsdown may have inlined) because plugins can dynamic-require at runtime. Why: Excluding "likely bundled" deps would risk "Cannot find module" in packaged app. If we ever get static analysis of plugin dist/ to know what is never required at runtime, we could shrink the copy; not a priority.

Longer-term

  • Desktop: Universal/fat macOS binary (single .app with arm64+x64) is possible via lipo or desktop packaging targets but adds build time and complexity; separate DMGs are acceptable for now.
  • CI: Consider caching desktop native rebuilds per arch to speed up release matrix.