Skip to content

architecture.mdx

Latest commit

 

History

History
615 lines (471 loc) · 28.8 KB

architecture.mdx

File metadata and controls

615 lines (471 loc) · 28.8 KB
title System Architecture
sidebarTitle Architecture
description High-level overview of the Milady AI system architecture, layers, plugin system, and cross-platform delivery.

Milady is a cross-platform AI agent runtime built on top of elizaOS. It delivers the same backend across CLI, web, desktop (Electrobun), and mobile (Capacitor) through a layered architecture. This page covers the full system from entry points through runtime lifecycle, plugin loading, API surface, and build tooling.

High-Level Diagram

+---------------------------------------------------------------+
|                        Frontend Layer                          |
|  React SPA (Vite) + Capacitor (iOS/Android) + Electrobun (Desktop) |
|  Tabs: Chat, Character, Wallets, Knowledge, Social, Apps,     |
|         Settings, Advanced (Plugins, Skills, Triggers, DB, ...) |
+----------------------------+----------------------------------+
                             | HTTP + WebSocket
                             v
+---------------------------------------------------------------+
|                       CLI Layer                                |
|  Commander-based CLI (entry.ts -> run-main.ts)                |
|  Commands: start, setup, configure, config, update, ...       |
|  Profile system, dotenv loading, log-level control            |
+----------------------------+----------------------------------+
                             |
                             v
+---------------------------------------------------------------+
|                    Runtime Layer (elizaOS)                     |
|  AgentRuntime + Plugin system                                 |
|  9 core plugins + optional plugins + Milady plugin            |
|  Providers, Actions, Services, Evaluators                     |
+----------------------------+----------------------------------+
                             |
                             v
+---------------------------------------------------------------+
|                     API Server Layer                           |
|  Node.js HTTP server (port 31337 dev / 2138 prod)             |
|  REST endpoints (/api/*) + WebSocket (/ws)                    |
|  SSE log streaming, static UI assets                          |
+---------------------------------------------------------------+
                             |
                             v
+---------------------------------------------------------------+
|                    Storage & Config Layer                      |
|  ~/.milady/milady.json  (Zod-validated config)              |
|  PGLite (default) or PostgreSQL (database)                    |
|  ~/.milady/workspace/ (agent files, skills)               |
|  ~/.milady/plugins/  (custom, ejected, installed)            |
+---------------------------------------------------------------+

Entry Points and Boot Sequence

Milady has two primary entry points, plus a dev-server variant:

CLI Entry (packages/app-core/src/entry.ts)

The CLI entry point is the bootstrap for all Milady operations. Built by tsdown into dist/entry.js and invoked by the milady.mjs wrapper script.

The boot sequence is:

  1. Set process title to milady (visible in ps output)
  2. Normalize environment -- honor --no-color, --debug, --verbose flags by setting NO_COLOR, FORCE_COLOR, and LOG_LEVEL in process.env
  3. Suppress native logging -- set NODE_LLAMA_CPP_LOG_LEVEL to match the chosen log level, suppressing noisy tokenizer warnings at default verbosity
  4. Parse CLI profiles -- extract --profile <name> from process.argv via parseCliProfileArgs(), apply the profile environment via applyCliProfileEnv()
  5. Delegate to Commander -- dynamically import src/cli/run-main.ts and call runCli(process.argv)
// packages/app-core/src/entry.ts — simplified bootstrap
process.title = "milady";
const parsed = parseCliProfileArgs(process.argv);
if (parsed.profile) applyCliProfileEnv({ profile: parsed.profile });
import("./cli/run-main")
  .then(({ runCli }) => runCli(process.argv))

run-main.ts continues by loading .env files via dotenv, normalizing API key environment variables (e.g., Z_AI_API_KEY to ZAI_API_KEY), building the Commander program tree, and dispatching to the requested subcommand.

Library Entry (packages/app-core/src/index.ts)

The public API surface exported by the miladyai npm package. It re-exports:

  • Config types from packages/app-core/src/config/types.ts -- the full MiladyConfig type and its nested schemas
  • Restart infrastructure -- RESTART_EXIT_CODE, requestRestart(), setRestartHandler(), and the RestartHandler type

This allows other packages (e.g., the Electrobun app, the dev-server) to programmatically control the runtime without importing internal modules.

Dev-Server Entry (packages/app-core/src/runtime/dev-server.ts)

A combined dev server that starts the elizaOS runtime in headless mode and wires it into the API server. Used during development (bun run dev):

  1. Loads .env files for parity with CLI mode
  2. Creates the runtime via startEliza({ headless: true })
  3. Starts the API server and connects the runtime
  4. Registers a restart handler that stops the old runtime, creates a new one, and hot-swaps references
  5. Includes exponential backoff retry logic (1s, 2s, 4s, ..., 30s cap) for runtime bootstrap failures

Runtime Lifecycle

The startEliza() function in packages/app-core/src/runtime/eliza.ts orchestrates the full lifecycle from config loading through agent operation.

Initialization Phase

Read `~/.milady/milady.json` via `loadMiladyConfig()`. If no config exists, fall back to an empty `MiladyConfig` object with all-optional fields. If no agent name is configured and stdin is a TTY, run the interactive onboarding flow: welcome banner, name selection (4 random + custom), personality/style preset, AI provider selection, and wallet setup. In headless mode (GUI), this is deferred to the web UI. Push connector secrets (`DISCORD_API_TOKEN`, `TELEGRAM_BOT_TOKEN`, etc.) from config into `process.env`. Auto-resolve Discord Application ID from bot token via the Discord API. Propagate cloud, x402, and database config into env vars. Construct an elizaOS `Character` object from the Milady config: agent name, bio, system prompt, style, adjectives, topics, message examples, and API key secrets. Ensure `~/.milady/workspace/` exists with bootstrap files. Create the `~/.milady/plugins/custom/` directory for drop-in plugins. Instantiate the bridge plugin that provides workspace context, session keys, emotes, custom actions, trigger tasks, and autonomous state tracking to the elizaOS runtime. Run the full plugin resolution pipeline (detailed below). Plugins are loaded in parallel for faster startup. Instantiate `AgentRuntime` with the character, Milady plugin, and all resolved plugins. Pre-register `plugin-sql` (with PGLite recovery) and `plugin-local-embedding` before other plugins to avoid race conditions. Call `runtime.initialize()` which registers all remaining plugins in parallel, starts services, and sets up the agent loop. Enable trajectory logging. Warm up the AgentSkillsService asynchronously. Launch the HTTP/WebSocket API server on port 2138 (default, or 31337 in dev mode). Register the restart handler for hot-reload support. Load webhook and hook configurations from the workspace. Fire the `gateway.startup` hook event.

Running Phase

During normal operation, the runtime:

  • Processes incoming messages from all connected channels (web UI, Discord, Telegram, etc.)
  • Runs the agent loop: evaluate message, select action, execute action, store memory
  • Broadcasts status updates to WebSocket clients every 5 seconds
  • Streams agent events (autonomy loop iterations, heartbeats) in real time
  • Executes scheduled jobs via the cron plugin
  • Manages the knowledge base and RAG pipeline

Shutdown Phase

Graceful shutdown is triggered by SIGINT or SIGTERM:

  1. Stop the sandbox manager (if active)
  2. Call runtime.stop() to close database connections, cancel timers, and clean up
  3. Exit the process with code 0

Restart Mechanism

Restart is pluggable via setRestartHandler():

Environment Handler Behavior
CLI (default) Exit with code 75 The runner script (scripts/run-node.mjs) catches exit code 75, optionally rebuilds if sources changed, and relaunches
Dev-server / API Hot-swap Stops the current runtime, reloads config from disk, creates a new AgentRuntime, and swaps the reference in the API server
Electrobun AgentManager.restart() Uses the Electrobun app's native restart mechanism

elizaOS Integration Layer

Milady extends elizaOS through several integration points:

Plugin System Bridge

The createMiladyPlugin() function (packages/app-core/src/runtime/eliza-plugin.ts) creates a standard elizaOS Plugin that bridges Milady-specific functionality into the runtime:

Providers (injected into every LLM context):

  • channelProfileProvider -- channel-specific personality profiles
  • workspaceProvider -- workspace file context
  • adminTrustProvider -- admin trust level for privileged operations
  • autonomousStateProvider -- current autonomous agent state
  • sessionKeyProvider -- session bridge for cross-channel continuity
  • uiCatalogProvider -- available UI components the agent can render
  • customActionsProvider -- user-defined custom action descriptions

Actions (agent-invokable operations):

  • restartAction -- trigger a runtime restart
  • sendMessageAction -- send a proactive message to a channel
  • triggerTaskAction -- execute a trigger task
  • emoteAction -- play an emote animation on the 3D avatar
  • Custom actions loaded from ~/.milady/workspace/custom-actions/

Init hooks:

  • Register trigger task workers
  • Start autonomous state tracking
  • Bind custom action runtime references
  • Remove PLAY_EMOTE action when character.settings.DISABLE_EMOTES is set

Error Boundaries

Every plugin is wrapped with error boundaries before registration (wrapPluginWithErrorBoundary()):

  • init() wrapper -- catches crashes during plugin initialization, logs the error, and allows the agent to continue in degraded mode
  • Provider wrappers -- catches crashes in provider get() calls (invoked every conversation turn), returns an error marker instead of crashing the agent

Actions are not wrapped because elizaOS's own action dispatch already has error boundaries.

Method Binding Fixes

The runtime applies compatibility patches after creation:

  • installRuntimeMethodBindings() -- binds getConversationLength() to the runtime instance, fixing private-field access errors when plugins store and later invoke the method without a receiver
  • installActionAliases() -- adds backward-compatible action aliases (e.g., CODE_TASK maps to CREATE_TASK)

Plugin Loading Pipeline

Plugin resolution (resolvePlugins()) handles three tiers of plugins through a multi-stage pipeline:

Stage 1: Collect Plugin Names

collectPluginNames() builds a Set<string> of package names to load:

  1. Start with all 9 core plugins (CORE_PLUGINS)
  2. Add plugins from the allow list (config.plugins.allow) -- these are additive, not exclusive
  3. Add connector plugins for any configured channels (Discord, Telegram, Slack, etc.) using the CHANNEL_PLUGIN_MAP
  4. Add model-provider plugins when their API key environment variables are detected (e.g., ANTHROPIC_API_KEY triggers @elizaos/plugin-anthropic)
  5. Handle Eliza Cloud mode -- when cloud is active, remove direct AI provider plugins since the cloud plugin handles all model calls
  6. Add plugins from config.plugins.entries with enabled !== false
  7. Add plugins from feature flags (config.features)
  8. Add user-installed plugins from config.plugins.installs
  9. Enforce the shell feature gate -- remove plugin-shell if config.features.shellEnabled === false

Stage 2: Discover Drop-In Plugins

Scan filesystem directories for additional plugins:

  • Ejected plugins (~/.milady/plugins/ejected/) -- override npm/core versions
  • Custom plugins (~/.milady/plugins/custom/) -- user-created plugins
  • Extra paths from config.plugins.load.paths -- additional directories

Each immediate subdirectory is treated as a plugin package. The plugin name comes from package.json or the directory name. Deny-list and core-collision filtering is applied.

Stage 3: Resolve and Import

All plugins are loaded in parallel via Promise.all(). For each plugin:

  1. Check for ejected plugin at its install path (highest priority)
  2. Check for workspace plugin override (for development)
  3. Check for installed plugin at its recorded install path, with fallback to node_modules
  4. Fall back to npm import() by package name

The findRuntimePluginExport() function handles multiple export patterns:

  • export default plugin (preferred)
  • export const plugin = ...
  • Named exports ending in Plugin
  • CJS compatibility (module.exports = { name, description })

Stage 4: Post-Processing

After loading, the pipeline:

  • Wraps each plugin with error boundaries
  • Repairs broken install records (stale paths) and persists the fix
  • Runs version-skew diagnostics (diagnoseNoAIProvider())
  • Logs a summary: loaded count, failed count, and failure reasons

Pre-Registration Order

Two plugins require special ordering to avoid race conditions:

  1. plugin-sql -- registered first so the database adapter is ready before other plugins call runtime.db. Includes PGLite corruption recovery (backs up corrupt data, resets, retries once).
  2. plugin-local-embedding -- registered second so its TEXT_EMBEDDING handler (priority 10) is available before the cloud plugin's handler (priority 0), preventing unintended paid API calls during startup.

Action, Provider, and Evaluator System

Milady inherits elizaOS's three-part extension model:

Providers

Providers inject context into every LLM prompt. Each provider returns a ProviderResult containing text that the agent sees when composing responses.

interface Provider {
  name: string;
  description: string;
  get(runtime: IAgentRuntime, message: Memory, state: State): Promise<ProviderResult>;
}

Milady registers 7+ providers through the Milady plugin, plus any providers from loaded plugins.

Actions

Actions are operations the agent can invoke based on conversation context. Each action defines:

  • name -- unique identifier (e.g., RESTART, SEND_MESSAGE, PLAY_EMOTE)
  • similes -- alternative names the agent can use
  • validate() -- whether the action can run in the current context
  • handler() -- the execution logic
  • examples -- few-shot examples for the LLM

Evaluators

Evaluators run after each conversation turn to assess agent performance and update state. They are defined by plugins and can trigger actions, update memory, or modify agent behavior.

Memory and State Management

Database Layer

Milady uses @elizaos/plugin-sql as the database adapter with two backend options:

Backend Config Use Case
PGLite (default) database.provider: "pglite" Zero-config embedded PostgreSQL. Data stored in ~/.milady/workspace/.eliza/.elizadb/
PostgreSQL database.provider: "postgres" External PostgreSQL server for production deployments

PGLite includes automatic corruption recovery: if initialization fails with a known error pattern (WASM abort, migration schema failure), the data directory is backed up and recreated.

Embedding System

Local embeddings for memory search use @elizaos/plugin-local-embedding with configurable parameters:

Setting Default Description
embedding.model nomic-embed-text-v1.5.Q5_K_M.gguf GGUF model file name
embedding.gpuLayers auto (Apple Silicon), 0 (others) GPU acceleration layers
embedding.dimensions 384 (capped at boot) Embedding vector dimensions
embedding.contextSize Auto-detected Maximum input token context

On macOS with Apple Silicon, Metal GPU acceleration is enabled by default with mmap disabled to prevent known compatibility issues.

Trajectory Logging

The trajectory logger (@elizaos/plugin-trajectory-logger) records agent decision paths for debugging and reinforcement learning. The runtime waits up to 3 seconds for the service to register, then enables it by default.

API Server Architecture

The API server (packages/app-core/src/api/server.ts) is a raw Node.js http.createServer() instance -- no Express or framework overhead.

Server State

The server maintains a ServerState object tracking:

  • Active AgentRuntime reference (swappable on restart)
  • Agent lifecycle state: not_started | starting | running | paused | stopped | restarting | error
  • Plugin and skill registries
  • Log and event ring buffers
  • Chat room/user connections
  • Cloud, sandbox, training, and registry service handles
  • WebSocket broadcast functions

Route Architecture

Routes are organized into domain-specific handler modules:

Module Prefix Purpose
agent-admin-routes.ts /api/agent/* Agent name, bio, system prompt, style, examples
agent-lifecycle-routes.ts /api/agent/* Start, stop, restart, status
auth-routes.ts /api/auth/* Token validation, onboarding tokens
autonomy-routes.ts /api/autonomy/* Autonomous mode state
character-routes.ts /api/character/* Character CRUD, AI-assisted generation
cloud-routes.ts /api/cloud/* Eliza Cloud integration
knowledge-routes.ts /api/knowledge/* Document upload, search, RAG management
models-routes.ts /api/models/* Model provider discovery and selection
plugin-validation.ts /api/plugins/* Plugin config validation
registry-routes.ts /api/registry/* Plugin registry browsing
training-routes.ts /api/training/* Fine-tuning orchestration
trigger-routes.ts /api/triggers/* Trigger task management
wallet-routes.ts /api/wallet/* Wallet addresses, balances, NFTs
apps-routes.ts /api/apps/* elizaOS app marketplace

Authentication

When MILADY_API_TOKEN is set (or auto-generated for non-loopback binds), all API requests must include a valid bearer token:

Authorization: Bearer <token>

If the API is bound to a non-loopback address (MILADY_API_BIND is not 127.0.0.1 or localhost) and no token is set, the server auto-generates a temporary token and logs it to stderr.

WebSocket Event System

The API server exposes a WebSocket endpoint at /ws on the same port as the API (31337 in dev, 2138 in production).

Event Types

Event Type Direction Description
status Server -> Client Agent status broadcast (every 5 seconds)
agent_event Server -> Client Runtime autonomy loop events with run/seq tracking
heartbeat_event Server -> Client Agent heartbeat with status, duration, channel info
training_event Server -> Client Fine-tuning progress updates
proactive_message Server -> Client Agent-initiated messages
conversation_update Server -> Client New messages in active conversations
ping / pong Bidirectional Keepalive mechanism
set_active_conversation Client -> Server Tell the server which conversation the UI is viewing

Event Envelope

All server-sent events use a standard envelope:

{
  type: "agent_event" | "heartbeat_event" | "training_event",
  version: 1,
  eventId: string,
  ts: number,
  runId?: string,
  seq?: number,
  stream?: string,
  sessionKey?: string,
  agentId?: string,
  roomId?: UUID,
  payload: object
}

Events are buffered in a ring buffer on the server so newly connected clients can receive recent history.

Service Architecture

Milady's packages/app-core/src/services/ directory contains standalone service modules:

Service File Purpose
Plugin Installer plugin-installer.ts Install, uninstall, and manage plugins from npm and git
Self-Updater self-updater.ts Detect install method and run the appropriate upgrade command
Update Checker update-checker.ts Query npm registry for new versions on the configured channel
Update Notifier update-notifier.ts Fire-and-forget background check that prints a one-line notice
App Manager app-manager.ts Launch and manage elizaOS marketplace apps
Sandbox Engine sandbox-engine.ts Container-based code execution sandboxing
Sandbox Manager sandbox-manager.ts Docker container lifecycle for sandboxed execution
Registry Client registry-client.ts elizaOS plugin registry API client
Skill Marketplace skill-marketplace.ts Search, install, and manage skills from the marketplace
Training Service training-service.ts Fine-tuning orchestration with Ollama integration
Agent Export agent-export.ts Export agent state (character, knowledge, config) as portable archives
Version Compat version-compat.ts Detect version skew between @elizaos/core and plugins
Core Eject core-eject.ts Eject core plugins for local modification
Plugin Eject plugin-eject.ts Eject installed plugins for local overrides
Skill Catalog skill-catalog-client.ts Skills registry API client for browsing and installing
MCP Marketplace mcp-marketplace.ts MCP server discovery and marketplace integration

Configuration Cascade

Milady resolves configuration through a priority cascade:

Highest priority
  1. CLI flags (--debug, --verbose, --no-color, --profile)
  2. Environment variables (MILADY_*, ANTHROPIC_API_KEY, etc.)
  3. .env file (loaded by dotenv in run-main.ts)
  4. Config file (~/.milady/milady.json)
  5. Profile-specific overrides (--profile <name>)
  6. Built-in defaults (hardcoded in Zod schemas)
Lowest priority

State Directory Resolution

The state directory (~/.milady/) is resolved via:

  1. MILADY_STATE_DIR environment variable (if set)
  2. Default: ~/.milady/ ($HOME/.milady/)

The config file path follows a similar pattern:

  1. MILADY_CONFIG_PATH environment variable (if set)
  2. Default: <stateDir>/milady.json

Config File Structure

Milady uses a Zod-validated JSON configuration file (milady.json):

The config is organized into typed sections: - `agent` / `agents` -- Agent identity, model, personality, sandbox settings - `env` -- Environment variable overrides (persisted API keys, wallet keys) - `gateway` -- Gateway ports, bind mode, TLS, auth - `plugins` -- Plugin allow/deny lists, install records, load paths - `connectors` -- Messaging connectors (Discord, Telegram, Slack, etc.) - `tools` -- Tool profiles, exec config, media understanding - `hooks` -- Webhook mappings, Gmail integration - `messages` -- TTS, queue, broadcast, response prefix - `database` -- Provider selection (PGLite/Postgres), connection config - `cloud` -- Eliza Cloud integration settings - `models` -- Custom model providers, Bedrock discovery - `update` -- Release channel, check intervals - `embedding` -- Local embedding model, GPU layers, dimensions - `features` -- Feature flags (shell, vision, browser, etc.) - `skills` -- Bundled skill allow/deny lists, extra directories - `logging` -- Log level configuration All config sections are validated against Zod schemas at load time. Invalid fields produce warnings but do not prevent startup. The config file includes metadata tracking the last Milady version that wrote it and when.

Security Boundaries

API Authentication

  • Token-based auth via MILADY_API_TOKEN (bearer token on all /api/* requests)
  • Auto-generated token when binding to non-loopback addresses
  • WebSocket connections validated against the same token

Sandbox System

Three sandbox modes control code execution isolation:

Mode Description
off No sandboxing (default)
light Audit logging of all fetch calls, token replacement tracking
standard / max Docker container isolation with configurable image, memory limits, CPU caps, and network restrictions

Sandbox events are recorded to a SandboxAuditLog for security review.

Plugin Security

  • Deny list (config.plugins.deny) blocks specific plugins from loading
  • Core collision detection prevents custom plugins from shadowing core plugins
  • Error boundaries isolate plugin crashes from the main runtime
  • Config allowlist prevents API endpoints from writing sensitive env vars (e.g., MILADY_API_TOKEN is stripped from config patches)

Secret Management

  • API keys and wallet private keys stored in config.env are hydrated into process.env at startup
  • Sensitive values in plugin parameters are masked in API responses (**** or sk-a...z123)
  • The @elizaos/plugin-secrets-manager provides runtime secret access with scoping

Cross-Platform Delivery

The same backend serves all platforms:

Platform Delivery Notes
CLI milady start Direct Node.js process, runner script handles restart exit codes
Web Browser at http://localhost:2138 Vite dev proxy in development
Desktop Electrobun wrapper Bundles the API server + UI, uses the Electrobun updater for native updates
iOS Capacitor Connects to local/remote API
Android Capacitor Connects to local/remote API

The frontend API client (apps/app/src/api-client.ts) is a thin fetch wrapper plus WebSocket for real-time chat and events.

Gateway

The Gateway runs on port 18789 (default) and provides:

  • Multiplexed WebSocket + HTTP on a single port
  • TLS support (auto-generated or custom certificates)
  • Authentication (token or password mode)
  • mDNS/Bonjour discovery for local network agent discovery
  • Tailscale integration (serve/funnel modes)
  • OpenAI-compatible HTTP endpoints (/v1/chat/completions, /v1/responses)

Build System

Tooling

Tool Purpose
tsdown TypeScript bundler (builds src/ into dist/)
Vite Frontend dev server and production bundler
Biome Linter and formatter (lint, format scripts)
Vitest Unit and integration testing
TypeScript Type checking (tsc --noEmit)

Workspace Structure

The monorepo uses Bun workspaces:

milady/
  packages/
    app-core/       # Core runtime, CLI, API, services (source of truth)
      src/
        cli/        # Commander CLI commands
        api/        # Dashboard API server
        runtime/    # Agent loader, plugin system
        config/     # Plugin auto-enable, config schemas
        connectors/ # Connector integration code
        services/   # Business logic (plugin installer, updater, etc.)
    agent/          # Upstream elizaOS agent (core plugins, auto-enable maps)
    plugin-wechat/  # WeChat connector plugin (@miladyai/plugin-wechat)
    ui/             # Shared UI component library
    shared/         # Shared utilities
    vrm-utils/      # VRM avatar utilities
  apps/
    app/            # React SPA frontend (Vite)
      electrobun/   # Electrobun desktop wrapper
    homepage/       # Marketing site
  deploy/
    cloud-agent-template/  # Cloud deployment template
  docs/             # Mintlify documentation
  scripts/          # Build, dev, and utility scripts
  test/             # E2E and integration tests

Build Pipeline

# Full build: TypeScript + build info + frontend
bun run build
# → tsdown (src/ → dist/)
# → scripts/write-build-info.ts (version metadata)
# → cd apps/app && bun run build (Vite frontend)

The scripts/rt.sh helper resolves the best available runtime (bun or Node.js with tsx) for running TypeScript files directly during development.

Engine Requirements

Milady requires Node.js 22+ (specified in package.json engines field). The node-llama-cpp dependency for local embeddings benefits from native compilation on the target platform.

See the [Configuration Schema Reference](/config-schema) for a complete field-by-field breakdown of every config section.