Server/client auth split: API keys, serve command, login flow

[christso]

Objective

Implement API key authentication with a server/client split. The server issues and validates API keys; the CLI authenticates against the server. This validates the architectural split between AgentV server and client.

Design

Key Format

sk-av-<random> (e.g., sk-av-a1b2c3d4e5f6...) — prefixed for easy identification, random suffix via crypto.

Storage

Both files live in ~/.agentv/ (home directory, never in project, never committed):

File	Owner	Purpose
~/.agentv/keys.json	Server	Issued API keys (array of { key, name, created })
~/.agentv/credentials.json	Client	Stored login credential ({ api_key, server_url })

Server

• agentv serve — new command, runs the Hono API server headless (no UI)
• Extract existing Hono API routes from studio command into a shared server module
• studio continues to work standalone (embedded server mode, zero breaking change)
• studio --server <url> connects to a running agentv serve instance instead

Auth Endpoints

Endpoint	Method	Auth	Purpose
POST /api/auth/keys	POST	Bearer token (admin)	Create a new API key
GET /api/auth/keys	GET	Bearer token (admin)	List keys (names + created, not the key values)
DELETE /api/auth/keys/:name	DELETE	Bearer token (admin)	Revoke a key

Server validates incoming Authorization: Bearer <key> against ~/.agentv/keys.json on every request.

CLI Commands

agentv auth create-key --name <name>    # → POST /api/auth/keys → prints key once
agentv auth login                       # → prompts for key + server URL, stores to ~/.agentv/credentials.json
agentv auth login --api-key <key>       # → non-interactive login
agentv auth whoami                      # → GET /api/auth/whoami → prints key name + server

Client Auth Precedence

1. AGENTV_API_KEY env var (CI/CD)
2. --api-key CLI flag
3. ~/.agentv/credentials.json

Bootstrap Problem

The first key must be created without auth. Options:

• agentv serve --init generates a bootstrap admin key on first start, prints it to stdout
• Or: agentv auth create-key works locally (file write) when run on the same machine as the server

Recommend the --init approach since it keeps key creation server-side.

Acceptance Signals

• agentv serve starts headless API server
• agentv serve --init generates and prints bootstrap key
• POST /api/auth/keys creates a key, returns it once
• agentv auth login stores credential to ~/.agentv/credentials.json
• All existing API routes (results, feedback, etc.) reject unauthenticated requests
• agentv studio still works standalone (embedded server, no auth required)
• agentv studio --server <url> connects to remote server with stored credential
• AGENTV_API_KEY env var works for CI

Non-Goals

• User/role management (single admin role for now)
• OAuth / browser pairing flow (API key only for v1)
• Token refresh / expiry (keys are long-lived)
• Database storage (JSON file is sufficient)
• HTTPS / TLS (assumed behind reverse proxy for remote deployments)

Industry Precedent

Research across 6 eval frameworks (Braintrust, Promptfoo, DeepEval, LangWatch, LangSmith, W&B) confirms:

• All use API keys, none use username/password for CLI auth
• LangWatch is the closest precedent: self-hosted server with project-scoped sk-lw-* keys validated by DB lookup, separate web UI auth plane
• Authorization: Bearer <key> is the most standard header convention
• Env var fallback (*_API_KEY) is universal for CI/CD

See: agentevals-research/research/ for full findings.

Implementation Notes

• Extract Hono routes from apps/cli/src/commands/results/serve.ts into a shared module (e.g., packages/core/src/server/)
• serve command imports server module directly
• studio command imports server module + serves React SPA on top
• Auth middleware: Hono middleware that reads Authorization header, looks up in keys.json
• Key generation: crypto.randomBytes(24).toString('hex') → sk-av-<hex>