Epic: Configuration + Adapter Discovery System

[prosdev]

🎯 Overview

Enable per-project MCP adapter configuration and discovery with automatic Cursor integration, centralized storage, and a unified CLI experience.

Key Insight: Users install @lytics/dev-agent globally once, then configure adapters per-project. Each project's .dev-agent/config.json defines which adapters to load, and dev-agent mcp init automatically adds the project to Cursor's MCP config.

🏗️ Architecture

Core Components

1. Centralized Storage - All repository indexes stored globally in ~/.dev-agent/indexes/ (enables cross-repo search, clean project dirs)
2. Project Configuration - Per-project .dev-agent/config.json defines adapters and settings
3. Adapter Discovery - Automatic discovery and loading of built-in, npm, and local adapters
4. CLI Management - Unified commands for MCP setup and adapter management

User Flow

# One-time global install
npm install -g @lytics/dev-agent

# Per-project setup
cd my-project
dev-agent init              # Create config + index repository
dev-agent mcp init         # Add to Cursor MCP config
dev-agent adapter install @company/mcp-adapter-jira  # Install custom adapter

📋 Sub-Issues

This epic is broken down into three focused sub-issues:

• Foundation: Storage + Config System #52: Foundation - Storage + Config System - Centralized storage, config schema, lazy loading
• Adapters: Discovery + Loading System #53: Adapters - Discovery + Loading System - Adapter discovery algorithm, loading mechanism
• UX: CLI Management Commands #54: UX - CLI Management Commands - User-facing commands for MCP + adapter management

✅ Success Criteria

• Users can run dev-agent init && dev-agent mcp init to fully configure a project
• Custom adapters can be installed via dev-agent adapter install
• Team members can clone and get same adapter setup
• Different projects can have different adapters from same global install
• No manual editing of ~/.cursor/mcp.json required
• Indexes stored centrally, enabling cross-repository search
• Built-in adapters can be disabled per-project

🔗 Related Issues

• Depends on: Explore + GitHub Adapters #30 (Explore + GitHub Adapters) - establishes adapter patterns
• Blocks: Documentation + Examples #32 (Documentation) - need to document new setup flow
• Blocks: Integration Testing + Release #33 (Integration Testing) - need to test adapter discovery
• Related: feat(mcp): add structured metadata field to MCP tool responses #51 (Metadata field) - will enhance adapter responses

Estimate: 3-4 days (across 3 sub-issues)
Priority: High (blocks documentation and release)
Milestone: v1.0 - MCP Integration Complete

[prosdev]

💾 Storage Strategy: Centralized Indexes

Overview

All repository indexes are stored globally in ~/.dev-agent/indexes/ rather than per-project. This enables:

• ✅ Clean project directories (no large files to gitignore)
• ✅ Cross-repository search and analysis
• ✅ Shared indexes across clones
• ✅ Easy storage management and cleanup
• ✅ Survives project moves/deletions

Directory Structure

~/.dev-agent/
  indexes/
    a1b2c3d4/              # Frontend (git remote hash)
      vectors.lance        # Code vectors
      github-state.json    # GitHub cache
      metadata.json        # Repository metadata
    e5f6g7h8/              # Backend
      vectors.lance
      github-state.json
      metadata.json
  
  cache/
    embeddings-model/      # Shared ML model (~100MB)
    github-api/            # GitHub API response cache
  
  config/
    global.json            # Global settings (optional)

Storage Location Algorithm

function getStoragePath(repositoryPath: string): string {
  // 1. Try git remote (stable across clones)
  const gitRemote = await getGitRemote(repositoryPath);
  
  if (gitRemote) {
    // Normalize: git@github.com:company/repo.git → company/repo
    const normalized = normalizeGitRemote(gitRemote);
    const hash = crypto.createHash('md5')
      .update(normalized)
      .digest('hex')
      .slice(0, 8);
    
    return path.join(os.homedir(), '.dev-agent/indexes', hash);
  }
  
  // 2. Fallback: absolute path hash (for non-git repos)
  const pathHash = crypto.createHash('md5')
    .update(path.resolve(repositoryPath))
    .digest('hex')
    .slice(0, 8);
  
  return path.join(os.homedir(), '.dev-agent/indexes', pathHash);
}

Metadata File

Each index includes metadata for identification:

// ~/.dev-agent/indexes/a1b2c3d4/metadata.json
{
  "version": "1.0",
  "repository": {
    "path": "/Users/you/workspace/frontend",
    "remote": "git@github.com:company/frontend.git",
    "branch": "main",
    "lastCommit": "abc123..."
  },
  "indexed": {
    "timestamp": "2025-11-25T12:00:00Z",
    "files": 243,
    "components": 1847,
    "size": 52428800
  },
  "config": {
    "languages": ["typescript", "javascript"],
    "excludePatterns": ["**/node_modules/**"]
  }
}

Memory Management

class MCPServer {
  private indexer?: RepositoryIndexer;
  private lastAccessed = Date.now();
  private readonly IDLE_TIMEOUT = 5 * 60 * 1000; // 5 minutes
  
  async ensureIndexer(): Promise<RepositoryIndexer> {
    if (!this.indexer) {
      // Lazy load on first use
      const storagePath = getStoragePath(this.repositoryPath);
      
      this.indexer = new RepositoryIndexer({
        repositoryPath: this.repositoryPath,
        vectorStorePath: path.join(storagePath, 'vectors.lance'),
      });
      
      await this.indexer.initialize();
      this.logger.info('Loaded indexes', { storagePath });
    }
    
    this.lastAccessed = Date.now();
    return this.indexer;
  }
  
  // Auto-unload after idle period
  startIdleMonitor() {
    setInterval(() => {
      const idleTime = Date.now() - this.lastAccessed;
      
      if (idleTime > this.IDLE_TIMEOUT && this.indexer) {
        this.indexer.close();
        this.indexer = undefined;
        this.logger.info('Unloaded indexes (idle timeout)', {
          idleMinutes: Math.floor(idleTime / 60000)
        });
      }
    }, 60000); // Check every minute
  }
}

CLI Storage Management

# Show storage usage and repository list
dev-agent storage info

Storage Location: ~/.dev-agent/indexes/
Total Size: 230 MB

Repositories:
  ID        Repository          Size    Last Indexed    Last Accessed
  a1b2c3d4  company/frontend    50 MB   2 hours ago     Active
  e5f6g7h8  company/backend     80 MB   5 mins ago      Active
  x9y8z7w6  old-project         100 MB  30 days ago     30 days ago

# Clean stale indexes
dev-agent storage clean

? Clean indexes not accessed in: (Select)
  ○ 7 days
  ○ 30 days
  ● 90 days

Found 1 stale index:
  x9y8z7w6  old-project  100 MB  Last accessed: 90 days ago

? Remove stale indexes? (Y/n) y
✓ Removed 1 index (100 MB freed)

# Force rebuild specific repository
dev-agent storage rebuild

? Which repository:
  ● Current (company/frontend)
  ○ All repositories
  ○ Select multiple

✓ Rebuilt company/frontend (50 MB, 243 files, 1847 components)

# Manually add/link repository
dev-agent storage link /path/to/other/clone

✓ Linked to existing index a1b2c3d4 (company/frontend)
  Same git remote detected, using shared storage

# Export indexes (backup or transfer)
dev-agent storage export frontend-indexes.tar.gz

✓ Exported company/frontend to frontend-indexes.tar.gz (50 MB)

# Import indexes (restore or transfer)
dev-agent storage import frontend-indexes.tar.gz

✓ Imported company/frontend to ~/.dev-agent/indexes/a1b2c3d4/

Cross-Repository Features

Centralized storage enables powerful cross-repo capabilities:

1. Search Across All Repositories

# Search all indexed repositories
dev-agent search --all "authentication middleware"

Found in 3 repositories:

company/frontend (2 results):
  • src/lib/auth.ts:45
    export function authMiddleware()
  
company/backend (5 results):
  • src/middleware/auth.ts:12
    export const authenticate = async ()
  • src/guards/jwt.guard.ts:23
    class JwtAuthGuard implements CanActivate

company/shared-lib (1 result):
  • packages/auth/src/index.ts:8
    export { AuthService }

2. Find Code Patterns Across Organization

# Find similar implementations across repos
dev-agent explore pattern --all "rate limiting"

Pattern found in 4 repositories:

company/api-gateway:
  • src/middleware/rate-limit.ts (Redis-based)
  
company/backend:
  • src/utils/throttle.ts (In-memory cache)

company/microservice-a:
  • lib/limiter.js (Token bucket)

# Suggests: Consolidate to shared package?

3. Dependency Analysis Across Organization

# See which repos use a package
dev-agent analyze dependencies "express"

express@4.18.2 used in:
  ✓ company/backend
  ✓ company/api-gateway
  ✓ company/admin-api

express@4.17.1 (outdated) used in:
  ⚠️ company/legacy-service
  ⚠️ company/old-api

Recommendation: Upgrade 2 repositories to 4.18.2

4. Monorepo Intelligence

# Understand monorepo structure
dev-agent analyze monorepo

Analyzing company/monorepo:
  Packages: 23
  Shared dependencies: 45
  Circular dependencies: 0 ✓
  
Package groups:
  • @company/ui-* (8 packages)
  • @company/api-* (5 packages)
  • @company/core-* (10 packages)

Implementation Tasks

• Update RepositoryIndexer to accept storage path
• Implement storage path resolver (git remote → hash)
• Create metadata.json for each index
• Implement lazy loading in MCP server
• Add idle timeout and auto-unload
• Create dev-agent storage command group
• Add cross-repository search capability
• Handle index migration (move existing to centralized)
• Update documentation

Migration Path

For users with existing project-local indexes:

# Automatically detect and migrate
dev-agent storage migrate

Found local indexes:
  ✓ /Users/you/frontend/.dev-agent/vectors.lance (50 MB)
  ✓ /Users/you/backend/.dev-agent/vectors.lance (80 MB)

? Migrate to centralized storage? (Y/n) y

✓ Migrated frontend → ~/.dev-agent/indexes/a1b2c3d4/
✓ Migrated backend → ~/.dev-agent/indexes/e5f6g7h8/
✓ Updated configs to use centralized storage

? Remove old local indexes? (Y/n) y
✓ Removed local indexes (130 MB freed from projects)

Benefits Recap

For Users:

• No large files in Git repos
• Indexes survive project deletion
• Fast when switching between clones
• Easy cleanup of old projects

For Teams:

• Cross-repository search
• Organization-wide code patterns
• Dependency analysis
• Standardization insights

For System:

• Centralized management
• Deduplication potential
• Easier backup/export
• Clear storage monitoring

[prosdev]

Status Update:

Parts of this epic have been implemented:

• ✅ Centralized storage (~/.dev-agent/indexes/)
• ✅ Per-project config (.dev-agent/config.json)
• ✅ dev init creates config
• ✅ dev mcp install --cursor adds to Cursor config
• ✅ Built-in adapters can be enabled/disabled

Not yet implemented:

• 🔲 Custom adapter discovery (npm packages)
• 🔲 dev adapter install command
• 🔲 Dynamic adapter loading at runtime

This is lower priority now that the core MCP integration works well. Moving to backlog.