From 7b007da8ea8f3b41b507c30df33ae75aebf8dbc8 Mon Sep 17 00:00:00 2001 From: Michelangelo Mori <328978+blkt@users.noreply.github.com> Date: Wed, 3 Dec 2025 12:15:48 +0100 Subject: [PATCH 1/9] Add ToolHive Registry documentation This change adds a few short pages about ToolHive Registry covering high-level flows, configuration, deployment, and local execution and development. --- .../guides-registry/api-reference.mdx | 286 ++++++++++++++++ .../guides-registry/configuration.mdx | 179 ++++++++++ docs/toolhive/guides-registry/database.mdx | 242 ++++++++++++++ docs/toolhive/guides-registry/deployment.mdx | 308 ++++++++++++++++++ docs/toolhive/guides-registry/index.mdx | 20 ++ docs/toolhive/guides-registry/intro.mdx | 101 ++++++ docs/toolhive/guides-registry/local-dev.mdx | 108 ++++++ docusaurus.config.ts | 6 +- sidebars.ts | 18 + 9 files changed, 1267 insertions(+), 1 deletion(-) create mode 100644 docs/toolhive/guides-registry/api-reference.mdx create mode 100644 docs/toolhive/guides-registry/configuration.mdx create mode 100644 docs/toolhive/guides-registry/database.mdx create mode 100644 docs/toolhive/guides-registry/deployment.mdx create mode 100644 docs/toolhive/guides-registry/index.mdx create mode 100644 docs/toolhive/guides-registry/intro.mdx create mode 100644 docs/toolhive/guides-registry/local-dev.mdx diff --git a/docs/toolhive/guides-registry/api-reference.mdx b/docs/toolhive/guides-registry/api-reference.mdx new file mode 100644 index 00000000..7060810a --- /dev/null +++ b/docs/toolhive/guides-registry/api-reference.mdx @@ -0,0 +1,286 @@ +--- +title: API reference +description: + Reference documentation for the ToolHive Registry API server endpoints +--- + +The Registry API server implements the official MCP Registry API v0.1 +specification, along with extension endpoints for registry management. + +## Base URL + +All API endpoints are prefixed with the server's base URL. By default, the +server runs on port 8080: + +``` +http://localhost:8080 +``` + +## Registry API (v0.1) + +The core MCP Registry API endpoints for discovering and accessing MCP servers. + +### List all servers + +List all available MCP servers in the registry. + +```http +GET /registry/v0.1/servers +GET /registry/{registryName}/v0.1/servers +``` + +**Query parameters:** + +- `cursor` (optional, string): Pagination cursor for retrieving the next set of + results +- `limit` (optional, integer): Maximum number of items to return +- `search` (optional, string): Search servers by name (substring match) +- `updated_since` (optional, string): Filter servers updated since timestamp + (RFC3339 datetime format, e.g., `2025-08-07T13:15:04.280Z`) +- `version` (optional, string): Filter by version (`latest` for latest version, + or an exact version like `1.2.3`) + +**Response:** + +```json +{ + "servers": [ + { + "name": "example/filesystem", + "description": "Example filesystem MCP server", + "version": "1.0.0" + } + ], + "metadata": { + "nextCursor": "", + "count": 1 + } +} +``` + +### List server versions + +List all versions of a specific server. + +```http +GET /registry/v0.1/servers/{name}/versions +GET /registry/{registryName}/v0.1/servers/{name}/versions +``` + +**Parameters:** + +- `name` (path, required): Server name (URL-encoded) +- `registryName` (path, optional): Registry name (only for registry-specific + endpoint) + +**Response:** + +```json +{ + "servers": [ + { + "name": "example/filesystem", + "version": "1.0.0", + "description": "Example filesystem MCP server" + }, + { + "name": "example/filesystem", + "version": "0.9.0", + "description": "Example filesystem MCP server" + } + ], + "metadata": { + "nextCursor": "", + "count": 2 + } +} +``` + +### Get server version + +Get details for a specific server version. + +```http +GET /registry/v0.1/servers/{name}/versions/{version} +GET /registry/{registryName}/v0.1/servers/{name}/versions/{version} +``` + +**Parameters:** + +- `name` (path, required): Server name (URL-encoded) +- `version` (path, required): Version identifier (use `latest` for the latest + version) +- `registryName` (path, optional): Registry name (only for registry-specific + endpoint) + +**Response:** + +```json +{ + "server": { + "name": "example/filesystem", + "version": "1.0.0", + "description": "Example filesystem MCP server", + "transport": "stdio", + "command": ["node", "server.js"], + "env": { + "NODE_ENV": "production" + } + }, + "meta": {} +} +``` + +### Delete server version + +Delete a server version from a managed registry. + +```http +DELETE /registry/{registryName}/v0.1/servers/{name}/versions/{version} +``` + +**Parameters:** + +- `registryName` (path, required): Registry name +- `name` (path, required): Server name (URL-encoded) +- `version` (path, required): Version identifier + +**Response:** + +- `204 No Content` on success + +**Error responses:** + +- `400 Bad Request` - Invalid request parameters +- `401 Unauthorized` - Invalid or no credentials provided +- `403 Forbidden` - Registry is not a managed registry (cannot delete from + read-only registries) +- `404 Not Found` - Registry or server version not found +- `500 Internal Server Error` - Server error + +### Publish server + +Publish a new server version to the registry. + +```http +POST /{registryName}/v0.1/publish +``` + +**Parameters:** + +- `registryName` (path, required): Registry name + +**Request body:** + +```json +{ + "name": "example/filesystem", + "version": "1.0.0", + "description": "Example filesystem MCP server", + "transport": "stdio", + "command": ["node", "server.js"], + "env": { + "NODE_ENV": "production" + } +} +``` + +**Response:** + +- `201 Created` - Server version published successfully + +**Error responses:** + +- `400 Bad Request` - Invalid request body or missing required fields +- `401 Unauthorized` - Invalid or no credentials provided +- `403 Forbidden` - Registry is not a managed registry (cannot publish to + read-only registries) +- `404 Not Found` - Registry not found +- `409 Conflict` - Version already exists +- `500 Internal Server Error` - Server error + +:::info[Legacy endpoint] + +The endpoint `POST /registry/v0.1/publish` returns `501 Not Implemented`. Use +the registry-specific endpoint `POST /{registryName}/v0.1/publish` instead. + +::: + +## Operational endpoints + +Endpoints for monitoring and operational purposes. + +### Health check + +Check if the server is running. + +```http +GET /health +``` + +**Response:** + +- `200 OK` if the server is running + +### Readiness check + +Check if the server is ready to serve requests. + +```http +GET /readiness +``` + +**Response:** + +- `200 OK` if the server is ready +- `503 Service Unavailable` if the server is not ready + +### Version information + +Get version information about the server. + +```http +GET /version +``` + +**Response:** + +```json +{ + "version": "0.1.0", + "commit": "abc123", + "buildDate": "2024-01-01T00:00:00Z" +} +``` + +## Error responses + +All endpoints may return standard HTTP error responses: + +- `400 Bad Request` - Invalid request parameters or request body +- `401 Unauthorized` - Invalid or no credentials provided +- `403 Forbidden` - Operation not allowed (e.g., attempting to delete or publish + to a read-only registry) +- `404 Not Found` - Resource not found (registry, server, or version) +- `409 Conflict` - Resource conflict (e.g., version already exists) +- `500 Internal Server Error` - Server error +- `501 Not Implemented` - Endpoint not supported +- `503 Service Unavailable` - Service temporarily unavailable + +Error responses include a JSON body with error details: + +```json +{ + "error": "Server not found" +} +``` + +## API specification + +For the complete API specification, see the +[MCP Registry API specification](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/api/openapi.yaml). + +## Next steps + +- [Configure the server](./configuration.mdx) to set up your data sources +- [Deploy the server](./deployment.mdx) in your environment diff --git a/docs/toolhive/guides-registry/configuration.mdx b/docs/toolhive/guides-registry/configuration.mdx new file mode 100644 index 00000000..8dca3afd --- /dev/null +++ b/docs/toolhive/guides-registry/configuration.mdx @@ -0,0 +1,179 @@ +--- +title: Configuration +description: + How to configure the ToolHive Registry API server with data sources and sync + policies +--- + +All configuration is done via YAML files. The server requires a `--config` flag +pointing to a YAML configuration file. + +## Configuration file structure + +```yaml title="config.yaml" +# Registry name/identifier (optional, defaults to "default") +registryName: my-registry + +# Data source configuration (required) +source: + # Source type: git, api, or file + type: git + + # Data format: toolhive (native) or upstream (MCP registry format) + format: toolhive + + # Source-specific configuration + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json + +# Automatic sync policy (required) +syncPolicy: + # Sync interval (e.g., "30m", "1h", "24h") + interval: '30m' + +# Optional: Server filtering +filter: + names: + include: ['official/*'] + exclude: ['*/deprecated'] + tags: + include: ['production'] + exclude: ['experimental'] + +# Optional: Database configuration +database: + host: localhost + port: 5432 + user: registry + database: registry + sslMode: require + maxOpenConns: 25 + maxIdleConns: 5 + connMaxLifetime: '5m' +``` + +## Command-line flags + +| Flag | Description | Required | Default | +| ----------- | ------------------------------- | -------- | ------- | +| `--config` | Path to YAML configuration file | Yes | - | +| `--address` | Server listen address | No | `:8080` | + +## Data sources + +The server supports three data source types, each with its own configuration +options. + +### Git repository source + +Clone and sync from Git repositories. Ideal for version-controlled registries. + +```yaml title="config-git.yaml" +source: + type: git + format: toolhive + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json +``` + +**Configuration options:** + +- `repository` (required): Git repository URL +- `branch` (optional): Branch name to use (defaults to `main`) +- `tag` (optional): Tag name to pin to a specific version +- `commit` (optional): Commit SHA to pin to a specific commit +- `path` (required): Path to the registry file within the repository + +:::tip + +You can use `branch`, `tag`, or `commit` to pin to a specific version. If +multiple are specified, `commit` takes precedence over `tag`, which takes +precedence over `branch`. + +::: + +### API endpoint source + +Sync from upstream MCP Registry APIs. Supports federation and aggregation +scenarios. + +```yaml title="config-api.yaml" +source: + type: api + format: upstream + api: + endpoint: https://registry.example.com/registry/v0.1/servers + headers: + Authorization: Bearer YOUR_TOKEN +``` + +**Configuration options:** + +- `endpoint` (required): URL of the upstream MCP Registry API endpoint +- `headers` (optional): HTTP headers to include in requests (useful for + authentication) + +### Local file source + +Read from filesystem. Ideal for local development and testing. + +```yaml title="config-file.yaml" +source: + type: file + format: toolhive + file: + path: /path/to/registry.json +``` + +**Configuration options:** + +- `path` (required): Path to the registry file on the filesystem + +## Sync policy + +Configure automatic synchronization of registry data. + +```yaml +syncPolicy: + # Sync interval (e.g., "30m", "1h", "24h") + interval: '30m' +``` + +The `interval` field specifies how often the server should fetch updates from +the data source. Use Go duration format (e.g., `"30m"`, `"1h"`, `"24h"`). + +## Server filtering + +Optionally filter which servers are exposed through the API. + +```yaml +filter: + names: + include: ['official/*'] + exclude: ['*/deprecated'] + tags: + include: ['production'] + exclude: ['experimental'] +``` + +**Filter options:** + +- `names.include`: List of name patterns to include (supports wildcards) +- `names.exclude`: List of name patterns to exclude (supports wildcards) +- `tags.include`: List of tags that servers must have +- `tags.exclude`: List of tags that servers must not have + +## Database configuration + +The server optionally supports PostgreSQL database connectivity for storing +registry state and metadata. See the [Database configuration](./database.mdx) +guide for detailed information. + +## Next steps + +- [Deploy the server](./deployment.mdx) with your configuration +- [Configure database storage](./database.mdx) for production use diff --git a/docs/toolhive/guides-registry/database.mdx b/docs/toolhive/guides-registry/database.mdx new file mode 100644 index 00000000..e5686842 --- /dev/null +++ b/docs/toolhive/guides-registry/database.mdx @@ -0,0 +1,242 @@ +--- +title: Database configuration +description: + How to configure PostgreSQL database storage and migrations for the ToolHive + Registry API server +--- + +The Registry API server optionally supports PostgreSQL database connectivity for +storing registry state and metadata. This enables persistence across restarts +and provides a foundation for advanced features. + +## Configuration + +### Basic database configuration + +```yaml title="config.yaml" +database: + host: localhost + port: 5432 + user: registry + database: registry + sslMode: require + maxOpenConns: 25 + maxIdleConns: 5 + connMaxLifetime: '5m' +``` + +### Configuration fields + +| Field | Type | Required | Default | Description | +| ----------------- | ------ | -------- | --------- | -------------------------------------------------------------------------- | +| `host` | string | Yes | - | Database server hostname or IP address | +| `port` | int | Yes | - | Database server port | +| `user` | string | Yes | - | Database username for normal operations | +| `migrationUser` | string | No | `user` | Database username for running migrations (should have elevated privileges) | +| `database` | string | Yes | - | Database name | +| `sslMode` | string | No | `require` | SSL mode (`disable`, `require`, `verify-ca`, `verify-full`) | +| `maxOpenConns` | int | No | `25` | Maximum number of open connections to the database | +| `maxIdleConns` | int | No | `5` | Maximum number of idle connections in the pool | +| `connMaxLifetime` | string | No | `5m` | Maximum lifetime of a connection (e.g., "1h", "30m") | + +\* Password configuration is required but has multiple sources (see Password +Security below) + +## Password security + +The server supports secure password management with separate credentials for +migrations and normal operations. This follows the principle of least privilege +by using elevated privileges only when necessary. + +Password configuration is done using a +[Postgres Password File](https://www.postgresql.org/docs/current/libpq-pgpass.html) +and exporting the `PGPASSFILE` environment variable. + +### Recommended setup + +For production deployments, use separate database users: + +1. **Application user** (`user`): Limited privileges for normal operations + - SELECT, INSERT, UPDATE, DELETE on application tables + - No schema modification privileges + +2. **Migration user** (`migrationUser`): Elevated privileges for migrations + - CREATE, ALTER, DROP on schemas and tables + - Used only during migration operations + +### Example configuration with separate users + +```yaml title="config-production.yaml" +database: + host: db.example.com + port: 5432 + user: db_app + migrationUser: db_migrator + database: registry + sslMode: verify-full +``` + +Store passwords in files with restricted permissions: + +```bash +# Create password files +echo "db.example.com:5432:registry:db_app:app_password" > /etc/secrets/pgpassfile +echo "db.example.com:5432:registry:db_migrator:migrator_password" >> /etc/secrets/pgpassfile + +# Mandatory: restrict permissions, will be ignored otherwise +chmod 600 /etc/secrets/pgpassfile +``` + +You can find more details about user creation and initial configuration in this +[test file](https://github.com/stacklok/toolhive-registry-server/blob/301ccf4e3ad13daad28be7b669d8e5fca337ec14/cmd/thv-registry-api/app/serve_test.go#L56-L103). + +## Database migrations + +The server uses database migrations to manage schema changes. Migrations run +automatically on startup, but you can also run them manually. + +### Automatic migrations + +By default, the server runs migrations automatically when it starts: + +1. Connects to the database using the migration user credentials +2. Checks the current migration version +3. Applies any pending migrations +4. Switches to the application user for normal operations + +This ensures the database schema is always up to date. + +### Manual migrations + +You can run migrations manually using the CLI: + +#### Run migrations + +```bash +thv-registry-api migrate up --config config.yaml [--yes] +``` + +The `--yes` flag skips the confirmation prompt. + +#### Rollback migrations + +```bash +thv-registry-api migrate down --config config.yaml --num-steps N [--yes] +``` + +The `--num-steps` parameter specifies how many migration steps to roll back. + +### Migration user privileges + +The migration user needs the following privileges: + +- CREATE, ALTER, DROP on the target database +- Ability to create and modify tables, indexes, and other schema objects +- SELECT, INSERT, UPDATE, DELETE on the migration tracking table + +Example SQL to create a migration user: + +```sql +DO $$ +DECLARE + migrator_user TEXT := 'db_migrator'; + migrator_password TEXT := 'migrator_password'; + db_name TEXT := 'registry'; +BEGIN + EXECUTE format('CREATE USER %I WITH PASSWORD %L', migrator_user, migrator_password); + EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', db_name, migrator_user); + EXECUTE format('GRANT CREATE ON SCHEMA public TO %I', migrator_user); + EXECUTE format('GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO %I', migrator_user); +END +$$; +``` + +### Application user privileges + +The application user needs limited privileges for normal operations: + +- SELECT, INSERT, UPDATE, DELETE on application tables +- No schema modification privileges + +Example SQL to create an application user: + +```sql +DO $$ +DECLARE + app_user TEXT := 'db_app'; + app_password TEXT := 'app_password'; + db_name TEXT := 'registry'; +BEGIN + CREATE ROLE toolhive_registry_server; + EXECUTE format('CREATE USER %I WITH PASSWORD %L', app_user, app_password); + EXECUTE format('GRANT toolhive_registry_server TO %I', app_user); + EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', db_name, app_user); +END +$$; +``` + +## SSL/TLS configuration + +Configure SSL/TLS for secure database connections: + +- `disable`: No SSL (not recommended for production) +- `require`: Require SSL (default) +- `verify-ca`: Require SSL and verify CA certificate +- `verify-full`: Require SSL and verify both CA and server hostname + +For production, use `verify-full`: + +```yaml +database: + sslMode: verify-full +``` + +## Connection pooling + +Tune connection pool settings for your workload: + +```yaml +database: + maxOpenConns: 25 # Maximum open connections + maxIdleConns: 5 # Maximum idle connections + connMaxLifetime: '5m' # Maximum connection lifetime +``` + +**Guidelines:** + +- `maxOpenConns`: Set based on your database server's connection limits +- `maxIdleConns`: Typically 20-25% of `maxOpenConns` +- `connMaxLifetime`: Set to less than your database server's connection timeout + +## Troubleshooting + +### Connection errors + +If you encounter connection errors: + +1. Verify database credentials are correct +2. Check network connectivity to the database server +3. Ensure the database server allows connections from your host +4. Verify SSL/TLS configuration matches your database server settings + +### Migration errors + +If migrations fail: + +1. Check that the migration user has sufficient privileges +2. Verify the database exists and is accessible +3. Check migration logs for specific error messages +4. Ensure no other processes are modifying the schema concurrently + +### Permission errors + +If you see permission errors during normal operations: + +1. Verify the application user has the required privileges +2. Check that migrations completed successfully +3. Ensure the application user can access all required tables + +## Next steps + +- [Deploy the server](./deployment.mdx) with database configuration +- [Configure data sources](./configuration.mdx) to populate the registry diff --git a/docs/toolhive/guides-registry/deployment.mdx b/docs/toolhive/guides-registry/deployment.mdx new file mode 100644 index 00000000..c0f56d89 --- /dev/null +++ b/docs/toolhive/guides-registry/deployment.mdx @@ -0,0 +1,308 @@ +--- +title: Deployment +description: + How to deploy the ToolHive Registry API server in Kubernetes, Docker, or + Docker Compose +--- + +The Registry API server can be deployed in various environments, from local +development to production Kubernetes clusters. + +## Kubernetes deployment + +The Registry API is designed to run as a sidecar container alongside the +ToolHive Operator's MCPRegistry controller. + +### Basic deployment + +```yaml title="deployment.yaml" +apiVersion: apps/v1 +kind: Deployment +metadata: + name: registry-api +spec: + replicas: 1 + selector: + matchLabels: + app: registry-api + template: + metadata: + labels: + app: registry-api + spec: + containers: + - name: registry-api + image: ghcr.io/stacklok/toolhive/thv-registry-api:latest + args: + - serve + - --config=/etc/registry/config.yaml + ports: + - containerPort: 8080 + name: http + volumeMounts: + - name: config + mountPath: /etc/registry + livenessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 30 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /readiness + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + volumes: + - name: config + configMap: + name: registry-api-config +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: registry-api-config +data: + config.yaml: | + registryName: my-registry + source: + type: git + format: toolhive + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json + syncPolicy: + interval: "15m" +--- +apiVersion: v1 +kind: Service +metadata: + name: registry-api +spec: + selector: + app: registry-api + ports: + - port: 80 + targetPort: 8080 + protocol: TCP + type: ClusterIP +``` + +Apply the deployment: + +```bash +kubectl apply -f deployment.yaml +``` + +### With database + +If you're using a database, you'll need to configure database credentials. Use a +Secret for sensitive information: + +```yaml title="deployment-with-db.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: registry-api-db-secret +type: Opaque +stringData: + password: your-database-password + migration-password: your-migration-password +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: registry-api +spec: + template: + spec: + containers: + - name: registry-api + image: ghcr.io/stacklok/toolhive/thv-registry-api:latest + args: + - serve + - --config=/etc/registry/config.yaml + env: + - name: THV_DATABASE_PASSWORD + valueFrom: + secretKeyRef: + name: registry-api-db-secret + key: password + - name: THV_DATABASE_MIGRATION_PASSWORD + valueFrom: + secretKeyRef: + name: registry-api-db-secret + key: migration-password + volumeMounts: + - name: config + mountPath: /etc/registry + volumes: + - name: config + configMap: + name: registry-api-config +``` + +## Docker deployment + +### Build the image + +```bash +task build-image +``` + +### Run with Git source + +```bash +docker run -v $(pwd)/examples:/config \ + ghcr.io/stacklok/toolhive/thv-registry-api:latest \ + serve --config /config/config-git.yaml +``` + +### Run with file source + +Mount a local registry file: + +```bash +docker run -v $(pwd)/examples:/config \ + -v /path/to/registry.json:/data/registry.json \ + ghcr.io/stacklok/toolhive/thv-registry-api:latest \ + serve --config /config/config-file.yaml +``` + +### Run with database password from environment + +```bash +docker run -v $(pwd)/examples:/config \ + -e THV_DATABASE_PASSWORD=your-password \ + ghcr.io/stacklok/toolhive/thv-registry-api:latest \ + serve --config /config/config-database-dev.yaml +``` + +## Docker Compose deployment + +A complete Docker Compose setup is provided that includes PostgreSQL and the API +server with automatic migrations. + +### Quick start + +Start all services (PostgreSQL + API with automatic migrations): + +```bash +docker-compose up +``` + +Run in detached mode: + +```bash +docker-compose up -d +``` + +View logs: + +```bash +docker-compose logs -f registry-api +``` + +Stop all services: + +```bash +docker-compose down +``` + +Stop and remove volumes (WARNING: deletes database data): + +```bash +docker-compose down -v +``` + +### Architecture + +The docker-compose.yaml includes two services: + +1. **postgres** - PostgreSQL 18 database server +2. **registry-api** - Main API server (runs migrations automatically on startup) + +**Service startup flow:** + +``` +postgres (healthy) → registry-api (runs migrations, then starts) +``` + +### Configuration + +- Config file: `examples/config-docker.yaml` +- Sample data: `examples/registry-sample.json` +- Database passwords: Set via environment variables in docker-compose.yaml + - `THV_DATABASE_PASSWORD`: Application user password + - `THV_DATABASE_MIGRATION_PASSWORD`: Migration user password + +The setup demonstrates: + +- Database-backed registry storage with separate users for migrations and + operations +- Automatic schema migrations on startup using elevated privileges +- Normal operations using limited database privileges (principle of least + privilege) +- File-based data source (for demo purposes) +- Proper service dependencies and health checks + +### Accessing the API + +Once running, the API is available at http://localhost:8080 + +```bash +# List all servers +curl http://localhost:8080/registry/v0.1/servers + +# Get specific server version +curl http://localhost:8080/registry/v0.1/servers/example%2Ffilesystem/versions/latest +``` + +### Customization + +To use your own registry data: + +1. Edit `examples/registry-sample.json` with your MCP servers +2. Or change the source configuration in `examples/config-docker.yaml` +3. Restart: `docker-compose restart registry-api` + +### Database access + +The Docker Compose setup creates three database users: + +- `registry`: Superuser (for administration) +- `db_migrator`: Migration user with schema modification privileges +- `db_app`: Application user with limited data access privileges + +To connect to the PostgreSQL database directly: + +```bash +# As superuser (for administration) +docker exec -it toolhive-registry-postgres psql -U registry -d registry + +# As application user +docker exec -it toolhive-registry-postgres psql -U db_app -d registry + +# From host machine +PGPASSWORD=registry_password psql -h localhost -U registry -d registry +PGPASSWORD=app_password psql -h localhost -U db_app -d registry +``` + +## Health checks + +The server provides health check endpoints: + +- `GET /health` - Health check (returns 200 if server is running) +- `GET /readiness` - Readiness check (returns 200 if server is ready to serve + requests) + +Use these endpoints for Kubernetes liveness and readiness probes, or for +monitoring and load balancer health checks. + +## Next steps + +- [Configure the API endpoints](./api-reference.mdx) to understand available + operations +- [Set up database storage](./database.mdx) for production deployments diff --git a/docs/toolhive/guides-registry/index.mdx b/docs/toolhive/guides-registry/index.mdx new file mode 100644 index 00000000..26bd986d --- /dev/null +++ b/docs/toolhive/guides-registry/index.mdx @@ -0,0 +1,20 @@ +--- +title: ToolHive Registry API Server +description: + How-to guides for using the ToolHive Registry API server to discover and + access MCP servers. +--- + +import DocCardList from '@theme/DocCardList'; + +## Introduction + +The ToolHive Registry API server implements the official +[Model Context Protocol (MCP) Registry API specification](https://modelcontextprotocol.io/development/roadmap#registry). +It provides a standardized REST API for discovering and accessing MCP servers +from multiple backend sources, including Kubernetes clusters, Git repositories, +API endpoints, and local files. + +## Contents + + diff --git a/docs/toolhive/guides-registry/intro.mdx b/docs/toolhive/guides-registry/intro.mdx new file mode 100644 index 00000000..321939fa --- /dev/null +++ b/docs/toolhive/guides-registry/intro.mdx @@ -0,0 +1,101 @@ +--- +title: Overview +description: + How to use the ToolHive Registry API server to discover and access MCP servers +--- + +The ToolHive Registry API server (`thv-registry-api`) is a standards-compliant +implementation of the MCP Registry API specification. It provides a REST API for +discovering and accessing MCP servers from multiple backend sources. + +## Overview + +The Registry API server aggregates MCP server metadata from various sources and +exposes it through a standardized API. When you start the server, it: + +1. Loads configuration from a YAML file +2. Runs database migrations automatically (if database is configured) +3. Immediately fetches registry data from the configured source +4. Starts background sync coordinator for automatic updates +5. Serves MCP Registry API endpoints on the configured address + +```mermaid +flowchart TB + subgraph Sources["Registry Sources"] + Managed["Managed Registry"] + API["Upstream Registry"] + Kubernetes["Kubernetes cluster"] + Git["Git Repository"] + File["Local File"] + end + + subgraph Server["Registry API Server"] + Config["Configuration"] + Sync["Sync Manager"] + Storage["Storage Layer"] + APIHandler["API Handler"] + end + + subgraph Clients["Clients"] + ToolHive["ToolHive"] + MCPClient["MCP Clients"] + end + + Sources -->|sync| Server + Config --> Sync + Sync --> Storage + Storage --> APIHandler + APIHandler -->|REST API| Clients +``` + +## Features + +- **Standards-compliant**: Implements the official MCP Registry API + specification +- **Multiple registry sources**: Git repositories, API endpoints, and local + files +- **Automatic synchronization**: Background sync with configurable intervals and + retry logic +- **Container-ready**: Designed for deployment in Kubernetes clusters, but can + be deployed anywhere +- **Flexible deployment**: Works standalone or as part of ToolHive + infrastructure +- **Production-ready**: Built-in health checks, graceful shutdown, and sync + status persistence + +## Registry sources + +The server supports three registry source types: + +1. **Managed Registry** - A fully-managed MCP Registry + - Ideal for private repositories + - Automatically exposes entries following upstream MCP Registry format + - Supports adding new MCP servers via `/publish` endpoint + +2. **Upstream Registry** - Sync from upstream MCP Registry APIs + - Supports federation and aggregation scenarios + - Format conversion from upstream to ToolHive format + - Does not support publishing + +3. **Kubernetes Cluster** - Automatically creates registry entries for running + workloads + - Ideal to quickly grant access to running MCP servers to knowledge workers + - Useful for bigger organizations where MCP server developers differ from + users + - Supports adding new MCP servers via `/publish` endpoint + +4. **Git Repository** - Clone and sync from Git repositories + - Supports branch, tag, or commit pinning + - Ideal for version-controlled registries + - Does not support publishing + +5. **Local File** - Read from filesystem + - Ideal for local development and testing + - Supports mounted volumes in containers + - Does not support publishing + +## Next steps + +- [Install the Registry API server](./install.mdx) to get started +- [Configure registry sources](./configuration.mdx) to set up your registry +- [Deploy the server](./deployment.mdx) in your environment diff --git a/docs/toolhive/guides-registry/local-dev.mdx b/docs/toolhive/guides-registry/local-dev.mdx new file mode 100644 index 00000000..1221b87a --- /dev/null +++ b/docs/toolhive/guides-registry/local-dev.mdx @@ -0,0 +1,108 @@ +--- +title: Local development +description: How to install and run the ToolHive Registry API server locally +--- + +## Prerequisites + +- Go 1.23 or later (for building from source) +- [Task](https://taskfile.dev) for build automation (for building from source) +- [Docker](https://docs.docker.com/desktop/) to run integration tests + +## Building from source + +To build the binary from source: + +```bash +# Build the binary +task build +``` + +This creates the `thv-registry-api` binary in your current directory. + +## Running the server + +All configuration is done via YAML configuration files. The server requires a +`--config` flag pointing to a YAML configuration file. + +### Quick start with Git source + +```bash +thv-registry-api serve --config examples/config-git.yaml +``` + +### Quick start with local file + +```bash +thv-registry-api serve --config examples/config-file.yaml +``` + +### Quick start with API endpoint + +```bash +thv-registry-api serve --config examples/config-api.yaml +``` + +The server starts on port 8080 by default. Use `--address :PORT` to customize +the listen address: + +```bash +thv-registry-api serve --config config.yaml --address :9090 +``` + +## What happens when the server starts + +When you start the server, it performs the following steps: + +1. Loads configuration from the specified YAML file +2. Runs database migrations automatically (if database is configured) +3. Immediately fetches registry data from the configured source +4. Starts background sync coordinator for automatic updates +5. Serves MCP Registry API endpoints on the configured address + +## Available commands + +The `thv-registry-api` CLI provides the following commands: + +### Start the API server + +```bash +thv-registry-api serve --config config.yaml [--address :8080] +``` + +### Database migrations + +Manually run database migrations: + +```bash +# Run migrations +thv-registry-api migrate up --config config.yaml [--yes] + +# Rollback migrations +thv-registry-api migrate down --config config.yaml --num-steps N [--yes] +``` + +See the [Database configuration](./database.mdx) guide for more details on using +migration commands. + +### Version information + +Display version information: + +```bash +thv-registry-api version [--format json] +``` + +### Help + +Show help for commands: + +```bash +thv-registry-api --help +thv-registry-api --help +``` + +## Next steps + +- [Configure the server](./configuration.mdx) to set up your data sources +- [Deploy the server](./deployment.mdx) in your environment diff --git a/docusaurus.config.ts b/docusaurus.config.ts index 607581db..513b1ba1 100644 --- a/docusaurus.config.ts +++ b/docusaurus.config.ts @@ -175,6 +175,10 @@ const config: Config = { label: 'Virtual MCP Server', to: 'toolhive/guides-vmcp', }, + { + label: 'ToolHive Registry', + to: 'toolhive/guides-registry', + }, ], }, { @@ -191,7 +195,7 @@ const config: Config = { to: 'toolhive/reference/api', }, { - label: 'ToolHive registry schema', + label: 'ToolHive Registry schema', to: 'toolhive/reference/registry-schema', }, { diff --git a/sidebars.ts b/sidebars.ts index 9d3d0d85..dc8d82c2 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -183,6 +183,24 @@ const sidebars: SidebarsConfig = { ], }, + { + type: 'category', + label: 'Guides: Registry Server', + description: + 'How to deploy and use the ToolHive Registry server to discover and access MCP servers', + link: { + type: 'doc', + id: 'toolhive/guides-registry/index', + }, + items: [ + 'toolhive/guides-registry/intro', + 'toolhive/guides-registry/configuration', + 'toolhive/guides-registry/api-reference', + 'toolhive/guides-registry/database', + 'toolhive/guides-registry/local-dev', + ], + }, + { type: 'category', label: 'Concepts', From 7b0b4c4d4f9b48a6b14a7aa6048f07c8db33069f Mon Sep 17 00:00:00 2001 From: Michelangelo Mori <328978+blkt@users.noreply.github.com> Date: Wed, 3 Dec 2025 17:37:25 +0100 Subject: [PATCH 2/9] WIP --- .../guides-registry/configuration.mdx | 4 - docs/toolhive/guides-registry/deployment.mdx | 276 ++++-------------- docs/toolhive/guides-registry/intro.mdx | 4 +- docs/toolhive/guides-registry/local-dev.mdx | 108 ------- sidebars.ts | 2 +- 5 files changed, 56 insertions(+), 338 deletions(-) delete mode 100644 docs/toolhive/guides-registry/local-dev.mdx diff --git a/docs/toolhive/guides-registry/configuration.mdx b/docs/toolhive/guides-registry/configuration.mdx index 8dca3afd..5cadad87 100644 --- a/docs/toolhive/guides-registry/configuration.mdx +++ b/docs/toolhive/guides-registry/configuration.mdx @@ -107,15 +107,11 @@ source: format: upstream api: endpoint: https://registry.example.com/registry/v0.1/servers - headers: - Authorization: Bearer YOUR_TOKEN ``` **Configuration options:** - `endpoint` (required): URL of the upstream MCP Registry API endpoint -- `headers` (optional): HTTP headers to include in requests (useful for - authentication) ### Local file source diff --git a/docs/toolhive/guides-registry/deployment.mdx b/docs/toolhive/guides-registry/deployment.mdx index c0f56d89..1fe6ecd6 100644 --- a/docs/toolhive/guides-registry/deployment.mdx +++ b/docs/toolhive/guides-registry/deployment.mdx @@ -1,8 +1,6 @@ --- -title: Deployment -description: - How to deploy the ToolHive Registry API server in Kubernetes, Docker, or - Docker Compose +title: Deployment in Kubernetes +description: How to deploy the ToolHive Registry API server in Kubernetes --- The Registry API server can be deployed in various environments, from local @@ -10,10 +8,14 @@ development to production Kubernetes clusters. ## Kubernetes deployment -The Registry API is designed to run as a sidecar container alongside the -ToolHive Operator's MCPRegistry controller. +The Registry API is designed to run as an independent deployment, possibly +alongside the ToolHive Operator. -### Basic deployment +Although it is possible to run ToolHive Registry to use an in-memory store, it +is unreliable to run multiple replicas as they would not share state, and we +recommend running it with a proper Postgres database. + +### Deployment Example ```yaml title="deployment.yaml" apiVersion: apps/v1 @@ -32,16 +34,25 @@ spec: spec: containers: - name: registry-api - image: ghcr.io/stacklok/toolhive/thv-registry-api:latest + image: ghcr.io/stacklok/thv-registry-api:latest args: - serve - --config=/etc/registry/config.yaml + env: + - name: PGPASSFILE + value: /pgpass/.pgpass ports: - containerPort: 8080 name: http volumeMounts: - name: config - mountPath: /etc/registry + mountPath: /etc/registry/config.yaml + subPath: config.yaml + readOnly: true + - name: pgpass + mountPath: /etc/registry/pgpass + subPath: pgpass + readOnly: true livenessProbe: httpGet: path: /health @@ -58,6 +69,15 @@ spec: - name: config configMap: name: registry-api-config + items: + - key: config.yaml + path: config.yaml + - name: pgpass + secret: + secretName: registry-api-pgpass + items: + - key: pgpass + path: pgpass --- apiVersion: v1 kind: ConfigMap @@ -66,15 +86,34 @@ metadata: data: config.yaml: | registryName: my-registry - source: - type: git + registries: + - name: git-registry format: toolhive git: repository: https://github.com/stacklok/toolhive.git branch: main path: pkg/registry/data/registry.json - syncPolicy: - interval: "15m" + syncPolicy: + interval: "15m" + auth: + mode: anonymous + database: + host: db.example.com + port: 5432 + user: db_app + migrationUser: db_migrator + database: registry + sslMode: verify-full +--- +apiVersion: v1 +kind: Secret +metadata: + name: registry-api-pgpass +type: Opaque +stringData: + pgpass: | + *:5432:registry:db_app:app_password + *:5432:registry:db_migrator:migrator_password --- apiVersion: v1 kind: Service @@ -95,214 +134,3 @@ Apply the deployment: ```bash kubectl apply -f deployment.yaml ``` - -### With database - -If you're using a database, you'll need to configure database credentials. Use a -Secret for sensitive information: - -```yaml title="deployment-with-db.yaml" -apiVersion: v1 -kind: Secret -metadata: - name: registry-api-db-secret -type: Opaque -stringData: - password: your-database-password - migration-password: your-migration-password ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: registry-api -spec: - template: - spec: - containers: - - name: registry-api - image: ghcr.io/stacklok/toolhive/thv-registry-api:latest - args: - - serve - - --config=/etc/registry/config.yaml - env: - - name: THV_DATABASE_PASSWORD - valueFrom: - secretKeyRef: - name: registry-api-db-secret - key: password - - name: THV_DATABASE_MIGRATION_PASSWORD - valueFrom: - secretKeyRef: - name: registry-api-db-secret - key: migration-password - volumeMounts: - - name: config - mountPath: /etc/registry - volumes: - - name: config - configMap: - name: registry-api-config -``` - -## Docker deployment - -### Build the image - -```bash -task build-image -``` - -### Run with Git source - -```bash -docker run -v $(pwd)/examples:/config \ - ghcr.io/stacklok/toolhive/thv-registry-api:latest \ - serve --config /config/config-git.yaml -``` - -### Run with file source - -Mount a local registry file: - -```bash -docker run -v $(pwd)/examples:/config \ - -v /path/to/registry.json:/data/registry.json \ - ghcr.io/stacklok/toolhive/thv-registry-api:latest \ - serve --config /config/config-file.yaml -``` - -### Run with database password from environment - -```bash -docker run -v $(pwd)/examples:/config \ - -e THV_DATABASE_PASSWORD=your-password \ - ghcr.io/stacklok/toolhive/thv-registry-api:latest \ - serve --config /config/config-database-dev.yaml -``` - -## Docker Compose deployment - -A complete Docker Compose setup is provided that includes PostgreSQL and the API -server with automatic migrations. - -### Quick start - -Start all services (PostgreSQL + API with automatic migrations): - -```bash -docker-compose up -``` - -Run in detached mode: - -```bash -docker-compose up -d -``` - -View logs: - -```bash -docker-compose logs -f registry-api -``` - -Stop all services: - -```bash -docker-compose down -``` - -Stop and remove volumes (WARNING: deletes database data): - -```bash -docker-compose down -v -``` - -### Architecture - -The docker-compose.yaml includes two services: - -1. **postgres** - PostgreSQL 18 database server -2. **registry-api** - Main API server (runs migrations automatically on startup) - -**Service startup flow:** - -``` -postgres (healthy) → registry-api (runs migrations, then starts) -``` - -### Configuration - -- Config file: `examples/config-docker.yaml` -- Sample data: `examples/registry-sample.json` -- Database passwords: Set via environment variables in docker-compose.yaml - - `THV_DATABASE_PASSWORD`: Application user password - - `THV_DATABASE_MIGRATION_PASSWORD`: Migration user password - -The setup demonstrates: - -- Database-backed registry storage with separate users for migrations and - operations -- Automatic schema migrations on startup using elevated privileges -- Normal operations using limited database privileges (principle of least - privilege) -- File-based data source (for demo purposes) -- Proper service dependencies and health checks - -### Accessing the API - -Once running, the API is available at http://localhost:8080 - -```bash -# List all servers -curl http://localhost:8080/registry/v0.1/servers - -# Get specific server version -curl http://localhost:8080/registry/v0.1/servers/example%2Ffilesystem/versions/latest -``` - -### Customization - -To use your own registry data: - -1. Edit `examples/registry-sample.json` with your MCP servers -2. Or change the source configuration in `examples/config-docker.yaml` -3. Restart: `docker-compose restart registry-api` - -### Database access - -The Docker Compose setup creates three database users: - -- `registry`: Superuser (for administration) -- `db_migrator`: Migration user with schema modification privileges -- `db_app`: Application user with limited data access privileges - -To connect to the PostgreSQL database directly: - -```bash -# As superuser (for administration) -docker exec -it toolhive-registry-postgres psql -U registry -d registry - -# As application user -docker exec -it toolhive-registry-postgres psql -U db_app -d registry - -# From host machine -PGPASSWORD=registry_password psql -h localhost -U registry -d registry -PGPASSWORD=app_password psql -h localhost -U db_app -d registry -``` - -## Health checks - -The server provides health check endpoints: - -- `GET /health` - Health check (returns 200 if server is running) -- `GET /readiness` - Readiness check (returns 200 if server is ready to serve - requests) - -Use these endpoints for Kubernetes liveness and readiness probes, or for -monitoring and load balancer health checks. - -## Next steps - -- [Configure the API endpoints](./api-reference.mdx) to understand available - operations -- [Set up database storage](./database.mdx) for production deployments diff --git a/docs/toolhive/guides-registry/intro.mdx b/docs/toolhive/guides-registry/intro.mdx index 321939fa..27d9b2a6 100644 --- a/docs/toolhive/guides-registry/intro.mdx +++ b/docs/toolhive/guides-registry/intro.mdx @@ -62,6 +62,8 @@ flowchart TB infrastructure - **Production-ready**: Built-in health checks, graceful shutdown, and sync status persistence +- **Kubernetes-aware**: Automatic synchronization with ToolHive Operator to keep + running MCP servers in sync with Registry ## Registry sources @@ -96,6 +98,6 @@ The server supports three registry source types: ## Next steps -- [Install the Registry API server](./install.mdx) to get started +- [Install the Registry API server](./deployment.mdx) to get started - [Configure registry sources](./configuration.mdx) to set up your registry - [Deploy the server](./deployment.mdx) in your environment diff --git a/docs/toolhive/guides-registry/local-dev.mdx b/docs/toolhive/guides-registry/local-dev.mdx deleted file mode 100644 index 1221b87a..00000000 --- a/docs/toolhive/guides-registry/local-dev.mdx +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: Local development -description: How to install and run the ToolHive Registry API server locally ---- - -## Prerequisites - -- Go 1.23 or later (for building from source) -- [Task](https://taskfile.dev) for build automation (for building from source) -- [Docker](https://docs.docker.com/desktop/) to run integration tests - -## Building from source - -To build the binary from source: - -```bash -# Build the binary -task build -``` - -This creates the `thv-registry-api` binary in your current directory. - -## Running the server - -All configuration is done via YAML configuration files. The server requires a -`--config` flag pointing to a YAML configuration file. - -### Quick start with Git source - -```bash -thv-registry-api serve --config examples/config-git.yaml -``` - -### Quick start with local file - -```bash -thv-registry-api serve --config examples/config-file.yaml -``` - -### Quick start with API endpoint - -```bash -thv-registry-api serve --config examples/config-api.yaml -``` - -The server starts on port 8080 by default. Use `--address :PORT` to customize -the listen address: - -```bash -thv-registry-api serve --config config.yaml --address :9090 -``` - -## What happens when the server starts - -When you start the server, it performs the following steps: - -1. Loads configuration from the specified YAML file -2. Runs database migrations automatically (if database is configured) -3. Immediately fetches registry data from the configured source -4. Starts background sync coordinator for automatic updates -5. Serves MCP Registry API endpoints on the configured address - -## Available commands - -The `thv-registry-api` CLI provides the following commands: - -### Start the API server - -```bash -thv-registry-api serve --config config.yaml [--address :8080] -``` - -### Database migrations - -Manually run database migrations: - -```bash -# Run migrations -thv-registry-api migrate up --config config.yaml [--yes] - -# Rollback migrations -thv-registry-api migrate down --config config.yaml --num-steps N [--yes] -``` - -See the [Database configuration](./database.mdx) guide for more details on using -migration commands. - -### Version information - -Display version information: - -```bash -thv-registry-api version [--format json] -``` - -### Help - -Show help for commands: - -```bash -thv-registry-api --help -thv-registry-api --help -``` - -## Next steps - -- [Configure the server](./configuration.mdx) to set up your data sources -- [Deploy the server](./deployment.mdx) in your environment diff --git a/sidebars.ts b/sidebars.ts index dc8d82c2..ae982e6e 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -197,7 +197,7 @@ const sidebars: SidebarsConfig = { 'toolhive/guides-registry/configuration', 'toolhive/guides-registry/api-reference', 'toolhive/guides-registry/database', - 'toolhive/guides-registry/local-dev', + 'toolhive/guides-registry/deployment', ], }, From 137beec840e5459749391b7f1089553b9529c304 Mon Sep 17 00:00:00 2001 From: Radoslav Dimitrov Date: Thu, 4 Dec 2025 02:41:50 +0200 Subject: [PATCH 3/9] Fix Registry API documentation to match current implementation Signed-off-by: Radoslav Dimitrov --- .../guides-registry/api-reference.mdx | 38 +++ .../guides-registry/configuration.mdx | 242 +++++++++++++----- docs/toolhive/guides-registry/database.mdx | 35 ++- docs/toolhive/guides-registry/deployment.mdx | 13 +- docs/toolhive/guides-registry/intro.mdx | 17 +- 5 files changed, 269 insertions(+), 76 deletions(-) diff --git a/docs/toolhive/guides-registry/api-reference.mdx b/docs/toolhive/guides-registry/api-reference.mdx index 7060810a..cff53e23 100644 --- a/docs/toolhive/guides-registry/api-reference.mdx +++ b/docs/toolhive/guides-registry/api-reference.mdx @@ -20,6 +20,44 @@ http://localhost:8080 The core MCP Registry API endpoints for discovering and accessing MCP servers. +The server provides two types of registry endpoints to support different use +cases: + +### Endpoint Types + +**1. Aggregated Registry Endpoints (Read-Only)** + +These endpoints provide a unified view across **all configured registries**. +Ideal for enterprise-wide discovery where you want to query all available MCP +servers from all sources (Git, API, File, Managed, Kubernetes). + +- Pattern: `/registry/v0.1/...` +- Example: `GET /registry/v0.1/servers` +- Use case: Enterprise UI showing complete catalog from all registries + +**2. Per-Registry Endpoints (Standards-Compliant)** + +These endpoints access a **specific registry** by name. Fully compatible with +the upstream MCP Registry API specification and support both read and write +operations (for managed registries). + +- Pattern: `/registry/{registryName}/v0.1/...` +- Example: `GET /registry/toolhive/v0.1/servers` +- Use case: Direct access to a specific registry (e.g., only upstream verified + MCPs) + +:::tip + +- Use aggregated endpoints (`/registry/v0.1/...`) when you want to discover + servers from all configured registries +- Use per-registry endpoints (`/registry/{registryName}/v0.1/...`) when you need + to: + - Query a specific registry + - Publish or delete servers (only supported for managed registries) + - Maintain compatibility with upstream MCP Registry API clients + +::: + ### List all servers List all available MCP servers in the registry. diff --git a/docs/toolhive/guides-registry/configuration.mdx b/docs/toolhive/guides-registry/configuration.mdx index 5cadad87..7f5dfebf 100644 --- a/docs/toolhive/guides-registry/configuration.mdx +++ b/docs/toolhive/guides-registry/configuration.mdx @@ -14,33 +14,36 @@ pointing to a YAML configuration file. # Registry name/identifier (optional, defaults to "default") registryName: my-registry -# Data source configuration (required) -source: - # Source type: git, api, or file - type: git - - # Data format: toolhive (native) or upstream (MCP registry format) - format: toolhive - - # Source-specific configuration - git: - repository: https://github.com/stacklok/toolhive.git - branch: main - path: pkg/registry/data/registry.json - -# Automatic sync policy (required) -syncPolicy: - # Sync interval (e.g., "30m", "1h", "24h") - interval: '30m' - -# Optional: Server filtering -filter: - names: - include: ['official/*'] - exclude: ['*/deprecated'] - tags: - include: ['production'] - exclude: ['experimental'] +# Registries configuration (required, can have multiple registries) +registries: + - name: toolhive + # Data format: toolhive (native) or upstream (MCP registry format) + format: toolhive + + # Git repository configuration + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json + + # Per-registry automatic sync policy (required for synced registries) + syncPolicy: + # Sync interval (e.g., "30m", "1h", "24h") + interval: '30m' + + # Optional: Per-registry server filtering + filter: + names: + include: ['official/*'] + exclude: ['*/deprecated'] + tags: + include: ['production'] + exclude: ['experimental'] + +# Authentication configuration (required) +auth: + # Mode: anonymous or oauth (defaults to oauth if not specified) + mode: anonymous # Optional: Database configuration database: @@ -63,21 +66,26 @@ database: ## Data sources -The server supports three data source types, each with its own configuration -options. +The server supports five registry types, each with its own configuration +options. You can configure multiple registries in a single server instance. ### Git repository source Clone and sync from Git repositories. Ideal for version-controlled registries. ```yaml title="config-git.yaml" -source: - type: git - format: toolhive - git: - repository: https://github.com/stacklok/toolhive.git - branch: main - path: pkg/registry/data/registry.json +registries: + - name: toolhive + format: toolhive + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json + syncPolicy: + interval: '30m' + +auth: + mode: anonymous ``` **Configuration options:** @@ -102,58 +110,168 @@ Sync from upstream MCP Registry APIs. Supports federation and aggregation scenarios. ```yaml title="config-api.yaml" -source: - type: api - format: upstream - api: - endpoint: https://registry.example.com/registry/v0.1/servers +registries: + - name: mcp-upstream + format: upstream + api: + endpoint: https://registry.modelcontextprotocol.io + syncPolicy: + interval: '1h' + +auth: + mode: anonymous ``` **Configuration options:** -- `endpoint` (required): URL of the upstream MCP Registry API endpoint +- `endpoint` (required): Base URL of the upstream MCP Registry API (without + path) +- `format` (required): Must be `upstream` for API sources + +:::note + +The server automatically appends the appropriate API paths (`/v0.1/servers`, +etc.) to the endpoint URL. + +::: ### Local file source Read from filesystem. Ideal for local development and testing. ```yaml title="config-file.yaml" -source: - type: file - format: toolhive - file: - path: /path/to/registry.json +registries: + - name: local + format: toolhive + file: + path: /data/registry.json + syncPolicy: + interval: '15m' + +auth: + mode: anonymous ``` **Configuration options:** - `path` (required): Path to the registry file on the filesystem +### Managed registry + +API-managed registry for directly publishing and deleting MCP servers via the +API. Does not sync from external sources. + +```yaml title="config-managed.yaml" +registries: + - name: internal + format: toolhive + managed: {} + +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: keycloak + issuerUrl: https://keycloak.example.com/realms/mcp + audience: registry-api +``` + +**Configuration options:** + +- `managed` (required): Empty object to indicate managed registry type +- No sync policy required (managed registries are updated via API, not synced) + +**Supported operations:** + +- `POST /{registryName}/v0.1/publish` - Publish new server versions +- `DELETE /{registryName}/v0.1/servers/{name}/versions/{version}` - Delete + versions +- `GET` operations for listing and retrieving servers + +### Kubernetes registry + +Discovers MCP servers from running Kubernetes deployments. Automatically creates +registry entries for deployed MCP servers in your cluster. + +```yaml title="config-kubernetes.yaml" +registries: + - name: k8s-cluster + format: toolhive + kubernetes: {} + +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: kubernetes + issuerUrl: https://kubernetes.default.svc.cluster.local + audience: https://kubernetes.default.svc.cluster.local + caCertPath: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt +``` + +**Configuration options:** + +- `kubernetes` (required): Empty object to indicate Kubernetes registry type +- No sync policy required (Kubernetes registries query live deployments + on-demand) + +**Use cases:** + +- Discover MCP servers deployed via ToolHive Operator +- Automatically expose running MCP servers to knowledge workers +- Separate MCP server development from user consumption + ## Sync policy -Configure automatic synchronization of registry data. +Configure automatic synchronization of registry data on a per-registry basis. +Only applicable to synced registries (Git, API, File). Not required for managed +or Kubernetes registries. ```yaml -syncPolicy: - # Sync interval (e.g., "30m", "1h", "24h") - interval: '30m' +registries: + - name: toolhive + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json + syncPolicy: + # Sync interval (e.g., "30m", "1h", "24h") + interval: '30m' ``` The `interval` field specifies how often the server should fetch updates from the data source. Use Go duration format (e.g., `"30m"`, `"1h"`, `"24h"`). +:::note + +Sync policy is per-registry and must be specified within each registry +configuration. Managed and Kubernetes registries do not require sync policies. + +::: + ## Server filtering -Optionally filter which servers are exposed through the API. +Optionally filter which servers are exposed through the API on a per-registry +basis. Only applicable to synced registries (Git, API, File). ```yaml -filter: - names: - include: ['official/*'] - exclude: ['*/deprecated'] - tags: - include: ['production'] - exclude: ['experimental'] +registries: + - name: toolhive + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json + syncPolicy: + interval: '30m' + filter: + names: + include: ['official/*'] + exclude: ['*/deprecated'] + tags: + include: ['production'] + exclude: ['experimental'] ``` **Filter options:** @@ -163,6 +281,12 @@ filter: - `tags.include`: List of tags that servers must have - `tags.exclude`: List of tags that servers must not have +:::note + +Filters are per-registry and specified within each registry configuration. + +::: + ## Database configuration The server optionally supports PostgreSQL database connectivity for storing diff --git a/docs/toolhive/guides-registry/database.mdx b/docs/toolhive/guides-registry/database.mdx index e5686842..629a788b 100644 --- a/docs/toolhive/guides-registry/database.mdx +++ b/docs/toolhive/guides-registry/database.mdx @@ -76,17 +76,46 @@ database: sslMode: verify-full ``` -Store passwords in files with restricted permissions: +Store passwords in a pgpass file with restricted permissions: ```bash -# Create password files +# Create pgpass file (recommended location: /etc/secrets/pgpassfile) echo "db.example.com:5432:registry:db_app:app_password" > /etc/secrets/pgpassfile echo "db.example.com:5432:registry:db_migrator:migrator_password" >> /etc/secrets/pgpassfile -# Mandatory: restrict permissions, will be ignored otherwise +# Mandatory: restrict permissions to 0600, will be ignored otherwise chmod 600 /etc/secrets/pgpassfile ``` +**Using the pgpass file:** + +Set the `PGPASSFILE` environment variable when running the server: + +```bash +# For standalone server +export PGPASSFILE=/etc/secrets/pgpassfile +thv-registry-api serve --config config.yaml + +# For Docker/Kubernetes +# Set the PGPASSFILE environment variable in your deployment configuration +# See deployment.mdx for examples +``` + +:::tip + +The pgpass file format is: `hostname:port:database:username:password` + +You can use wildcards (`*`) for any field except password. For example: + +- `*:5432:*:db_app:app_password` - matches any host or database +- `localhost:*:registry:db_app:app_password` - matches any port + +See the +[PostgreSQL documentation](https://www.postgresql.org/docs/current/libpq-pgpass.html) +for more details. + +::: + You can find more details about user creation and initial configuration in this [test file](https://github.com/stacklok/toolhive-registry-server/blob/301ccf4e3ad13daad28be7b669d8e5fca337ec14/cmd/thv-registry-api/app/serve_test.go#L56-L103). diff --git a/docs/toolhive/guides-registry/deployment.mdx b/docs/toolhive/guides-registry/deployment.mdx index 1fe6ecd6..08f18691 100644 --- a/docs/toolhive/guides-registry/deployment.mdx +++ b/docs/toolhive/guides-registry/deployment.mdx @@ -40,7 +40,7 @@ spec: - --config=/etc/registry/config.yaml env: - name: PGPASSFILE - value: /pgpass/.pgpass + value: /etc/secrets/pgpassfile ports: - containerPort: 8080 name: http @@ -50,8 +50,8 @@ spec: subPath: config.yaml readOnly: true - name: pgpass - mountPath: /etc/registry/pgpass - subPath: pgpass + mountPath: /etc/secrets/pgpassfile + subPath: pgpassfile readOnly: true livenessProbe: httpGet: @@ -76,8 +76,9 @@ spec: secret: secretName: registry-api-pgpass items: - - key: pgpass - path: pgpass + - key: pgpassfile + path: pgpassfile + defaultMode: 0600 --- apiVersion: v1 kind: ConfigMap @@ -111,7 +112,7 @@ metadata: name: registry-api-pgpass type: Opaque stringData: - pgpass: | + pgpassfile: | *:5432:registry:db_app:app_password *:5432:registry:db_migrator:migrator_password --- diff --git a/docs/toolhive/guides-registry/intro.mdx b/docs/toolhive/guides-registry/intro.mdx index 27d9b2a6..51785263 100644 --- a/docs/toolhive/guides-registry/intro.mdx +++ b/docs/toolhive/guides-registry/intro.mdx @@ -15,8 +15,9 @@ exposes it through a standardized API. When you start the server, it: 1. Loads configuration from a YAML file 2. Runs database migrations automatically (if database is configured) -3. Immediately fetches registry data from the configured source -4. Starts background sync coordinator for automatic updates +3. Immediately fetches registry data from the configured sources +4. Starts background sync coordinator for automatic updates (for synced + registries) 5. Serves MCP Registry API endpoints on the configured address ```mermaid @@ -52,22 +53,22 @@ flowchart TB - **Standards-compliant**: Implements the official MCP Registry API specification -- **Multiple registry sources**: Git repositories, API endpoints, and local - files +- **Multiple registry sources**: Git repositories, API endpoints, local files, + managed registries, and Kubernetes discovery - **Automatic synchronization**: Background sync with configurable intervals and - retry logic + retry logic for Git, API, and File sources - **Container-ready**: Designed for deployment in Kubernetes clusters, but can be deployed anywhere - **Flexible deployment**: Works standalone or as part of ToolHive infrastructure - **Production-ready**: Built-in health checks, graceful shutdown, and sync status persistence -- **Kubernetes-aware**: Automatic synchronization with ToolHive Operator to keep - running MCP servers in sync with Registry +- **Kubernetes-aware**: Automatic discovery of MCP servers deployed via ToolHive + Operator ## Registry sources -The server supports three registry source types: +The server supports five registry source types: 1. **Managed Registry** - A fully-managed MCP Registry - Ideal for private repositories From af073c2f37710f0552be62646ff7a2ef5effeb95 Mon Sep 17 00:00:00 2001 From: Michelangelo Mori <328978+blkt@users.noreply.github.com> Date: Thu, 4 Dec 2025 14:17:22 +0100 Subject: [PATCH 4/9] Further fixes --- .../guides-registry/api-reference.mdx | 324 ------------------ .../guides-registry/configuration.mdx | 15 +- docs/toolhive/guides-registry/database.mdx | 4 +- docs/toolhive/guides-registry/deployment.mdx | 21 +- docs/toolhive/guides-registry/index.mdx | 8 +- docs/toolhive/guides-registry/intro.mdx | 14 +- sidebars.ts | 1 - 7 files changed, 36 insertions(+), 351 deletions(-) delete mode 100644 docs/toolhive/guides-registry/api-reference.mdx diff --git a/docs/toolhive/guides-registry/api-reference.mdx b/docs/toolhive/guides-registry/api-reference.mdx deleted file mode 100644 index cff53e23..00000000 --- a/docs/toolhive/guides-registry/api-reference.mdx +++ /dev/null @@ -1,324 +0,0 @@ ---- -title: API reference -description: - Reference documentation for the ToolHive Registry API server endpoints ---- - -The Registry API server implements the official MCP Registry API v0.1 -specification, along with extension endpoints for registry management. - -## Base URL - -All API endpoints are prefixed with the server's base URL. By default, the -server runs on port 8080: - -``` -http://localhost:8080 -``` - -## Registry API (v0.1) - -The core MCP Registry API endpoints for discovering and accessing MCP servers. - -The server provides two types of registry endpoints to support different use -cases: - -### Endpoint Types - -**1. Aggregated Registry Endpoints (Read-Only)** - -These endpoints provide a unified view across **all configured registries**. -Ideal for enterprise-wide discovery where you want to query all available MCP -servers from all sources (Git, API, File, Managed, Kubernetes). - -- Pattern: `/registry/v0.1/...` -- Example: `GET /registry/v0.1/servers` -- Use case: Enterprise UI showing complete catalog from all registries - -**2. Per-Registry Endpoints (Standards-Compliant)** - -These endpoints access a **specific registry** by name. Fully compatible with -the upstream MCP Registry API specification and support both read and write -operations (for managed registries). - -- Pattern: `/registry/{registryName}/v0.1/...` -- Example: `GET /registry/toolhive/v0.1/servers` -- Use case: Direct access to a specific registry (e.g., only upstream verified - MCPs) - -:::tip - -- Use aggregated endpoints (`/registry/v0.1/...`) when you want to discover - servers from all configured registries -- Use per-registry endpoints (`/registry/{registryName}/v0.1/...`) when you need - to: - - Query a specific registry - - Publish or delete servers (only supported for managed registries) - - Maintain compatibility with upstream MCP Registry API clients - -::: - -### List all servers - -List all available MCP servers in the registry. - -```http -GET /registry/v0.1/servers -GET /registry/{registryName}/v0.1/servers -``` - -**Query parameters:** - -- `cursor` (optional, string): Pagination cursor for retrieving the next set of - results -- `limit` (optional, integer): Maximum number of items to return -- `search` (optional, string): Search servers by name (substring match) -- `updated_since` (optional, string): Filter servers updated since timestamp - (RFC3339 datetime format, e.g., `2025-08-07T13:15:04.280Z`) -- `version` (optional, string): Filter by version (`latest` for latest version, - or an exact version like `1.2.3`) - -**Response:** - -```json -{ - "servers": [ - { - "name": "example/filesystem", - "description": "Example filesystem MCP server", - "version": "1.0.0" - } - ], - "metadata": { - "nextCursor": "", - "count": 1 - } -} -``` - -### List server versions - -List all versions of a specific server. - -```http -GET /registry/v0.1/servers/{name}/versions -GET /registry/{registryName}/v0.1/servers/{name}/versions -``` - -**Parameters:** - -- `name` (path, required): Server name (URL-encoded) -- `registryName` (path, optional): Registry name (only for registry-specific - endpoint) - -**Response:** - -```json -{ - "servers": [ - { - "name": "example/filesystem", - "version": "1.0.0", - "description": "Example filesystem MCP server" - }, - { - "name": "example/filesystem", - "version": "0.9.0", - "description": "Example filesystem MCP server" - } - ], - "metadata": { - "nextCursor": "", - "count": 2 - } -} -``` - -### Get server version - -Get details for a specific server version. - -```http -GET /registry/v0.1/servers/{name}/versions/{version} -GET /registry/{registryName}/v0.1/servers/{name}/versions/{version} -``` - -**Parameters:** - -- `name` (path, required): Server name (URL-encoded) -- `version` (path, required): Version identifier (use `latest` for the latest - version) -- `registryName` (path, optional): Registry name (only for registry-specific - endpoint) - -**Response:** - -```json -{ - "server": { - "name": "example/filesystem", - "version": "1.0.0", - "description": "Example filesystem MCP server", - "transport": "stdio", - "command": ["node", "server.js"], - "env": { - "NODE_ENV": "production" - } - }, - "meta": {} -} -``` - -### Delete server version - -Delete a server version from a managed registry. - -```http -DELETE /registry/{registryName}/v0.1/servers/{name}/versions/{version} -``` - -**Parameters:** - -- `registryName` (path, required): Registry name -- `name` (path, required): Server name (URL-encoded) -- `version` (path, required): Version identifier - -**Response:** - -- `204 No Content` on success - -**Error responses:** - -- `400 Bad Request` - Invalid request parameters -- `401 Unauthorized` - Invalid or no credentials provided -- `403 Forbidden` - Registry is not a managed registry (cannot delete from - read-only registries) -- `404 Not Found` - Registry or server version not found -- `500 Internal Server Error` - Server error - -### Publish server - -Publish a new server version to the registry. - -```http -POST /{registryName}/v0.1/publish -``` - -**Parameters:** - -- `registryName` (path, required): Registry name - -**Request body:** - -```json -{ - "name": "example/filesystem", - "version": "1.0.0", - "description": "Example filesystem MCP server", - "transport": "stdio", - "command": ["node", "server.js"], - "env": { - "NODE_ENV": "production" - } -} -``` - -**Response:** - -- `201 Created` - Server version published successfully - -**Error responses:** - -- `400 Bad Request` - Invalid request body or missing required fields -- `401 Unauthorized` - Invalid or no credentials provided -- `403 Forbidden` - Registry is not a managed registry (cannot publish to - read-only registries) -- `404 Not Found` - Registry not found -- `409 Conflict` - Version already exists -- `500 Internal Server Error` - Server error - -:::info[Legacy endpoint] - -The endpoint `POST /registry/v0.1/publish` returns `501 Not Implemented`. Use -the registry-specific endpoint `POST /{registryName}/v0.1/publish` instead. - -::: - -## Operational endpoints - -Endpoints for monitoring and operational purposes. - -### Health check - -Check if the server is running. - -```http -GET /health -``` - -**Response:** - -- `200 OK` if the server is running - -### Readiness check - -Check if the server is ready to serve requests. - -```http -GET /readiness -``` - -**Response:** - -- `200 OK` if the server is ready -- `503 Service Unavailable` if the server is not ready - -### Version information - -Get version information about the server. - -```http -GET /version -``` - -**Response:** - -```json -{ - "version": "0.1.0", - "commit": "abc123", - "buildDate": "2024-01-01T00:00:00Z" -} -``` - -## Error responses - -All endpoints may return standard HTTP error responses: - -- `400 Bad Request` - Invalid request parameters or request body -- `401 Unauthorized` - Invalid or no credentials provided -- `403 Forbidden` - Operation not allowed (e.g., attempting to delete or publish - to a read-only registry) -- `404 Not Found` - Resource not found (registry, server, or version) -- `409 Conflict` - Resource conflict (e.g., version already exists) -- `500 Internal Server Error` - Server error -- `501 Not Implemented` - Endpoint not supported -- `503 Service Unavailable` - Service temporarily unavailable - -Error responses include a JSON body with error details: - -```json -{ - "error": "Server not found" -} -``` - -## API specification - -For the complete API specification, see the -[MCP Registry API specification](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/api/openapi.yaml). - -## Next steps - -- [Configure the server](./configuration.mdx) to set up your data sources -- [Deploy the server](./deployment.mdx) in your environment diff --git a/docs/toolhive/guides-registry/configuration.mdx b/docs/toolhive/guides-registry/configuration.mdx index 7f5dfebf..deff678a 100644 --- a/docs/toolhive/guides-registry/configuration.mdx +++ b/docs/toolhive/guides-registry/configuration.mdx @@ -1,8 +1,7 @@ --- title: Configuration description: - How to configure the ToolHive Registry API server with data sources and sync - policies + How to configure the ToolHive Registry Server with registry and sync policies --- All configuration is done via YAML files. The server requires a `--config` flag @@ -17,8 +16,8 @@ registryName: my-registry # Registries configuration (required, can have multiple registries) registries: - name: toolhive - # Data format: toolhive (native) or upstream (MCP registry format) - format: toolhive + # Data format: upstream (MCP registry format) or toolhive (legacy) + format: upstream # Git repository configuration git: @@ -64,7 +63,7 @@ database: | `--config` | Path to YAML configuration file | Yes | - | | `--address` | Server listen address | No | `:8080` | -## Data sources +## Registries The server supports five registry types, each with its own configuration options. You can configure multiple registries in a single server instance. @@ -142,7 +141,7 @@ Read from filesystem. Ideal for local development and testing. ```yaml title="config-file.yaml" registries: - name: local - format: toolhive + format: upstream file: path: /data/registry.json syncPolicy: @@ -164,7 +163,7 @@ API. Does not sync from external sources. ```yaml title="config-managed.yaml" registries: - name: internal - format: toolhive + format: upstream managed: {} auth: @@ -242,7 +241,7 @@ registries: ``` The `interval` field specifies how often the server should fetch updates from -the data source. Use Go duration format (e.g., `"30m"`, `"1h"`, `"24h"`). +the registry. Use Go duration format (e.g., `"30m"`, `"1h"`, `"24h"`). :::note diff --git a/docs/toolhive/guides-registry/database.mdx b/docs/toolhive/guides-registry/database.mdx index 629a788b..ea618f4b 100644 --- a/docs/toolhive/guides-registry/database.mdx +++ b/docs/toolhive/guides-registry/database.mdx @@ -2,10 +2,10 @@ title: Database configuration description: How to configure PostgreSQL database storage and migrations for the ToolHive - Registry API server + Registry server --- -The Registry API server optionally supports PostgreSQL database connectivity for +The Registry server optionally supports PostgreSQL database connectivity for storing registry state and metadata. This enables persistence across restarts and provides a foundation for advanced features. diff --git a/docs/toolhive/guides-registry/deployment.mdx b/docs/toolhive/guides-registry/deployment.mdx index 08f18691..fd4493f8 100644 --- a/docs/toolhive/guides-registry/deployment.mdx +++ b/docs/toolhive/guides-registry/deployment.mdx @@ -1,14 +1,14 @@ --- title: Deployment in Kubernetes -description: How to deploy the ToolHive Registry API server in Kubernetes +description: How to deploy the ToolHive Registry server in Kubernetes --- -The Registry API server can be deployed in various environments, from local +The Registry server can be deployed in various environments, from local development to production Kubernetes clusters. ## Kubernetes deployment -The Registry API is designed to run as an independent deployment, possibly +The Registry server is designed to run as an independent deployment, possibly alongside the ToolHive Operator. Although it is possible to run ToolHive Registry to use an in-memory store, it @@ -17,6 +17,17 @@ recommend running it with a proper Postgres database. ### Deployment Example +Below is an example Kubernetes Deployment configuring ToolHive Registry server +to expose a single static registry based on a Git repository. This example +assumes that a Postgres database is available at `db.example.com` and the +necessary users for migration and application execution are configured and able +to connect to a `registry` database. + +For further details about user grants read the +[Migration user privileges](./database.mdx#migration-user-privileges) and +[Application user privileges](./database.mdx#application-user-privileges) +sections. + ```yaml title="deployment.yaml" apiVersion: apps/v1 kind: Deployment @@ -113,8 +124,8 @@ metadata: type: Opaque stringData: pgpassfile: | - *:5432:registry:db_app:app_password - *:5432:registry:db_migrator:migrator_password + postgres:5432:registry:db_app:app_password + postgres:5432:registry:db_migrator:migrator_password --- apiVersion: v1 kind: Service diff --git a/docs/toolhive/guides-registry/index.mdx b/docs/toolhive/guides-registry/index.mdx index 26bd986d..8bdc6923 100644 --- a/docs/toolhive/guides-registry/index.mdx +++ b/docs/toolhive/guides-registry/index.mdx @@ -1,15 +1,15 @@ --- -title: ToolHive Registry API Server +title: ToolHive Registry Server description: - How-to guides for using the ToolHive Registry API server to discover and - access MCP servers. + How-to guides for using the ToolHive Registry server to discover and access + MCP servers. --- import DocCardList from '@theme/DocCardList'; ## Introduction -The ToolHive Registry API server implements the official +The ToolHive Registry server implements the official [Model Context Protocol (MCP) Registry API specification](https://modelcontextprotocol.io/development/roadmap#registry). It provides a standardized REST API for discovering and accessing MCP servers from multiple backend sources, including Kubernetes clusters, Git repositories, diff --git a/docs/toolhive/guides-registry/intro.mdx b/docs/toolhive/guides-registry/intro.mdx index 51785263..e0763ee4 100644 --- a/docs/toolhive/guides-registry/intro.mdx +++ b/docs/toolhive/guides-registry/intro.mdx @@ -1,16 +1,16 @@ --- title: Overview description: - How to use the ToolHive Registry API server to discover and access MCP servers + How to use the ToolHive Registry server to discover and access MCP servers --- -The ToolHive Registry API server (`thv-registry-api`) is a standards-compliant -implementation of the MCP Registry API specification. It provides a REST API for -discovering and accessing MCP servers from multiple backend sources. +The ToolHive Registry server is a standards-compliant implementation of the MCP +Registry API specification. It provides a REST API for discovering and accessing +MCP servers from multiple backend sources. ## Overview -The Registry API server aggregates MCP server metadata from various sources and +The Registry server aggregates MCP server metadata from various sources and exposes it through a standardized API. When you start the server, it: 1. Loads configuration from a YAML file @@ -30,7 +30,7 @@ flowchart TB File["Local File"] end - subgraph Server["Registry API Server"] + subgraph Server["Registry Server"] Config["Configuration"] Sync["Sync Manager"] Storage["Storage Layer"] @@ -99,6 +99,6 @@ The server supports five registry source types: ## Next steps -- [Install the Registry API server](./deployment.mdx) to get started +- [Deploy the Registry server](./deployment.mdx) to get started - [Configure registry sources](./configuration.mdx) to set up your registry - [Deploy the server](./deployment.mdx) in your environment diff --git a/sidebars.ts b/sidebars.ts index ae982e6e..ce384bcc 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -195,7 +195,6 @@ const sidebars: SidebarsConfig = { items: [ 'toolhive/guides-registry/intro', 'toolhive/guides-registry/configuration', - 'toolhive/guides-registry/api-reference', 'toolhive/guides-registry/database', 'toolhive/guides-registry/deployment', ], From d5de6d21c9858a2cc898dd6df1927d96bb5f93f8 Mon Sep 17 00:00:00 2001 From: Radoslav Dimitrov Date: Thu, 4 Dec 2025 16:29:47 +0200 Subject: [PATCH 5/9] Split the authentication docs Signed-off-by: Radoslav Dimitrov --- .../guides-registry/authentication.mdx | 463 ++++++++++++++++++ .../guides-registry/configuration.mdx | 33 +- sidebars.ts | 1 + 3 files changed, 483 insertions(+), 14 deletions(-) create mode 100644 docs/toolhive/guides-registry/authentication.mdx diff --git a/docs/toolhive/guides-registry/authentication.mdx b/docs/toolhive/guides-registry/authentication.mdx new file mode 100644 index 00000000..c97eb5ef --- /dev/null +++ b/docs/toolhive/guides-registry/authentication.mdx @@ -0,0 +1,463 @@ +--- +title: Authentication configuration +description: + How to configure authentication for the ToolHive Registry server with + anonymous and OAuth modes +--- + +The Registry server provides secure-by-default authentication, defaulting to +OAuth mode to protect your registry. You can configure authentication to fit +different deployment scenarios, from development environments to production +deployments with enterprise identity providers. + +## Authentication modes + +The server supports two authentication modes configured via the required `auth` +section in your configuration file: + +- **OAuth** (default): Secure authentication using JWT tokens from identity + providers +- **Anonymous**: No authentication (development/testing only) + +:::info[Secure by default] + +The server defaults to **OAuth mode** when no explicit auth configuration is +provided. This secure-by-default posture ensures your registry is protected +unless you explicitly choose anonymous mode for development scenarios. + +::: + +## OAuth authentication + +OAuth mode (the default) validates JWT tokens from identity providers. This +enables enterprise authentication with providers like Keycloak, Auth0, Okta, +Azure AD, Kubernetes service accounts, or any OIDC-compliant service. + +### Basic OAuth configuration + +```yaml title="config-oauth.yaml" +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: keycloak + issuerUrl: https://keycloak.example.com/realms/mcp + audience: registry-api +``` + +### OAuth configuration fields + +| Field | Type | Required | Default | Description | +| ----------------- | -------- | -------- | ----------------------------------------- | -------------------------------------------------- | +| `mode` | string | Yes | `oauth` | Authentication mode (`oauth` or `anonymous`) | +| `resourceUrl` | string | Yes | - | The URL of the registry resource being protected | +| `realm` | string | No | `mcp-registry` | OAuth realm identifier | +| `scopesSupported` | []string | No | `[mcp-registry:read, mcp-registry:write]` | Supported OAuth scopes | +| `publicPaths` | []string | No | `[]` | Additional paths accessible without authentication | +| `providers` | array | Yes | - | List of OAuth/OIDC identity providers | + +### Provider configuration fields + +| Field | Type | Required | Description | +| ------------------ | ------ | -------- | ----------------------------------------------------------------------- | +| `name` | string | Yes | Provider identifier for logging and monitoring | +| `issuerUrl` | string | Yes | OAuth/OIDC issuer URL (e.g., `https://keycloak.example.com/realms/mcp`) | +| `audience` | string | Yes | Expected audience claim in the JWT token | +| `clientId` | string | No | OAuth client ID (for token introspection) | +| `clientSecretFile` | string | No | Path to file containing client secret (for token introspection) | +| `caCertPath` | string | No | Path to CA certificate for TLS verification | + +### Complete OAuth configuration example + +```yaml title="config-oauth-complete.yaml" +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + realm: mcp-registry + scopesSupported: + - mcp-registry:read + - mcp-registry:write + publicPaths: + - /custom-health + - /metrics + providers: + - name: keycloak-prod + issuerUrl: https://keycloak.example.com/realms/production + audience: registry-api + clientId: registry-client + clientSecretFile: /etc/secrets/keycloak-secret + caCertPath: /etc/ssl/certs/keycloak-ca.crt + - name: keycloak-staging + issuerUrl: https://keycloak.example.com/realms/staging + audience: registry-api-staging +``` + +## Kubernetes authentication + +For Kubernetes deployments, you can configure OAuth to validate service account +tokens. This provides automatic, zero-config authentication for workloads +running in the cluster. + +### Kubernetes provider configuration + +```yaml title="config-k8s-auth.yaml" +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: kubernetes + issuerUrl: https://kubernetes.default.svc + audience: https://kubernetes.default.svc + caCertPath: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt +``` + +:::tip[In-cluster service DNS] + +The correct Kubernetes API server URL for in-cluster access is +`https://kubernetes.default.svc` (not +`https://kubernetes.default.svc.cluster.local`). + +::: + +### How Kubernetes authentication works + +1. Workloads in the cluster mount service account tokens automatically at + `/var/run/secrets/kubernetes.io/serviceaccount/token` +2. Clients send these tokens in the `Authorization: Bearer ` header +3. The server validates tokens using the Kubernetes API server's public keys +4. Access is granted based on the service account's identity and token claims + +### Kubernetes deployment example + +When deploying in Kubernetes, the service account CA certificate is +automatically mounted: + +```yaml title="deployment-k8s-auth.yaml" +apiVersion: apps/v1 +kind: Deployment +metadata: + name: registry-api +spec: + replicas: 1 + selector: + matchLabels: + app: registry-api + template: + metadata: + labels: + app: registry-api + spec: + serviceAccountName: registry-api + containers: + - name: registry-api + image: ghcr.io/stacklok/thv-registry-api:latest + args: + - serve + - --config=/etc/registry/config.yaml + volumeMounts: + - name: config + mountPath: /etc/registry/config.yaml + subPath: config.yaml + readOnly: true + # Service account token and CA cert are mounted automatically by Kubernetes + volumes: + - name: config + configMap: + name: registry-api-config +``` + +The service account token and CA certificate are automatically mounted at: + +- Token: `/var/run/secrets/kubernetes.io/serviceaccount/token` +- CA cert: `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` + +## Provider-specific examples + +### Keycloak + +```yaml +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: keycloak + issuerUrl: https://keycloak.example.com/realms/YOUR_REALM + audience: registry-api +``` + +The `issuerUrl` should point to your Keycloak realm. The realm name is part of +the URL path. + +### Auth0 + +```yaml +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: auth0 + issuerUrl: https://YOUR_DOMAIN.auth0.com/ + audience: https://registry.example.com +``` + +For Auth0, the `issuerUrl` is your Auth0 domain. The `audience` should match the +API identifier configured in your Auth0 dashboard. + +### Azure AD + +```yaml +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: azure-ad + issuerUrl: https://login.microsoftonline.com/YOUR_TENANT_ID/v2.0 + audience: api://YOUR_CLIENT_ID +``` + +For Azure AD, the `issuerUrl` includes your tenant ID, and the `audience` +typically uses the `api://` scheme with your application's client ID. + +### Okta + +```yaml +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: okta + issuerUrl: https://YOUR_DOMAIN.okta.com/oauth2/default + audience: api://default +``` + +For Okta, the `issuerUrl` points to your Okta authorization server. Use +`/oauth2/default` for the default authorization server or +`/oauth2/YOUR_AUTH_SERVER_ID` for custom servers. + +## Anonymous authentication + +Anonymous mode disables authentication entirely, allowing unrestricted access to +all registry endpoints. This is only suitable for development, testing, or +internal deployments where authentication is handled at a different layer (e.g., +network policies, VPN, or reverse proxy). + +### Anonymous configuration + +```yaml title="config-anonymous.yaml" +auth: + mode: anonymous +``` + +:::danger[No access control] + +Anonymous mode provides **no access control**. Only use it in trusted +environments or when other security measures are in place. **Do not use +anonymous mode in production.** + +::: + +### Anonymous use cases + +- Local development and testing +- Internal deployments behind corporate firewalls +- Read-only public registries +- Environments with external authentication (reverse proxy, API gateway) + +## Default public paths + +The following endpoints are **always accessible without authentication**, +regardless of the auth mode: + +- `/health` - Health check endpoint +- `/readiness` - Readiness probe endpoint +- `/version` - Version information +- `/.well-known/*` - OAuth discovery endpoints (RFC 9728) + +You can configure additional public paths using the `publicPaths` field in your +OAuth configuration. + +## RFC 9728 OAuth discovery + +The server implements [RFC 9728](https://www.rfc-editor.org/rfc/rfc9728.html) +for OAuth Protected Resource Metadata, enabling clients to automatically +discover authentication requirements. + +### Discovery endpoint + +Clients can discover the server's OAuth configuration at: + +```text +GET /.well-known/oauth-protected-resource +``` + +### Example discovery response + +```json +{ + "resource": "https://registry.example.com", + "authorization_servers": [ + "https://keycloak.example.com/realms/production", + "https://keycloak.example.com/realms/staging" + ], + "scopes_supported": ["mcp-registry:read", "mcp-registry:write"], + "bearer_methods_supported": ["header"], + "resource_documentation": "https://docs.example.com/registry" +} +``` + +This allows OAuth clients to automatically configure themselves without manual +setup, improving interoperability and reducing configuration errors. + +## Testing authentication + +### Using curl with a bearer token + +```bash +TOKEN="your-jwt-token-here" + +curl -H "Authorization: Bearer $TOKEN" \ + https://registry.example.com/default/v0.1/servers +``` + +### Using kubectl with Kubernetes service accounts + +```bash +# Get the service account token +TOKEN=$(kubectl get secret \ + -n \ + -o jsonpath='{.data.token}' | base64 -d) + +# Make authenticated request +curl -H "Authorization: Bearer $TOKEN" \ + https://registry.example.com/default/v0.1/servers +``` + +### Testing token validation + +To verify your token is valid: + +1. Decode the JWT at [jwt.io](https://jwt.io/) (don't paste production tokens!) +2. Check the `iss` (issuer) matches your configured `issuerUrl` +3. Check the `aud` (audience) matches your configured `audience` +4. Check the `exp` (expiration) is in the future + +## Choosing an authentication mode + +| Mode | Security | Complexity | Best for | +| --------- | -------- | ---------- | ----------------------------------------------- | +| OAuth | High | Medium | Production deployments, enterprise environments | +| Anonymous | None | None | Development, testing, internal trusted networks | + +**Recommendations:** + +- **Production deployments**: Always use OAuth mode with your organization's + identity provider +- **Kubernetes deployments**: Use OAuth with Kubernetes provider for automatic + authentication +- **Development/testing**: Use anonymous mode for local development only +- **Public read-only registries**: Use OAuth mode with rate limiting; avoid + anonymous in production + +## Security considerations + +### Token validation + +All OAuth providers validate: + +- Token signature using the provider's public keys (fetched from issuer's JWKS + endpoint) +- Token expiration (`exp` claim) +- Audience claim (`aud`) matches configuration +- Issuer (`iss`) matches the configured provider + +### HTTPS requirements + +Always use HTTPS in production to protect tokens in transit: + +```yaml +auth: + mode: oauth + oauth: + resourceUrl: https://registry.example.com # Use HTTPS + providers: + - issuerUrl: https://keycloak.example.com/realms/mcp # Use HTTPS +``` + +### Token storage + +- Never log or persist JWT tokens in plaintext +- Use short-lived tokens when possible (e.g., 1 hour) +- Implement token refresh flows for long-running clients +- Rotate client secrets regularly if using `clientSecretFile` + +### Custom CA certificates + +If your identity provider uses a custom CA certificate, specify the `caCertPath` +in your provider configuration: + +```yaml +providers: + - name: internal-keycloak + issuerUrl: https://keycloak.internal.example.com/realms/mcp + audience: registry-api + caCertPath: /etc/ssl/certs/internal-ca.crt +``` + +## Troubleshooting + +### 401 Unauthorized errors + +If you receive 401 Unauthorized responses: + +1. **Verify the token is valid**: Check expiration, issuer, and audience claims +2. **Check the Authorization header**: Must be `Authorization: Bearer ` +3. **Verify issuer URL**: Must exactly match the `iss` claim in your token +4. **Check audience**: The `aud` claim must match your configured `audience` +5. **Review server logs**: Look for specific validation error messages + +### Token validation errors + +If you see authentication failures in server logs: + +1. **Issuer URL accessibility**: Verify the server can reach the issuer's JWKS + endpoint (typically `${issuerUrl}/.well-known/openid-configuration`) +2. **Network connectivity**: Check DNS resolution and firewall rules +3. **CA certificates**: If using custom CAs, ensure `caCertPath` is correct +4. **Token expiration**: Ensure your token hasn't expired (check `exp` claim) + +### Provider connectivity issues + +If the server cannot connect to the OAuth provider: + +1. Verify network connectivity to the issuer URL from the server +2. Check DNS resolution for the provider's domain +3. Ensure firewall rules allow outbound HTTPS (port 443) to the provider +4. For Kubernetes providers, verify the API server is reachable at + `https://kubernetes.default.svc` +5. Check CA certificate configuration if using self-signed certificates + +### Multiple providers not working + +If tokens from some providers work but others don't: + +1. Verify each provider's configuration independently using curl +2. Check that audience claims differ between providers if needed (or are + intentionally the same) +3. Ensure issuer URLs are correct and include the full path (including realm for + Keycloak) +4. Review server logs to identify which specific provider validation is failing +5. Test each provider's JWKS endpoint accessibility: + `curl ${issuerUrl}/.well-known/openid-configuration` + +## Next steps + +- [Configure database storage](./database.mdx) for production deployments +- [Deploy the server](./deployment.mdx) with authentication enabled +- [Configure registry sources](./configuration.mdx) for your environment diff --git a/docs/toolhive/guides-registry/configuration.mdx b/docs/toolhive/guides-registry/configuration.mdx index deff678a..987324d1 100644 --- a/docs/toolhive/guides-registry/configuration.mdx +++ b/docs/toolhive/guides-registry/configuration.mdx @@ -40,8 +40,8 @@ registries: exclude: ['experimental'] # Authentication configuration (required) +# See authentication.mdx for detailed configuration options auth: - # Mode: anonymous or oauth (defaults to oauth if not specified) mode: anonymous # Optional: Database configuration @@ -168,12 +168,7 @@ registries: auth: mode: oauth - oauth: - resourceUrl: https://registry.example.com - providers: - - name: keycloak - issuerUrl: https://keycloak.example.com/realms/mcp - audience: registry-api + # See authentication.mdx for OAuth configuration details ``` **Configuration options:** @@ -201,13 +196,7 @@ registries: auth: mode: oauth - oauth: - resourceUrl: https://registry.example.com - providers: - - name: kubernetes - issuerUrl: https://kubernetes.default.svc.cluster.local - audience: https://kubernetes.default.svc.cluster.local - caCertPath: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + # See authentication.mdx for Kubernetes OAuth configuration details ``` **Configuration options:** @@ -286,6 +275,21 @@ Filters are per-registry and specified within each registry configuration. ::: +## Authentication configuration + +The server supports multiple authentication modes to fit different deployment +scenarios. All authentication configuration is done via the `auth` section in +your configuration file. + +Available modes: + +- **Anonymous**: No authentication (development/testing) +- **OAuth with Kubernetes**: Service account token validation +- **OAuth with generic providers**: Integration with Keycloak, Auth0, Okta, etc. + +See the [Authentication configuration](./authentication.mdx) guide for detailed +information on configuring each mode. + ## Database configuration The server optionally supports PostgreSQL database connectivity for storing @@ -294,5 +298,6 @@ guide for detailed information. ## Next steps +- [Configure authentication](./authentication.mdx) for secure access - [Deploy the server](./deployment.mdx) with your configuration - [Configure database storage](./database.mdx) for production use diff --git a/sidebars.ts b/sidebars.ts index ce384bcc..c436767c 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -195,6 +195,7 @@ const sidebars: SidebarsConfig = { items: [ 'toolhive/guides-registry/intro', 'toolhive/guides-registry/configuration', + 'toolhive/guides-registry/authentication', 'toolhive/guides-registry/database', 'toolhive/guides-registry/deployment', ], From 8834f70d55d353b9eb8c8ed743cc5f89a3631181 Mon Sep 17 00:00:00 2001 From: Michelangelo Mori <328978+blkt@users.noreply.github.com> Date: Thu, 4 Dec 2025 16:34:06 +0100 Subject: [PATCH 6/9] Improve deployment instructions for Registry --- docs/toolhive/guides-registry/deployment.mdx | 63 +++++++++++++------- 1 file changed, 42 insertions(+), 21 deletions(-) diff --git a/docs/toolhive/guides-registry/deployment.mdx b/docs/toolhive/guides-registry/deployment.mdx index fd4493f8..3020ddef 100644 --- a/docs/toolhive/guides-registry/deployment.mdx +++ b/docs/toolhive/guides-registry/deployment.mdx @@ -18,10 +18,12 @@ recommend running it with a proper Postgres database. ### Deployment Example Below is an example Kubernetes Deployment configuring ToolHive Registry server -to expose a single static registry based on a Git repository. This example -assumes that a Postgres database is available at `db.example.com` and the -necessary users for migration and application execution are configured and able -to connect to a `registry` database. +to expose a single static registry based on a Git repository. + +This example assumes that a Postgres database is available at `db.example.com` +and the necessary users for migration and application execution are configured +and able to connect to a `registry` database. It also assumes that you have a +keycloak instance configured to act as identity provider. For further details about user grants read the [Migration user privileges](./database.mdx#migration-user-privileges) and @@ -43,26 +45,38 @@ spec: labels: app: registry-api spec: + initContainers: + - name: pgpass-fixer + image: alpine:3 + command: + - /bin/sh + - -c + - cp /cfg/* /etc/ && chmod 0600 /etc/pgpass && chown 65532:65532 + /etc/pgpass + volumeMounts: + - name: etc + mountPath: /etc + - name: config + mountPath: /cfg/config.yaml + subPath: config.yaml + - name: pgpass + mountPath: /cfg/pgpass + subPath: pgpass containers: - name: registry-api - image: ghcr.io/stacklok/thv-registry-api:latest + image: ghcr.io/stacklok/toolhive-registry-server/thv-registry-api:latest args: - serve - - --config=/etc/registry/config.yaml + - --config=/etc/config.yaml env: - name: PGPASSFILE - value: /etc/secrets/pgpassfile + value: /etc/pgpass ports: - containerPort: 8080 name: http volumeMounts: - - name: config - mountPath: /etc/registry/config.yaml - subPath: config.yaml - readOnly: true - - name: pgpass - mountPath: /etc/secrets/pgpassfile - subPath: pgpassfile + - name: etc + mountPath: /etc readOnly: true livenessProbe: httpGet: @@ -77,6 +91,8 @@ spec: initialDelaySeconds: 5 periodSeconds: 5 volumes: + - name: etc + emptyDir: {} - name: config configMap: name: registry-api-config @@ -87,9 +103,8 @@ spec: secret: secretName: registry-api-pgpass items: - - key: pgpassfile - path: pgpassfile - defaultMode: 0600 + - key: pgpass + path: pgpass --- apiVersion: v1 kind: ConfigMap @@ -108,7 +123,13 @@ data: syncPolicy: interval: "15m" auth: - mode: anonymous + mode: oauth + oauth: + resourceUrl: https://registry.example.com + providers: + - name: keycloak + issuerUrl: https://keycloak.example.com/realms/mcp + audience: registry-api database: host: db.example.com port: 5432 @@ -123,9 +144,9 @@ metadata: name: registry-api-pgpass type: Opaque stringData: - pgpassfile: | - postgres:5432:registry:db_app:app_password - postgres:5432:registry:db_migrator:migrator_password + pgpass: | + db.example.com:5432:registry:db_app:app_password + db.example.com:5432:registry:db_migrator:migrator_password --- apiVersion: v1 kind: Service From 126068cb7984e6053a85d45fdbb8afe0fec62b64 Mon Sep 17 00:00:00 2001 From: Michelangelo Mori <328978+blkt@users.noreply.github.com> Date: Fri, 5 Dec 2025 09:00:16 +0100 Subject: [PATCH 7/9] PR comments --- docs/toolhive/guides-registry/index.mdx | 2 +- docs/toolhive/guides-registry/intro.mdx | 31 ++++++++++++------------- 2 files changed, 16 insertions(+), 17 deletions(-) diff --git a/docs/toolhive/guides-registry/index.mdx b/docs/toolhive/guides-registry/index.mdx index 8bdc6923..d5573f57 100644 --- a/docs/toolhive/guides-registry/index.mdx +++ b/docs/toolhive/guides-registry/index.mdx @@ -10,7 +10,7 @@ import DocCardList from '@theme/DocCardList'; ## Introduction The ToolHive Registry server implements the official -[Model Context Protocol (MCP) Registry API specification](https://modelcontextprotocol.io/development/roadmap#registry). +[Model Context Protocol (MCP) Registry API specification](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/api/generic-registry-api.md). It provides a standardized REST API for discovering and accessing MCP servers from multiple backend sources, including Kubernetes clusters, Git repositories, API endpoints, and local files. diff --git a/docs/toolhive/guides-registry/intro.mdx b/docs/toolhive/guides-registry/intro.mdx index e0763ee4..67ad97a4 100644 --- a/docs/toolhive/guides-registry/intro.mdx +++ b/docs/toolhive/guides-registry/intro.mdx @@ -21,32 +21,32 @@ exposes it through a standardized API. When you start the server, it: 5. Serves MCP Registry API endpoints on the configured address ```mermaid -flowchart TB - subgraph Sources["Registry Sources"] - Managed["Managed Registry"] - API["Upstream Registry"] +flowchart LR + subgraph Sources["Registry sources"] + direction LR + Managed["Managed registry"] + API["Upstream registry"] Kubernetes["Kubernetes cluster"] - Git["Git Repository"] - File["Local File"] + Git["Git repository"] + File["Local file"] end - - subgraph Server["Registry Server"] + subgraph Server["Registry server"] + direction TB Config["Configuration"] - Sync["Sync Manager"] - Storage["Storage Layer"] - APIHandler["API Handler"] + Sync["Sync manager"] + Storage["Storage layer"] + APIHandler["API handler"] end - subgraph Clients["Clients"] + direction LR ToolHive["ToolHive"] - MCPClient["MCP Clients"] + MCPClient["MCP clients"] end - Sources -->|sync| Server Config --> Sync Sync --> Storage Storage --> APIHandler - APIHandler -->|REST API| Clients + Server -->|REST API| Clients ``` ## Features @@ -99,6 +99,5 @@ The server supports five registry source types: ## Next steps -- [Deploy the Registry server](./deployment.mdx) to get started - [Configure registry sources](./configuration.mdx) to set up your registry - [Deploy the server](./deployment.mdx) in your environment From 9ee753fc954a1522c33c5a5bfb9f08b05a467ae5 Mon Sep 17 00:00:00 2001 From: Michelangelo Mori <328978+blkt@users.noreply.github.com> Date: Fri, 5 Dec 2025 11:11:42 +0100 Subject: [PATCH 8/9] Add docs on workload discovery This commit adds documentation regarding workload discovery, required annotations, and deployment configuration. It also adds a couple links back to other sections of ToolHive Operator and Virtual MCP. --- .../guides-registry/configuration.mdx | 77 +++++++++++---- docs/toolhive/guides-registry/deployment.mdx | 95 +++++++++++++++++++ docs/toolhive/guides-registry/intro.mdx | 2 +- 3 files changed, 155 insertions(+), 19 deletions(-) diff --git a/docs/toolhive/guides-registry/configuration.mdx b/docs/toolhive/guides-registry/configuration.mdx index 987324d1..9d0dc781 100644 --- a/docs/toolhive/guides-registry/configuration.mdx +++ b/docs/toolhive/guides-registry/configuration.mdx @@ -82,9 +82,6 @@ registries: path: pkg/registry/data/registry.json syncPolicy: interval: '30m' - -auth: - mode: anonymous ``` **Configuration options:** @@ -116,9 +113,6 @@ registries: endpoint: https://registry.modelcontextprotocol.io syncPolicy: interval: '1h' - -auth: - mode: anonymous ``` **Configuration options:** @@ -134,7 +128,7 @@ etc.) to the endpoint URL. ::: -### Local file source +### File source Read from filesystem. Ideal for local development and testing. @@ -146,14 +140,31 @@ registries: path: /data/registry.json syncPolicy: interval: '15m' +``` -auth: - mode: anonymous +Alternatively, the source can be a custom URL. + +```yaml title="config-file.yaml" +registries: + - name: remote + format: upstream + file: + url: https://www.example.com/registry.json + timeout: 5s + syncPolicy: + interval: '15m' ``` +The fields `file` and `url` are mutually exclusive. + **Configuration options:** -- `path` (required): Path to the registry file on the filesystem +- `path` (required): Path to the registry file on the filesystem. Mutually + exclusive with `url` +- `url` (required): URL is the HTTP/HTTPS URL to fetch the registry file from. + Mutually exclusive with `file` +- `timeout` (optional): The timeout for HTTP requests when using URL, defaults + to 30s, ignored when `file` is specified ### Managed registry @@ -165,10 +176,6 @@ registries: - name: internal format: upstream managed: {} - -auth: - mode: oauth - # See authentication.mdx for OAuth configuration details ``` **Configuration options:** @@ -193,10 +200,6 @@ registries: - name: k8s-cluster format: toolhive kubernetes: {} - -auth: - mode: oauth - # See authentication.mdx for Kubernetes OAuth configuration details ``` **Configuration options:** @@ -205,6 +208,44 @@ auth: - No sync policy required (Kubernetes registries query live deployments on-demand) +:::info[How does it work?] + +Kubernetes workload discovery works by looking for annotations in a specific set +of workloads. The types being watched are +[`MCPServer`](../guides-k8s/run-mcp-k8s.mdx), +[`MCPRemoteProxy`](../guides-k8s/remote-mcp-proxy.mdx), and +[`VirtualMCPServer`](../guides-vmcp/configuration.mdx). + +The Registry Server will receive events when a resource among those listed above +is annotated with the following annotations + +```yaml +apiVersion: toolhive.stacklok.dev/v1alpha1 +kind: MCPServer +metadata: + name: my-mcp-server + namespace: production + annotations: + toolhive.stacklok.dev/registry-export: 'true' + toolhive.stacklok.dev/registry-url: 'https://mcp.example.com/servers/my-mcp-server' + toolhive.stacklok.dev/registry-description: | + Production MCP server for code analysis +spec: + # ... MCP server spec +``` + +| Annotation | Required | Description | +| -------------------------------------------- | -------- | ----------------------------------------- | +| `toolhive.stacklok.dev/registry-export` | Yes | Must be `"true"` to include in registry | +| `toolhive.stacklok.dev/registry-url` | Yes | The external endpoint URL for this server | +| `toolhive.stacklok.dev/registry-description` | Yes | Override the description in registry | + +This feature requires the Registry Server to be granted access to those +resources via a Service Account, check the details in the +[deployment section](./deployment.mdx#workload-discovery). + +::: + **Use cases:** - Discover MCP servers deployed via ToolHive Operator diff --git a/docs/toolhive/guides-registry/deployment.mdx b/docs/toolhive/guides-registry/deployment.mdx index 3020ddef..bade3e8c 100644 --- a/docs/toolhive/guides-registry/deployment.mdx +++ b/docs/toolhive/guides-registry/deployment.mdx @@ -167,3 +167,98 @@ Apply the deployment: ```bash kubectl apply -f deployment.yaml ``` + +## Workload Discovery + +Kubernetes workload discovery works by looking for annotations in a specific set +of workloads. The types being watched are +[`MCPServer`](../guides-k8s/run-mcp-k8s.mdx), +[`MCPRemoteProxy`](../guides-k8s/remote-mcp-proxy.mdx), and +[`VirtualMCPServer`](../guides-vmcp/configuration.mdx). + +This feature requires the Registry Server to be granted access to those +resources via a Service Account like the following + +```yaml title="registry-service-account.yaml" +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + toolhive.stacklok.io/registry-name: example-registry + name: example-registry-registry-api + namespace: toolhive-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + labels: + toolhive.stacklok.io/registry-name: example-registry + name: example-registry-registry-api + namespace: toolhive-system +rules: + - apiGroups: + - toolhive.stacklok.dev + resources: + - mcpservers + - mcpremoteproxies + - virtualmcpservers + verbs: + - get + - list + - watch + - apiGroups: + - '' + resources: + - services + verbs: + - get + - list + - watch + - apiGroups: + - '' + resources: + - configmaps + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - coordination.k8s.io + resources: + - leases + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - '' + resources: + - events + verbs: + - create + - patch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + labels: + toolhive.stacklok.io/registry-name: example-registry + name: example-registry-registry-api + namespace: toolhive-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: example-registry-registry-api +subjects: + - kind: ServiceAccount + name: example-registry-registry-api + namespace: toolhive-system +``` diff --git a/docs/toolhive/guides-registry/intro.mdx b/docs/toolhive/guides-registry/intro.mdx index 67ad97a4..ac4347da 100644 --- a/docs/toolhive/guides-registry/intro.mdx +++ b/docs/toolhive/guides-registry/intro.mdx @@ -85,7 +85,7 @@ The server supports five registry source types: - Ideal to quickly grant access to running MCP servers to knowledge workers - Useful for bigger organizations where MCP server developers differ from users - - Supports adding new MCP servers via `/publish` endpoint + - Does not support publishing 4. **Git Repository** - Clone and sync from Git repositories - Supports branch, tag, or commit pinning From 3f81cafd07388a16df17c4bb2713877db1a0a819 Mon Sep 17 00:00:00 2001 From: Michelangelo Mori <328978+blkt@users.noreply.github.com> Date: Fri, 5 Dec 2025 11:38:15 +0100 Subject: [PATCH 9/9] Add cross references back to Registry Server --- docs/toolhive/guides-k8s/remote-mcp-proxy.mdx | 4 ++++ docs/toolhive/guides-k8s/run-mcp-k8s.mdx | 4 ++++ docs/toolhive/guides-vmcp/configuration.mdx | 6 ++++++ 3 files changed, 14 insertions(+) diff --git a/docs/toolhive/guides-k8s/remote-mcp-proxy.mdx b/docs/toolhive/guides-k8s/remote-mcp-proxy.mdx index 6495a04d..b179074a 100644 --- a/docs/toolhive/guides-k8s/remote-mcp-proxy.mdx +++ b/docs/toolhive/guides-k8s/remote-mcp-proxy.mdx @@ -649,6 +649,10 @@ to learn how to connect to remote MCP proxies using different clients. Learn how to customize MCP tools using [filters and overrides](./customize-tools.mdx). +Discover your deployed MCP servers automatically using the +[Kubernetes registry](../guides-registry/configuration.mdx#kubernetes-registry) +feature in the ToolHive Registry Server. + ## Related information - [Kubernetes CRD reference](../reference/crd-spec.mdx) - Full MCPRemoteProxy diff --git a/docs/toolhive/guides-k8s/run-mcp-k8s.mdx b/docs/toolhive/guides-k8s/run-mcp-k8s.mdx index 6f3a2cf8..c15b7dcc 100644 --- a/docs/toolhive/guides-k8s/run-mcp-k8s.mdx +++ b/docs/toolhive/guides-k8s/run-mcp-k8s.mdx @@ -455,6 +455,10 @@ Collect telemetry data from your MCP servers by following the [Telemetry and metrics](./telemetry-and-metrics.mdx) guide. Configure audit logging by following the [Set up logging](./logging.mdx) guide. +Discover your deployed MCP servers automatically using the +[Kubernetes registry](../guides-registry/configuration.mdx#kubernetes-registry) +feature in the ToolHive Registry Server. + ## Related information - [Kubernetes CRD reference](../reference/crd-spec.mdx) - Reference for the diff --git a/docs/toolhive/guides-vmcp/configuration.mdx b/docs/toolhive/guides-vmcp/configuration.mdx index 0e84e8ed..5480a04b 100644 --- a/docs/toolhive/guides-vmcp/configuration.mdx +++ b/docs/toolhive/guides-vmcp/configuration.mdx @@ -98,6 +98,12 @@ Key status fields: | `backendCount` | Number of discovered backend MCP servers | | `discoveredBackends` | Details about each backend and its auth type | +## Next steps + +Discover your deployed MCP servers automatically using the +[Kubernetes registry](../guides-registry/configuration.mdx#kubernetes-registry) +feature in the ToolHive Registry Server. + ## Related information - [VirtualMCPServer CRD specification](../reference/crd-spec.mdx#virtualmcpserver)