AI-powered specialist skill for ClickHouse database optimization, diagnostics, and operations.
Author: André Pretto
ClickHouse Expert is a Claude Code skill that provides expert-level guidance for ClickHouse at scale. It consists of specialized agents, diagnostic queries, and reference documentation that enable Claude to assist with query optimization, data modeling, observability, and SRE operations.
+============================================================================+
| CLICKHOUSE EXPERT (CHE) |
+============================================================================+
| |
| SLASH COMMANDS |
| +----------------------------------------------------------------------+ |
| | /ch-full-audit <-- Comprehensive audit (security+metrics+perf) | |
| +----------------------------------------------------------------------+ |
| | /ch-observe | /ch-model | /ch-optimize | /ch-ingest | |
| | /ch-sre | /ch-govern | /ch-security | /ch-ddl | |
| | /ch-schema-doc| /ch-query | /ch-client | | |
| +----------------------------------------------------------------------+ |
| | |
| v |
| SPECIALIZED AGENTS |
| +----------------------------------------------------------------------+ |
| | FULL AUDIT AGENT | |
| | Orchestrates: Security -> Metrics -> Optimization -> Memory Analysis | |
| +----------------------------------------------------------------------+ |
| | | |
| | +---------------+ +---------------+ +---------------+ +------------+ | |
| | | Security | | Observability | | Data Modeling | | Query | | |
| | | Audit Agent | | Agent | | Agent | | Optim. | | |
| | | | | | | | | Agent | | |
| | | - Users | | - query_log | | - MergeTree | | - EXPLAIN | | |
| | | - Roles | | - parts | | - Indexes | | - JOINs | | |
| | | - Quotas | | - processes | | - Codecs | | - PREWHERE | | |
| | +---------------+ +---------------+ +---------------+ +------------+ | |
| | | |
| | +---------------+ +---------------+ +---------------+ +------------+ | |
| | | Ingestion | | Governance | | SRE | | Schema | | |
| | | Agent | | Agent | | Operations | | Doc Agent | | |
| | | | | | | Agent | | | | |
| | | - Kafka/S3 | | - Lineage | | - Backup/DR | | - Reverse | | |
| | | - Batching | | - COMMENT ON | | - Quotas | | Engineer | | |
| | | - Async | | - Versioning | | - Workloads | | - Catalogs | | |
| | +---------------+ +---------------+ +---------------+ +------------+ | |
| | | |
| | +---------------+ +---------------+ | |
| | | Query Builder | | DDL Extractor | | |
| | | Agent | | Agent | | |
| | | | | | | |
| | | - Templates | | - Export DDL | | |
| | | - Analytics | | - Versioning | | |
| | | - Optimize | | - Backup | | |
| | +---------------+ +---------------+ | |
| +----------------------------------------------------------------------+ |
| | |
| v |
| KNOWLEDGE BASE |
| +----------------------------------------------------------------------+ |
| | +------------------+ +------------------+ +------------------+ | |
| | | references/ | | queries/ | | scripts/ | | |
| | | | | | | | | |
| | | - Schema Design | | - Analytics | | - schema-audit | | |
| | | - Analytics | | - Performance | | - schema-doc | | |
| | | Patterns | | - Diagnostics | | | | |
| | | - Client Conn | | - Templates | | | | |
| | | - Environment | | - Domain | | | | |
| | +------------------+ +------------------+ +------------------+ | |
| +----------------------------------------------------------------------+ |
| | |
+============================================================================+
|
v
+--------------------------------+
| CLICKHOUSE SERVER |
| |
| +-------------------------+ |
| | system.* tables | |
| | - query_log | |
| | - parts | |
| | - processes | |
| | - replicas | |
| | - query_views_log | |
| +-------------------------+ |
| |
| Cloud | Cluster | Single Node |
+--------------------------------+
| Requirement | Version | Description |
|---|---|---|
| Docker | 20.10+ | Required for ClickHouse client |
| Claude Code | Latest | Anthropic's CLI for Claude |
| ClickHouse Server | 23.8+ | Cloud, Cluster, or Single Node |
| Environment | Engine Default | Connection |
|---|---|---|
| ClickHouse Cloud | SharedMergeTree | Port 9440 (TLS) |
| Self-hosted Cluster | ReplicatedMergeTree | Port 9000 |
| Single Node | MergeTree | Port 9000 |
git clone https://github.com/your-username/che.git
cd che# Copy the example environment file
cp clickhouse.env.example clickhouse.env
# Edit with your credentials
nano clickhouse.env # or your preferred editorExample clickhouse.env:
# Single Node / Cluster
CH_HOST=localhost
CH_PORT=9000
CH_USER=default
CH_PASSWORD=your_password_here
# ClickHouse Cloud (uncomment if using cloud)
# CH_HOST=your-instance.clickhouse.cloud
# CH_PORT=9440
# CH_SECURE=--secure
# Apple Silicon Macs (uncomment to suppress warnings)
# CH_PLATFORM=--platform=linux/amd64source clickhouse.env && docker run --rm $CH_PLATFORM clickhouse/clickhouse-client \
--host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
--query "SELECT version()"source clickhouse.env && docker run --rm $CH_PLATFORM clickhouse/clickhouse-client \
--host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
--query "SELECT CASE WHEN getSetting('cloud_mode')='1' THEN 'cloud' \
WHEN (SELECT count() FROM system.replicas)>0 THEN 'cluster' \
ELSE 'single' END AS env, version()"Use these commands in Claude Code to invoke specialized agents:
| Command | Agent | Description | Example |
|---|---|---|---|
/ch-full-audit |
Full Audit | Comprehensive audit: security, metrics, optimization, memory | /ch-full-audit |
| Command | Agent | Description | Example |
|---|---|---|---|
/ch-security |
Security Audit | Users, roles, privileges, quotas | /ch-security audit users and roles |
/ch-observe |
Observability | Query performance, parts health, diagnostics | /ch-observe why are queries slow? |
/ch-model |
Data Modeling | Table design, engines, indexes, codecs | /ch-model design events table |
/ch-optimize |
Query Optimization | EXPLAIN, JOINs, PREWHERE, aggregations | /ch-optimize this JOIN is slow |
/ch-ingest |
Ingestion | Kafka/S3, batching, async inserts | /ch-ingest setup Kafka consumer |
/ch-govern |
Governance | Lineage, documentation, schema versioning | /ch-govern document this table |
/ch-sre |
SRE Operations | Backups, DR, workload management | /ch-sre backup strategy |
/ch-schema-doc |
Schema Documentation | Reverse-engineer, generate catalogs | /ch-schema-doc analytics |
/ch-ddl |
DDL Extractor | Export DDLs to files, schema versioning | /ch-ddl extract analytics |
/ch-query |
Query Builder | Generate, optimize, save queries | /ch-query retention analysis |
/ch-client |
Language Clients | Python, Go, JS, Java connection examples | /ch-client Python async example |
# Start Claude Code in the project directory
claude
# Run comprehensive audit (recommended first step)
> /ch-full-audit
# Security-focused audit
> /ch-security audit users without password
# Invoke observability agent
> /ch-observe analyze slow queries in the last hour
# Design a new table
> /ch-model design a user_events table with high write volume
# Optimize a problematic query
> /ch-optimize SELECT * FROM events JOIN users ON events.user_id = users.id
# Generate analytics queries
> /ch-query create DAU and retention queries for my events table
# Export DDLs to files
> /ch-ddl extract all databases# Run security audit in batch mode
claude -p "/ch-security audit users without password" --output-format json > security-report.json
# Run full audit with Plan Mode (read-only)
claude --permission-mode plan -p "/ch-full-audit all databases"
# Quick health check
claude -p "/ch-observe check parts health"File: agents/full-audit-agent.md
Orchestrates comprehensive ClickHouse audits across all domains.
Capabilities:
- Security audit (users, roles, privileges, quotas)
- Database metrics extraction (sizes, parts, compression)
- Query optimization analysis (slow queries, patterns)
- Memory-intensive query identification
- Unified audit report with severity indicators
Workflow Phases:
- Environment Detection → Cloud/Cluster/Single
- Security Audit → Users, roles, admin privileges
- Database Metrics → Sizes, parts health, compression
- Query Optimization → Slow queries, missing indexes
- Memory Analysis → High-memory patterns, recommendations
File: agents/security-audit-agent.md
Audits users, roles, privileges, and access control.
Capabilities:
- User authentication analysis
- Role hierarchy mapping
- Privilege audit (admin detection)
- Quota monitoring and alerts
- Security risk identification
- Best practices recommendations
File: agents/observability-agent.md
Diagnoses query performance and system health via system tables.
Capabilities:
- Query performance analysis (
system.query_log) - Parts health monitoring (
system.parts) - Real-time process monitoring (
system.processes) - Replication lag detection
- Resource usage analysis
- Memory debugging
File: agents/data-modeling-agent.md
Designs optimal table schemas for ClickHouse.
Capabilities:
- MergeTree engine selection
- ORDER BY key design
- Skip index recommendations
- Codec optimization
- Partitioning strategies
File: agents/query-optimization-agent.md
Improves query performance through analysis and rewriting.
Capabilities:
- EXPLAIN plan interpretation
- JOIN algorithm selection
- PREWHERE optimization
- Aggregation tuning
- Query rewriting patterns
File: agents/ingestion-agent.md
Configures data ingestion pipelines.
Capabilities:
- Batch insert optimization
- Async insert configuration
- Kafka consumer setup
- S3 integration
- "Too many parts" remediation
File: agents/governance-agent.md
Manages schema documentation and data lineage.
Capabilities:
- Schema documentation (
COMMENT ON) - Data lineage tracking
- Migration management
- Data contract validation
- Access control documentation
File: agents/sre-operations-agent.md
Handles operational tasks and disaster recovery.
Capabilities:
- Native backup/restore
- Disaster recovery procedures
- Workload management
- Resource quotas
- Capacity planning
File: agents/schema-documentation-agent.md
Generates documentation from existing schemas.
Capabilities:
- Reverse-engineer schemas
- Generate database catalogs
- Analyze ORDER BY rationale
- Compression reports
- Identify documentation gaps
File: agents/query-builder-agent.md
Creates and manages optimized queries.
Capabilities:
- Schema analysis
- Relationship discovery
- Optimized query generation
- Query library management
- Analytics patterns (DAU, retention, funnel)
File: agents/ddl-extractor-agent.md
Exports DDL definitions to organized files.
Capabilities:
- Extract CREATE statements for all object types
- Organize by database and object type
- Support for tables, MVs, views, dictionaries, functions
- Schema versioning and backup
File: agents/language-clients-agent.md
Provides connection examples for various programming languages.
Capabilities:
- Python (clickhouse-connect, clickhouse-driver)
- Go (clickhouse-go)
- JavaScript/TypeScript
- Java
- Connection pooling and best practices
che/
├── CLAUDE.md # Claude Code project instructions
├── SKILL.md # Main skill definition
├── README.md # This documentation
├── clickhouse.env.example # Environment template
├── clickhouse.env # Your credentials (gitignored)
│
├── .claude/
│ └── commands/ # Slash command definitions
│ ├── ch-full-audit.md # /ch-full-audit - comprehensive audit
│ ├── ch-security.md # /ch-security - security audit
│ ├── ch-observe.md # /ch-observe command
│ ├── ch-model.md # /ch-model command
│ ├── ch-optimize.md # /ch-optimize command
│ ├── ch-ingest.md # /ch-ingest command
│ ├── ch-govern.md # /ch-govern command
│ ├── ch-sre.md # /ch-sre command
│ ├── ch-schema-doc.md # /ch-schema-doc command
│ ├── ch-ddl.md # /ch-ddl command
│ ├── ch-query.md # /ch-query command
│ └── ch-client.md # /ch-client command
│
├── agents/ # Specialized agent definitions
│ ├── README.md # Agent routing guide
│ ├── full-audit-agent.md # Comprehensive audit orchestrator
│ ├── security-audit-agent.md # Users, roles, privileges
│ ├── observability-agent.md
│ ├── data-modeling-agent.md
│ ├── query-optimization-agent.md
│ ├── ingestion-agent.md
│ ├── governance-agent.md
│ ├── sre-operations-agent.md
│ ├── schema-documentation-agent.md
│ ├── ddl-extractor-agent.md # DDL export to files
│ ├── query-builder-agent.md
│ ├── language-clients-agent.md
│ └── context7-integration.md # Context7 MCP docs access
│
├── references/ # Reference documentation
│ ├── client-connection.md # Docker client connection guide
│ ├── environment-detection.md # Cloud/Cluster/Single detection
│ ├── schema-design-guide.md # Table design best practices
│ ├── analytics-patterns.md # Common analytics patterns
│ ├── performance-guide.md # Query cache, profiling, parallelism, indexes
│ ├── functions-reference.md # Array, date, string, JSON, aggregate functions
│ └── sql-reference.md # SQL statements, data types, operators
│
├── queries/ # Query library
│ ├── README.md # Query index
│ ├── analytics/ # Business intelligence queries
│ │ ├── dau.sql
│ │ ├── retention.sql
│ │ └── funnel.sql
│ ├── performance/ # Performance monitoring
│ │ └── slow_queries.sql
│ ├── diagnostics/ # System health
│ │ └── parts_health.sql
│ └── templates/ # Reusable patterns
│ ├── aggregation.sql
│ ├── time_series.sql
│ ├── joins.sql
│ ├── window_functions.sql
│ └── cte_strategies.sql
│
├── scripts/ # Diagnostic SQL scripts
│ ├── schema-audit.sql # Full schema audit
│ └── schema-doc.sql # Schema documentation extraction
│
└── docs/ # Additional documentation
├── cloudflare_radar.md
└── routeviews_mrt.md
# All databases
source clickhouse.env && docker run -i --rm $CH_PLATFORM clickhouse/clickhouse-client \
--host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
--param_target_db="" \
--multiquery < scripts/schema-audit.sql
# Specific database
source clickhouse.env && docker run -i --rm $CH_PLATFORM clickhouse/clickhouse-client \
--host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
--param_target_db="analytics" \
--multiquery < scripts/schema-audit.sql# With parameters
source clickhouse.env && docker run -i --rm $CH_PLATFORM clickhouse/clickhouse-client \
--host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
--param_database="your_database" \
--param_table="your_table" \
--param_days="30" \
< queries/analytics/dau.sqlClickHouse supports multiple engine families for different use cases:
| Engine | Use Case | Key Feature |
|---|---|---|
| MergeTree | General analytics | Primary engine, sorted storage |
| ReplacingMergeTree | Deduplication | Removes duplicates by key during merge |
| SummingMergeTree | Metric aggregation | Auto-sums numeric columns |
| AggregatingMergeTree | Pre-aggregation | Stores intermediate aggregate states |
| CollapsingMergeTree | Mutable data | Sign column (+1/-1) for updates |
| VersionedCollapsingMergeTree | Versioned mutations | Sign + version for out-of-order |
| Engine | Source | Use Case |
|---|---|---|
| Kafka | Apache Kafka | Real-time streaming |
| MySQL | MySQL/MariaDB | Database sync |
| PostgreSQL | PostgreSQL | Database sync |
| S3 | Amazon S3 / MinIO | Object storage |
| HDFS | Hadoop | Big data ecosystem |
| JDBC/ODBC | Any database | Generic connectivity |
| Engine | Use Case |
|---|---|
| Log | Temporary tables, staging |
| TinyLog | Tiny datasets (<1MB) |
| StripeLog | Parallel reads needed |
| Engine | Purpose |
|---|---|
| Distributed | Query routing across shards |
| Dictionary | O(1) lookups, replace JOINs |
| Memory | RAM-only (extremely fast, volatile) |
| Buffer | Write buffering before flush |
| MaterializedView | Auto-updated aggregations |
| Null | Discard data (useful for MVs) |
Tip: Use
/ch-modelfor detailed engine selection guidance.
The agents include comprehensive coverage of advanced ClickHouse topics:
- Query Execution Stages - Parser, Analyzer, Planner, Pipeline, Executor
- EXPLAIN Commands - AST, QUERY TREE, PLAN, PIPELINE analysis
- JOIN Algorithms - Hash, Parallel Hash, Grace Hash, Full Sorting Merge, Partial Merge, Direct
- Time Series Gap Filling - WITH FILL, INTERPOLATE, FROM/TO/STEP
- TTL Management - Table/Column TTL, Conditional TTL, Rollup, Storage Tiering
- Cascading Materialized Views - Multi-stage aggregation pipelines
- Deduplication Strategies - ReplacingMergeTree, CollapsingMergeTree, VersionedCollapsing
- Lightweight DELETE - Fast row masking with
_row_existscolumn - Lightweight UPDATE - On-the-fly mutations with immediate visibility
- ALTER Mutations - Heavy mutations for guaranteed data removal
- Best Practices - When to use each method
- Memory Debugging - Systematic diagnosis of memory issues
- Mutation Management - Monitor, troubleshoot, and optimize mutations
- Backup/Restore - Environment-specific backup strategies
- Query Cache - Cache SELECT results with configurable TTL
- Query Profiling - Sampling profiler with flame graph generation
- Query Parallelism - Processing lanes, max_threads tuning
- Data Skipping Indexes - minmax, set, bloom_filter, tokenbf, ngrambf
- Userspace Page Cache - For remote/cloud storage optimization
- Array Functions - arrayMap, arrayFilter, arrayJoin, groupArray
- Date/Time Functions - toStartOf*, dateDiff, formatDateTime, timezones
- String Functions - substring, split, replace, regex
- JSON Functions - JSONExtract*, JSON_VALUE, simpleJSON*
- Aggregate Combinators - -If, -Array, -State, -Merge, -ForEach, -Distinct
- Table Functions - s3, mysql, postgresql, file, url, remote
- Window Functions - row_number, rank, lag, lead, running totals
The comprehensive audit runs through 5 phases:
┌─────────────────────────────────────────────────────────────┐
│ FULL AUDIT WORKFLOW │
├─────────────────────────────────────────────────────────────┤
│ │
│ Phase 1: Environment Detection │
│ └─> Detect Cloud/Cluster/Single, list databases │
│ │
│ Phase 2: Security Audit │
│ └─> Users, authentication, roles, privileges, quotas │
│ │
│ Phase 3: Database Metrics │
│ └─> Table sizes, parts health, compression, MVs, disk │
│ │
│ Phase 4: Query Optimization │
│ └─> Slow queries, patterns, missing indexes, PREWHERE │
│ │
│ Phase 5: Memory Analysis │
│ └─> High-memory queries, patterns, recommendations │
│ │
│ Output: Unified report with 🔴/🟡/✅ severity indicators │
└─────────────────────────────────────────────────────────────┘
| Scenario | Recommended Command |
|---|---|
| First time on a ClickHouse deployment | /ch-full-audit |
| Security compliance review | /ch-security |
| Performance investigation | /ch-observe + /ch-optimize |
| Memory issues | /ch-observe (memory debugging section) |
| Schema documentation | /ch-schema-doc |
| Backup/DR planning | /ch-sre |
SELECT query, query_duration_ms, read_rows, memory_usage
FROM system.query_log
WHERE type = 'QueryFinish'
ORDER BY query_duration_ms DESC
LIMIT 10;SELECT database, table, count() AS parts, sum(rows) AS rows,
formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY parts DESC;SELECT query_id, user, elapsed, query
FROM system.processes;SELECT view_name, avg(view_duration_ms) AS avg_ms, count() AS inserts
FROM system.query_views_log
GROUP BY view_name
ORDER BY avg_ms DESC;The audit agents use consistent severity indicators:
| Indicator | Meaning | Action Required |
|---|---|---|
| 🔴 | Critical issue | Immediate action required |
| 🟡 | Warning | Plan remediation |
| ✅ | Healthy | No action needed |
| 💡 | Optimization hint | Consider improvement |
| Caution | Review and monitor |
| Metric | Warning | Critical |
|---|---|---|
| Parts per table | > 300 | > 1000 |
| Memory per query | > 5GB | > 20GB |
| Compression ratio | > 0.3 | > 0.5 |
| Quota usage | > 70% | > 90% |
| Disk free | < 20% | < 10% |
| Anti-Pattern | Why It's Bad | Solution |
|---|---|---|
| Row-by-row inserts | Creates too many parts | Batch 10k-100k rows |
SELECT * on wide tables |
Full column scan | Specify needed columns |
| Ignoring "Too many parts" | Leads to performance degradation | Fix ingestion immediately |
| MVs on high-frequency sources | Write amplification | Design for write patterns |
| ReplacingMergeTree for real-time dedup | Merge is async | Use FINAL or redesign |
| Users without password | Security vulnerability | Use sha256/bcrypt auth |
HOST ANY for production users |
Unrestricted access | Restrict by IP/hostname |
Use Plan Mode for safe analysis without modifications:
# Start in Plan Mode
claude --permission-mode plan
# Or toggle during session with Shift+TabPlan Mode is ideal for:
- Initial codebase/database exploration
- Security audits (read-only)
- Performance analysis
- Compliance reviews
Run audits in scripts and CI/CD pipelines:
# JSON output for parsing
claude -p "/ch-full-audit" --output-format json > audit.json
# Text output
claude -p "/ch-security audit admin users" --output-format text
# Streaming JSON for real-time processing
claude -p "/ch-observe check parts health" --output-format stream-json# Resume last session
claude --continue
# Resume specific session
claude --resume security-audit
# Name sessions for later
> /rename security-audit-2024For up-to-date ClickHouse documentation, use the Context7 MCP:
mcp__context7__get-library-docs
context7CompatibleLibraryID: "websites/clickhouse"
topic: "MergeTree engine configuration"
| Property | Value |
|---|---|
| Library ID | websites/clickhouse |
| Snippets | 32,445 |
| Trust Score | 10/10 |
| Issue | Solution |
|---|---|
| Connection refused | Check host/port, firewall rules |
| Authentication failed | Verify user/password in clickhouse.env |
| SSL/TLS errors | Add --secure flag for cloud |
| "Input device is not a TTY" | Remove -t flag when using --query |
| Platform mismatch warning (ARM64 Mac) | Set CH_PLATFORM=--platform=linux/amd64 |
- Follow the existing file structure and naming conventions
- Keep diagnostic SQL queries tested and production-ready
- Update README when adding new reference documents
- Maintain the query file header format
André Pretto
MIT License - See LICENSE file for details.