Skip to content

Repository files navigation

ClickHouse Expert (CHE)

AI-powered specialist skill for ClickHouse database optimization, diagnostics, and operations.

Author: André Pretto

ClickHouse Expert is a Claude Code skill that provides expert-level guidance for ClickHouse at scale. It consists of specialized agents, diagnostic queries, and reference documentation that enable Claude to assist with query optimization, data modeling, observability, and SRE operations.


Architecture Overview

+============================================================================+
|                          CLICKHOUSE EXPERT (CHE)                           |
+============================================================================+
|                                                                            |
|  SLASH COMMANDS                                                            |
|  +----------------------------------------------------------------------+  |
|  | /ch-full-audit  <-- Comprehensive audit (security+metrics+perf)      |  |
|  +----------------------------------------------------------------------+  |
|  | /ch-observe   | /ch-model    | /ch-optimize | /ch-ingest             |  |
|  | /ch-sre       | /ch-govern   | /ch-security | /ch-ddl                |  |
|  | /ch-schema-doc| /ch-query    | /ch-client   |                        |  |
|  +----------------------------------------------------------------------+  |
|                                    |                                       |
|                                    v                                       |
|  SPECIALIZED AGENTS                                                        |
|  +----------------------------------------------------------------------+  |
|  | FULL AUDIT AGENT                                                     |  |
|  | Orchestrates: Security -> Metrics -> Optimization -> Memory Analysis |  |
|  +----------------------------------------------------------------------+  |
|  |                                                                      |  |
|  | +---------------+ +---------------+ +---------------+ +------------+ |  |
|  | | Security      | | Observability | | Data Modeling | | Query      | |  |
|  | | Audit Agent   | | Agent         | | Agent         | | Optim.     | |  |
|  | |               | |               | |               | | Agent      | |  |
|  | | - Users       | | - query_log   | | - MergeTree   | | - EXPLAIN  | |  |
|  | | - Roles       | | - parts       | | - Indexes     | | - JOINs    | |  |
|  | | - Quotas      | | - processes   | | - Codecs      | | - PREWHERE | |  |
|  | +---------------+ +---------------+ +---------------+ +------------+ |  |
|  |                                                                      |  |
|  | +---------------+ +---------------+ +---------------+ +------------+ |  |
|  | | Ingestion     | | Governance    | | SRE           | | Schema     | |  |
|  | | Agent         | | Agent         | | Operations    | | Doc Agent  | |  |
|  | |               | |               | | Agent         | |            | |  |
|  | | - Kafka/S3    | | - Lineage     | | - Backup/DR   | | - Reverse  | |  |
|  | | - Batching    | | - COMMENT ON  | | - Quotas      | |   Engineer | |  |
|  | | - Async       | | - Versioning  | | - Workloads   | | - Catalogs | |  |
|  | +---------------+ +---------------+ +---------------+ +------------+ |  |
|  |                                                                      |  |
|  | +---------------+ +---------------+                                  |  |
|  | | Query Builder | | DDL Extractor |                                  |  |
|  | | Agent         | | Agent         |                                  |  |
|  | |               | |               |                                  |  |
|  | | - Templates   | | - Export DDL  |                                  |  |
|  | | - Analytics   | | - Versioning  |                                  |  |
|  | | - Optimize    | | - Backup      |                                  |  |
|  | +---------------+ +---------------+                                  |  |
|  +----------------------------------------------------------------------+  |
|                                    |                                       |
|                                    v                                       |
|  KNOWLEDGE BASE                                                            |
|  +----------------------------------------------------------------------+  |
|  | +------------------+ +------------------+ +------------------+        |  |
|  | | references/      | | queries/         | | scripts/         |        |  |
|  | |                  | |                  | |                  |        |  |
|  | | - Schema Design  | | - Analytics      | | - schema-audit   |        |  |
|  | | - Analytics      | | - Performance    | | - schema-doc     |        |  |
|  | |   Patterns       | | - Diagnostics    | |                  |        |  |
|  | | - Client Conn    | | - Templates      | |                  |        |  |
|  | | - Environment    | | - Domain         | |                  |        |  |
|  | +------------------+ +------------------+ +------------------+        |  |
|  +----------------------------------------------------------------------+  |
|                                    |                                       |
+============================================================================+
                                     |
                                     v
                    +--------------------------------+
                    |      CLICKHOUSE SERVER         |
                    |                                |
                    |  +-------------------------+   |
                    |  |    system.* tables      |   |
                    |  |    - query_log          |   |
                    |  |    - parts              |   |
                    |  |    - processes          |   |
                    |  |    - replicas           |   |
                    |  |    - query_views_log    |   |
                    |  +-------------------------+   |
                    |                                |
                    | Cloud | Cluster | Single Node  |
                    +--------------------------------+

Requirements

Software Prerequisites

Requirement Version Description
Docker 20.10+ Required for ClickHouse client
Claude Code Latest Anthropic's CLI for Claude
ClickHouse Server 23.8+ Cloud, Cluster, or Single Node

Environment Support

Environment Engine Default Connection
ClickHouse Cloud SharedMergeTree Port 9440 (TLS)
Self-hosted Cluster ReplicatedMergeTree Port 9000
Single Node MergeTree Port 9000

Quick Start

1. Clone the Repository

git clone https://github.com/your-username/che.git
cd che

2. Configure Connection

# Copy the example environment file
cp clickhouse.env.example clickhouse.env

# Edit with your credentials
nano clickhouse.env  # or your preferred editor

Example clickhouse.env:

# Single Node / Cluster
CH_HOST=localhost
CH_PORT=9000
CH_USER=default
CH_PASSWORD=your_password_here

# ClickHouse Cloud (uncomment if using cloud)
# CH_HOST=your-instance.clickhouse.cloud
# CH_PORT=9440
# CH_SECURE=--secure

# Apple Silicon Macs (uncomment to suppress warnings)
# CH_PLATFORM=--platform=linux/amd64

3. Test Connection

source clickhouse.env && docker run --rm $CH_PLATFORM clickhouse/clickhouse-client \
  --host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
  --query "SELECT version()"

4. Detect Environment

source clickhouse.env && docker run --rm $CH_PLATFORM clickhouse/clickhouse-client \
  --host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
  --query "SELECT CASE WHEN getSetting('cloud_mode')='1' THEN 'cloud' \
    WHEN (SELECT count() FROM system.replicas)>0 THEN 'cluster' \
    ELSE 'single' END AS env, version()"

Slash Commands

Use these commands in Claude Code to invoke specialized agents:

Comprehensive Audit (Recommended Starting Point)

Command Agent Description Example
/ch-full-audit Full Audit Comprehensive audit: security, metrics, optimization, memory /ch-full-audit

Specialized Commands

Command Agent Description Example
/ch-security Security Audit Users, roles, privileges, quotas /ch-security audit users and roles
/ch-observe Observability Query performance, parts health, diagnostics /ch-observe why are queries slow?
/ch-model Data Modeling Table design, engines, indexes, codecs /ch-model design events table
/ch-optimize Query Optimization EXPLAIN, JOINs, PREWHERE, aggregations /ch-optimize this JOIN is slow
/ch-ingest Ingestion Kafka/S3, batching, async inserts /ch-ingest setup Kafka consumer
/ch-govern Governance Lineage, documentation, schema versioning /ch-govern document this table
/ch-sre SRE Operations Backups, DR, workload management /ch-sre backup strategy
/ch-schema-doc Schema Documentation Reverse-engineer, generate catalogs /ch-schema-doc analytics
/ch-ddl DDL Extractor Export DDLs to files, schema versioning /ch-ddl extract analytics
/ch-query Query Builder Generate, optimize, save queries /ch-query retention analysis
/ch-client Language Clients Python, Go, JS, Java connection examples /ch-client Python async example

Usage Examples

# Start Claude Code in the project directory
claude

# Run comprehensive audit (recommended first step)
> /ch-full-audit

# Security-focused audit
> /ch-security audit users without password

# Invoke observability agent
> /ch-observe analyze slow queries in the last hour

# Design a new table
> /ch-model design a user_events table with high write volume

# Optimize a problematic query
> /ch-optimize SELECT * FROM events JOIN users ON events.user_id = users.id

# Generate analytics queries
> /ch-query create DAU and retention queries for my events table

# Export DDLs to files
> /ch-ddl extract all databases

Headless Mode (CI/CD & Automation)

# Run security audit in batch mode
claude -p "/ch-security audit users without password" --output-format json > security-report.json

# Run full audit with Plan Mode (read-only)
claude --permission-mode plan -p "/ch-full-audit all databases"

# Quick health check
claude -p "/ch-observe check parts health"

Specialized Agents

Full Audit Agent ⭐

File: agents/full-audit-agent.md

Orchestrates comprehensive ClickHouse audits across all domains.

Capabilities:

  • Security audit (users, roles, privileges, quotas)
  • Database metrics extraction (sizes, parts, compression)
  • Query optimization analysis (slow queries, patterns)
  • Memory-intensive query identification
  • Unified audit report with severity indicators

Workflow Phases:

  1. Environment Detection → Cloud/Cluster/Single
  2. Security Audit → Users, roles, admin privileges
  3. Database Metrics → Sizes, parts health, compression
  4. Query Optimization → Slow queries, missing indexes
  5. Memory Analysis → High-memory patterns, recommendations

Security Audit Agent

File: agents/security-audit-agent.md

Audits users, roles, privileges, and access control.

Capabilities:

  • User authentication analysis
  • Role hierarchy mapping
  • Privilege audit (admin detection)
  • Quota monitoring and alerts
  • Security risk identification
  • Best practices recommendations

Observability Agent

File: agents/observability-agent.md

Diagnoses query performance and system health via system tables.

Capabilities:

  • Query performance analysis (system.query_log)
  • Parts health monitoring (system.parts)
  • Real-time process monitoring (system.processes)
  • Replication lag detection
  • Resource usage analysis
  • Memory debugging

Data Modeling Agent

File: agents/data-modeling-agent.md

Designs optimal table schemas for ClickHouse.

Capabilities:

  • MergeTree engine selection
  • ORDER BY key design
  • Skip index recommendations
  • Codec optimization
  • Partitioning strategies

Query Optimization Agent

File: agents/query-optimization-agent.md

Improves query performance through analysis and rewriting.

Capabilities:

  • EXPLAIN plan interpretation
  • JOIN algorithm selection
  • PREWHERE optimization
  • Aggregation tuning
  • Query rewriting patterns

Ingestion Agent

File: agents/ingestion-agent.md

Configures data ingestion pipelines.

Capabilities:

  • Batch insert optimization
  • Async insert configuration
  • Kafka consumer setup
  • S3 integration
  • "Too many parts" remediation

Governance Agent

File: agents/governance-agent.md

Manages schema documentation and data lineage.

Capabilities:

  • Schema documentation (COMMENT ON)
  • Data lineage tracking
  • Migration management
  • Data contract validation
  • Access control documentation

SRE Operations Agent

File: agents/sre-operations-agent.md

Handles operational tasks and disaster recovery.

Capabilities:

  • Native backup/restore
  • Disaster recovery procedures
  • Workload management
  • Resource quotas
  • Capacity planning

Schema Documentation Agent

File: agents/schema-documentation-agent.md

Generates documentation from existing schemas.

Capabilities:

  • Reverse-engineer schemas
  • Generate database catalogs
  • Analyze ORDER BY rationale
  • Compression reports
  • Identify documentation gaps

Query Builder Agent

File: agents/query-builder-agent.md

Creates and manages optimized queries.

Capabilities:

  • Schema analysis
  • Relationship discovery
  • Optimized query generation
  • Query library management
  • Analytics patterns (DAU, retention, funnel)

DDL Extractor Agent

File: agents/ddl-extractor-agent.md

Exports DDL definitions to organized files.

Capabilities:

  • Extract CREATE statements for all object types
  • Organize by database and object type
  • Support for tables, MVs, views, dictionaries, functions
  • Schema versioning and backup

Language Clients Agent

File: agents/language-clients-agent.md

Provides connection examples for various programming languages.

Capabilities:

  • Python (clickhouse-connect, clickhouse-driver)
  • Go (clickhouse-go)
  • JavaScript/TypeScript
  • Java
  • Connection pooling and best practices

Project Structure

che/
├── CLAUDE.md                    # Claude Code project instructions
├── SKILL.md                     # Main skill definition
├── README.md                    # This documentation
├── clickhouse.env.example       # Environment template
├── clickhouse.env               # Your credentials (gitignored)
│
├── .claude/
│   └── commands/                # Slash command definitions
│       ├── ch-full-audit.md     # /ch-full-audit - comprehensive audit
│       ├── ch-security.md       # /ch-security - security audit
│       ├── ch-observe.md        # /ch-observe command
│       ├── ch-model.md          # /ch-model command
│       ├── ch-optimize.md       # /ch-optimize command
│       ├── ch-ingest.md         # /ch-ingest command
│       ├── ch-govern.md         # /ch-govern command
│       ├── ch-sre.md            # /ch-sre command
│       ├── ch-schema-doc.md     # /ch-schema-doc command
│       ├── ch-ddl.md            # /ch-ddl command
│       ├── ch-query.md          # /ch-query command
│       └── ch-client.md         # /ch-client command
│
├── agents/                      # Specialized agent definitions
│   ├── README.md                # Agent routing guide
│   ├── full-audit-agent.md      # Comprehensive audit orchestrator
│   ├── security-audit-agent.md  # Users, roles, privileges
│   ├── observability-agent.md
│   ├── data-modeling-agent.md
│   ├── query-optimization-agent.md
│   ├── ingestion-agent.md
│   ├── governance-agent.md
│   ├── sre-operations-agent.md
│   ├── schema-documentation-agent.md
│   ├── ddl-extractor-agent.md   # DDL export to files
│   ├── query-builder-agent.md
│   ├── language-clients-agent.md
│   └── context7-integration.md  # Context7 MCP docs access
│
├── references/                  # Reference documentation
│   ├── client-connection.md     # Docker client connection guide
│   ├── environment-detection.md # Cloud/Cluster/Single detection
│   ├── schema-design-guide.md   # Table design best practices
│   ├── analytics-patterns.md    # Common analytics patterns
│   ├── performance-guide.md     # Query cache, profiling, parallelism, indexes
│   ├── functions-reference.md   # Array, date, string, JSON, aggregate functions
│   └── sql-reference.md         # SQL statements, data types, operators
│
├── queries/                     # Query library
│   ├── README.md                # Query index
│   ├── analytics/               # Business intelligence queries
│   │   ├── dau.sql
│   │   ├── retention.sql
│   │   └── funnel.sql
│   ├── performance/             # Performance monitoring
│   │   └── slow_queries.sql
│   ├── diagnostics/             # System health
│   │   └── parts_health.sql
│   └── templates/               # Reusable patterns
│       ├── aggregation.sql
│       ├── time_series.sql
│       ├── joins.sql
│       ├── window_functions.sql
│       └── cte_strategies.sql
│
├── scripts/                     # Diagnostic SQL scripts
│   ├── schema-audit.sql         # Full schema audit
│   └── schema-doc.sql           # Schema documentation extraction
│
└── docs/                        # Additional documentation
    ├── cloudflare_radar.md
    └── routeviews_mrt.md

Running Diagnostic Scripts

Full Schema Audit

# All databases
source clickhouse.env && docker run -i --rm $CH_PLATFORM clickhouse/clickhouse-client \
  --host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
  --param_target_db="" \
  --multiquery < scripts/schema-audit.sql

# Specific database
source clickhouse.env && docker run -i --rm $CH_PLATFORM clickhouse/clickhouse-client \
  --host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
  --param_target_db="analytics" \
  --multiquery < scripts/schema-audit.sql

Run Query Templates

# With parameters
source clickhouse.env && docker run -i --rm $CH_PLATFORM clickhouse/clickhouse-client \
  --host $CH_HOST --user $CH_USER --password $CH_PASSWORD $CH_SECURE \
  --param_database="your_database" \
  --param_table="your_table" \
  --param_days="30" \
  < queries/analytics/dau.sql

Table Engines Reference

ClickHouse supports multiple engine families for different use cases:

MergeTree Family (Production)

Engine Use Case Key Feature
MergeTree General analytics Primary engine, sorted storage
ReplacingMergeTree Deduplication Removes duplicates by key during merge
SummingMergeTree Metric aggregation Auto-sums numeric columns
AggregatingMergeTree Pre-aggregation Stores intermediate aggregate states
CollapsingMergeTree Mutable data Sign column (+1/-1) for updates
VersionedCollapsingMergeTree Versioned mutations Sign + version for out-of-order

Integration Engines (External Sources)

Engine Source Use Case
Kafka Apache Kafka Real-time streaming
MySQL MySQL/MariaDB Database sync
PostgreSQL PostgreSQL Database sync
S3 Amazon S3 / MinIO Object storage
HDFS Hadoop Big data ecosystem
JDBC/ODBC Any database Generic connectivity

Log Engines (Simple Storage)

Engine Use Case
Log Temporary tables, staging
TinyLog Tiny datasets (<1MB)
StripeLog Parallel reads needed

Special Engines

Engine Purpose
Distributed Query routing across shards
Dictionary O(1) lookups, replace JOINs
Memory RAM-only (extremely fast, volatile)
Buffer Write buffering before flush
MaterializedView Auto-updated aggregations
Null Discard data (useful for MVs)

Tip: Use /ch-model for detailed engine selection guidance.


Advanced Topics Covered

The agents include comprehensive coverage of advanced ClickHouse topics:

Query Optimization

  • Query Execution Stages - Parser, Analyzer, Planner, Pipeline, Executor
  • EXPLAIN Commands - AST, QUERY TREE, PLAN, PIPELINE analysis
  • JOIN Algorithms - Hash, Parallel Hash, Grace Hash, Full Sorting Merge, Partial Merge, Direct
  • Time Series Gap Filling - WITH FILL, INTERPOLATE, FROM/TO/STEP

Data Modeling

  • TTL Management - Table/Column TTL, Conditional TTL, Rollup, Storage Tiering
  • Cascading Materialized Views - Multi-stage aggregation pipelines
  • Deduplication Strategies - ReplacingMergeTree, CollapsingMergeTree, VersionedCollapsing

Data Mutations

  • Lightweight DELETE - Fast row masking with _row_exists column
  • Lightweight UPDATE - On-the-fly mutations with immediate visibility
  • ALTER Mutations - Heavy mutations for guaranteed data removal
  • Best Practices - When to use each method

Operations

  • Memory Debugging - Systematic diagnosis of memory issues
  • Mutation Management - Monitor, troubleshoot, and optimize mutations
  • Backup/Restore - Environment-specific backup strategies

Performance & Caching

  • Query Cache - Cache SELECT results with configurable TTL
  • Query Profiling - Sampling profiler with flame graph generation
  • Query Parallelism - Processing lanes, max_threads tuning
  • Data Skipping Indexes - minmax, set, bloom_filter, tokenbf, ngrambf
  • Userspace Page Cache - For remote/cloud storage optimization

Functions Reference

  • Array Functions - arrayMap, arrayFilter, arrayJoin, groupArray
  • Date/Time Functions - toStartOf*, dateDiff, formatDateTime, timezones
  • String Functions - substring, split, replace, regex
  • JSON Functions - JSONExtract*, JSON_VALUE, simpleJSON*
  • Aggregate Combinators - -If, -Array, -State, -Merge, -ForEach, -Distinct
  • Table Functions - s3, mysql, postgresql, file, url, remote
  • Window Functions - row_number, rank, lag, lead, running totals

Audit Workflows

Full Audit Workflow (/ch-full-audit)

The comprehensive audit runs through 5 phases:

┌─────────────────────────────────────────────────────────────┐
│                    FULL AUDIT WORKFLOW                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Phase 1: Environment Detection                              │
│  └─> Detect Cloud/Cluster/Single, list databases            │
│                                                              │
│  Phase 2: Security Audit                                     │
│  └─> Users, authentication, roles, privileges, quotas       │
│                                                              │
│  Phase 3: Database Metrics                                   │
│  └─> Table sizes, parts health, compression, MVs, disk      │
│                                                              │
│  Phase 4: Query Optimization                                 │
│  └─> Slow queries, patterns, missing indexes, PREWHERE      │
│                                                              │
│  Phase 5: Memory Analysis                                    │
│  └─> High-memory queries, patterns, recommendations         │
│                                                              │
│  Output: Unified report with 🔴/🟡/✅ severity indicators    │
└─────────────────────────────────────────────────────────────┘

When to Use Each Audit

Scenario Recommended Command
First time on a ClickHouse deployment /ch-full-audit
Security compliance review /ch-security
Performance investigation /ch-observe + /ch-optimize
Memory issues /ch-observe (memory debugging section)
Schema documentation /ch-schema-doc
Backup/DR planning /ch-sre

Essential Diagnostic Queries

Top Slow Queries

SELECT query, query_duration_ms, read_rows, memory_usage
FROM system.query_log
WHERE type = 'QueryFinish'
ORDER BY query_duration_ms DESC
LIMIT 10;

Parts Health

SELECT database, table, count() AS parts, sum(rows) AS rows,
       formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY parts DESC;

Active Processes

SELECT query_id, user, elapsed, query
FROM system.processes;

Materialized View Latency

SELECT view_name, avg(view_duration_ms) AS avg_ms, count() AS inserts
FROM system.query_views_log
GROUP BY view_name
ORDER BY avg_ms DESC;

Audit Severity Indicators

The audit agents use consistent severity indicators:

Indicator Meaning Action Required
🔴 Critical issue Immediate action required
🟡 Warning Plan remediation
Healthy No action needed
💡 Optimization hint Consider improvement
⚠️ Caution Review and monitor

Audit Thresholds

Metric Warning Critical
Parts per table > 300 > 1000
Memory per query > 5GB > 20GB
Compression ratio > 0.3 > 0.5
Quota usage > 70% > 90%
Disk free < 20% < 10%

Anti-Patterns to Avoid

Anti-Pattern Why It's Bad Solution
Row-by-row inserts Creates too many parts Batch 10k-100k rows
SELECT * on wide tables Full column scan Specify needed columns
Ignoring "Too many parts" Leads to performance degradation Fix ingestion immediately
MVs on high-frequency sources Write amplification Design for write patterns
ReplacingMergeTree for real-time dedup Merge is async Use FINAL or redesign
Users without password Security vulnerability Use sha256/bcrypt auth
HOST ANY for production users Unrestricted access Restrict by IP/hostname

Claude Code Integration

Plan Mode (Read-Only Analysis)

Use Plan Mode for safe analysis without modifications:

# Start in Plan Mode
claude --permission-mode plan

# Or toggle during session with Shift+Tab

Plan Mode is ideal for:

  • Initial codebase/database exploration
  • Security audits (read-only)
  • Performance analysis
  • Compliance reviews

Headless Mode (Automation)

Run audits in scripts and CI/CD pipelines:

# JSON output for parsing
claude -p "/ch-full-audit" --output-format json > audit.json

# Text output
claude -p "/ch-security audit admin users" --output-format text

# Streaming JSON for real-time processing
claude -p "/ch-observe check parts health" --output-format stream-json

Session Management

# Resume last session
claude --continue

# Resume specific session
claude --resume security-audit

# Name sessions for later
> /rename security-audit-2024

Context7 Integration

For up-to-date ClickHouse documentation, use the Context7 MCP:

mcp__context7__get-library-docs
  context7CompatibleLibraryID: "websites/clickhouse"
  topic: "MergeTree engine configuration"
Property Value
Library ID websites/clickhouse
Snippets 32,445
Trust Score 10/10

Troubleshooting

Issue Solution
Connection refused Check host/port, firewall rules
Authentication failed Verify user/password in clickhouse.env
SSL/TLS errors Add --secure flag for cloud
"Input device is not a TTY" Remove -t flag when using --query
Platform mismatch warning (ARM64 Mac) Set CH_PLATFORM=--platform=linux/amd64

Contributing

  1. Follow the existing file structure and naming conventions
  2. Keep diagnostic SQL queries tested and production-ready
  3. Update README when adding new reference documents
  4. Maintain the query file header format

Author

André Pretto


License

MIT License - See LICENSE file for details.


Resources

About

Clickhouse Expert AI

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors