feat: write event data to clickhouse by iand · Pull Request #438 · FilOzone/dealbot

iand · 2026-04-08T16:03:20Z

Adds a ClickHouse integration to the backend that captures individual deal, retrieval, and data retention observations as queryable analytical records

Three tables are written to: data_storage_checks, retrieval_checks, and data_retention_challenges. Writes are buffered and flushed in the background; failures are logged and swallowed so ClickHouse issues never affect job execution.

When CLICKHOUSE_URL is unset the integration is a no-op, so the app works unchanged without it configured. Tables are created automatically on startup. A ClickHouse instance is included in the local Kind cluster for development.

Four Prometheus metrics expose ClickHouse health: flush duration, error count, buffer depth, rows inserted.

For #426

Copilot

Pull request overview

Adds a ClickHouse integration to the backend to emit high-dimensional analytical “check” events (deal storage, retrieval, and data retention) into queryable tables, plus local Kind resources to run ClickHouse in development.

Changes:

Added a global NestJS ClickhouseModule/ClickhouseService with buffered inserts, periodic flushing, and Prometheus metrics.
Emitted ClickHouse rows from DealService, RetrievalService, and DataRetentionService (with corresponding test wiring updates).
Added local kustomize overlay resources + Make targets to deploy/reset/shell into ClickHouse.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Locks `@clickhouse/client` dependency additions.
Makefile	Adds `clickhouse-reset` and `clickhouse-shell` dev targets.
kustomize/overlays/local/kustomization.yaml	Includes the new local ClickHouse overlay.
kustomize/overlays/local/clickhouse/service.yaml	Exposes ClickHouse ports (8123/9000) in-cluster.
kustomize/overlays/local/clickhouse/pvc.yaml	Persists ClickHouse data locally via PVC.
kustomize/overlays/local/clickhouse/kustomization.yaml	Wires ClickHouse overlay resources together.
kustomize/overlays/local/clickhouse/initdb-configmap.yaml	Initializes the `dealbot` ClickHouse database in local dev.
kustomize/overlays/local/clickhouse/deployment.yaml	Deploys ClickHouse server in the local overlay.
kustomize/overlays/local/backend-resources-local.yaml	Sets local backend resource requests/limits.
kustomize/overlays/local/backend-configmap-local.yaml	Enables ClickHouse in local overlay and sets probe location.
apps/backend/src/worker.module.ts	Imports `ClickhouseModule` into the worker app.
apps/backend/src/retrieval/retrieval.service.ts	Inserts retrieval check rows into ClickHouse.
apps/backend/src/retrieval/retrieval.service.spec.ts	Mocks `ClickhouseService` and config for updated DI.
apps/backend/src/deal/deal.service.ts	Inserts storage check rows into ClickHouse; adjusts `pieceId` assignment logic.
apps/backend/src/deal/deal.service.spec.ts	Mocks `ClickhouseService` for updated DI.
apps/backend/src/data-retention/data-retention.service.ts	Inserts retention challenge rows into ClickHouse.
apps/backend/src/data-retention/data-retention.service.spec.ts	Adds a `ClickhouseService` mock to constructor wiring.
apps/backend/src/config/app.config.ts	Adds ClickHouse env vars and `probeLocation` to app config.
apps/backend/src/clickhouse/clickhouse.service.ts	Implements ClickHouse client lifecycle, buffering, flushing, and metrics.
apps/backend/src/clickhouse/clickhouse.schema.ts	Defines ClickHouse DDL for the three analytical tables.
apps/backend/src/clickhouse/clickhouse.module.ts	Registers Prometheus metrics + service as a global module.
apps/backend/src/clickhouse/clickhouse.config.ts	Reads ClickHouse config from environment.
apps/backend/src/app.module.ts	Imports `ClickhouseModule` into the API app.
apps/backend/package.json	Adds `@clickhouse/client` dependency.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iand · 2026-04-09T14:02:28Z

I will look at the copilot issues on Monday

SgtPooki

a few changes requested, a lot of general comments and responses to copilot things, I want to run the local cluster and see how things fare.

SgtPooki · 2026-04-10T14:37:54Z

ran this locally and it seems to be working fine, but I would like to see the try/catch addressed. I might get to this before you do, we'll see..

I did add #443 to address some errors i saw during startup, but not blocking at all

BigLep

A few comments when looking at the schema. Will leave to engineers to give the approval on the PR.

iand · 2026-04-13T11:28:25Z

I think I have resolved all the straightforward issues.

Added comments to the schema.
Renamed ttfb_ms column to first_byte_ms
Add sp_id columns to record the storage provider id
Made stylistic changes for guarding against undefined variables
Moved clickhouse config into app config for consistency

There were some review comments by copilot that were erroneous, probably due to clickhouse specifics:

Datetime(3) resolution - the 3 indicates millisecond resolution which is what we want
ORDER BY on tables - not needed in any recent version of clickhouse

Some comments that need more discussion/response:

migrate() errors in onModuleInit - I added try/catch to log error then rethrow it. Are you ok with this approach @SgtPooki
add rows back to byTable - I think it's ok as-is @SgtPooki
retry_count - I left this in because it could be useful, but it's marginal
data_retention_challenges schema - made a suggestion for schema change @BigLep

iand · 2026-04-24T09:04:50Z

@SgtPooki I've hopefully addressed your latest comments

SgtPooki

@iand the only thing left is to move forward with your schema updates to get you unblocked (see #438 (comment)), and also the retry_count can likely be dropped (see #438 (comment))

iand · 2026-04-24T14:33:53Z

Thanks @SgtPooki, I removed the retry_count columns as suggested.

Final decision is needed by @BigLep on the data_retention_challenges schema

iand · 2026-04-24T15:03:20Z

All changes from review comments are addressed now

iand · 2026-04-27T08:02:11Z

Is there anything remaining blocking this from being approved and merged @SgtPooki ?

iand · 2026-04-27T10:54:55Z

Actually, one change we could make to the retrieval_checks table is to add a retrieval_type column

retrieval_type            LowCardinality(String),  -- 'deal' | 'sample'

The two values are:

deal - a retrieval triggered from a known deal id
sample - a retrieval triggered from sampling a random piece, in this case deal_id will be null

Not I used sample here rather than anonymous as used in #427 and #459 since I think it better conveys the concept, but we could align on anon if preferred

/cc @BigLep @dennis-tra @SgtPooki

SgtPooki · 2026-04-27T12:59:47Z

Not I used sample here rather than anonymous as used in #427 and #459 since I think it better conveys the concept, but we could align on anon if preferred

I'm good with either sample or anon, we just need to make sure we're using the same language.

* chore: add Copilot PR review instructions Adds .github/copilot-instructions.md to guide GitHub Copilot's PR review behavior toward high-signal feedback and away from CI-duplicate noise. Process: - Reviewed Copilot's review-platform constraints (4000-char base-branch read, Comment-only review, no merge gating, no external link following) plus Google/Microsoft/OWASP/NIST review literature. - Analyzed 319 Copilot inline comments across the last 150 dealbot PRs to identify which areas Copilot reviews well (job-state consistency, test/fixture-contract drift, multi-network behavior, quoted SQL identifiers, redaction) versus where it overreaches (generic-SQL assumptions on ClickHouse code in PRs #438 and #485, low-priority frontend optimization comments). - Iterated through rounds of adversarial review (self-review against the evidence, then a second-opinion review by Codex) to tighten wording, fit the 4000-byte budget, and encode dealbot-specific invariants. Encoded: - Repository context: monorepo layout, Postgres = source of truth, ClickHouse cluster/schema/migrations owned by an external team (dealbot reviews payload correctness and operational impact, not schema/retention/ops design). - Core invariants: at most one job per SP per check type per network; jobs fail only on execution failure, not on negative check results; scheduling/cleanup/filtering/queue execution stay consistent across the same SP set and network. - Blocker/Important priorities aligned to observed high-value comment themes. - Do-Not-Comment list to suppress CI-duplicate noise (Biome, build, typecheck, test, Docker already enforced in CI). Final size: 3890/4000 bytes. * chore: address Copilot feedback on review instructions - Clarify ClickHouse ownership: DDL (clickhouse.schema.ts) and event payloads are owned in-repo and in-scope for review; only cluster ops/retention/infra tuning are externally owned. Earlier wording could have suppressed legitimate schema review (Copilot caught this). - Add Prometheus as a source of truth alongside Postgres, and discourage adding new persisted DB state without need. - Align Comment Format header with the Blocker/Important priority scheme instead of a free-form `Severity:` label. - Drop the generic performance Important bullet; not backed by the PR-comment evidence and frees bytes for the above. Final size: 3950/4000 bytes.

FilOzzy added this to FOC Apr 8, 2026

github-project-automation Bot moved this to 📌 Triage in FOC Apr 8, 2026

iand changed the title ~~Clickhouse data layer~~ feat: write event data to clickhouse Apr 8, 2026

iand force-pushed the clickhouse-data-layer branch 4 times, most recently from ae0768d to 2197aaf Compare April 9, 2026 10:41

iand mentioned this pull request Apr 9, 2026

Provide hook to write high dimensionality data to a custom clickhouse isntance #426

Closed

iand force-pushed the clickhouse-data-layer branch from cd6ce7b to 8777da9 Compare April 9, 2026 11:16

dennis-tra reviewed Apr 9, 2026

View reviewed changes

Comment thread apps/backend/src/clickhouse/clickhouse.config.ts Outdated

Comment thread apps/backend/src/clickhouse/clickhouse.schema.ts Outdated

Comment thread apps/backend/src/clickhouse/clickhouse.service.ts

iand marked this pull request as ready for review April 9, 2026 13:33

Copilot AI review requested due to automatic review settings April 9, 2026 13:33

Copilot started reviewing on behalf of iand April 9, 2026 13:34 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

BigLep assigned iand Apr 9, 2026

BigLep moved this from 📌 Triage to 🔎 Awaiting review in FOC Apr 9, 2026

BigLep requested review from BigLep and SgtPooki April 9, 2026 19:39

SgtPooki requested changes Apr 9, 2026

View reviewed changes

github-project-automation Bot moved this from 🔎 Awaiting review to ⌨️ In Progress in FOC Apr 9, 2026

SgtPooki mentioned this pull request Apr 10, 2026

fix: wait for clickhouse and postgres #443

Merged

BigLep reviewed Apr 12, 2026

View reviewed changes

iand added 5 commits April 20, 2026 10:44

fix: add missing backend-resources-local.yaml

b4a2180

feat: add clickhouse to local kind cluster for development

bcf1978

feat: add initial clickhouse module

2beed2e

feat: perform clickhouse migration on app startup

70c6cb2

feat: wire up writes to cliickhouse

32e649a

iand added 5 commits April 20, 2026 10:44

feat: add sp_id to clickhouse schema

731dfc7

chore: stylistic changes for guarding against undefined variables

497ca04

chore: move clickhouse config into app config for consistency

ff0e6fc

chore: log errors with clickhouse migration in module startup

4871d7f

chore: expand documentation to include clickhouse

0534a17

iand force-pushed the clickhouse-data-layer branch from aae939b to 0534a17 Compare April 20, 2026 09:46

SgtPooki self-requested a review April 21, 2026 19:46

SgtPooki requested changes Apr 21, 2026

View reviewed changes

iand added 3 commits April 24, 2026 09:47

chore: address review comments

addbd2e

merge upstream/main

ed0fc29

chore: fix import order in worker module and add clickhouse mock

00cd978

SgtPooki requested changes Apr 24, 2026

View reviewed changes

chore: drop retry_count from ClickHouse tables

240c799

feat: expand data_retention_challenges schema

19f9658

silent-cipher mentioned this pull request Apr 24, 2026

docs: clarify data retention metrics units and calculation details #476

Merged

SgtPooki requested a review from BigLep April 27, 2026 12:59

SgtPooki approved these changes Apr 27, 2026

View reviewed changes

SgtPooki merged commit 6edbc88 into FilOzone:main Apr 27, 2026
7 checks passed

github-project-automation Bot moved this from ⌨️ In Progress to 🎉 Done in FOC Apr 27, 2026

iand mentioned this pull request Apr 28, 2026

feat: add retrieval_type column to retrieval_checks clickhouse table #485

Closed

FilOzzy mentioned this pull request Apr 28, 2026

chore: release to production (main) #458

Merged

SgtPooki mentioned this pull request Apr 28, 2026

chore: add Copilot PR review instructions #489

Merged

4 tasks

BigLep linked an issue May 1, 2026 that may be closed by this pull request

Provide hook to write high dimensionality data to a custom clickhouse isntance #426

Closed

Conversation

iand commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iand commented Apr 9, 2026

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SgtPooki commented Apr 10, 2026

Uh oh!

BigLep left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iand commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iand commented Apr 24, 2026

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

iand commented Apr 24, 2026

Uh oh!

iand commented Apr 24, 2026

Uh oh!

iand commented Apr 27, 2026

Uh oh!

iand commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SgtPooki commented Apr 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

iand commented Apr 8, 2026 •

edited

Loading

iand commented Apr 13, 2026 •

edited

Loading

iand commented Apr 27, 2026 •

edited

Loading