feat(managed-warehouse): duckling schema prep for streamed persons

[fuziontech]

Problem

The duckling persons backfill (duckling_persons_backfill) is batch: a daily sensor re-exports yesterday's persons from ClickHouse (person FINAL join person_distinct_id2 FINAL), deletes the day in DuckLake, and re-registers fresh parquet. Data is up to a day stale and every top-up re-reads full person state. We want persons on the same streaming path events already use (millpond Kafka to DuckLake, then viaduck CDC fanout), but the batch table's denormalized shape (one row per distinct_id, properties stamped on) can't be produced by either tool, and deletes/merges arrive on the changelog topics as versioned upserts, not CDC deletes.

This PR is the duckling-side prep for that migration. It does not change any existing behavior.

Changes

• PERSONS_STREAMING_TABLE_DDL + PERSONS_DISTINCT_IDS_TABLE_DDL: the two raw current-state tables a viaduck full_cdc pipeline upserts into (keys (team_id, id) and (team_id, distinct_id)). Unlike the batch table, the raw persons table carries is_deleted: tombstones arrive as upserts, never deletes.
• PERSONS_DENORMALIZED_VIEW_DDL: a view over the two raw tables reproducing the batch persons table's exact column shape (EXPECTED_DUCKLAKE_PERSONS_COLUMNS), tombstones filtered. Reader compatibility after cutover.
• persons_distinct_ids_table_name(): the name derivation, rule-for-rule identical to the duckgres control plane's distinctIDsTableName (see linked duckgres PR), so viaduck's discovery-served write target and this view's join target can never drift apart.
• ensure_persons_streaming_schema() wired to a new create_persons_streaming_schema config flag on the persons backfill asset: an operator preps a duckling from Launchpad with a scratch view name for validation, then swaps the view onto the canonical name at cutover. Raw tables take _raw names so the batch table keeps the canonical name until then.
• posthog/dags/PERSONS_STREAMING_MIGRATION.md: the per-team cutover runbook (prep, repoint the control-plane override, final top-up, validate, swap, disable backfill), rollback, and the open decisions (view naming, the viaduck rowid-reuse flush-window issue, update fan-out read cost).

Topology after cutover:

flowchart LR
    K1[clickhouse_person topic]:::phYellow --> M{{millpond}}:::phBlue
    K2[clickhouse_person_distinct_id topic]:::phYellow --> M
    M --> L[(shared changelog DuckLake)]:::phGray
    L --> V{{viaduck full_cdc x2 pipelines}}:::phBlue
    V --> R1[(persons_<suffix>_raw)]:::phGray
    V --> R2[(persons_distinct_ids_<suffix>_raw)]:::phGray
    R1 --> VW[(persons_<suffix> view)]:::phGray
    R2 --> VW
    VW --> Q[SQL editor / HogQL readers]:::phRed
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;

Loading

The full historical backfill stays batch (Kafka retention bounds millpond; viaduck discovery seeds at source head). Streaming replaces only the daily top-up.

Cross-repo companions: duckgres PR serving persons_distinct_ids_table in discovery (PostHog/duckgres#1026), and PostHog/viaduck#67 adding discovery.table_field so a persons pipeline routes to the persons table fields instead of events_table.

How did you test this code?

Agent-run testing, no manual testing claimed:

• New TestPersonsStreamingSchema in posthog/dags/test_events_backfill_to_duckling.py, following the file's existing in-memory duckdb DDL pattern. The regressions each catches: (1) the parameterized derivation cases lock the name rule against drift from the duckgres control plane's derivation (drift means viaduck writes one table while the view joins another); (2) the view test catches a dropped is_deleted filter, a wrong join key, or column drift from the batch shape, by inserting live + tombstoned persons/mappings and asserting the projected rows and column set.
• I could not run the repo pytest suite on this VM (no dev venv set up, disk-constrained), so I validated the DDL and view semantics standalone against duckdb 1.5.5 (the version the duckgres upgrade targets) with the same operations the test performs: table creation, view creation, tombstone filtering, and the column-set match all pass. ruff check and ruff format pass on both changed files. Full pytest runs in CI.

Automatic notifications

• Publish to changelog?

Docs update

Runbook added at posthog/dags/PERSONS_STREAMING_MIGRATION.md; README_DUCKLINGS.md links to it.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Shelley (Claude) on an exe.dev VM, directed by the assignee. Skills invoked: /writing-tests (gated the new tests to the three named regression shapes above) and /writing-user-facing-copy (runbook voice: no em-dashes, sentence case).

Decisions worth knowing: the two-tables-plus-a-view design is forced, since neither millpond (insert-only) nor viaduck (row-level routing, no joins) can emit the batch's denormalized join shape. Raw tables take _raw names pre-cutover specifically so the batch table keeps the canonical name and the cutover is a view swap rather than a reader migration; the alternative (leave the canonical name on the raw table, migrate readers to a new view name) is documented as an open decision in the runbook. The view-name-as-config approach lets operators validate streamed counts against the batch table before any reader-visible change. The deployment side lives in PostHog/charts#13846 (viaduck persons pipeline ApplicationSets, shipped disabled).

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[github-actions]

Hey @fuziontech! 👋

It looks like your git author email on this PR isn't your @posthog.com address (fuziontech@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs