dataraum-testdata

Synthetic operating-model data with an answer key — a corpus whose metrics, structure and injected defects are all known, so anything computed over it can be graded rather than believed.

docs/operating-model.md is the plan: the six performance dimensions, the family framework, the oracle contract, and the order of work.

Architecture

The generator uses an event-driven cascade model where business events produce numerically consistent data across all tables:

• Operating chain: Customer → sales order → order line (units × price) → AR invoice → receipt, gated by each entity's validity window
• Revenue cycle: Order lines → revenue and COGS journal entries → cash receipts → bank transactions
• Stock subledger: Order line issues stock → replenishment receives it on a vendor bill → the bill settles like any other payable
• Expenditure cycle: Purchase invoices → AP journal entries → vendor payments → bank transactions
• Operating events: Monthly payroll, rent, depreciation, insurance, misc expenses — all sized off the scale anchor
• Trial balance: Derived from actual cumulative GL entries (not approximated)

This produces closed-loop accounting — GL entries, invoices, payments, bank transactions, and trial balance are all numerically consistent and traceable back to the originating business event. Because revenue and cost of sale both derive from the order line (units × unit_price, units × standard_cost), contribution margin per customer and per product group is exact, not estimated.

The stock subledger closes the same loop on the asset side: opening + receipts − issues ± adjustments = closing holds per product, location and period, and Σ position value equals the GL inventory balance exactly. That is what makes CCC = DIO + DSO − DPO an answer key rather than a plausible number.

Quick Start

# Generate clean baseline data
testdata generate --scenario month-end-close --strategy clean --output ./output/clean --seed 42

# Generate data with realistic entropy injections
testdata generate --scenario month-end-close --strategy medium --output ./output/medium --seed 42

# List available scenarios
testdata list-scenarios

# Describe a scenario
testdata describe --scenario month-end-close

Strategies

Strategy	Description
clean	No injections — baseline data
low	Subtle issues (2-5% rates)
medium	Realistic problems (~11 injection types)
high	Severe quality issues across all layers

Scale Profiles

How big the firm is, and what shape its populations have. Declared per scenario as generator.scale_profile and stamped into the corpus identity.

Profile	Customers	Products / groups	Suppliers	Orders/yr
tiny	16	9 / 4	20	~2.7K
mid	400	240 / 12	120	~14.8K
large	4,000	1,200 / 24	600	~148K

month-end-close uses mid — the reference company. The other scenarios are about schema shape rather than population size and stay on tiny, which is also the generator's own default so calling it directly in a test suite stays cheap.

Counts alone are not the point. 400 uniform customers make concentration risk exactly as unmeasurable as 16 do, so each profile carries its distributions: revenue per customer Pareto (the top fifth holds 53% of revenue at mid), order value log-normal, a declared fraction of the catalogue priced below the contribution threshold, and entity birth and death — customers.created_date / churned_date and products.launched_date / discontinued_date, which gate the orders and not just the master row.

The profile also sizes the expense base: operating expense is a declared share of the contribution the order lines actually produce, so the bottom line is a property of the firm rather than of a row count. See docs/operating-model.md §9.

Normalization Levels

The normalization setting in the scenario YAML controls table structure:

Level	Tables	Analogue
full	17 (default)	ERP schema export
partial	14	Reporting views — merges three parent-child pairs
flat	13	Analyst spreadsheet — inlines lookup tables
single	1	One mega-table

Counts follow the family registry (default_tables()), so they grow with the corpus.

Set via generator.normalization in config/scenarios/month_end_close.yaml.

Output

Each generation produces:

• CSV files — one per table (varies by normalization level)
• manifest.yaml — corpus identity, file list, row counts
• entropy_map.yaml — defect ground truth: every injection with its target rows, layer, defect class and severity
• ground_truth.yaml — known-correct financial metrics (see Ground Truth)
• metadata_truth.yaml — structural ground truth (see Metadata Truth)

Corpus identity

output/ is gitignored and stays that way. A corpus is not an artifact to preserve; it is a function of its parameters, and every output file carries them as a corpus: block:

corpus:
  id: 524369963d65
  generator: dataraum-testdata
  version: 0.2.0
  scenario: month-end-close
  strategy: clean
  seed: 42
  months: 12
  fiscal_start: '2025-01-01'
  normalization: full
  families: [core_ledger, operating_chain, inventory]
  lever: null

Pin the id, regenerate when you want the bytes, and assert which corpus you graded against. The digest is sha256 over exactly the fields shown, so you can recompute it rather than trust it. Two directories with the same id hold the same data; a different id means something upstream moved — most often a new family, which changes the corpus under an unchanged seed by design. intervention.yaml additionally names counterfactual_corpus_id, so a lever's baseline pair can be verified rather than assumed.

Defect labels are consumer-agnostic

An injection records what was broken, never which detector should catch it: layer (structural / semantic / value / computational), defect (the class, e.g. referential_integrity), defect_detail (the form it took) and injection_type (the injector that produced it). Mapping a defect onto the machinery meant to catch it is the consumer's job. A strategy may set consumer_hint: on an injection to carry its own label through; the generator never reads it.

Metadata Truth

metadata_truth.yaml is the structural ground truth: the FK topology, table and column roles, stock vs flow, metric additivity, cycles and the conformed-dimension matrix. The generator knows every one of these answers from the models and the generator design, so anything that recovers structure from the data can be graded against it, the way entropy_map.yaml grades defect detection. Authored in src/testdata/metadata_truth.py; generated — do not hand-edit.

Section	Keyed by	Content
metric_additivity	metric/measure name	drill additivity verdict (categorical_additive / time_additive + reason) with a determinism tag (function_symmetry = assertable, label_dependent = diagnostic)
stock_flow	table.column	additive (per-period flow) vs point_in_time (stock/level)
reconciles_structurally	table.column	measures that reconcile against a finer event fact
relationships	—	the true FK topology ({from, to} qualified names)
table_roles	—	facts / dimensions / ambiguous table lists
semantic_roles	role	measure / timestamp column lists
business_concepts	table.column	required measure→concept bindings
cycles	—	business cycles the corpus supports (canonical_type, key_tables, required)

Remap-safety. The truth is authored at canonical (full / snake_case) names and rewritten to match each run's exported schema: table names follow the normalization table_mapping (like InjectionRegistry.remap_tables), column names follow the column_style. A cross-table FK that a merge collapses into one table is dropped (no longer discoverable); a genuine self-FK (chart_of_accounts.parent_id) is kept. As with entropy_map.yaml, column renames introduced by normalization merges (e.g. payments.amount → invoice_data.payment_amount) are not reflected — merged columns keep canonical names. Multi-source runs write one canonical top-level file, mirroring the top-level entropy_map.yaml.

Tables

Every corpus carries these; the probe tables (addresses, orders, deliveries, ref_entities, ref_activity, measure_probes, formula_probes) materialize only when a strategy injects into them. Row counts are for a 12-month clean run at seed 42 on the mid profile (month-end-close's default).

Table	~Rows	Description
chart_of_accounts	61	Account hierarchy (5 types)
journal_entries	73K	General ledger entries (event-driven)
journal_lines	158K	Debit/credit lines (balanced per entry)
invoices	16.8K	Vendor bills — category splits expense from goods
payments	14.9K	Invoice payments (paid + partial)
bank_transactions	26.7K	Bank statement (derived from cash events)
fx_rates	472	Weekly exchange rates (8 currency pairs)
trial_balance	336	Per-period movement (a flow)
balance_sheet	112	Carry-forward ending balance (a stock)
customers / products	400 / 240	Master data with validity windows — the Demand and Offer ladders
sales_orders / sales_order_lines	14.8K / 23.6K	The operating chain's event grain
ar_invoices / receipts	14.8K / 11.4K	The AR side — what DSO measures
stock_movements	31.8K	Stock subledger — signed receipts, issues, adjustments
inventory_positions	5.8K	Closing stock at (product, location, period) — a stock

On tiny the same corpus is roughly a quarter the size in orders and an order of magnitude smaller in master data.

Development

uv sync
uv run pytest tests/ -v

Schema Variants

Beyond normalization levels, the library provides additional transforms:

Column naming styles (apply_column_style):

• snake_case — default (identity)
• camelCase — JavaScript/API style
• PascalCase — C#/.NET style
• legacy — abbreviated uppercase (ERP-style: DR_AMT, ACCT_NO, CC)

Key strategies (apply_key_strategy):

• surrogate — default (identity, e.g. JE-0001)
• natural — prefix-based (e.g. JE-00001)
• uuid — deterministic UUIDs (seeded)
• composite — table-prefixed (journal_entries::JE-0001)

Pivots (standalone functions):

• pivot_trial_balance_wide — accounts as rows, periods as columns
• pivot_journal_lines_wide — single amount + side column instead of separate debit/credit

Scenarios

Scenario	Sources	Description
month-end-close	1	12-month fiscal year, 8 tables, standard ERP export
erp-migration	1	6-month migration window, high entropy, partial normalization
multi-system-recon	3	Same events exported through ERP (legacy), banking (PascalCase), AP system (camelCase)

Multi-Source Scenarios

Multi-source scenarios split tables across separate "data sources" with different schema conventions. Each source gets its own subdirectory, manifest, and column naming.

testdata generate --scenario multi-system-recon --strategy clean --output ./output/multi --seed 42

Output:

output/
├── erp_export/          # chart_of_accounts, journal_*, trial_balance (legacy columns)
├── banking_feed/        # bank_transactions, fx_rates (PascalCase columns)
├── ap_system/           # invoices, payments (camelCase columns)
├── sources.yaml         # source index
├── entropy_map.yaml     # injection ground truth
└── ground_truth.yaml    # financial ground truth

Every source carries the same corpus id — they are one set of events exported through different conventions, which is the premise a reconciliation scenario rests on.

Define sources in scenario YAML:

sources:
  erp_export:
    tables: [chart_of_accounts, journal_entries, journal_lines, trial_balance]
    column_style: legacy
    key_strategy: surrogate
    format: csv

Ground Truth

Each scenario run computes ground_truth.yaml. Every metric ships with the definition that produced it, so grading is mechanical: a consumer's alternative matches a named variant instead of reading as a delta to be argued.

metrics:
  - id: dpo
    title: Days payable outstanding
    unit: days
    kind: ratio                # → window: recompute (never average monthly ratios)
    grains: [month, year]
    definition: "ap_balance[end of w] / purchases[w] * days[w]"
    scope: "Closing payable over the window's purchases; 0.0 when purchases are 0."
    values:
      month: {2025-01: 16.3, …}
      year:  {2025: 47.2}
    variants:
      - id: dpo_on_total_expenses
        definition: "ap_balance[end of w] / expenses[w] * days[w]"
        rationale: "what a consumer without a separable purchases figure necessarily computes"
        values: {…}

24 metrics, 5 variants. kind fixes the window rule a consumer needs to read a quarter out of monthly values: a flow/count sums, a stock takes the window's last period, a ratio is recomputed on the window's own aggregates. basis is derived everywhere today — a future figure synthesized without a real basis is marked, not footnoted. The registry is src/testdata/oracle.py; publishing a metric without values, or values without a definition, raises.

Also in the file: invariants — journal balanced, TB balanced, invoice-payment matched, bank reconciliation rate, inventory roll-forward, inventory-to-GL tie — each naming the family that guarantees it and stating reproducibly what must hold, so a consumer can re-check rather than take a bare true on faith. And injection impact (estimated deviations, reported only against a defined metric or a declared integrity surface). Which corpus the numbers are true of is the corpus: stamp's job, so seed, strategy, months and fiscal start appear there once.

Per-entity unit metrics are first-class. db1, db1_pct, units_sold, order_count, revenue and cogs all publish at customer and product_group grain, derived from the order lines — a GL posting cannot be attributed to a customer, which is why the operating chain exists.

Two DPOs, both correct. dpo divides the payable by purchases (vendor-bill credits to AP) — the textbook definition, computable only since goods bills became separable from expense bills. The dpo_on_total_expenses variant carries the total-expense denominator. They are named alternatives, not one right and one wrong answer; cash_conversion_cycle uses the first and publishes a variant on the second. CCC is composed from the published, rounded DIO/DSO/DPO, so recombining them reproduces it exactly.

Gross profit means revenue less cost of sale. It carried the operating-income formula until the contract pinned a definition beside it — which is the failure the contract exists to prevent. operating_income, gross_margin and operating_margin are published separately. Entity-grain revenue sums to the operating_revenue variant, not to revenue: the gap is 43xx other income, which belongs to no customer.

Free cash flow is a cold start. FCF is negative at every profile while operating profit is positive, and the arithmetic is right: the corpus begins with zero receivables and zero stock, so the first fiscal year absorbs a full AR balance and a full inventory position into working capital. A property of a corpus that starts from nothing, not a defect — but worth knowing before benchmarking it against a going concern.

Backlog

The ordered plan is docs/operating-model.md §10 — next up is the rest of the family registry (§3), which has to land before the Supply, Capacity and Throughput families. Independent of that:

• Format profiles (DATEV, SAP, Salesforce, HubSpot) via YAML config + OpenAPI specs