diff --git a/docs/.vuepress/cli-version.json b/docs/.vuepress/cli-version.json index 7230463..e1f6b59 100644 --- a/docs/.vuepress/cli-version.json +++ b/docs/.vuepress/cli-version.json @@ -1,6 +1,6 @@ { - "supportedCliVersion": "0.12.0", - "supportedCliInstallSpec": "data-product-forge==0.12.0", + "supportedCliVersion": "0.13.0", + "supportedCliInstallSpec": "data-product-forge==0.13.0", "supportedFluidContractVersion": "0.7.5", "quickstartScaffoldVersion": "0.7.2", "supportedSdkVersion": "0.10.0", diff --git a/docs/.vuepress/config.ts b/docs/.vuepress/config.ts index cec0357..3fd303e 100644 --- a/docs/.vuepress/config.ts +++ b/docs/.vuepress/config.ts @@ -222,6 +222,7 @@ export default defineUserConfig({ children: [ '/cli/ai.md', '/cli/agents.md', + '/cli/mission.md', '/cli/mcp.md', '/cli/memory.md', '/cli/stats.md', @@ -446,6 +447,7 @@ export default defineUserConfig({ text: 'Project', children: [ '/contributing.md', + '/RELEASE_NOTES_0.13.0.md', '/RELEASE_NOTES_0.12.0.md', '/RELEASE_NOTES_0.11.0.md', '/RELEASE_NOTES_0.10.0.md', diff --git a/docs/RELEASE_NOTES_0.13.0.md b/docs/RELEASE_NOTES_0.13.0.md new file mode 100644 index 0000000..d538cec --- /dev/null +++ b/docs/RELEASE_NOTES_0.13.0.md @@ -0,0 +1,307 @@ +# Fluid Forge Docs Baseline: CLI `0.13.0` + +**Release Date:** July 18, 2026 +**Status:** Current stable docs baseline (supersedes [`0.12.0`](./RELEASE_NOTES_0.12.0.md)) + +## Headline + +`0.13.0` is the **verifiable-autonomy + declarative-packaging** release. + +The headline is a new top-level command: [**`fluid mission`**](./cli/mission.md). A mission is a +YAML file pairing a plain-language goal with **deterministic success criteria** — and the criteria, +not the model, decide when the work is done. `fluid mission check` runs those criteria with **zero +LLM calls** and drops straight into CI. `fluid mission run` adds the autonomous +VERIFY → PLAN → EXECUTE → GATE → PROGRESS loop, in which the model plans and edits but has no +mechanism to terminate the run. + +Alongside it, **declarative packaging modes** land in the `0.7.6` **preview** schema: a `packaging` +block declares, per infrastructure container, whether this product **owns** it (`isolated`) or +writes into a pre-existing **platform-owned pool** (`shared`). A shared container is emitted as an +OpenTofu *data source* plus leaf-only resources — so a tenant's product structurally cannot destroy +a pool. `0.7.5` remains the stable default; `0.7.6` is opt-in. + +And a correctness fix worth reading if you grant cross-project or cross-account access: +**`accessPolicy` is now the IaC access-grant surface.** The GCP plugin previously read +`metadata.policies`, which **no shipped schema permits** — so the documented cross-project mechanism +was not expressible in a contract that passes `fluid validate`. See +[accessPolicy](#accesspolicy-is-now-the-iac-access-grant-surface) below. + +`pip install --upgrade data-product-forge`. + +::: tip Who should upgrade +Anyone who wants agentic contract work that a CI gate can actually verify (`fluid mission`); +platform teams running **multi-tenant shared infrastructure pools** on AWS / GCP / Snowflake +(packaging modes); anyone granting **cross-project or cross-account** access from a contract (the +`accessPolicy` fix); and anyone using the **semantic layer** — several correctness fixes below +changed the numbers a governed query returns. +::: + +## What changed in `v0.13.0` + +### Added — `fluid mission` (declarative, verifiable goals) + +[`fluid mission`](./cli/mission.md) has four subcommands: + +| Subcommand | What it does | +| --- | --- | +| `fluid mission check [contract]` | Run the spec's success criteria against the on-disk contract and render a scorecard. **Zero LLM calls** — safe as a standalone CI gate. | +| `fluid mission run [contract]` | Run the mission autonomously until the code-owned checks pass or a ceiling fires. `--resume` re-enters an unfinished run. | +| `fluid mission trust ` | One-time, direnv-style approval of a workspace spec — pins its `sha256`. | +| `fluid mission list` | Available specs (built-in / user-global / workspace) with trust status. | + +Exit codes: `0` scorecard green, `1` scorecard red, `2` harness error. + +Two flagship missions ship built in: **`gdpr-clean`** (every PII column classified with provenance, +every MCP-exposed port carrying an `agentPolicy` with retention limits) and **`quality-coverage`** +(every exposed output port carries at least one data-quality rule). + +**Only code-owned checks can declare success.** There is no "finish" action available to the model. +Every cycle re-reads and re-hashes the contract on disk and re-runs the criteria against it. Three +consequences: + +- **Resume is free.** VERIFY is idempotent and reads only from disk, so a paused, stalled, or + crashed run re-enters at VERIFY with no replay machinery — the scorecard is simultaneously the + termination authority and the resume pointer. +- **Self-healing is free.** Failing-check diagnostics are recycled verbatim as the next cycle's + repair feedback. +- **Anti-gaming is enforced.** Every proposed write passes the destructive gate before it lands, so + a model cannot satisfy "every column has a description" by deleting columns. The gate **fails + closed**: on a non-TTY it resolves to deny, and `--yes` never approves a destructive diff. + +Success criteria come in three v1 check types — `validate` (the same in-process validation +`fluid validate` runs), `ai_ready`, and `predicate`. The `predicate` mini-language is **deliberately +frozen**: dotted paths, `[*]` array fan-out, and the operators +`eq / ne / lt / lte / gt / gte / exists / contains`. Nothing more. + +**`.fluid/missions/` is a trust boundary.** A spec configures autonomous execution — tool allowlist, +gate mode, budgets, and the goal text that reaches the planner — so a cloned repo must not silently +control any of it. Built-ins and user-global specs (`~/.fluid/missions/`) are trusted implicitly; +workspace specs and arbitrary paths require `fluid mission trust`, pinned by the file's `sha256` +(a changed file needs re-approval). There is no bypass environment variable. + +Budgets (`max_usd`, `max_iterations`, `max_wall_seconds`) are hard and cumulative — spend is +re-summed from on-disk receipts each cycle, so pause/resume cannot reset it. Overshoot is bounded +but nonzero: one in-flight call can cross the line. + +Receipts land in `/.fluid/missions//` (`manifest.json`, `scorecard.json`, and +per-cycle `scorecard.json` / `cost.json` / `plan.json`), mirroring the existing +`.fluid/agents//` convention. + +### Added — declarative packaging modes (`0.7.6` preview) + +A new `packaging` block in the **`0.7.6` preview** schema declares container ownership: + +```yaml +fluidVersion: "0.7.6" # preview — 0.7.5 remains the stable default +packaging: + mode: shared # isolated | shared + pool: analytics-pool-eu # REQUIRED whenever any container resolves 'shared' + containers: # per-kind overrides win over `mode` + warehouse: isolated # hybrid tier: shared database, own warehouse +``` + +- **`isolated`** emits an owned OpenTofu **resource** — today's exact emit. +- **`shared`** emits a **data source** referencing a pre-existing, platform-owned pool, plus + leaf-only owned resources (tables, prefixed objects, scoped grants). The product writes into the + pool but cannot destroy it. +- Six container kinds: `bucket`, `database`, `dataset`, `schema`, `warehouse`, `cluster`. (`cluster` + accepts `shared` only in v1 — dedicated Confluent cluster provisioning is not yet supported.) +- `binding.packaging` overrides the contract-wide block per exposure. Precedence is + `binding.packaging` > top-level `packaging` > absent. +- **An absent `packaging` block is a distinct LEGACY sentinel**, not a synonym for `isolated`: the + IaC emit is **byte-identical** to pre-packaging releases, golden-pinned. Every existing contract + is unaffected. +- `pool` propagates as the `fluid_pool` label/tag for cost attribution. Optional `poolManifest` + snapshots the platform team's pool file into the bundle so `bundleDigest` covers it. + +**Ownership-transition guard.** Changing a container's mode changes who owns it, but OpenTofu only +sees a resource that left the configuration and plans a **destroy** — on a pool, that reaches every +other tenant's data. So the ownership model is diffed against `tofu state list` **before** +`tofu plan`, and a transition fails closed with copy-pasteable `tofu state rm` remediation: + +- **`isolated` → `shared`** (owned → referenced) is **always blocked**. There is no flag; the only + correct move is to drop the resource from state first, which touches zero bytes of infrastructure. +- **`shared` → `isolated`** (referenced → owned) requires the new + [`fluid apply --adopt-shared-container`](./cli/apply.md#packaging-modes) flag, which + emits a structured `packaging_adoption_override` audit event — the same discipline as + `--allow-data-loss`. Without the gate, brownfield adoption would `tofu import` the platform's pool + into this product's state with `force_destroy` restored. + +Detection is deliberately conservative — over-flagging asks a human to look; under-flagging destroys +a pool. Structured `packaging_transition_blocked` / `packaging_adoption_override` events are emitted +for CI log scrapers. The existing data-loss gate is untouched and remains the unconditional last +line. LEGACY contracts resolve to the sentinel and can never transition, so the guard is a provable +no-op for every pre-existing contract. + +**`plan.json` tells the truth.** Under `packaging.mode: shared` the plan no longer lists a create +action for a container the product does not own; a packaging summary is included in the +digest-bound artifact a human approves. + +See [`fluid generate iac`](./cli/generate-iac.md#packaging-modes) for the emitted +shapes per provider. + +### `accessPolicy` is now the IaC access-grant surface + +**Read this if you grant cross-project (GCP) or cross-account (AWS) access from a contract.** + +The GCP IaC plugin read `contract.metadata.policies` to emit BigQuery `access[]` entries and GCS IAM +members. That surface is **not in any shipped schema** — every version from `0.7.1` to `0.7.6` +declares `metadata` with `additionalProperties: false` and no `policies` property — so a contract +carrying it fails `fluid validate`: + +``` +metadata: Additional properties are not allowed ('policies' was unexpected) +``` + +`fluid generate iac` does not run schema validation, so the emit path worked while the contract was +unusable everywhere else. **Users could not express a cross-project grant in a contract that +validates.** + +The fix reads the schema-valid, already-documented root-level **`accessPolicy`** surface: + +```yaml +accessPolicy: + grants: + - principal: "serviceAccount:consumer@other-project.iam.gserviceaccount.com" + permissions: [read, select, query] + - principal: "group:data-analytics@company.com" + permissions: [read] +``` + +- **`metadata.policies` still emits**, for back-compat with out-of-tree contracts — but it remains + schema-invalid and is **deprecated**. Both surfaces are read (rather than either/or) so a contract + mid-migration does not silently drop half its grants; duplicates collapse. **Migrate to + `accessPolicy`.** +- **A latent bug is fixed for free.** The legacy reader classified a principal by `"@" in principal` + — user if it had an `@`, group otherwise. Group addresses contain `@` too, so **every group was + emitted as a BigQuery `user_by_email` entry**. `accessPolicy` principals carry an explicit + `user:` / `group:` / `serviceAccount:` prefix, so the type is *declared* rather than guessed. + Unprefixed legacy values keep the exact old inference, so existing emitted ACLs do not silently + change — **declare `group:` to get a group entry.** +- **Snowflake is deliberately not unified here.** `security.access_control.grants` is Snowflake-native + RBAC and does not map onto `accessPolicy`'s `{principal, permissions, resources}` without real + loss (masking / row-access policies have no equivalent at all). That surface needs its own schema + decision; it is tracked, not papered over. + +See [GCP provider — access grants](./providers/gcp.md#access-grants-accesspolicy). + +### Semantic layer — correctness and round-tripping + +Several fixes here **change the numbers a governed query returns**. Re-check any dashboards built on +`exposes[].semantics`. + +- **Double aggregation fixed.** Forge-emitted measures copied whole aggregate expressions + (`SUM(amount)`) into `measures[].expr` next to an inferred `agg`, so both consumers aggregated + twice — the MCP query compiler rendered `SUM(SUM(amount))` (invalid SQL on every engine) and the + dbt MetricFlow bridge exported the same double wrap. A shared builder now splits single-aggregate + expressions into `agg` + inner expr (`COUNT(*)` → count over 1; `COUNT(DISTINCT x)` → + `count_distinct`, previously misfiled as plain `count`). +- **`metrics[].filter` is honored on the governed query path.** It was silently ignored by the MCP + query compiler, so `completed_revenue = sum(amount)` filtered to `status = 'completed'` returned + **unfiltered** numbers via the query tool while the dbt export honored the same filter — two + consumers, two answers, one contract. The predicate is now allowlist-validated and applied + parenthesized to the `WHERE`; a filter that fails the allowlist raises `QueryValidationError` + without echoing the raw text. **Fail closed, never wrong.** +- **`agg: percentile` now carries a percentile.** The `0.7.6` preview schema gains + `measures[].aggParams` (`percentile`, `useDiscretePercentile`), mirroring + dbt-semantic-interfaces `agg_params`. The MCP compiler renders + `PERCENTILE_CONT(p) / PERCENTILE_DISC(p) WITHIN GROUP (ORDER BY expr)` and fails closed on + BigQuery / Athena (no grouped ordered-set percentile there); the dbt bridge maps `aggParams` + through. Both share a pinned default of `0.5`. GA `0.7.5` is untouched. +- **Time grains have one source of truth.** The vocabulary lived in three hand-maintained copies + that had measurably drifted (whether `hr` / `mins` were accepted depended on which module saw the + value first). The interview path now normalizes free-form input (`daily` → `day`) instead of + emitting enum-invalid contracts, and omits unrecognized grains rather than passing them through. +- **`defaultAggTimeDimension` is populated** by both producers — the field was write-never while the + dbt bridge read it and silently fell back. +- **Governance round-trips through dbt.** `metrics[].owner` emits as the metric's + `config.meta.owner`; semantics `tags` / `labels` emit as the semantic model's + `config.meta.fluid_tags` / `fluid_labels` (namespaced so they never collide with your own dbt meta + conventions). The manifest importer recovers all three, so the governance surface now survives + contract → dbt → contract round trips. Verified against a real `dbt parse` (dbt-core 1.11.11 + + duckdb). +- **`fluid import dbt` now imports the semantic layer.** Manifest `semantic_models` and `metrics` + (dbt manifest v10+) map into `exposes[].semantics`, closing the round trip: `fluid generate` has + exported a semantics block to MetricFlow YAML since `0.12.0`, and re-importing that project + previously lost it. See [`fluid import`](./cli/import.md). +- **Every template product ships a queryable semantics block.** Template-mode contracts previously + carried no `exposes[].semantics` at all — no MCP query tool (the gateway hides the query + capability without metrics/measures/dimensions), no MetricFlow export, and a validator warning — + until a human authored the block by hand. All five templates (`analytics`, `etl_pipeline`, + `ml_pipeline`, `starter`, `streaming`) now derive a conservative, deterministic block from their + columns. +- **Apache Ossie (OSI) resync.** The OSI port tracked OSI `v0.1.1` before the project moved to + Apache Ossie (incubating). The dialect vocabulary gains `BIGQUERY` and `MAQL`, `vendor_name` is + free-form per the spec, and `ai_context` accepts the spec's plain-string form. The + `*.semantics.osi.yaml` sidecar is now a **spec-conformant interchange document** with the proper + `{version, semantic_model: []}` root wrapper (it was previously a bare model dump that strict + validators rejected, and that nothing could consume). New + `fluid forge data-model --osi-sidecar-format json` emits the shape **dbt Core (v1.12+) reads + natively** — drop it in your dbt project's `OSI/` directory to query the semantic model through + dbt with no conversion step. + +### Security + +- **HIGH — mission run-id path traversal.** `MissionRunStore.run_dir` joined an unvalidated `run_id` + onto the missions root, and `--resume` sourced that id from a **workspace-resident** + `manifest.json` — attacker-controlled content in a cloned repo. `../../../../ESCAPED` resolved + fully outside the workspace, and a manifest could also declare an identity other than the + directory it lived in and win the newest-first sort with a chosen `started_at`. The spec trust + model did not cover this (it pins the *spec* file, not run manifests), and a built-in spec is + trusted with no prompt — so the exact `--resume` command the CLI prints reached the escaped write + with no approval. Fixed at the single chokepoint both entry points funnel through: a run id is now + an opaque **single path segment**, and the directory name is authoritative. +- **Two IAM-widening defects in shared-mode emit** (found and fixed within the packaging lane before + release). The prefix scoping that shared mode exists to provide **failed open** when the contract + omitted `location.path`, so a tenant's grant landed **bucket-wide on a platform-owned pool**: the + `s3:ListBucket` narrowing applied only when a path was present, and `s3:GetObject` fell back to + the whole bucket. Separately, an authoritative `aws_s3_bucket_policy` would have **replaced the + pool's own policy**. Both now fail closed. On GCP, a shared bucket's IAM member conditions escape + contract content so a crafted `path` cannot close the CEL string literal and widen the grant, and + a shared BigQuery dataset drops its authoritative `access[]` block (it would rewrite the pool's + whole ACL and evict other tenants) in favour of per-table + `google_bigquery_table_iam_member`. +- **`fluid import dbt` hardened against Jinja** in recovered free-text fields. + +### Fixed & internal + +- **Import adoption safety.** Every IaC plugin's `discover_imports` is gated on the packaging + resolution, so a shared pool can never be `tofu import`ed into a product's state. Leaf resources + inside a pool stay importable. +- **One canonical shared-bucket name derivation.** A `{{ env.* }}` bucket previously declared the + `data.aws_s3_bucket` lookup under one key and referenced it under another — a dangling reference + that failed `tofu validate`. Both sides now route through one resolver that fails closed + (`PackagingError` kind `shared-bucket-unresolved`) when a pool bucket cannot be resolved to a + concrete name. +- **Backend state key.** A packaging-bearing contract defaults to + `fluid//terraform.tfstate`; contracts with no packaging block keep today's key exactly. +- **Agent-loop transport fixes.** Three defects made tool use non-functional against every shipped + provider (a sentinel URL POSTed over HTTP, litellm's proxy MCP bridge pulling in an absent + `fastapi`, and an Anthropic-vs-OpenAI tool-shape mismatch that produced no error and no tool + calls). All three are fixed; the loop gains additive `tool_allowlist` and `goal_scope` parameters + that default to today's exact behaviour. +- **Verification.** A bilateral cross-account Lake Formation + S3 policy live test on LocalStack, + with a causal control proving the emitted bucket policy is the deciding control; and a + cross-project BigQuery `dataset.access` live test on the BQ emulator, with the emulator's lack of + validation/enforcement pinned executably rather than assumed. + +## Compatibility + +- **No breaking changes to existing contracts.** `0.7.5` remains the stable default for untagged + contracts. `0.7.6` is bundled as **opt-in preview** — a contract must declare + `fluidVersion: "0.7.6"` to use the `packaging` block or `measures[].aggParams`. Preview versions + are never the silent default. +- **IaC emit is byte-identical** for every contract without a `packaging` block. That path is a + distinct LEGACY sentinel, golden-pinned — not a re-derivation. +- **`metadata.policies` is deprecated.** It still emits, but it has never been schema-valid and + fails `fluid validate`. Migrate to root-level `accessPolicy`. If you relied on the old + `"@"`-based principal inference, note that unprefixed values keep the exact legacy behaviour — + declare `group:` explicitly to get a group entry. +- **Semantic-layer numbers may change.** The double-aggregation, metric-filter, and percentile fixes + correct results that were previously wrong (invalid SQL, unfiltered totals, or a hardcoded + median). Re-check anything built on `exposes[].semantics`. +- **`fluid mission run` needs a configured LLM provider.** `fluid mission check` does not — it is + zero-LLM by design and is the half you want in CI. +- **SDK / custom-scaffold:** unchanged (`data-product-forge-sdk 0.10.0`, + `data-product-forge-custom-scaffold 0.4.0`). +- **Install:** `pip install --upgrade data-product-forge` → `0.13.0`. diff --git a/docs/cli/README.md b/docs/cli/README.md index 1368063..633685e 100644 --- a/docs/cli/README.md +++ b/docs/cli/README.md @@ -126,6 +126,7 @@ ODPS and ODCS are **spec exporters**, not cloud providers — they serialize a c | [`fluid diff`](./diff.md) | Detect drift from deployed state | | [`fluid test`](./test.md) | Validate the contract against live resources | | [`fluid verify`](./verify.md) | Verify deployed resources still match the contract | +| [`fluid mission`](./mission.md) | Declarative goals with deterministic success criteria; `mission check` is a zero-LLM CI gate | ## Project & Workspace diff --git a/docs/cli/apply.md b/docs/cli/apply.md index a37dbd6..1bccae5 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -39,6 +39,36 @@ The build-augmented modes (`amend-and-build`, `replace-and-build`) run the confi | `--allow-data-loss` | Required to run `replace` / `replace-and-build` when `FLUID_ENV != dev` **or** the target already has rows. Two independent risk surfaces (env + population) → two-factor opt-in. Never default. | | `--no-verify-plan-binding` | **Emergency escape hatch.** Skip the `bundleDigest` / `planDigest` verification that stage 7 normally enforces on a saved plan. Logged at `WARNING` so audit trails catch it. Use only during documented DR procedures. | | `--no-verify-federation` | **Emergency escape hatch.** Skip the federated-`consumes[]` upstream-digest gate (drift between a pinned `upstreamDigest` and the live upstream). Logged at `WARNING` for audit. A distinct trust domain from plan binding — each gate has its own narrowly-scoped waiver. | +| `--adopt-shared-container` | *(since 0.13.0)* Confirm taking **ownership** of a container this contract previously referenced as a shared pool (`packaging` `shared` → `isolated`). Emits a structured `packaging_adoption_override` audit event; the data-loss gate still applies. See [Packaging modes](#packaging-modes). | + +## Packaging modes + +::: tip Opt-in, new in `0.13.0` +Only relevant to contracts that declare `fluidVersion: "0.7.6"` **and** carry a `packaging` block. A contract without one resolves to the LEGACY sentinel, can never transition, and applies exactly as it did before `0.13.0`. +::: + +A `packaging` block declares whether this product **owns** each infrastructure container (`isolated`) or writes into a pre-existing, platform-owned **pool** (`shared`). Full reference: [`fluid generate iac` — Packaging modes](./generate-iac.md#packaging-modes). + +Changing a container's mode changes *who owns it*, but OpenTofu only sees a resource that left the configuration and plans a **destroy** — on a shared pool that reaches every other tenant's data. So `apply` diffs the resolved ownership model against `tofu state list` **before** `tofu plan`: + +| Transition | Behaviour | +| --- | --- | +| `isolated` → `shared` (owned → referenced) | **Always blocked** — there is no flag. `apply` fails closed and prints copy-pasteable `tofu -chdir= state rm
` commands. State surgery touches zero bytes of infrastructure; re-run `apply` afterwards. | +| `shared` → `isolated` (referenced → owned) | Requires `--adopt-shared-container`. Without the gate, brownfield adoption would `tofu import` the platform's pool into this product's state with `force_destroy` restored — the exact blast radius the feature exists to close. | + +```bash +# Blocked — drop the resource from state first, then re-run. +fluid apply runtime/plan.json --provider aws --yes +# → packaging_transition_blocked: aws_s3_bucket.data (owned → referenced) +# → tofu -chdir=.fluid/iac/aws/ state rm aws_s3_bucket.data + +# Taking ownership of a previously-shared container (audited) +fluid apply runtime/plan.json --provider aws --yes --adopt-shared-container +``` + +The printed `tofu state rm` commands include `-chdir` pointing at the per-contract working directory (`.fluid/iac///`), so they run against the right state without you having to find it. + +Structured `packaging_transition_blocked` / `packaging_adoption_override` events are emitted for CI log scrapers. This guard runs **earlier** than, and is independent of, the data-loss gate — that gate remains the unconditional last line. ## Plan binding diff --git a/docs/cli/generate-iac.md b/docs/cli/generate-iac.md index 1566cce..e7fd2e5 100644 --- a/docs/cli/generate-iac.md +++ b/docs/cli/generate-iac.md @@ -61,6 +61,68 @@ Engine selection is automatic and per-provider (`apply.py::resolve_apply_engine` Catalog metadata that previously lived in the retired `glue` and `snowflake_horizon` publish-side registrars is now emitted into `aws_glue_catalog_table.parameters` and `snowflake_table` column comments directly — one source of truth, drift-detected by `tofu plan`. See [catalog overview](./catalogs/overview.md#retired-registrars-glue-snowflake-horizon). +## Packaging modes + +::: tip Opt-in, new in `0.13.0` +The `packaging` block lives in the **`0.7.6` preview** schema. `0.7.5` remains the stable default — a contract must declare `fluidVersion: "0.7.6"` to use it. **A contract with no `packaging` block emits byte-identically to pre-`0.13.0` releases** (that path is a distinct LEGACY sentinel, golden-pinned — not a re-derivation), so nothing existing changes. +::: + +`packaging` declares, per infrastructure **container**, whether this product **owns** it or writes into a pre-existing, platform-owned **pool**: + +```yaml +fluidVersion: "0.7.6" +packaging: + mode: shared # isolated | shared — blanket default for every kind + pool: analytics-pool-eu # REQUIRED whenever any container resolves 'shared' + poolManifest: platform/pools.yaml # optional; snapshotted into the bundle + containers: # per-kind overrides win over `mode` + warehouse: isolated # hybrid tier: shared database, own warehouse +``` + +| Mode | Emits | +| --- | --- | +| `isolated` | An **owned OpenTofu resource** — this product creates and can destroy the container. Today's exact emit. | +| `shared` | An OpenTofu **data source** referencing the platform-owned pool, plus **leaf-only** owned resources (tables, prefixed objects, scoped grants). The product writes into the pool but structurally cannot destroy it. | + +Six container kinds are accepted: + +| Kind | Resource | +| --- | --- | +| `bucket` | `aws_s3_bucket` / `google_storage_bucket` | +| `database` | `snowflake_database` **and** `aws_glue_catalog_database` | +| `dataset` | `google_bigquery_dataset` | +| `schema` | `snowflake_schema` | +| `warehouse` | `snowflake_warehouse` — `isolated` gives per-product cost attribution | +| `cluster` | Confluent environment/cluster — **`shared` only in v1**; the resolver rejects `isolated` (dedicated-cluster provisioning is not yet supported) | + +`binding.packaging` overrides the contract-wide block per exposure. Precedence is `binding.packaging` > top-level `packaging` > absent-LEGACY. `pool` propagates as the `fluid_pool` label/tag for cost attribution. + +### What `shared` changes per provider + +| Provider | Referenced (shared) behaviour | +| --- | --- | +| `aws` | The bucket becomes `data.aws_s3_bucket` with **no `force_destroy`**. A referenced Glue database is addressed by literal name (`hashicorp/aws` ships `aws_glue_catalog_database` as a resource only — no data source). Lake Formation `registerLocation` scopes to `location.path`, and registers **nothing** when a pooled bucket has no prefix; the bucket policy's `ListBucket` statement gains an `s3:prefix` condition. | +| `gcp` | Dataset and bucket become data sources. A shared dataset **drops its authoritative `access[]` block** (it would rewrite the pool's whole ACL and evict other tenants) in favour of per-table `google_bigquery_table_iam_member`. A shared bucket's IAM members gain an object-prefix CEL condition. | +| `snowflake` | A referenced database/schema emits **neither a resource nor a data source** (Snowflake's data sources are thin), so consumers inline the literal name. An `isolated` warehouse gets a dedicated `snowflake_warehouse`. | + +### Ownership transitions + +Changing a container's mode changes *who owns it* — but OpenTofu only sees a resource that left the configuration and plans a **destroy**. On a shared pool, that reaches every other tenant's data. So the ownership model is diffed against `tofu state list` **before** `tofu plan`, and a transition fails closed with copy-pasteable `tofu state rm` remediation. + +| Transition | Behaviour | +| --- | --- | +| `isolated` → `shared` (owned → referenced) | **Always blocked.** There is no flag. Drop the resource from state first — `tofu state rm` touches zero bytes of infrastructure. | +| `shared` → `isolated` (referenced → owned) | Requires [`fluid apply --adopt-shared-container`](./apply.md#packaging-modes), which emits a structured audit event. | + +Detection is deliberately conservative: over-flagging asks a human to look, under-flagging destroys a pool. LEGACY contracts resolve to the sentinel and can never transition, so the guard is a provable no-op for every pre-existing contract. Structured `packaging_transition_blocked` / `packaging_adoption_override` events are emitted for CI log scrapers. + +**Import adoption is gated too.** Every provider plugin's `discover_imports` hook is gated on the packaging resolution, so a shared pool can never be `tofu import`ed into a product's state. Leaf resources inside a pool stay importable. + +### Plan truthfulness and state key + +- Under `packaging.mode: shared`, `plan.json` **no longer lists a create action** for a container the product does not own, and carries a packaging summary — the digest-bound artifact a human approves now tells the truth. +- A packaging-bearing contract defaults its backend state key to `fluid//terraform.tfstate`. Contracts with no packaging block keep today's key exactly. + ## Operational requirements - **`tofu ≥ 1.6.0` on `PATH`.** `require_tofu_version()` catches the silent `terraform`-on-`PATH`-as-`tofu` mixup at apply time. diff --git a/docs/cli/mission.md b/docs/cli/mission.md new file mode 100644 index 0000000..d13e98b --- /dev/null +++ b/docs/cli/mission.md @@ -0,0 +1,236 @@ +# `fluid mission` + +Declarative, verifiable goals for a data product. A **mission** is a YAML file that pairs a +plain-language `goal` with **deterministic `success_criteria`** — and the criteria, not the model, +decide when the mission is done. + +`fluid mission check` runs those criteria with **zero LLM calls**, so it drops straight into CI. +`fluid mission run` adds the autonomous loop: the model plans and edits, the code-owned checks +re-run against the re-read contract on disk after every cycle, and the run terminates only when +they pass (or when a budget, iteration, or stall ceiling fires). + +::: tip New in `0.13.0` +`fluid mission` ships in `v0.13.0`. See [Release Notes `0.13.0`](../RELEASE_NOTES_0.13.0.md). +::: + +## Syntax + +```bash +fluid mission [options] +``` + +Exit codes are shared across `check` and `run`: + +| Code | Meaning | +| --- | --- | +| `0` | Scorecard green — every non-advisory criterion passed. | +| `1` | Scorecard red — at least one non-advisory criterion failed. | +| `2` | Harness error — bad spec, untrusted spec, unreadable contract, or a failed run. | + +## Subcommands + +### `fluid mission check [contract]` + +Load the spec (trust-gated), re-read the on-disk contract, run every success criterion, and render +a scorecard. **No LLM is involved** — this is a pure gate and safe to run in CI. + +| Argument / Option | Description | +| --- | --- | +| `` | Mission name (e.g. `quality-coverage`) or a path to a spec YAML file. | +| `[contract]` | Contract to verify. Default `contract.fluid.yaml`. | +| `--json` | Emit the scorecard as JSON instead of the rendered table. | + +### `fluid mission run [contract]` + +Run the mission autonomously: **VERIFY → PLAN → EXECUTE → GATE → PROGRESS**, repeating until the +deterministic checks pass or a ceiling fires. Needs a configured LLM provider. + +| Argument / Option | Description | +| --- | --- | +| `` | Mission name or spec YAML path. | +| `[contract]` | Contract to work on. Default `contract.fluid.yaml`. | +| `--resume` | Re-enter the newest unfinished run for this mission. VERIFY is idempotent, so resuming just re-verifies what is on disk — there is no replay. | +| `--run-id ` | Target a specific mission run id. | +| `--llm-provider ` | LLM provider (default: your configured provider). | +| `--llm-model ` | LLM model id. | +| `--workspace ` | Workspace root for receipts and tool confinement (default: auto-detected). | +| `--json` | Emit the outcome as JSON. | + +### `fluid mission trust ` + +One-time, direnv-style approval of a workspace spec: records the file's `sha256` in +`~/.fluid/mission_trust.json`. Editing the file requires re-approval. Built-ins and user-global +specs (`~/.fluid/missions/`) are trusted implicitly — trusting them is a no-op. + +### `fluid mission list` + +List every discoverable mission spec (built-in, user-global, workspace) with its trust status and +description. + +## Examples + +```bash +fluid mission list # what's available, and is it trusted? +fluid mission check quality-coverage # zero-LLM gate against ./contract.fluid.yaml +fluid mission check gdpr-clean ./contract.fluid.yaml --json # machine-readable, for CI +fluid mission trust .fluid/missions/my-mission.yaml # approve a workspace spec (pins its sha256) +fluid mission run gdpr-clean # autonomous loop until the checks pass +fluid mission run gdpr-clean --resume # re-enter a paused run +``` + +## Built-in missions + +Two flagship missions ship with the CLI. + +| Name | Goal | +| --- | --- | +| `gdpr-clean` | Every PII column is classified with provenance, every MCP-exposed port carries an `agentPolicy` with retention limits, and the contract validates. | +| `quality-coverage` | Every exposed output port carries at least one data-quality rule and the contract validates against its declared `fluidVersion` schema. | + +## Spec format + +A mission spec is snake_case YAML. `name`, `description`, `goal`, and `success_criteria` are the +substance; `budgets`, `gates`, `tools`, and `plan_hint` configure the autonomous runner. Unknown +keys are rejected loudly — a typo must never silently weaken a criterion. + +```yaml +name: quality-coverage +description: Reach data-quality rule coverage on every output port. +goal: > + Every exposed output port carries at least one data-quality rule and + the contract validates against its declared fluidVersion schema. +success_criteria: + - check: validate + - check: predicate + path: "exposes[*].contract.dq.rules" + op: exists +budgets: + max_usd: 3.00 + max_iterations: 4 + max_wall_seconds: 1200 +gates: + destructive: ask +tools: + allow: + - discover_workspace + - read_sample_schema + - validate_contract + - propose_contract +plan_hint: + - inspect + - add_dq_rules +``` + +### `success_criteria` + +Every non-advisory criterion must pass. Set `advisory: true` to report a criterion on the scorecard +without gating on it. Three check types ship in v1: + +| `check` | What it does | +| --- | --- | +| `validate` | Runs the same in-process schema validation `fluid validate` runs, with exit-0 semantics. | +| `ai_ready` | Reuses the AI-readiness agent. `require` accepts `sensitive_exposes_annotated` and `missing_descriptions`. | +| `predicate` | Evaluates a dotted path over the contract dict. | + +The `predicate` mini-language is **deliberately frozen**: dotted paths, `[*]` array fan-out, and the +operators `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `exists`, `contains`. No filters, no functions, no +extensibility hooks. Fan-out is fail-closed — a port with no `dq.rules`, an empty list, or no ports +at all fails the criterion. + +```yaml + - check: predicate + path: "exposes[*].policy.agentPolicy.retentionPolicy.maxRetentionDays" + op: lte + value: 30 +``` + +### `budgets` + +| Key | Meaning | +| --- | --- | +| `max_usd` | Hard per-run spend ceiling. Re-summed from on-disk receipts each cycle, so pause/resume cannot reset spend. | +| `max_iterations` | Maximum outer cycles. | +| `max_wall_seconds` | Deadline checked before every step and every check; the remaining time becomes the per-call LLM timeout. | + +Overshoot is bounded but nonzero — one in-flight call can cross the line. + +### `gates` + +`gates.destructive` is `ask` or `deny`. On a non-TTY it resolves to **deny** — the gate fails +closed, and `--yes` never approves a destructive diff. This is what stops a model from satisfying +"every column has a description" by deleting columns. + +### `tools` + +`tools.allow` is an allowlist, intersected with the live tool registry inside the agent loop. The +planner may reorder or drop `plan_hint` steps; it can never add a tool that is not on the list. + +## How the loop terminates + +The load-bearing inversion: **only the code-owned checks may declare success.** There is no "finish" +action available to the model. Every cycle begins by re-reading and re-hashing the contract on disk +and re-running the criteria against it — so: + +- **Resume is free.** VERIFY is idempotent and reads only from disk. A paused, stalled, or crashed + run re-enters at VERIFY with no replay machinery; the scorecard is simultaneously the termination + authority and the resume pointer. +- **Self-healing is free.** Failing-check diagnostics are recycled verbatim as the next cycle's + repair feedback. Verification failure *is* the repair prompt. +- **Every proposed write passes the destructive gate** before it lands. + +A run's `status` stays inside `running | paused | complete | failed`. A `paused` run carries a +`pause_reason` of `stalled`, `budget`, `timeout`, `iterations`, or `gate_rejected`. + +## Trust boundary + +A mission spec configures autonomous execution — tool allowlist, gate mode, budgets, and the goal +text that reaches the planner. A cloned repo shipping `.fluid/missions/` must therefore not silently +control any of it. + +| Source | Trust | +| --- | --- | +| Built-in (shipped with the CLI) | Trusted implicitly. | +| `~/.fluid/missions/` (user-global, outside any repo) | Trusted implicitly. | +| `.fluid/missions/` (workspace) and arbitrary paths | **Requires `fluid mission trust`**, pinned by the file's `sha256`. | + +The gate **fails closed**: an unpinned or changed spec is refused with a +`mission_untrusted_spec_refused` WARNING before anything the spec configures takes effect. A changed +file needs re-approval. There is no bypass environment variable — `fluid mission trust` is the only +way to approve. + +## Receipts + +`fluid mission run` writes to `/.fluid/missions//`, mirroring the +`.fluid/agents//` convention: + +``` +.fluid/missions// + manifest.json run-level state (the resume pointer) + scorecard.json latest VERIFY result + cycles//scorecard.json per-cycle scorecard + cycles//cost.json per-cycle cost receipt + cycles//plan.json the planner's step list +``` + +A run id is an opaque **single path segment**, validated as such — this is a security boundary, not +cosmetics ([`0.13.0` security fixes](../RELEASE_NOTES_0.13.0.md#security)). + +## Using `mission check` in CI + +`check` is the CI-safe half: no LLM, no network, no credentials. + +```yaml +- name: Mission gate + run: | + pip install data-product-forge==0.13.0 + fluid mission check quality-coverage contract.fluid.yaml +``` + +Non-zero exit fails the job. Add `--json` if you want to post the scorecard as a check annotation. + +## See also + +- [`fluid validate`](./validate.md) — the schema-validation stage the `validate` check wraps +- [`fluid agents`](./agents.md) — the sibling `.fluid/agents//` receipt stack +- [`fluid stats`](./stats.md) — aggregate spend across runs +- [Release Notes `0.13.0`](../RELEASE_NOTES_0.13.0.md) diff --git a/docs/providers/aws.md b/docs/providers/aws.md index 92e765f..9cb427b 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -403,6 +403,17 @@ When you run `fluid apply` on an AWS contract, the provider creates: | **Glue Table** | `bitcoin_prices` — External table pointing to S3 | | **Athena Workgroup** | Query engine configured for the region | +### Shared vs. isolated containers + +::: tip New in `0.13.0` (`0.7.6` preview, opt-in) +By default this product **owns** the S3 bucket and Glue database it creates. A +[`packaging` block](../cli/generate-iac.md#packaging-modes) can instead declare them `shared` — a +pre-existing, platform-owned pool that the product writes into but **cannot destroy**. A shared +bucket becomes `data.aws_s3_bucket` with no `force_destroy`; a shared Glue database is addressed by +literal name; Lake Formation `registerLocation` scopes to `location.path` and registers nothing when +a pooled bucket has no prefix. Contracts with no `packaging` block emit exactly as before. +::: + ### What the Pipeline Produces After a successful run, the pipeline writes real data: diff --git a/docs/providers/gcp.md b/docs/providers/gcp.md index 2707da2..9f16266 100644 --- a/docs/providers/gcp.md +++ b/docs/providers/gcp.md @@ -433,6 +433,74 @@ accessPolicy: > roles applied outside the contract. Use `resources` on a grant to scope a > principal to a specific expose. +#### Access grants (`accessPolicy`) + +::: tip Changed in `0.13.0` +`accessPolicy` is now the **IaC access-grant surface** as well as the +`fluid policy-compile` surface. The GCP IaC plugin previously read +`metadata.policies` to emit BigQuery `access[]` entries and GCS IAM members — a +key **no shipped schema permits**, so a contract carrying it fails `fluid validate` +(`metadata: Additional properties are not allowed ('policies' was unexpected)`). +Because `fluid generate iac` does not run schema validation, that emit path worked +while the contract was unusable everywhere else. See +[Release Notes `0.13.0`](../RELEASE_NOTES_0.13.0.md#accesspolicy-is-now-the-iac-access-grant-surface). +::: + +A principal is `:`, and the **type is declared, not guessed**: + +| Prefix | BigQuery dataset `access[]` field | GCS IAM member | +| --- | --- | --- | +| `user:` | `user_by_email` | `user:` | +| `group:` | `group_by_email` | `group:` | +| `serviceAccount:` | `user_by_email` — BigQuery's own convention for SA identities, and what makes a cross-project grant work | `serviceAccount:` | +| `domain:` | `user_by_email` | `domain:` | + +##### Cross-project grants + +This is how you grant a consumer in another GCP project read access to a dataset +this product owns — expressible in a contract that passes `fluid validate`: + +```yaml +accessPolicy: + grants: + - principal: "serviceAccount:consumer@other-project.iam.gserviceaccount.com" + permissions: [read, select, query] + - principal: "group:partner-analytics@other-company.com" + permissions: [read] +``` + +`fluid generate iac --provider gcp` compiles these into the dataset's `access[]` +block (and GCS IAM members where the exposure is object storage). + +##### `metadata.policies` is deprecated + +The legacy `metadata.policies` mapping still emits, so existing out-of-tree +contracts keep working — but it has never been schema-valid and **fails +`fluid validate`**. Migrate to `accessPolicy`. + +Both surfaces are read (rather than either/or), so a contract mid-migration does +not silently drop half its grants; duplicate grants collapse. + +::: warning Groups were previously emitted as users +The legacy reader classified a principal by whether it contained an `@` — user if +yes, group if no. Group addresses contain `@` too, so **every group was emitted as +a BigQuery `user_by_email` entry**. Unprefixed legacy values keep that exact +inference so existing emitted ACLs do not silently change. Declare `group:` +explicitly to get a group entry. +::: + +### Shared vs. isolated containers + +::: tip New in `0.13.0` (`0.7.6` preview, opt-in) +By default this product **owns** the BigQuery dataset and GCS bucket it creates. A +[`packaging` block](../cli/generate-iac.md#packaging-modes) can instead declare them `shared` — a +pre-existing, platform-owned pool the product writes into but **cannot destroy**. A shared dataset +and bucket become OpenTofu data sources; the dataset **drops its authoritative `access[]` block** +(it would rewrite the pool's whole ACL and evict other tenants) in favour of per-table +`google_bigquery_table_iam_member`, and a shared bucket's IAM members gain an object-prefix CEL +condition. Contracts with no `packaging` block emit exactly as before. +::: + --- ## Loading Data