Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/.vuepress/cli-version.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"supportedCliVersion": "0.12.0",
"supportedCliInstallSpec": "data-product-forge==0.12.0",
"supportedCliVersion": "0.13.0",
"supportedCliInstallSpec": "data-product-forge==0.13.0",
"supportedFluidContractVersion": "0.7.5",
"quickstartScaffoldVersion": "0.7.2",
"supportedSdkVersion": "0.10.0",
Expand Down
2 changes: 2 additions & 0 deletions docs/.vuepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,7 @@ export default defineUserConfig({
children: [
'/cli/ai.md',
'/cli/agents.md',
'/cli/mission.md',
'/cli/mcp.md',
'/cli/memory.md',
'/cli/stats.md',
Expand Down Expand Up @@ -446,6 +447,7 @@ export default defineUserConfig({
text: 'Project',
children: [
'/contributing.md',
'/RELEASE_NOTES_0.13.0.md',
'/RELEASE_NOTES_0.12.0.md',
'/RELEASE_NOTES_0.11.0.md',
'/RELEASE_NOTES_0.10.0.md',
Expand Down
307 changes: 307 additions & 0 deletions docs/RELEASE_NOTES_0.13.0.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ ODPS and ODCS are **spec exporters**, not cloud providers — they serialize a c
| [`fluid diff`](./diff.md) | Detect drift from deployed state |
| [`fluid test`](./test.md) | Validate the contract against live resources |
| [`fluid verify`](./verify.md) | Verify deployed resources still match the contract |
| [`fluid mission`](./mission.md) | Declarative goals with deterministic success criteria; `mission check` is a zero-LLM CI gate |

## Project & Workspace

Expand Down
30 changes: 30 additions & 0 deletions docs/cli/apply.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,36 @@ The build-augmented modes (`amend-and-build`, `replace-and-build`) run the confi
| `--allow-data-loss` | Required to run `replace` / `replace-and-build` when `FLUID_ENV != dev` **or** the target already has rows. Two independent risk surfaces (env + population) → two-factor opt-in. Never default. |
| `--no-verify-plan-binding` | **Emergency escape hatch.** Skip the `bundleDigest` / `planDigest` verification that stage 7 normally enforces on a saved plan. Logged at `WARNING` so audit trails catch it. Use only during documented DR procedures. |
| `--no-verify-federation` | **Emergency escape hatch.** Skip the federated-`consumes[]` upstream-digest gate (drift between a pinned `upstreamDigest` and the live upstream). Logged at `WARNING` for audit. A distinct trust domain from plan binding — each gate has its own narrowly-scoped waiver. |
| `--adopt-shared-container` | *(since 0.13.0)* Confirm taking **ownership** of a container this contract previously referenced as a shared pool (`packaging` `shared` → `isolated`). Emits a structured `packaging_adoption_override` audit event; the data-loss gate still applies. See [Packaging modes](#packaging-modes). |

## Packaging modes

::: tip Opt-in, new in `0.13.0`
Only relevant to contracts that declare `fluidVersion: "0.7.6"` **and** carry a `packaging` block. A contract without one resolves to the LEGACY sentinel, can never transition, and applies exactly as it did before `0.13.0`.
:::

A `packaging` block declares whether this product **owns** each infrastructure container (`isolated`) or writes into a pre-existing, platform-owned **pool** (`shared`). Full reference: [`fluid generate iac` — Packaging modes](./generate-iac.md#packaging-modes).

Changing a container's mode changes *who owns it*, but OpenTofu only sees a resource that left the configuration and plans a **destroy** — on a shared pool that reaches every other tenant's data. So `apply` diffs the resolved ownership model against `tofu state list` **before** `tofu plan`:

| Transition | Behaviour |
| --- | --- |
| `isolated` → `shared` (owned → referenced) | **Always blocked** — there is no flag. `apply` fails closed and prints copy-pasteable `tofu -chdir=<workdir> state rm <address>` commands. State surgery touches zero bytes of infrastructure; re-run `apply` afterwards. |
| `shared` → `isolated` (referenced → owned) | Requires `--adopt-shared-container`. Without the gate, brownfield adoption would `tofu import` the platform's pool into this product's state with `force_destroy` restored — the exact blast radius the feature exists to close. |

```bash
# Blocked — drop the resource from state first, then re-run.
fluid apply runtime/plan.json --provider aws --yes
# → packaging_transition_blocked: aws_s3_bucket.data (owned → referenced)
# → tofu -chdir=.fluid/iac/aws/<product-id> state rm aws_s3_bucket.data

# Taking ownership of a previously-shared container (audited)
fluid apply runtime/plan.json --provider aws --yes --adopt-shared-container
```

The printed `tofu state rm` commands include `-chdir` pointing at the per-contract working directory (`.fluid/iac/<provider>/<id>/`), so they run against the right state without you having to find it.

Structured `packaging_transition_blocked` / `packaging_adoption_override` events are emitted for CI log scrapers. This guard runs **earlier** than, and is independent of, the data-loss gate — that gate remains the unconditional last line.

## Plan binding

Expand Down
62 changes: 62 additions & 0 deletions docs/cli/generate-iac.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,68 @@ Engine selection is automatic and per-provider (`apply.py::resolve_apply_engine`

Catalog metadata that previously lived in the retired `glue` and `snowflake_horizon` publish-side registrars is now emitted into `aws_glue_catalog_table.parameters` and `snowflake_table` column comments directly — one source of truth, drift-detected by `tofu plan`. See [catalog overview](./catalogs/overview.md#retired-registrars-glue-snowflake-horizon).

## Packaging modes

::: tip Opt-in, new in `0.13.0`
The `packaging` block lives in the **`0.7.6` preview** schema. `0.7.5` remains the stable default — a contract must declare `fluidVersion: "0.7.6"` to use it. **A contract with no `packaging` block emits byte-identically to pre-`0.13.0` releases** (that path is a distinct LEGACY sentinel, golden-pinned — not a re-derivation), so nothing existing changes.
:::

`packaging` declares, per infrastructure **container**, whether this product **owns** it or writes into a pre-existing, platform-owned **pool**:

```yaml
fluidVersion: "0.7.6"
packaging:
mode: shared # isolated | shared — blanket default for every kind
pool: analytics-pool-eu # REQUIRED whenever any container resolves 'shared'
poolManifest: platform/pools.yaml # optional; snapshotted into the bundle
containers: # per-kind overrides win over `mode`
warehouse: isolated # hybrid tier: shared database, own warehouse
```

| Mode | Emits |
| --- | --- |
| `isolated` | An **owned OpenTofu resource** — this product creates and can destroy the container. Today's exact emit. |
| `shared` | An OpenTofu **data source** referencing the platform-owned pool, plus **leaf-only** owned resources (tables, prefixed objects, scoped grants). The product writes into the pool but structurally cannot destroy it. |

Six container kinds are accepted:

| Kind | Resource |
| --- | --- |
| `bucket` | `aws_s3_bucket` / `google_storage_bucket` |
| `database` | `snowflake_database` **and** `aws_glue_catalog_database` |
| `dataset` | `google_bigquery_dataset` |
| `schema` | `snowflake_schema` |
| `warehouse` | `snowflake_warehouse` — `isolated` gives per-product cost attribution |
| `cluster` | Confluent environment/cluster — **`shared` only in v1**; the resolver rejects `isolated` (dedicated-cluster provisioning is not yet supported) |

`binding.packaging` overrides the contract-wide block per exposure. Precedence is `binding.packaging` > top-level `packaging` > absent-LEGACY. `pool` propagates as the `fluid_pool` label/tag for cost attribution.

### What `shared` changes per provider

| Provider | Referenced (shared) behaviour |
| --- | --- |
| `aws` | The bucket becomes `data.aws_s3_bucket` with **no `force_destroy`**. A referenced Glue database is addressed by literal name (`hashicorp/aws` ships `aws_glue_catalog_database` as a resource only — no data source). Lake Formation `registerLocation` scopes to `location.path`, and registers **nothing** when a pooled bucket has no prefix; the bucket policy's `ListBucket` statement gains an `s3:prefix` condition. |
| `gcp` | Dataset and bucket become data sources. A shared dataset **drops its authoritative `access[]` block** (it would rewrite the pool's whole ACL and evict other tenants) in favour of per-table `google_bigquery_table_iam_member`. A shared bucket's IAM members gain an object-prefix CEL condition. |
| `snowflake` | A referenced database/schema emits **neither a resource nor a data source** (Snowflake's data sources are thin), so consumers inline the literal name. An `isolated` warehouse gets a dedicated `snowflake_warehouse`. |

### Ownership transitions

Changing a container's mode changes *who owns it* — but OpenTofu only sees a resource that left the configuration and plans a **destroy**. On a shared pool, that reaches every other tenant's data. So the ownership model is diffed against `tofu state list` **before** `tofu plan`, and a transition fails closed with copy-pasteable `tofu state rm` remediation.

| Transition | Behaviour |
| --- | --- |
| `isolated` → `shared` (owned → referenced) | **Always blocked.** There is no flag. Drop the resource from state first — `tofu state rm` touches zero bytes of infrastructure. |
| `shared` → `isolated` (referenced → owned) | Requires [`fluid apply --adopt-shared-container`](./apply.md#packaging-modes), which emits a structured audit event. |

Detection is deliberately conservative: over-flagging asks a human to look, under-flagging destroys a pool. LEGACY contracts resolve to the sentinel and can never transition, so the guard is a provable no-op for every pre-existing contract. Structured `packaging_transition_blocked` / `packaging_adoption_override` events are emitted for CI log scrapers.

**Import adoption is gated too.** Every provider plugin's `discover_imports` hook is gated on the packaging resolution, so a shared pool can never be `tofu import`ed into a product's state. Leaf resources inside a pool stay importable.

### Plan truthfulness and state key

- Under `packaging.mode: shared`, `plan.json` **no longer lists a create action** for a container the product does not own, and carries a packaging summary — the digest-bound artifact a human approves now tells the truth.
- A packaging-bearing contract defaults its backend state key to `fluid/<safe-id>/terraform.tfstate`. Contracts with no packaging block keep today's key exactly.

## Operational requirements

- **`tofu ≥ 1.6.0` on `PATH`.** `require_tofu_version()` catches the silent `terraform`-on-`PATH`-as-`tofu` mixup at apply time.
Expand Down
Loading