From b4b24cd6d34e9a2f78e89addcef1f4272b2e739d Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Tue, 26 May 2026 21:36:01 -0500 Subject: [PATCH 01/20] docs: rename plan + postgres-first-class design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add three planning docs for the dataform → sqlanvil work: - CLAUDE.md: project guidance for Claude Code sessions - docs/postgres_first_class_design.md: spec for Postgres + Supabase adapters (replaces antigravity assessment's Phase 3+ framing) - docs/rename_checklist.md: exhaustive rename surface map Co-Authored-By: Claude Opus 4.7 --- CLAUDE.md | 104 +++++++++ docs/postgres_first_class_design.md | 342 ++++++++++++++++++++++++++++ docs/rename_checklist.md | 198 ++++++++++++++++ 3 files changed, 644 insertions(+) create mode 100644 CLAUDE.md create mode 100644 docs/postgres_first_class_design.md create mode 100644 docs/rename_checklist.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..ed513357 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,104 @@ +# CLAUDE.md + +Guidance for Claude Code when working in the `sqlanvil/` project. + +## What This Is + +**sqlanvil** is Ivan's fork of [`dataform-co/dataform`](https://github.com/dataform-co/dataform), renamed and being repositioned as an open-source SQL workflow tool that runs against **both BigQuery and PostgreSQL/Supabase** (upstream Dataform OSS dropped Postgres support some time ago). + +- **Upstream**: `git@github.com:dataform-co/dataform.git` (Google's Dataform OSS — low activity since GA in BigQuery) +- **Origin**: `git@github.com:ihistand/sqlanvil.git` +- **Marketing site**: sibling repo `../sqlanvil-com/` — static HTML on Vercel (project: `sqlanvil-com`, team `Zlu36JPJdwPqwMeAWAqISllx`) + +## Stack + +- **Language**: TypeScript +- **Build**: Bazel (via Bazelisk) — old-style `WORKSPACE`, not `MODULE.bazel` +- **Protos**: protobuf (`protos/*.proto`) for core/configs/db_adapter/etc. +- **Target warehouses**: BigQuery (working), PostgreSQL (being reintegrated) +- **No npm `scripts`** in `package.json` — everything runs through Bazel. + +## Layout + +``` +core/ Compiler + action types (table/view/incremental/assertion/operation/notebook/declaration) +cli/ CLI entrypoint (cli/index.ts) and per-adapter glue (cli/api/dbadapters/) +protos/ Protobuf definitions for core/configs/execution/db_adapter +api/ Legacy directory — currently holds restored Postgres adapter files awaiting relocation +tools/ Bazel rules + the Postgres docker test fixture (tools/postgres/postgres_fixture.ts) +tests/ Integration specs (bigquery + postgres) against real warehouses +docs/ Reference + the two new Antigravity design docs (see below) +examples/ Sample Dataform projects +scripts/ `./scripts/run` is the CLI entrypoint wrapper +``` + +## Common Commands + +```bash +# One-time +npm i -g @bazel/bazelisk +sudo sysctl -w kern.maxfiles=65536 # Mac only — Bazel hits the default fd limit + +# Run the CLI (substitute for `dataform` from @dataform/cli) +./scripts/run help +./scripts/run compile path/to/project + +# Tests +bazel test //... # everything +bazel test //core/... # core only +bazel test //cli:index_test # CLI integration (needs GCP creds — see contributing.md) +``` + +## Design Directive — Rename Is Mandatory + +The fork must be fully renamed from `dataform` → `sqlanvil` before public-facing artifacts (npm packages, CLI binary, docs site, marketing) ship. Reason: avoid trademark conflict with Google's "Dataform" product. Scope of rename: + +- Proto package names (`dataform.*` → `sqlanvil.*`) +- npm package names (`@dataform/*` → `@sqlanvil/*` or unscoped `sqlanvil-*`) +- CLI binary (`dataform` → `sqlanvil`) +- Config files (`dataform.json` → `sqlanvil.json`, `workflow_settings.yaml` keys) +- Internal class names referencing `Dataform` +- Docs site references + +The rename is a hard prerequisite, not nice-to-have. Sequence it before — or in parallel with — the Postgres adapter work. + +## Design Directive — Postgres Is First-Class + +The Postgres adapter is **not** a BigQuery adapter with translated SQL. It generates idiomatic Postgres DDL/DML. Typical sqlanvil users may never have touched BigQuery; they should never see BigQuery quirks like `CREATE PRIMARY KEY mykey NOT ENFORCED`, `OPTIONS(...)` table options, `PARTITION BY`/`CLUSTER BY` clauses, or BigQuery's `MERGE` dialect. + +Two adapter variants ship: + +- **`postgres`** — standard Postgres. Idiomatic DDL, `INSERT ... ON CONFLICT` for upserts, native `CREATE INDEX`, tablespaces, fillfactor, partitioning via `PARTITION BY RANGE/LIST/HASH`. +- **`supabase`** — extends `postgres` with Supabase-specific surface area: RLS policies in actions, `auth.users` references, Realtime publications, `pgvector` indexes, `pg_cron` scheduling, Supabase Wrappers (FDW) declarations. + +Implications for the Antigravity reintegration doc: Phase 3 ("Interface Alignment") is **under-scoped**. It frames the work as making the restored adapter implement BigQuery's `IDbAdapter` interface. Real work also includes: + +- Postgres-specific action config blocks (e.g., `postgres: { tablespace, fillfactor, indexes, partition }` parallel to the existing `bigquery: { partitionBy, clusterBy, ... }`). +- A separate Postgres SQL generator path in `core/compilation_sql/` rather than reusing BigQuery's. +- Config schema additions in `protos/configs.proto` for both variants. + +## Active Work — Postgres Reintegration + +Current branch: `restore-postgres-adapter`. Recent commits (Ivan's, on top of upstream): + +1. `a220e2ed` — restored the Postgres adapter files from git history into `api/dbadapters/postgres.ts` and `api/utils/postgres.ts` (won't compile as-is) +2. `fcca60c1` — added `docs/postgres_reintegration_assessment.md` (Antigravity) +3. `1636e275` — added `docs/hybrid_warehouses_supabase_bigquery.md` (Antigravity) + +The two new design docs are Antigravity-authored and **load-bearing for the next sprint**: + +- **`docs/postgres_reintegration_assessment.md`** — 5-phase, ~1-2 day plan to make the restored adapter compile and wire into the CLI. Phases: deps (`pg`, `pg-query-stream`) → relocate `api/` → `cli/api/` → align `IDbAdapter` interface (implement `executeRaw`, `deleteTable`, full `ITableMetadata`) → branch CLI on `projectConfig.warehouse === "postgres"` → Bazel/docker fixture verification. +- **`docs/hybrid_warehouses_supabase_bigquery.md`** — marketing/architecture doc: three patterns for combining Supabase + BigQuery (federated queries, sequential pipeline, Supabase Wrappers / FDW). Reference material, no implementation required. + +When working on Postgres reintegration, **follow the assessment doc's phase order** — deps before relocation before interface work — because each phase's tests depend on the previous one passing through Bazel. + +## Fork Hygiene + +- Upstream changes still merge in cleanly today; the longer Ivan diverges (renames, Postgres adapter, future Supabase-specific features), the harder this gets. When pulling upstream, prefer rebasing feature branches onto `upstream/main` over merge commits to keep the history readable. +- Renaming Dataform → sqlanvil should be done in one sweep (package names, proto packages, CLI binary, docs) rather than incrementally — partial renames create grep ambiguity. + +## Things To Know + +- The `api/` directory at the repo root is **legacy**. The active CLI/adapter layout lives under `cli/api/`. Restored Postgres files are in the legacy location and need to move (see Phase 2 of the assessment doc). +- Integration tests need real warehouses: BigQuery creds in `test_credentials/bigquery.json`, Postgres via Docker container started by `tools/postgres/postgres_fixture.ts` inside the Bazel sandbox. +- The fork still references `@dataform/...` in many places — rename surface area is large. diff --git a/docs/postgres_first_class_design.md b/docs/postgres_first_class_design.md new file mode 100644 index 00000000..1e9055d5 --- /dev/null +++ b/docs/postgres_first_class_design.md @@ -0,0 +1,342 @@ +# Postgres-First-Class Adapter Design + +**Status:** Draft +**Replaces (in scope):** `docs/postgres_reintegration_assessment.md` Phase 3 framing +**Complements:** `docs/hybrid_warehouses_supabase_bigquery.md` (architectural patterns) — this doc is the implementation spec + +## 0. TL;DR + +The restored Postgres adapter must not be a BigQuery adapter with translated SQL. sqlanvil ships two warehouse variants — `postgres` (standard) and `supabase` (Postgres + Supabase platform features) — both generating idiomatic SQL with native action config blocks. A user who has never touched BigQuery must never see BigQuery quirks (`NOT ENFORCED` PKs, `OPTIONS(...)` table options, `PARTITION BY DATE_TRUNC(...)` clauses, `MERGE` dialect). + +## 1. Why Not a BigQuery-Shaped Swap + +The Antigravity assessment frames reintegration as: implement `IDbAdapter` for Postgres, branch on `projectConfig.warehouse`, done. That is necessary but not sufficient. It leaves three structural problems: + +1. **SQL generation is BQ-shaped.** `core/compilation_sql/` was written around BigQuery's dialect (`MERGE`, `CREATE OR REPLACE TABLE ... OPTIONS(...)`, partitioning DSL). A `case warehouse` branch inside each generator produces brittle, half-translated SQL. +2. **Action config blocks are BQ-shaped.** `table.bigquery = { partitionBy, clusterBy, requirePartitionFilter, ... }` exposes BigQuery concepts. Postgres equivalents (`tablespace`, `fillfactor`, native `PARTITION BY RANGE/LIST/HASH`, btree/gin/gist/hnsw indexes) don't map. Forcing them through BQ-shaped fields is leaky. +3. **Supabase isn't just "Postgres on a host."** RLS, Realtime publications, `auth.users` integration, pgvector, pg_cron, Supabase Wrappers — none of these are addressable via a generic Postgres adapter, and all of them are why someone picks Supabase. + +## 2. Adapter Architecture + +``` +PostgresDbAdapter (postgres) + │ + ├── implements IDbAdapter + ├── uses node-postgres (`pg`, `pg-query-stream`) + ├── delegates SQL generation to PostgresSqlGenerator + └── connection: standard libpq DSN / JDBC-style credential + +SupabaseDbAdapter extends PostgresDbAdapter (supabase) + │ + ├── inherits all Postgres behavior + ├── adds: RLS introspection, Realtime publication management, + │ Supabase Wrapper foreign-server discovery + ├── delegates SQL generation to SupabaseSqlGenerator + │ (extends PostgresSqlGenerator) + └── connection: Supabase project credentials + (project_ref + service_role_key OR direct DB url + service_role for RLS bypass) +``` + +Both adapters live under `cli/api/dbadapters/`. SQL generators live under `core/compilation_sql/postgres/` and `core/compilation_sql/supabase/`. + +### 2.1 `IDbAdapter` Surface + +Methods the BigQuery adapter exposes today (`executeRaw`, `tables`, `deleteTable`, `prepareSchema`, `dryRun`, ...) must all be implementable against `pg` without semantic distortion. Where BigQuery returns BQ-specific metadata (`tableType: "VIEW" | "TABLE" | "MATERIALIZED_VIEW" | "EXTERNAL"`), Postgres returns its equivalent set (`tableType: "TABLE" | "VIEW" | "MATERIALIZED_VIEW" | "FOREIGN_TABLE" | "PARTITIONED_TABLE"`). The `ITableMetadata` interface should be a union over warehouse-specific extensions, not a lowest-common-denominator struct. + +## 3. Action Config Schema + +### 3.1 Proto changes (`protos/configs.proto`) + +Add two new message types alongside the existing `BigQueryOptions`: + +```proto +message PostgresOptions { + // Physical storage + string tablespace = 1; + uint32 fillfactor = 2; + bool unlogged = 3; + + // Partitioning (native Postgres declarative partitioning) + message Partition { + enum Kind { RANGE = 0; LIST = 1; HASH = 2; } + Kind kind = 1; + repeated string columns = 2; + } + Partition partition = 4; + + // Indexes + message Index { + string name = 1; + repeated string columns = 2; + enum Method { BTREE = 0; HASH = 1; GIN = 2; GIST = 3; BRIN = 4; } + Method method = 3; + string where = 4; // partial index predicate + bool unique = 5; + repeated string include = 6; // INCLUDE columns + } + repeated Index indexes = 5; + + // Materialized view options + bool with_data = 6; // WITH DATA / WITH NO DATA on initial creation + string refresh_policy = 7; // "manual" | "on_dependency_change" +} + +message SupabaseOptions { + // Standard Postgres options apply + PostgresOptions postgres = 1; + + // Supabase platform + bool publish_to_realtime = 2; // ALTER PUBLICATION supabase_realtime + bool enable_rls = 3; // ALTER TABLE ... ENABLE ROW LEVEL SECURITY + string owner_role = 4; // typically "postgres" or "service_role" + + // pgvector convenience (otherwise expressible via PostgresOptions.indexes) + message VectorConfig { + string column = 1; + uint32 dimensions = 2; + enum IndexType { IVFFLAT = 0; HNSW = 1; } + IndexType index_type = 3; + map params = 4; // ivfflat: lists; hnsw: m, ef_construction + } + repeated VectorConfig vectors = 5; +} +``` + +### 3.2 TypeScript action surface + +```typescript +publish("daily_orders", { + type: "incremental", + uniqueKey: ["order_id"], + postgres: { + partition: { kind: "range", columns: ["order_date"] }, + indexes: [ + { name: "ix_daily_orders_customer", columns: ["customer_id"], method: "btree" }, + { name: "ix_daily_orders_search", columns: ["description"], method: "gin" } + ] + } +}).query(ctx => `SELECT ... FROM ${ctx.ref("raw_orders")} WHERE order_date >= ${ctx.incremental() ? "(SELECT MAX(order_date) FROM ${ctx.self()})" : "'2020-01-01'"}`); +``` + +Compare to existing BigQuery shape: + +```typescript +publish("daily_orders", { + type: "incremental", + bigquery: { partitionBy: "DATE(order_date)", clusterBy: ["customer_id"], requirePartitionFilter: true } +}).query(...); +``` + +Each warehouse owns its own config namespace. Compilation errors if the wrong block is used against the wrong warehouse. + +## 4. SQL Generation Differences + +| Concern | BigQuery (existing) | Postgres (this spec) | +| :--- | :--- | :--- | +| Create table | `CREATE OR REPLACE TABLE \`x.y.z\` OPTIONS(...) AS SELECT ...` | `CREATE TABLE schema.tbl (...); INSERT INTO ...` (atomic via `BEGIN; DROP IF EXISTS; CREATE; INSERT; COMMIT;`) | +| Replace table | Single-statement `CREATE OR REPLACE TABLE` | Transactional drop + create + populate | +| Incremental upsert | `MERGE ... USING ... WHEN MATCHED THEN UPDATE ... WHEN NOT MATCHED THEN INSERT` | `INSERT ... ON CONFLICT (cols) DO UPDATE SET ...` | +| View | `CREATE OR REPLACE VIEW` | `CREATE OR REPLACE VIEW` (works in PG) | +| Materialized view | `CREATE MATERIALIZED VIEW` (auto-refresh) | `CREATE MATERIALIZED VIEW ... WITH [NO] DATA;` + explicit `REFRESH MATERIALIZED VIEW [CONCURRENTLY]` | +| Partitioning | `PARTITION BY DATE(col)`, `PARTITION BY RANGE_BUCKET(col, ...)` | `PARTITION BY RANGE/LIST/HASH (cols)` + `CREATE TABLE part PARTITION OF parent FOR VALUES ...` | +| Clustering | `CLUSTER BY col1, col2` (storage layout) | No direct equivalent. Closest: `CLUSTER table USING index` (one-shot reorder) + appropriate btree index. Not silently translated. | +| Primary key | `CREATE PRIMARY KEY ... NOT ENFORCED` (informational only in BQ) | `PRIMARY KEY (...)` — actually enforced | +| Assertions | `SELECT ... FROM x WHERE failing_condition` | Same shape; can additionally compile to `CHECK` constraints when user opts in | + +**Rule of thumb:** if BigQuery has a concept Postgres lacks (clustering, NOT ENFORCED PKs, `OPTIONS(description = ...)`), the Postgres generator either translates to the nearest meaningful equivalent **and warns in compilation output**, or refuses with a clear error pointing at `postgres:`-namespaced alternatives. + +## 5. New Supabase-Native Action Types + +Add to `core/actions/`: + +### 5.1 `rlsPolicy` + +```typescript +publish("orders_policy", { + type: "rlsPolicy", + table: "orders", + name: "users_see_own_orders", + command: "select", // "all" | "select" | "insert" | "update" | "delete" + roles: ["authenticated"], + using: "user_id = auth.uid()", + withCheck: "user_id = auth.uid()" +}); +``` + +Compiles to: + +```sql +CREATE POLICY users_see_own_orders ON orders + FOR SELECT TO authenticated + USING (user_id = auth.uid()) + WITH CHECK (user_id = auth.uid()); +``` + +Refs into tables work via the standard action graph — declaring an RLS policy creates a dependency edge to the underlying table. + +### 5.2 `realtimePublication` + +```typescript +publish("orders_realtime", { + type: "realtimePublication", + table: "orders", + events: ["insert", "update", "delete"] +}); +``` + +Compiles to `ALTER PUBLICATION supabase_realtime ADD TABLE orders;` (with replica identity adjustments). + +### 5.3 `wrapper` (foreign-server / FDW) + +Pairs with the hybrid-warehouse doc's Pattern C (BigQuery → Supabase): + +```typescript +publish("bq_churn_predictions", { + type: "wrapper", + wrapper: "bigquery", + server: "bq_analytics", + options: { project: "my-bq-project", dataset: "models", table: "churn_predictions" } +}); +``` + +Compiles to `CREATE FOREIGN TABLE ... SERVER bq_analytics OPTIONS (...)`. + +### 5.4 `vectorIndex` + +Convenience wrapper for pgvector — could also be expressed as a `PostgresOptions.Index` with `method: HNSW`, but a dedicated type makes RAG pipelines first-class. + +## 6. Implementation Phases (Replaces Antigravity Phases 3-5) + +Phases 1-2 (deps + relocation) from the Antigravity doc stand as written. The remaining work is re-scoped: + +### Phase 3a — Adapter skeleton (1 day) +- Implement `PostgresDbAdapter` against `pg` and `pg-query-stream`. +- Map `IDbAdapter` methods to Postgres semantics. Where BigQuery returns warehouse-specific metadata, return Postgres equivalents — do not force into BQ shape. +- Credential format: standard `{ host, port, database, user, password, ssl }` (or DSN string). Separate from BigQuery's `{ projectId, credentials, location }`. + +### Phase 3b — Postgres SQL generator (2-3 days) +- New directory: `core/compilation_sql/postgres/`. +- One generator per action type (table, view, incremental, materialized view, operation, assertion, declaration). +- Tests: `core/compilation_sql/postgres/*_test.ts`. Snapshot output against expected idiomatic Postgres SQL. + +### Phase 3c — Action config schema additions (1 day) +- Add `PostgresOptions` and `SupabaseOptions` to `protos/configs.proto`. +- Surface in TypeScript action types alongside existing `bigquery:` namespace. +- Compilation errors for cross-warehouse misuse (`postgres:` block against `warehouse: "bigquery"`, etc.). + +### Phase 4 — CLI wiring (0.5 day) +- Branch `cli/index.ts` adapter instantiation on `projectConfig.warehouse ∈ {bigquery, postgres, supabase}`. +- Credential file format auto-detection. +- `dataform init` (rename target: `sqlanvil init`) templates for both new warehouses. + +### Phase 5 — Supabase variant (2-3 days) +- `SupabaseDbAdapter extends PostgresDbAdapter`. +- Supabase-specific SQL generator extensions. +- New action types: `rlsPolicy`, `realtimePublication`, `wrapper`, `vectorIndex`. +- Connection config supports both direct Postgres URL and Supabase project_ref + service_role_key. + +### Phase 6 — Integration tests (1-2 days) +- Extend `tools/postgres/postgres_fixture.ts` with a parallel `tools/supabase/` fixture (Docker-compose'd Supabase stack — `supabase/postgres` image + Realtime + PostgREST optional). +- Test specs: `tests/integration/postgres.spec.ts` (already restored), new `tests/integration/supabase.spec.ts`. + +**Revised total estimate: 7-10 engineering days.** Antigravity's 1-2 day estimate covers Phase 1-2 + a minimal Phase 3a only. + +## 7. Risks Specific to This Approach + +| Risk | Mitigation | +| :--- | :--- | +| Postgres + Supabase generator divergence over time | Supabase generator inherits from Postgres via class extension; share a fixture suite where behaviors overlap. | +| New action types (`rlsPolicy`, etc.) bloat the core graph | Gate them behind the Supabase variant; standard Postgres users never see them. | +| Connection-string format proliferation | Document one canonical format per variant; provide a `sqlanvil credentials check` subcommand that validates and reports which warehouse it inferred. | +| BigQuery users expecting feature parity (clustering, GA partition pruning) on Postgres | Compilation warnings + docs page mapping BQ concepts → Postgres equivalents, with explicit "no equivalent" markers. | + +## 8. Resolved Decisions + +### 8.1 Rename is mandatory and precedes public release + +The full `dataform` → `sqlanvil` rename happens before any public-facing artifact ships. Trademark risk from Google is the driver. Scope: + +- Proto package names: `dataform.proto.*` → `sqlanvil.proto.*`. Touches every `.proto` file's `package` line and every TS import of generated types. +- npm packages: `@dataform/core`, `@dataform/cli`, etc. → `@sqlanvil/core`, `@sqlanvil/cli`. Republish under new scope; old `@dataform/*` namespace was never Ivan's anyway. +- CLI binary: `dataform` → `sqlanvil`. Update `cli/BUILD`, `scripts/run`, install docs. +- Config files: `dataform.json` → `sqlanvil.json`. `workflow_settings.yaml` keys retained (already neutral). +- Internal class names: `IDataformConfig` → `ISqlanvilConfig`, etc. +- Docs, error messages, telemetry user-agent strings. + +Recommended sequencing: **rename first, in a single sweep PR on the `restore-postgres-adapter` branch**, then layer the adapter work on top. Rationale: a partial rename is worse than either state — grep ambiguity, broken imports, mixed branding. Get it done in one painful day. + +`sqlanvil-com/index.html` should also gain a one-line legal notice acknowledging the fork's origin per Apache 2.0 license terms (Dataform OSS is Apache-2.0; attribution is required, derivative naming is not — but credit upstream cleanly). + +### 8.2 Connection config is nested under `warehouse:` block + +Decision: **nested**. Shape: + +```yaml +# workflow_settings.yaml — BigQuery variant +warehouse: + kind: bigquery + project: my-bq-project + location: US + defaultDataset: analytics + +# workflow_settings.yaml — Postgres variant +warehouse: + kind: postgres + host: db.example.com + port: 5432 + database: analytics + user: sqlanvil_writer + password: ${PG_PASSWORD} # env interpolation + ssl: require + defaultSchema: public + +# workflow_settings.yaml — Supabase variant +warehouse: + kind: supabase + projectRef: abcdefghijklmnop # from supabase dashboard + serviceRoleKey: ${SUPABASE_SERVICE_ROLE_KEY} + defaultSchema: public + # alternative: direct DB URL bypassing PostgREST + # connectionString: postgresql://postgres:${PASSWORD}@db.${PROJECT_REF}.supabase.co:5432/postgres +``` + +Why nested over flat: + +- **Extensibility.** Adding AlloyDB / CockroachDB / Redshift later = add a new `kind` value, not invent new top-level keys. +- **No naming collisions.** Flat `warehouse: postgres` collides semantically with `defaultDatabase: postgres` (kind vs database name). Nested removes the ambiguity. +- **Grouping.** All connection-affecting fields live in one block. Credentials, defaults, dialect flags all co-located. +- **Validation.** Discriminated union on `kind` — strict per-variant field validation, no global field that's only meaningful for some warehouses. + +Proto representation (`protos/configs.proto`): + +```proto +message WarehouseConfig { + oneof connection { + BigQueryConnection bigquery = 1; + PostgresConnection postgres = 2; + SupabaseConnection supabase = 3; + } +} + +message BigQueryConnection { string project = 1; string location = 2; string default_dataset = 3; } +message PostgresConnection { string host = 1; uint32 port = 2; string database = 3; string user = 4; string password = 5; string ssl_mode = 6; string default_schema = 7; } +message SupabaseConnection { string project_ref = 1; string service_role_key = 2; string default_schema = 3; string connection_string = 4; /* optional override */ } +``` + +YAML parser uses the `kind:` tag to discriminate before unmarshalling into the appropriate `oneof` arm. + +### 8.3 No migration path needed from existing Dataform projects + +Acuantia (`~/projects/acuantia-gcp-dataform/`) and other BQ-only Dataform projects stay on Google Cloud / upstream Dataform. They are **not** migration targets. sqlanvil's audience is new personal/OSS projects, especially Supabase-backed ones like **listanvil** (in this monorepo at `../listanvil/`). + +This means: no need for `dataform-compat` translation layer, no need to accept `dataform.json` in addition to `sqlanvil.json`, no need to support `@dataform/...` action config blocks under the rename. Clean break. + +## 9. Recommended Branch Strategy + +Three sequential PRs on top of `restore-postgres-adapter`: + +1. **`rename/dataform-to-sqlanvil`** — pure rename, no behavior change. Mechanical. Reviewable as a diff against upstream. +2. **`adapter/postgres-first-class`** — Phases 1-4 of section 6. New proto messages, Postgres SQL generator, `PostgresDbAdapter`, CLI wiring. No Supabase code yet. +3. **`adapter/supabase-variant`** — Phase 5-6. `SupabaseDbAdapter`, new action types (`rlsPolicy`, `realtimePublication`, `wrapper`, `vectorIndex`), Supabase integration fixture. + +Each PR self-contained, mergeable independently. PR 1 unblocks any future public artifact; PRs 2-3 unblock listanvil-style projects using sqlanvil. diff --git a/docs/rename_checklist.md b/docs/rename_checklist.md new file mode 100644 index 00000000..db0aaaca --- /dev/null +++ b/docs/rename_checklist.md @@ -0,0 +1,198 @@ +# Rename Checklist: `dataform` → `sqlanvil` + +**Status:** Draft / Spec +**Branch target:** `rename/dataform-to-sqlanvil` (first of three PRs per `postgres_first_class_design.md` §9) +**Driver:** Trademark risk from Google's "Dataform" product. + +## Conventions + +- **Package scope:** `@dataform/...` → `@sqlanvil/...` (keep scoped, just swap the scope owner). +- **Bazel workspace name:** `df` → `sa` (matches the existing 2-char convention). Implies tsconfig path `df/*` → `sa/*` and every import `from "df/core/..."` → `from "sa/core/..."`. +- **Proto package:** `dataform` → `sqlanvil`. +- **Proto Java package:** `com.dataform.protos` → `com.sqlanvil.protos` (cosmetic; no Java consumers in this fork). +- **Proto Go package:** `github.com/dataform-co/dataform/protos/dataform` → `github.com/ihistand/sqlanvil/protos/sqlanvil`. +- **Config file:** `dataform.json` is removed entirely (already deprecated upstream; clean break — no `sqlanvil.json` deprecated-fallback hybrid). `workflow_settings.yaml` is the only project config going forward; the key names are already neutral. +- **CLI binary:** `dataform` → `sqlanvil` (binary name and `./scripts/run` references). +- **NOT renamed (legitimate upstream references):** the git remote `upstream` pointing at `github.com/dataform-co/dataform`, attribution lines in LICENSE-equivalent files, historical commit messages. + +## Categorized Checklist + +### A. Proto schema (8 files) + +``` +protos/configs.proto +protos/core.proto +protos/db_adapter.proto +protos/evaluation.proto +protos/execution.proto +protos/extension.proto +protos/jit.proto +protos/profiles.proto +``` + +In each: + +- [ ] `package dataform;` → `package sqlanvil;` +- [ ] `option java_package = "com.dataform.protos";` → `option java_package = "com.sqlanvil.protos";` (where present) +- [ ] `option go_package = "github.com/dataform-co/dataform/protos/dataform";` → `option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil";` (where present) +- [ ] Field-level: `string dataform_core_version` → `string sqlanvil_core_version` (in `configs.proto` line 18 and `core.proto` line 414). **This is a wire-format-breaking change** — fine, since we're publishing fresh. +- [ ] Comment-level: `// The desired dataform core version to compile against.` → `// The desired sqlanvil core version to compile against.` +- [ ] URL comments pointing at `cloud.google.com/dataform/docs/...` — **keep** if they reference legitimate BigQuery/Dataform partitioning docs that are still factually correct, OR replace with sqlanvil's own docs once written. For the rename PR, keep them — annotate them as TODO for the Postgres-first-class doc work. + +### B. Bazel — WORKSPACE / BUILD / .bzl (every BUILD file) + +- [ ] `WORKSPACE` line 1: `workspace(name = "df")` → `workspace(name = "sa")` +- [ ] `protos/BUILD:7,27`: target `dataform_proto` → `sqlanvil_proto` (and the `:dataform_proto` reference in same file) +- [ ] Directory rename: `packages/@dataform/` → `packages/@sqlanvil/` (rename `packages/@dataform/cli` → `packages/@sqlanvil/cli`, same for `core`) +- [ ] Every `//packages/@dataform/core:package_tar` label → `//packages/@sqlanvil/core:package_tar` (15+ occurrences across BUILD files in `core/`, `cli/`, `cli/api/`, `examples/`, and every `tests/**/BUILD`) +- [ ] Every `//packages/@dataform/cli:package_tar` label → `//packages/@sqlanvil/cli:package_tar` (3+ occurrences) +- [ ] `packages/@dataform/core/BUILD:47`: `package_name = "@dataform/core"` → `package_name = "@sqlanvil/core"` +- [ ] `packages/@dataform/cli/BUILD:56`: `package_name = "@dataform/cli"` → `package_name = "@sqlanvil/cli"` +- [ ] `packages/sample-extension/BUILD:46`: `package_name = "@dataform/sample-extension"` → `package_name = "@sqlanvil/sample-extension"` +- [ ] `//packages/@dataform:package.layer.json` → `//packages/@sqlanvil:package.layer.json` (and rename the directory layer file) +- [ ] `BUILD` (root) line 55: `# gazelle:prefix github.com/dataform-co/dataform` → `# gazelle:prefix github.com/ihistand/sqlanvil` +- [ ] `vscode/BUILD:46`: `dataform_logo.png` reference (and the actual file rename — see §G) +- [ ] `test_credentials/BUILD:12-14`: GCP KMS keyring references (`dataform-open-source`, `dataform-builder-key`, `dataform-builder-keyring`) — these point at `dataform-co`'s GCP project which Ivan can't access. **Delete or replace** with his own GCP KMS setup. For the rename PR: delete `test_credentials/BUILD` entirely; integration tests against BQ will need fresh credentials wiring anyway. + +### C. tsconfig + TypeScript imports + +- [ ] `tsconfig.json` `paths`: `"df/*"` → `"sa/*"` (and the three array entries underneath — same key) +- [ ] All TS imports across `core/`, `cli/`, `tests/`, `testing/`, `tools/`, `packages/` of the form `from "df/..."` → `from "sa/..."`. Mechanical sed: + +```bash +find . -name '*.ts' -not -path '*/node_modules/*' -not -path './bazel-*' \ + -exec sed -i '' 's|from "df/|from "sa/|g; s|require("df/|require("sa/|g; s|import("df/|import("sa/|g' {} + +``` + +Estimated occurrences: hundreds. Verify after with `grep -r '"df/' --include='*.ts'` returns nothing outside generated bazel output. + +### D. npm package metadata + +- [ ] `package.json` (root): add `"name": "sqlanvil"` if absent (currently the file has no `name` field — confirms it's a workspace root, fine to leave nameless, but add `"private": true` + repo URL). +- [ ] Subdirectory `package.json` files for the published packages — generated from `tools/gen-package-json/` templates? Confirm whether the templates live there and update them. Otherwise update each: + - `packages/@sqlanvil/cli/package.json` (post-directory-rename) — `"name": "@sqlanvil/cli"`, `"bin": { "sqlanvil": "..." }` + - `packages/@sqlanvil/core/package.json` — `"name": "@sqlanvil/core"` + - `packages/sample-extension/package.json` — `"name": "@sqlanvil/sample-extension"` +- [ ] `tests/integration/bigquery_project/package.json` — drop or update `@dataform/core` dependency to `@sqlanvil/core`. +- [ ] `tests/api/projects/common_v2/package.json` — same. +- [ ] `tests/api/projects/never_finishes_compiling/package.json` — same. +- [ ] `tests/api/projects/invalid_dataform_json/package.json` — see §F (test deletion). +- [ ] `vscode/package.json` — VSCode extension manifest. Rename: `name`, `displayName`, `publisher`, command IDs (`dataform.*` → `sqlanvil.*`), activation events. + +### E. CLI binary + scripts + +- [ ] `scripts/run`: + ```bash + bazel build //packages/@dataform/cli:bin → //packages/@sqlanvil/cli:bin + ./bazel-bin/packages/@dataform/cli/bin.sh "$@" → ./bazel-bin/packages/@sqlanvil/cli/bin.sh "$@" + ``` +- [ ] CLI binary `name` in `packages/@sqlanvil/cli/BUILD` — confirm the binary target name; rename anything `dataform_bin` / `dataform_cli` → `sqlanvil_bin` / `sqlanvil_cli`. +- [ ] `cli/yargswrapper.ts` / `cli/index.ts` `scriptName` calls (yargs program name shown in `--help`) — search for `.scriptName("dataform")` → `.scriptName("sqlanvil")`. +- [ ] Help text strings referencing "Dataform" → "sqlanvil". + +### F. Drop the `dataform.json` legacy path + +The upstream code already comments `dataform.json` as deprecated. Clean-break decision: remove it entirely instead of accepting a `sqlanvil.json` parallel. + +- [ ] `cli/index.ts:59` — delete the `dataformJsonPath` resolution. +- [ ] `cli/vm/compile.ts:83` — delete `global.dataformJson = ...` line. +- [ ] `cli/api/commands/init.ts:26-27` — delete the dataform.json branch from project scaffolding. +- [ ] `core/workflow_settings.ts:15-20` — remove dataform.json fallback; `workflow_settings.yaml` is the only path. +- [ ] `testing/run_core.ts:113` — delete the dataformJson global injection. +- [ ] `core/main_test.ts` — delete tests at lines 545, 570, 606, 636, 803 (dataform.json validation tests). Keep the workflow_settings.yaml tests. +- [ ] `cli/index_compile_test.ts:47` — remove the dataform.json fixture usage from this test. +- [ ] Test fixture directory `tests/api/projects/invalid_dataform_json/` — delete entirely (its purpose was testing the legacy path). +- [ ] `tests/api/BUILD:9-10` — remove the `invalid_dataform_json` references. + +### G. Static assets / branding + +- [ ] `vscode/dataform_logo.png` → `vscode/sqlanvil_logo.png`. Replace the image asset itself before publishing the VSCode extension (Ivan needs to design/source one). For the rename PR, a placeholder is fine. +- [ ] `static/` directory — audit for any other branded assets (icons, banners). +- [ ] `LICENSE` — verify Apache-2.0 attribution is intact. Add a `NOTICE` file (Apache-2.0 §4 requirement when distributing derivative works): "sqlanvil is a derivative of Dataform, originally developed by Dataform Co and contributed to by Google LLC, licensed under Apache License 2.0." Required, not optional. + +### H. Documentation + +- [ ] `readme.md` (root) — currently empty per the `Dataform Core` indexed content (the README still has upstream's content). Rewrite for sqlanvil's positioning. +- [ ] `contributing.md` — update build/test instructions, replace `dataform` CLI references. +- [ ] `docs/configs-reference.md` — references `dataform` semantics. Update for sqlanvil and per the Postgres-first-class spec. +- [ ] `docs/packages.md` — package author guide; update `@dataform/...` patterns to `@sqlanvil/...`. +- [ ] `docs/reference/` — all existing reference docs (likely auto-generated; update generator templates rather than the output). +- [ ] `docs/postgres_reintegration_assessment.md` — mark "SUPERSEDED by postgres_first_class_design.md" at the top. +- [ ] `docs/hybrid_warehouses_supabase_bigquery.md` line 25 — `dataform.json` reference, update. +- [ ] `CLAUDE.md` — already names sqlanvil throughout; no change. + +### I. Test credentials + CI + +- [ ] `cloudbuild-publish.yaml`, `cloudbuild-test.yaml`, `cloudbuild-version.yaml` — these are upstream's Cloud Build configs targeting dataform-co's GCP project. Ivan can't run them. **Delete or replace** with GitHub Actions targeting his own infra. For the rename PR: delete. +- [ ] `test_credentials/bigquery.json` (referenced from `cli/BUILD` per contributing.md) — replace with Ivan's own GCP service account if he wants integration tests against a real BQ project; otherwise delete and remove the dependency from `cli/BUILD`. + +### J. Code-level identifiers + +- [ ] Class / interface / type names containing `Dataform`: + - `IDataformConfig` (if exists) → `ISqlanvilConfig` + - Any `DataformError`, `DataformProject`, etc. → `Sqlanvil...` + - Grep: `grep -rn 'class.*Dataform\|interface.*Dataform\|type.*Dataform\|enum.*Dataform' --include='*.ts'` +- [ ] Variable names: `dataformJson`, `dataformCoreVersion`, etc. Search-and-replace where the meaning is unambiguous. +- [ ] String literals in error messages: `"Dataform compilation failed"` → `"sqlanvil compilation failed"`. Search: `grep -rn '"[^"]*Dataform[^"]*"' --include='*.ts'`. +- [ ] User-agent strings sent to BigQuery / external services — change so server-side logs distinguish sqlanvil from upstream Dataform. + +### K. Repo metadata + +- [ ] `.gitignore` — no dataform references expected, but verify. +- [ ] GitHub repo description (set via `gh repo edit ihistand/sqlanvil --description ...`). +- [ ] GitHub repo topics: drop `dataform`, add `sqlanvil`, `postgres`, `supabase`, `bigquery`, `data-pipeline`. + +## Execution Order Within the Rename PR + +1. **Mechanical first** (low-risk, high-volume): tsconfig + every `from "df/` → `from "sa/"` import, all `//packages/@dataform/` → `//packages/@sqlanvil/` Bazel labels, proto package names. Run `bazel build //...` and `bazel test //...` after each to catch breakage early. +2. **Directory moves**: `packages/@dataform/` → `packages/@sqlanvil/`, `vscode/dataform_logo.png` → `vscode/sqlanvil_logo.png`. Commit as separate logical step. +3. **Removals**: `dataform.json` code path (§F), Cloud Build configs, `test_credentials/BUILD` if not replaced. +4. **Identifiers and strings**: class/interface renames, error messages, help text. +5. **Docs + LICENSE NOTICE**. +6. **Final sweep**: `grep -ri 'dataform' . --include='*' --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=bazel-*` — every remaining hit must be either (a) a legitimate upstream/attribution reference, (b) inside a comment URL pointing at cloud.google.com docs that's still factually valid, or (c) in this rename checklist itself. + +## Verification Commands + +After the rename PR is drafted, run all of these. Each must pass. + +```bash +# 1. Build +bazel build //... + +# 2. Test +bazel test //... + +# 3. CLI smoke test +./scripts/run help +./scripts/run init /tmp/sqlanvil-test +ls /tmp/sqlanvil-test/ # should contain workflow_settings.yaml, NOT dataform.json + +# 4. No stray references +grep -rn 'dataform' . \ + --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=bazel-out \ + --exclude-dir=bazel-bin --exclude-dir=bazel-testlogs --exclude-dir=bazel-sqlanvil \ + | grep -v 'docs/rename_checklist.md' \ + | grep -v 'upstream' \ + | grep -v 'NOTICE' \ + | grep -v 'cloud.google.com/dataform' + +# Expected: empty (or only allow-listed legitimate references) + +# 5. No `df/` import paths remain +grep -rn '"df/' --include='*.ts' --exclude-dir=node_modules --exclude-dir='bazel-*' . +# Expected: empty +``` + +## Out of Scope (Defer to Later PRs) + +- Postgres adapter implementation — that's PR 2 (`adapter/postgres-first-class`). +- Supabase variant — PR 3 (`adapter/supabase-variant`). +- New proto messages (`PostgresOptions`, `SupabaseOptions`, `WarehouseConfig`) — also PR 2. +- VSCode extension actual feature work — only the rename happens here. +- Marketing site (`sqlanvil-com/`) content rewrite — separate concern. + +## Risk Notes + +- **Upstream merges become harder after this PR.** Every cherry-pick from `upstream/main` will conflict on package names, imports, BUILD labels. Mitigation: pull all desired upstream changes first, merge cleanly, then start the rename. +- **The wire-format change to `dataform_core_version` → `sqlanvil_core_version`** means any compiled graph proto file from upstream Dataform won't deserialize. Acceptable since sqlanvil isn't claiming proto compatibility with upstream. +- **Bazel cache invalidates entirely** after directory moves. First post-rename build will be slow (cold cache). From f084b0aeede33640ae66f3c937b7783013db101f Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Tue, 26 May 2026 21:37:30 -0500 Subject: [PATCH 02/20] =?UTF-8?q?refactor:=20rename=20proto=20packages=20d?= =?UTF-8?q?ataform=20=E2=86=92=20sqlanvil=20+=20WORKSPACE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - All 8 .proto files: package dataform → package sqlanvil - Java options: com.dataform.protos → com.sqlanvil.protos - Go options: github.com/dataform-co/dataform → github.com/ihistand/sqlanvil - Field rename (wire-breaking): dataform_core_version → sqlanvil_core_version (in configs.proto + core.proto) - WORKSPACE name: df → sa - Bazel target //protos:dataform_proto → //protos:sqlanvil_proto cloud.google.com/dataform/docs/... URL comments retained — factually correct references to BigQuery partitioning docs. Co-Authored-By: Claude Opus 4.7 --- WORKSPACE | 2 +- protos/BUILD | 4 ++-- protos/configs.proto | 10 +++++----- protos/core.proto | 8 ++++---- protos/db_adapter.proto | 2 +- protos/evaluation.proto | 4 ++-- protos/execution.proto | 4 ++-- protos/extension.proto | 6 +++--- protos/jit.proto | 2 +- protos/profiles.proto | 4 ++-- 10 files changed, 23 insertions(+), 23 deletions(-) diff --git a/WORKSPACE b/WORKSPACE index ee73c05f..077dc063 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -1,4 +1,4 @@ -workspace(name = "df") +workspace(name = "sa") load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") diff --git a/protos/BUILD b/protos/BUILD index 4466c887..d20be132 100644 --- a/protos/BUILD +++ b/protos/BUILD @@ -4,7 +4,7 @@ load("//tools:ts_proto_library.bzl", "ts_proto_library") package(default_visibility = ["//visibility:public"]) proto_library( - name = "dataform_proto", + name = "sqlanvil_proto", srcs = [ "configs.proto", "core.proto", @@ -24,6 +24,6 @@ proto_library( ts_proto_library( name = "ts", deps = [ - ":dataform_proto", + ":sqlanvil_proto", ], ) diff --git a/protos/configs.proto b/protos/configs.proto index ad8668c9..90526b27 100644 --- a/protos/configs.proto +++ b/protos/configs.proto @@ -1,12 +1,12 @@ syntax = "proto3"; -package dataform; +package sqlanvil; -option java_package = "com.dataform.protos"; +option java_package = "com.sqlanvil.protos"; option java_outer_classname = "ConfigsMeta"; option java_multiple_files = true; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; import "google/protobuf/struct.proto"; import "extension.proto"; @@ -14,8 +14,8 @@ import "extension.proto"; // Workflow Settings defines the contents of the `workflow_settings.yaml` // configuration file. message WorkflowSettings { - // The desired dataform core version to compile against. - string dataform_core_version = 1; + // The desired sqlanvil core version to compile against. + string sqlanvil_core_version = 1; // Required. The default Google Cloud project (database). string default_project = 2; diff --git a/protos/core.proto b/protos/core.proto index 66333201..9ea89fec 100644 --- a/protos/core.proto +++ b/protos/core.proto @@ -1,16 +1,16 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "configs.proto"; import "extension.proto"; import "google/protobuf/struct.proto"; -option java_package = "com.dataform.protos"; +option java_package = "com.sqlanvil.protos"; option java_outer_classname = "CoreMeta"; option java_multiple_files = true; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message ProjectConfig { string warehouse = 1; @@ -411,7 +411,7 @@ message CompiledGraph { GraphErrors graph_errors = 7; - string dataform_core_version = 10; + string sqlanvil_core_version = 10; repeated Target targets = 11; diff --git a/protos/db_adapter.proto b/protos/db_adapter.proto index 3760c672..c4cf49fc 100644 --- a/protos/db_adapter.proto +++ b/protos/db_adapter.proto @@ -1,6 +1,6 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "core.proto"; diff --git a/protos/evaluation.proto b/protos/evaluation.proto index 9b7dafe7..6748e34d 100644 --- a/protos/evaluation.proto +++ b/protos/evaluation.proto @@ -1,8 +1,8 @@ syntax="proto3"; -package dataform; +package sqlanvil; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message QueryEvaluationError { string message = 1; diff --git a/protos/execution.proto b/protos/execution.proto index b7578d17..d6e9c4d8 100644 --- a/protos/execution.proto +++ b/protos/execution.proto @@ -1,12 +1,12 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "core.proto"; import "db_adapter.proto"; import "google/protobuf/struct.proto"; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message RunConfig { repeated string actions = 1; diff --git a/protos/extension.proto b/protos/extension.proto index 8284bd95..8e31b550 100644 --- a/protos/extension.proto +++ b/protos/extension.proto @@ -1,12 +1,12 @@ syntax = "proto3"; -package dataform; +package sqlanvil; -option java_package = "com.dataform.protos"; +option java_package = "com.sqlanvil.protos"; option java_outer_classname = "ExtensionMeta"; option java_multiple_files = true; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; // Execution mode of compilation extension. enum ExtensionCompilationMode { diff --git a/protos/jit.proto b/protos/jit.proto index e86e6ba2..ad01ba24 100644 --- a/protos/jit.proto +++ b/protos/jit.proto @@ -1,6 +1,6 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "google/protobuf/empty.proto"; import "google/protobuf/struct.proto"; diff --git a/protos/profiles.proto b/protos/profiles.proto index 09eabcdb..f988dd43 100644 --- a/protos/profiles.proto +++ b/protos/profiles.proto @@ -1,8 +1,8 @@ syntax = "proto3"; -package dataform; +package sqlanvil; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message BigQuery { string project_id = 1; From 2148bb3ecef837236a3a29356afc9f3a7abdb8d5 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Tue, 26 May 2026 21:58:20 -0500 Subject: [PATCH 03/20] =?UTF-8?q?refactor:=20rename=20packages/@dataform?= =?UTF-8?q?=20=E2=86=92=20@sqlanvil=20+=20Bazel=20labels=20+=20delete=20de?= =?UTF-8?q?ad=20infra?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - git mv packages/@dataform → packages/@sqlanvil (3 subdirs: cli, core, package.layer.json; all symbol-pinned package_name fields updated) - Sed @dataform/ → @sqlanvil/ across all BUILD files, .bzl, scripts, .ts/.js/.sqlx test fixtures (40+ files) - Root BUILD: gazelle:prefix → github.com/ihistand/sqlanvil - .github/CODEOWNERS: dataform-co/dataform-reviewers → ihistand - scripts/run: build target → //packages/@sqlanvil/cli:bin - Test fixture emails: allusers@dataform.co → allusers@example.com (consistent across constants.js, .sqlx fixtures, and matching assertions in tests/api/projects.spec.ts) Deletions (dead infra targeting upstream's GCP project): - cloudbuild-publish.yaml, cloudbuild-test.yaml, cloudbuild-version.yaml (Google Cloud Build configs for dataform-co's project) - scripts/publish, scripts/update_test_credentials (publishing to upstream npm + KMS key access against dataform-co's project) - test_credentials/{BUILD,bigquery.json.enc} + all //test_credentials:bigquery.json refs from 4 BUILD files (integration tests against BQ will need fresh credential wiring) NOT renamed (preserved historical/upstream references): - contributing.md upstream issue tracker links - readme.md links to Google Cloud Dataform docs - .bazelrc commented-out remote cache (already disabled) - cli/credentials.ts docs link (TODO: update once sqlanvil docs exist) - All `dataform.*` proto namespace usages in TS — deferred to phase C Co-Authored-By: Claude Opus 4.7 --- .github/CODEOWNERS | 2 +- BUILD | 2 +- cli/BUILD | 5 +- cli/api/BUILD | 5 +- cli/api/commands/compile.ts | 6 +-- cli/api/utils/graphs.ts | 4 +- cli/index_compile_test.ts | 10 ++-- cli/index_project_test.ts | 2 +- cli/index_run_e2e_test.ts | 12 ++--- cli/index_test_base.ts | 2 +- cli/vm/compile.ts | 14 +++--- cli/vm/jit_worker.ts | 8 ++-- cloudbuild-publish.yaml | 35 -------------- cloudbuild-test.yaml | 12 ----- cloudbuild-version.yaml | 23 --------- contributing.md | 4 +- core/BUILD | 2 +- core/index.ts | 10 ++-- core/main.ts | 4 +- core/main_test.ts | 14 +++--- examples/BUILD | 7 ++- examples/examples_test.ts | 6 +-- packages/{@dataform => @sqlanvil}/BUILD | 0 packages/{@dataform => @sqlanvil}/cli/BUILD | 6 +-- .../{@dataform => @sqlanvil}/cli/index.ts | 0 .../cli/package.layer.json | 0 .../{@dataform => @sqlanvil}/cli/worker.ts | 0 packages/{@dataform => @sqlanvil}/core/BUILD | 4 +- .../{@dataform => @sqlanvil}/core/index.ts | 0 .../core/webpack.config.js | 2 +- .../package.layer.json | 0 packages/sample-extension/BUILD | 4 +- readme.md | 2 +- scripts/publish | 44 ------------------ scripts/run | 4 +- scripts/update_test_credentials | 41 ---------------- sqlx/format_test.ts | 2 +- sqlx/lexer_test.ts | 2 +- test_credentials/BUILD | 15 ------ test_credentials/bigquery.json.enc | Bin 2643 -> 0 bytes testing/index.ts | 2 +- testing/run_core.ts | 2 +- tests/api/api.spec.ts | 2 +- tests/api/projects.spec.ts | 6 +-- tests/api/projects/common_v2/BUILD | 2 +- .../definitions/example_backticks.sqlx | 2 +- .../common_v2/definitions/example_table.sqlx | 2 +- .../definitions/example_table_with_tags.sqlx | 4 +- .../projects/common_v2/includes/constants.js | 2 +- .../api/projects/invalid_dataform_json/BUILD | 2 +- .../projects/never_finishes_compiling/BUILD | 2 +- tests/integration/BUILD | 1 - tests/integration/bigquery.spec.ts | 2 +- tests/integration/bigquery_project/BUILD | 2 +- tests/integration/postgres.spec.ts | 2 +- tests/integration/postgres_project/BUILD | 2 +- vscode/README.md | 2 +- vscode/server.ts | 2 +- 58 files changed, 91 insertions(+), 265 deletions(-) delete mode 100644 cloudbuild-publish.yaml delete mode 100644 cloudbuild-test.yaml delete mode 100644 cloudbuild-version.yaml rename packages/{@dataform => @sqlanvil}/BUILD (100%) rename packages/{@dataform => @sqlanvil}/cli/BUILD (93%) rename packages/{@dataform => @sqlanvil}/cli/index.ts (100%) rename packages/{@dataform => @sqlanvil}/cli/package.layer.json (100%) rename packages/{@dataform => @sqlanvil}/cli/worker.ts (100%) rename packages/{@dataform => @sqlanvil}/core/BUILD (96%) rename packages/{@dataform => @sqlanvil}/core/index.ts (100%) rename packages/{@dataform => @sqlanvil}/core/webpack.config.js (87%) rename packages/{@dataform => @sqlanvil}/package.layer.json (100%) delete mode 100755 scripts/publish delete mode 100755 scripts/update_test_credentials delete mode 100644 test_credentials/BUILD delete mode 100644 test_credentials/bigquery.json.enc diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index a16cc846..ef278756 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1 +1 @@ -* @dataform-co/dataform-reviewers +* @ihistand diff --git a/BUILD b/BUILD index b427536a..993a6005 100644 --- a/BUILD +++ b/BUILD @@ -52,7 +52,7 @@ nodejs_binary( load("@bazel_gazelle//:def.bzl", "gazelle") -# gazelle:prefix github.com/dataform-co/dataform +# gazelle:prefix github.com/ihistand/sqlanvil # gazelle:proto package # gazelle:proto_group go_package gazelle(name = "gazelle") diff --git a/cli/BUILD b/cli/BUILD index 2eebdabb..3d61ae96 100644 --- a/cli/BUILD +++ b/cli/BUILD @@ -38,7 +38,7 @@ ts_library( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/cli:package_tar", + "//packages/@sqlanvil/cli:package_tar", ], ) @@ -69,8 +69,7 @@ ts_test_suite( ], data = [ ":node_modules", - "//packages/@dataform/core:package_tar", - "//test_credentials:bigquery.json", + "//packages/@sqlanvil/core:package_tar", "@nodejs//:node", "@nodejs//:npm", ], diff --git a/cli/api/BUILD b/cli/api/BUILD index 111ff2a8..ce1504de 100644 --- a/cli/api/BUILD +++ b/cli/api/BUILD @@ -51,7 +51,7 @@ ts_library( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) @@ -65,8 +65,7 @@ ts_test_suite( ], data = [ ":node_modules", - "//packages/@dataform/core:package_tar", - "//test_credentials:bigquery.json", + "//packages/@sqlanvil/core:package_tar", "@nodejs//:node", "@nodejs//:npm", ] + glob(["goldens/**"]), diff --git a/cli/api/commands/compile.ts b/cli/api/commands/compile.ts index ac863a9a..351fb276 100644 --- a/cli/api/commands/compile.ts +++ b/cli/api/commands/compile.ts @@ -48,7 +48,7 @@ export async function compile( }); if (compileConfig.verbose) { - print(`Using isolated environment for @dataform/core@${workflowSettingsDataformCoreVersion}\n`); + print(`Using isolated environment for @sqlanvil/core@${workflowSettingsDataformCoreVersion}\n`); print(`Copying project to temporary directory: ${temporaryProjectPath}\n`); } const copyStartTime = performance.now(); @@ -64,7 +64,7 @@ export async function compile( path.join(temporaryProjectPath, "package.json"), `{ "dependencies": { - "@dataform/core": "${workflowSettingsDataformCoreVersion}" + "@sqlanvil/core": "${workflowSettingsDataformCoreVersion}" } }` ); @@ -100,7 +100,7 @@ export async function compile( export class CompileChildProcess { public static forkProcess() { - // Runs the worker_bundle script we generate for the package (see packages/@dataform/cli/BUILD) + // Runs the worker_bundle script we generate for the package (see packages/@sqlanvil/cli/BUILD) // if it exists, otherwise run the bazel compile loader target. const findForkScript = () => { try { diff --git a/cli/api/utils/graphs.ts b/cli/api/utils/graphs.ts index 30271be3..f483eee0 100644 --- a/cli/api/utils/graphs.ts +++ b/cli/api/utils/graphs.ts @@ -21,7 +21,7 @@ function combineAllActions(graph: dataform.ICompiledGraph) { export function actionsByTarget(compiledGraph: dataform.ICompiledGraph) { const actionsMap = new Map(); combineAllActions(compiledGraph) - // Required for backwards compatibility with old versions of @dataform/core. + // Required for backwards compatibility with old versions of @sqlanvil/core. .filter(action => !!action.target) .forEach(action => { actionsMap.set(targetStringifier.stringify(action.target), action); @@ -31,7 +31,7 @@ export function actionsByTarget(compiledGraph: dataform.ICompiledGraph) { export function actionsByCanonicalTarget(compiledGraph: dataform.ICompiledGraph) { const actionsMap = new Map(); combineAllActions(compiledGraph) - // Required for backwards compatibility with old versions of @dataform/core. + // Required for backwards compatibility with old versions of @sqlanvil/core. .filter(action => !!action.canonicalTarget) .forEach(action => { actionsMap.set(targetStringifier.stringify(action.canonicalTarget), action); diff --git a/cli/index_compile_test.ts b/cli/index_compile_test.ts index c4966c5a..bf4181b8 100644 --- a/cli/index_compile_test.ts +++ b/cli/index_compile_test.ts @@ -39,7 +39,7 @@ suite("compile command", ({ afterEach }) => { path.join(projectDir, "package.json"), `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -58,9 +58,9 @@ suite("compile command", ({ afterEach }) => { (await getProcessResult(execFile(nodePath, [cliEntryPointPath, "compile", projectDir]))) .stderr ).contains( - "Could not find a recent installed version of @dataform/core in the project. Check that " + + "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + - "`@dataform/core` is specified in `package.json`. If using `package.json`, then run " + + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + "`dataform install`." ); }); @@ -115,7 +115,7 @@ suite("disable-assertions flag (compilation)", ({ afterEach, beforeEach }) => { packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -323,7 +323,7 @@ suite("extension config", ({ afterEach }) => { path.join(projectDir, "package.json"), `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); diff --git a/cli/index_project_test.ts b/cli/index_project_test.ts index 35059d0f..7c11ef76 100644 --- a/cli/index_project_test.ts +++ b/cli/index_project_test.ts @@ -59,7 +59,7 @@ suite("project ops", ({ afterEach }) => { packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); diff --git a/cli/index_run_e2e_test.ts b/cli/index_run_e2e_test.ts index da19b4af..ecf1bdf0 100644 --- a/cli/index_run_e2e_test.ts +++ b/cli/index_run_e2e_test.ts @@ -40,7 +40,7 @@ suite("run e2e", ({ afterEach }) => { packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -213,7 +213,7 @@ select 1 as \${dataform.projectConfig.vars.testVar2} packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -413,7 +413,7 @@ SELECT 1 as id packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -511,7 +511,7 @@ SELECT 1 as id packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -588,7 +588,7 @@ select 1 packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -671,7 +671,7 @@ select 2 packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); diff --git a/cli/index_test_base.ts b/cli/index_test_base.ts index b47f53fe..4c3fe1ce 100644 --- a/cli/index_test_base.ts +++ b/cli/index_test_base.ts @@ -6,4 +6,4 @@ export const DEFAULT_LOCATION = "US"; export const DEFAULT_RESERVATION = "projects/dataform-open-source/locations/us/reservations/dataform-test"; export const CREDENTIALS_PATH = path.resolve(process.env.RUNFILES, "df/test_credentials/bigquery.json"); -export const cliEntryPointPath = "cli/node_modules/@dataform/cli/bundle.js"; +export const cliEntryPointPath = "cli/node_modules/@sqlanvil/cli/bundle.js"; diff --git a/cli/vm/compile.ts b/cli/vm/compile.ts index 7faefb46..0dfc2f55 100644 --- a/cli/vm/compile.ts +++ b/cli/vm/compile.ts @@ -15,9 +15,9 @@ export function compile(compileConfig: dataform.ICompileConfig) { ) ) { throw new Error( - "Could not find a recent installed version of @dataform/core in the project. Check that " + + "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + - "`@dataform/core` is specified in `package.json`. If using `package.json`, then run " + + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + "`dataform install`." ); } @@ -34,7 +34,7 @@ export function compile(compileConfig: dataform.ICompileConfig) { } }); const compiler: CompilerFunction = indexGeneratorVm.run( - 'return require("@dataform/core").compiler', + 'return require("@sqlanvil/core").compiler', vmIndexFileName ); @@ -51,7 +51,7 @@ export function compile(compileConfig: dataform.ICompileConfig) { }, sourceExtensions: ["js", "sql", "sqlx", "yaml", "yml"], // vm2 3.11.3 strips file paths from V8 CallSite objects inside the sandbox, - // which breaks getCallerFile() in @dataform/core. Wrap each compiled module so + // which breaks getCallerFile() in @sqlanvil/core. Wrap each compiled module so // the current file path is exposed via a global, used as a fallback when the // stack-trace path is unavailable. The try/finally restores the previous value // to keep nested requires (macros) consistent. @@ -70,18 +70,18 @@ export function compile(compileConfig: dataform.ICompileConfig) { }); const dataformCoreVersion: string = userCodeVm.run( - 'return require("@dataform/core").version || "0.0.0"', + 'return require("@sqlanvil/core").version || "0.0.0"', vmIndexFileName ); if (semver.lt(dataformCoreVersion, "3.0.0-alpha.0")) { - throw new Error("@dataform/core ^3.0.0 required."); + throw new Error("@sqlanvil/core ^3.0.0 required."); } return userCodeVm.run( ` global.workflowSettingsYaml = (function() { try { return require("./workflow_settings.yaml"); } catch(e) { console.error("YAML require failed:", e); } })(); global.dataformJson = (function() { try { return require("./dataform.json"); } catch(e) {} })(); - return require("@dataform/core").main("${createCoreExecutionRequest(compileConfig)}") + return require("@sqlanvil/core").main("${createCoreExecutionRequest(compileConfig)}") `, vmIndexFileName ); diff --git a/cli/vm/jit_worker.ts b/cli/vm/jit_worker.ts index 5b24439c..3b14c623 100644 --- a/cli/vm/jit_worker.ts +++ b/cli/vm/jit_worker.ts @@ -30,9 +30,9 @@ export async function handleJitRequest(message: { if (!fs.existsSync(path.join(projectDir, "node_modules", "@dataform", "core", "bundle.js"))) { throw new Error( - "Could not find a recent installed version of @dataform/core in the project. Check that " + + "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + - "`@dataform/core` is specified in `package.json`. If using `package.json`, then run " + + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + "`dataform install`." ); } @@ -60,7 +60,7 @@ export async function handleJitRequest(message: { builtin: [], context: "sandbox", external: { - modules: ["@dataform/*"] + modules: ["@sqlanvil/*"] }, root: projectDir }, @@ -68,7 +68,7 @@ export async function handleJitRequest(message: { }); const jitCompileInVm = vm.run(` - const { jitCompiler } = require("@dataform/core"); + const { jitCompiler } = require("@sqlanvil/core"); global.require = require; diff --git a/cloudbuild-publish.yaml b/cloudbuild-publish.yaml deleted file mode 100644 index eaea1f75..00000000 --- a/cloudbuild-publish.yaml +++ /dev/null @@ -1,35 +0,0 @@ -steps: - - name: 'gcr.io/cloud-builders/gcloud' - script: | - #!/usr/bin/env bash - gcloud secrets versions access latest --secret=github-token-access --format='get(payload.data)' | tr '_-' '/+' | base64 -d > ~/token.txt - REPO_TOKEN="$(gcloud auth print-access-token)" NPM_TOKEN=$NPM_TOKEN ./scripts/create_npmrc - secretEnv: ['NPM_TOKEN'] - - name: gcr.io/cloud-builders/bazel:5.4.0 - script: | - #!/usr/bin/env bash - ./scripts/publish - - name: 'gcr.io/$PROJECT_ID/github' - entrypoint: 'bash' - args: - - '-c' - - | - set -e - version=$(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g") - echo "Creating release notes for $version" - gh auth login --with-token < ~/token.txt - gh release create $version --generate-notes -availableSecrets: - secretManager: - - versionName: projects/178487900909/secrets/npm-publish-token/versions/latest - env: 'NPM_TOKEN' -artifacts: - npmPackages: - - repository: 'https://us-central1-npm.pkg.dev/dataform-open-source/dataform-open-source' - packagePath: './bazel-bin/packages/@dataform/cli/package' - - repository: 'https://us-central1-npm.pkg.dev/dataform-open-source/dataform-open-source' - packagePath: './bazel-bin/packages/@dataform/core/package' -options: - machineType: E2_HIGHCPU_8 - requestedVerifyOption: VERIFIED -timeout: 3600s diff --git a/cloudbuild-test.yaml b/cloudbuild-test.yaml deleted file mode 100644 index 09207ec0..00000000 --- a/cloudbuild-test.yaml +++ /dev/null @@ -1,12 +0,0 @@ -steps: - - name: 'gcr.io/cloud-builders/gcloud' - script: | - #!/usr/bin/env bash - REPO_TOKEN="$(gcloud auth print-access-token)" NPM_TOKEN="test_only" ./scripts/create_npmrc - - name: gcr.io/cloud-builders/bazel:5.4.0 - script: | - #!/usr/bin/env bash - ./scripts/run_tests_on_cloudbuild -options: - machineType: E2_HIGHCPU_8 -timeout: 3600s diff --git a/cloudbuild-version.yaml b/cloudbuild-version.yaml deleted file mode 100644 index 2bbbf86a..00000000 --- a/cloudbuild-version.yaml +++ /dev/null @@ -1,23 +0,0 @@ -steps: -# We have github personal access token stored in `github-token-access` secret in our GCP project. -# This step downloads it and store it in token.txt file for later steps to use for authentication. -- name: gcr.io/cloud-builders/gcloud - entrypoint: 'bash' - args: [ '-c', "gcloud secrets versions access latest --secret=github-token-access --format='get(payload.data)' | tr '_-' '/+' | base64 -d > token.txt" ] -- name: gcr.io/cloud-builders/git - entrypoint: 'bash' - args: - - '-c' - - | - _GITHUB_USER=$_GITHUB_USER _GITHUB_EMAIL=$_GITHUB_EMAIL ./scripts/create_gh_pr -- name: 'gcr.io/$PROJECT_ID/github' - entrypoint: 'bash' - args: - - '-c' - - | - set -e - echo "Create PR..." - gh auth login --with-token < token.txt - gh pr create -t "Publishing Dataform security patches" -b "Updating NPM package version to $(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g")" -B $BRANCH_NAME -H $(cat git_branch_name.txt) -options: - automapSubstitutions: true diff --git a/contributing.md b/contributing.md index 36831d06..575e32a1 100644 --- a/contributing.md +++ b/contributing.md @@ -20,7 +20,7 @@ npm i -g @bazel/bazelisk ### Run the CLI -You can run the project as you would the `npm` installation of `@dataform/cli`, but replace `dataform` with `./scripts/run`. +You can run the project as you would the `npm` installation of `@sqlanvil/cli`, but replace `dataform` with `./scripts/run`. For example, to print out the default help information: @@ -34,7 +34,7 @@ _Note: If you are running Bazel on a **Mac**, this or any step that requires bui ### Test -The following command runs tests for @dataform/core: +The following command runs tests for @sqlanvil/core: ```bash bazel test //core/... diff --git a/core/BUILD b/core/BUILD index db9219e8..fbde4ff2 100644 --- a/core/BUILD +++ b/core/BUILD @@ -106,7 +106,7 @@ ts_test_suite( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", "//packages/sample-extension:package_tar", ], ) diff --git a/core/index.ts b/core/index.ts index fcade976..7fcdc802 100644 --- a/core/index.ts +++ b/core/index.ts @@ -8,7 +8,7 @@ import { dataform } from "df/protos/ts"; // Create static session object. // This hack just enforces the singleton session object to -// be the same, regardless of the @dataform/core package that is running. +// be the same, regardless of the @sqlanvil/core package that is running. function globalSession() { if (!(global as any)._DF_SESSION) { (global as any)._DF_SESSION = new Session(); @@ -24,10 +24,10 @@ const supportedFeatures = [dataform.SupportedFeatures.ARRAY_BUFFER_IPC]; // Note: for future backwards compatability breaking changes, the exported "version" variable should // be used instead. function indexFileGenerator() { - throw new Error("@dataform/cli ^3.0.0 required."); + throw new Error("@sqlanvil/cli ^3.0.0 required."); } -// These exports constitute the public API of @dataform/core. -// They must also be listed in packages/@dataform/core/index.ts. -// Changes to these will break @dataform/cli, so take care! +// These exports constitute the public API of @sqlanvil/core. +// They must also be listed in packages/@sqlanvil/core/index.ts. +// Changes to these will break @sqlanvil/cli, so take care! export { compiler, IDataformExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version }; diff --git a/core/main.ts b/core/main.ts index b4ead165..309c76ea 100644 --- a/core/main.ts +++ b/core/main.ts @@ -52,7 +52,7 @@ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | st }); // Initialize the compilation session. - const session = nativeRequire("@dataform/core").session as Session; + const session = nativeRequire("@sqlanvil/core").session as Session; session.init(compileRequest.compileConfig.projectDir, projectConfig, projectConfig); // Allow "includes" files to use the current session object. @@ -224,7 +224,7 @@ function dataformCompile(compileRequest: dataform.ICompileExecutionRequest, sess }); Object.assign(globalAny, topLevelIncludes); - // Bind various @dataform/core APIs to the 'global' object. + // Bind various @sqlanvil/core APIs to the 'global' object. globalAny.publish = session.publish.bind(session); globalAny.operate = session.operate.bind(session); globalAny.assert = session.assert.bind(session); diff --git a/core/main_test.ts b/core/main_test.ts index 1b53e0e9..59cb28c0 100644 --- a/core/main_test.ts +++ b/core/main_test.ts @@ -48,7 +48,7 @@ function toVerifiableAction(graph: dataform.ICompiledGraph, actionType: string): // INFO: if you want to see an overview of the tests in this file, press cmd-k-3 while in // VSCode, to collapse everything below the third level of indentation. -suite("@dataform/core", ({ afterEach }) => { +suite("@sqlanvil/core", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); suite("session", () => { @@ -1939,7 +1939,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = setUpProjectWithExtension(); const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", + name: "@sqlanvil/sample-extension", compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, }; @@ -1953,7 +1953,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = setUpProjectWithExtension(); const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", + name: "@sqlanvil/sample-extension", compilationMode: dataform.ExtensionCompilationMode.APPLICATION_CODE, }; @@ -1971,7 +1971,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", + name: "@sqlanvil/sample-extension", compilationMode: dataform.ExtensionCompilationMode.APPLICATION_CODE }; @@ -1991,7 +1991,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", + name: "@sqlanvil/sample-extension", compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, }; @@ -2020,7 +2020,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = setUpProjectWithExtension(); const request = coreExecutionRequestFromPath(projectDir, dataform.ProjectConfig.create({vars: {"throw-error": "true"}})); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", + name: "@sqlanvil/sample-extension", compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, }; @@ -2035,7 +2035,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = setUpProjectWithExtension(); const request = coreExecutionRequestFromPath(projectDir, dataform.ProjectConfig.create({vars: {"store-compile-error": "true"}})); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", + name: "@sqlanvil/sample-extension", compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, }; diff --git a/examples/BUILD b/examples/BUILD index 95e33682..b0c63fab 100644 --- a/examples/BUILD +++ b/examples/BUILD @@ -22,8 +22,7 @@ ts_test_suite( data = [ ":all_projects", ":node_modules", - "//packages/@dataform/core:package_tar", - "//test_credentials:bigquery.json", + "//packages/@sqlanvil/core:package_tar", "@nodejs//:node", ], deps = [ @@ -44,7 +43,7 @@ node_modules( deps = [ # The CLI bundle remains at this directory level's node_modules, whereas the core package # is copied into each example project's node_modules during testing. - "//packages/@dataform/cli:package_tar", - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/cli:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/examples/examples_test.ts b/examples/examples_test.ts index 5465e8f8..de51c405 100644 --- a/examples/examples_test.ts +++ b/examples/examples_test.ts @@ -7,7 +7,7 @@ import { dataform } from "df/protos/ts"; import { getProcessResult, nodePath, suite, test } from "df/testing"; suite("examples", { parallel: true }, () => { - const cliEntryPointPath = "examples/node_modules/@dataform/cli/bundle.js"; + const cliEntryPointPath = "examples/node_modules/@sqlanvil/cli/bundle.js"; ["stackoverflow_reporter", "extreme_weather_programming"].forEach(exampleProject => { test(`${exampleProject} runs`, async () => { @@ -19,8 +19,8 @@ suite("examples", { parallel: true }, () => { const projectDir = `examples/${exampleProject}_copy`; fs.copySync(originalProjectDir, projectDir, { dereference: true }); fs.copySync( - "examples/node_modules/@dataform/core", - `${projectDir}/node_modules/@dataform/core` + "examples/node_modules/@sqlanvil/core", + `${projectDir}/node_modules/@sqlanvil/core` ); // A blank `package.json` makes no `dataformCoreVersion` in `workflow_settings.yaml` be OK. // tslint:disable-next-line: tsr-detect-non-literal-fs-filename diff --git a/packages/@dataform/BUILD b/packages/@sqlanvil/BUILD similarity index 100% rename from packages/@dataform/BUILD rename to packages/@sqlanvil/BUILD diff --git a/packages/@dataform/cli/BUILD b/packages/@sqlanvil/cli/BUILD similarity index 93% rename from packages/@dataform/cli/BUILD rename to packages/@sqlanvil/cli/BUILD index 7a1d7085..016a4335 100644 --- a/packages/@dataform/cli/BUILD +++ b/packages/@sqlanvil/cli/BUILD @@ -53,13 +53,13 @@ externals = [ pkg_json( name = "json", - package_name = "@dataform/cli", + package_name = "@sqlanvil/cli", description = "Dataform command line interface.", external_deps = externals, layers = [ "//:package.json", - "//packages/@dataform:package.layer.json", - "//packages/@dataform/cli:package.layer.json", + "//packages/@sqlanvil:package.layer.json", + "//packages/@sqlanvil/cli:package.layer.json", ], main = "bundle.js", version = DF_VERSION, diff --git a/packages/@dataform/cli/index.ts b/packages/@sqlanvil/cli/index.ts similarity index 100% rename from packages/@dataform/cli/index.ts rename to packages/@sqlanvil/cli/index.ts diff --git a/packages/@dataform/cli/package.layer.json b/packages/@sqlanvil/cli/package.layer.json similarity index 100% rename from packages/@dataform/cli/package.layer.json rename to packages/@sqlanvil/cli/package.layer.json diff --git a/packages/@dataform/cli/worker.ts b/packages/@sqlanvil/cli/worker.ts similarity index 100% rename from packages/@dataform/cli/worker.ts rename to packages/@sqlanvil/cli/worker.ts diff --git a/packages/@dataform/core/BUILD b/packages/@sqlanvil/core/BUILD similarity index 96% rename from packages/@dataform/core/BUILD rename to packages/@sqlanvil/core/BUILD index 15edc71c..a60a5e59 100644 --- a/packages/@dataform/core/BUILD +++ b/packages/@sqlanvil/core/BUILD @@ -44,12 +44,12 @@ npm_package_bin( pkg_json( name = "json", - package_name = "@dataform/core", + package_name = "@sqlanvil/core", description = "Dataform core API.", external_deps = [], layers = [ "//:package.json", - "//packages/@dataform:package.layer.json", + "//packages/@sqlanvil:package.layer.json", ], main = "bundle.js", version = DF_VERSION, diff --git a/packages/@dataform/core/index.ts b/packages/@sqlanvil/core/index.ts similarity index 100% rename from packages/@dataform/core/index.ts rename to packages/@sqlanvil/core/index.ts diff --git a/packages/@dataform/core/webpack.config.js b/packages/@sqlanvil/core/webpack.config.js similarity index 87% rename from packages/@dataform/core/webpack.config.js rename to packages/@sqlanvil/core/webpack.config.js index 2e389a04..423a9b47 100644 --- a/packages/@dataform/core/webpack.config.js +++ b/packages/@sqlanvil/core/webpack.config.js @@ -5,7 +5,7 @@ module.exports = (env, argv) => { const config = { mode: argv.mode || "development", target: 'node', - entry: [path.resolve(process.env.RUNFILES, "df/packages/@dataform/core/index")], + entry: [path.resolve(process.env.RUNFILES, "df/packages/@sqlanvil/core/index")], output: { libraryTarget: "commonjs-module", }, diff --git a/packages/@dataform/package.layer.json b/packages/@sqlanvil/package.layer.json similarity index 100% rename from packages/@dataform/package.layer.json rename to packages/@sqlanvil/package.layer.json diff --git a/packages/sample-extension/BUILD b/packages/sample-extension/BUILD index b99f7f1b..e5b6ce30 100644 --- a/packages/sample-extension/BUILD +++ b/packages/sample-extension/BUILD @@ -43,12 +43,12 @@ npm_package_bin( pkg_json( name = "json", - package_name = "@dataform/sample-extension", + package_name = "@sqlanvil/sample-extension", description = "Sample extension.", external_deps = [], layers = [ "//:package.json", - "//packages/@dataform:package.layer.json", + "//packages/@sqlanvil:package.layer.json", ], main = "bundle.js", version = DF_VERSION, diff --git a/readme.md b/readme.md index 9c23579c..cc39d087 100644 --- a/readme.md +++ b/readme.md @@ -24,7 +24,7 @@ Follow the [quickstart guide](https://cloud.google.com/dataform/docs/quickstart) You can run Dataform locally using the Dataform CLI tool, which can be installed using the following command line. Follow the [CLI guide](https://cloud.google.com/dataform/docs/use-dataform-cli) to get started. ``` -npm i -g @dataform/cli +npm i -g @sqlanvil/cli ``` ## Useful Links diff --git a/scripts/publish b/scripts/publish deleted file mode 100755 index 8c585960..00000000 --- a/scripts/publish +++ /dev/null @@ -1,44 +0,0 @@ -#!/bin/bash -set -e - -# This can be uncommented once it no longer throws an error exit code. -# ./scripts/regenerate_docs - -# By default GCB uses master branch name. This code change it to main. -if [ "master" == "$(git branch --show-current)" ]; then - git branch -m main -fi - -if [ "$(git status --porcelain)" ]; then - echo "There are uncommitted changes; aborting." 1>&2 - exit 1 -fi - -# Run all the tests. -bazel run @nodejs//:yarn config set registry https://us-npm.pkg.dev/artifact-foundry-prod/ah-3p-staging-npm/ -bazel run //tools/registry-tools:switch_registry -- $(pwd)/yarn.lock https://us-npm.pkg.dev/artifact-foundry-prod/npm-3p-trusted/ -bazel run @nodejs//:yarn install -- --frozen-lockfile -bazel test //... --build_tests_only - -# After the code is build with Airlock dependencies, we change the registry to public npmjs -# to publish our npm package. -bazel run @nodejs//:npm config set registry https://registry.npmjs.org/ - -VERSION=$(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g") - -TAG=next - -# If the version is a normal release (1.2.3) and not pre-release e.g (1.2.3-alpha.1) then publish with the 'latest' tag. -if [[ "$VERSION" =~ [0-9]+\.[0-9]+\.[0-9]+$ ]]; then - TAG=latest - - if [ "main" != "$(git branch --show-current)" ]; then - echo "Not on the 'main' branch; aborting." 1>&2 - exit 1 - fi -fi - -echo "Publishing as '$TAG' based on version: $VERSION" - -bazel run packages/@dataform/cli:package.publish -- --tag=$TAG -bazel run packages/@dataform/core:package.publish -- --tag=$TAG diff --git a/scripts/run b/scripts/run index aae89b8a..03006a0b 100755 --- a/scripts/run +++ b/scripts/run @@ -1,5 +1,5 @@ #!/bin/bash set -e -bazel build //packages/@dataform/cli:bin -./bazel-bin/packages/@dataform/cli/bin.sh "$@" +bazel build //packages/@sqlanvil/cli:bin +./bazel-bin/packages/@sqlanvil/cli/bin.sh "$@" diff --git a/scripts/update_test_credentials b/scripts/update_test_credentials deleted file mode 100755 index 81a61e23..00000000 --- a/scripts/update_test_credentials +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash - -# Generates new test credentials. -# -# The script must be executed from the repository root folder. -# -# Dependencies: -# - gcloud -# - jq - -SECRET_JSON_PATH=test_credentials/secret.json -BIGQUERY_JSON_PATH=test_credentials/bigquery.json -BIGQUERY_JSON_ENC_PATH=test_credentials/bigquery.json.enc - -# Generate a new key for dataform-testing service account and download it. -gcloud iam service-accounts keys create "${SECRET_JSON_PATH}" \ - --iam-account=dataform-testing@dataform-open-source.iam.gserviceaccount.com \ - --project=dataform-open-source - -# Create bigquery.json for encryption. Basically we do the same thing -# as `dataform init-creds` will do but without creating dataform project. -cat < "${BIGQUERY_JSON_PATH}" -{ - "projectId": "dataform-open-source", - "credentials": $(jq -Rsa < ${SECRET_JSON_PATH}), - "location": "US" -} -EOF - -# Create encrypted secret -gcloud kms encrypt \ - --ciphertext-file="${BIGQUERY_JSON_ENC_PATH}" \ - --plaintext-file="${BIGQUERY_JSON_PATH}" \ - --project=dataform-open-source \ - --keyring=dataform-builder-keyring \ - --key=dataform-builder-key \ - --location=global - -# Cleanup secrets -rm -f "${SECRET_JSON_PATH}" -rm -f "${BIGQUERY_JSON_PATH}" diff --git a/sqlx/format_test.ts b/sqlx/format_test.ts index ccb2f38b..c170918e 100644 --- a/sqlx/format_test.ts +++ b/sqlx/format_test.ts @@ -5,7 +5,7 @@ import * as path from "path"; import { format, formatFile } from "df/sqlx/format"; import { suite, test } from "df/testing"; -suite("@dataform/sqlx", () => { +suite("@sqlanvil/sqlx", () => { suite("formatter", () => { test("correctly formats a simple SQLX file", async () => { const filePath = path.join(process.env.TEST_TMPDIR, "simple.sqlx"); diff --git a/sqlx/lexer_test.ts b/sqlx/lexer_test.ts index b5ea1fb1..b2e9b832 100644 --- a/sqlx/lexer_test.ts +++ b/sqlx/lexer_test.ts @@ -3,7 +3,7 @@ import { expect } from "chai"; import { SyntaxTreeNode, SyntaxTreeNodeType } from "df/sqlx/lexer"; import { suite, test } from "df/testing"; -suite("@dataform/sqlx", () => { +suite("@sqlanvil/sqlx", () => { suite("syntax tree construction", () => { test("SQL strings don't affect the tree", () => { const actual = SyntaxTreeNode.create("SELECT SUM(IF(track.event = 'example', 1, 0)) js { }"); diff --git a/test_credentials/BUILD b/test_credentials/BUILD deleted file mode 100644 index bd9edfa9..00000000 --- a/test_credentials/BUILD +++ /dev/null @@ -1,15 +0,0 @@ -load("//tools/gcloud:secrets.bzl", "gcloud_secret") - -package(default_visibility = ["//visibility:public"]) - -# You can update the credentials for testing by: -# * Ensuring you have the required permissions (at time of writing, you need to request a grant). -# * Run the "scripts/update_test_credentials" script. -gcloud_secret( - name = "bigquery.json", - testonly = 1, - ciphertext_file = ":bigquery.json.enc", - project = "dataform-open-source", - key = "dataform-builder-key", - keyring = "dataform-builder-keyring", -) diff --git a/test_credentials/bigquery.json.enc b/test_credentials/bigquery.json.enc deleted file mode 100644 index 7aa4ba8f28a4a5764317a0c2b5d68866fb59666d..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 2643 zcmV-Z3as@CBmfS&muS&|9cMs3rKP$p$el9jK$324hjeH@qPFwWxS(vMof4`P0NEdv zUtN=$8bi@CYj3a80?>PZD~B_K1?2_`sl0%mjWX_&Nvs-?S$^TauDeBx8bE4UzzNj5 z<@KGa^Vr4oJYotdrY|P)+X$pkoTA)bD8sa*S))kborN9BkEF2}%8b%1?O+ z^^#Ea25H+CU$iPJ6%8`?wi|;4afw`VlV5;BX?)CS0dM0(evWVfHaC6O_+2LJ`aJcqAT=pRIl}W8d2ANA%{Btt6x`W5 zrd8iEB#hTv3Jyt}d=M-(%5?a=_A~ zn@BzDK6KaqWNF8rU#EVG7u{Mr;Mkc$->1Qt_|@t%2lGE?XodBUhmR_dRjGs~9!30d zH?JM|W~A*fW|{vx9vkV<-|p|?)0N;~OJOYUTe}r9DQz6fht`A)@0$>;zUpO)$;2=) zKjB8*Q(cDcSxpE}t(?T~2`DW)62nQfZ2$-+{$aow*}$Y9n7&7xpu&Bk4(v{2Fm(rVKo1|&63(z8Zx6G;q%kk&e3qU zWAe0|l|1CKUI%+y{UbyCDTgZMV@`g-bH#2aW>LCu2Kq6Kby|>TQOvc$-oOI4?$dzf z&_A&Rm^A{mx7#~UWpRxByWTRhu0&Wj z`=_e&+BPsxC%M&iy(opM5@>os<(zS;Sxf=cr}UUZJwtup-Vd4!e55pjD;22uU)o(g zki)dds8=6~&qIzxBzSDYLQUrVKty*O5vw=m&Nq!!cGt((?^;n>_q;<%GbDQ0NXVTV z<_ij;Hvitx`*u?osh5+~ps}QD1LlhiGdT!{(r9M>L=ylKFc~GFnln@C^NCzRkK|uo z1;3nQc8?(W`XoHuS}TCUGa0AVXX8EWfJOZ(+WNAAMTnLEI9kcy#mAbrmPhf4do5o% z0i#*u7#Wcx*J8NwrUD2fkG}B99ydfIGuf)jM2a6dP}Z-rxL;K3aX{T@&grCKH&p;q{LB^z-uk z@rjgRmBHj%4j9Juj0ci1#qw`NVu+*!NXn1?MEyAIba5^36a9RP&A1t+qBT77DFF%G zi^Fg6F6g(0E(l{NU+oxe+K~gZ4xFT{Z5vi?8M(7ohdlfe{+-VLI$|lvv99MvvK)f3 zq)NT8%ngk8A@$bHJt!!1yy>C7vuuR_SUG<5! z?G5gALCUD(1z(+}SLQ>hWp+WDt@AV7Vs&QUD;SNE-q9IcH2)+35FPsTdC?`-SjWV_ z08!HV5~1cVF`|T=pmtGJIg^h^|i-{rtH;h z#q4oPv!cghbhI$eF2PrWt=4F;cqyU$462_Sp3A!J(}r9WS5U?j2O;*8zt(3oc`3S5 zQTIhm<6Fc+w)J(gL}v`egN=9w!xyXY^)BdMze&~bE4FqAUx}E~JShmxNx2AV@)Eyp zAP_|n%lNvHt5W2KO|Detcwb81pbk31fZ3>6h0Bo^Zz_tX>#BZg*`e)z8SuPZl<{Dl zt@Vv-PE^vIUugmX)FJ0>-3d!3|D>-29uQHiBBES*(~{nxfb2Vf#EhCmqzszrM7*BC zY>FLO?)t~8M}fX}%mjHPV|FP2+51;D)iYLdHiD~ z{17|K67fJT1o43qo^EDCP-U`v1$SGKU9%&CotPURybQGVG+h!mxILdYP2Hu|k+Z}+ zb6_=XctGsiG0hr-vAR4Lz**OcI)$+&?CTM1OCCn8l8+tyC5;P^*V5Lb{L;uoBwxB~Ku@?37yYmCj3?&!Y=P=yv^0A@v48eHD>3e3?L`bs zR;Z&&*s?Sd0(v)vnk##41zsxZQ?w=M>4@hQF!74BP!=x%mdL^ot9>cIAoHOWihias zj|gY3RvhHpaZHG-e(Z#@IBJ9yq07I34z;qCoZ;?QaAvw}oU|@M_3R^EMS$PWU5B_? z5grFfk71{-a({IKTp*>>lGsI12B6W8fM*!;y%bh72)W!_q*K(6n6R3%2YrT-xA2N~ zPopC|Q5uKiQOgZmg=HyB$&R_F+$zktb1Sn^4Rf&`BS^_F&<^hR#a>_3)_VlOxd)B^ zUI$QizP4#>2RnO;s45AV?O9}2D`{M9*%>4rM4{iACPuih5H*|XOENR5t`CDZ^+I!F znRUf*_)o7k^3C`tTp7!(cV83_rKNREJ>yw-g)^qkC9`A3o2icDY2OkhHdg*{s6uDUnwAEE@dK%$j|Ygcl#C5w(JLrxwCM? BI>i70 diff --git a/testing/index.ts b/testing/index.ts index 9779b639..414a1403 100644 --- a/testing/index.ts +++ b/testing/index.ts @@ -25,7 +25,7 @@ export const platformPath = () => { // Note: it would be more correct for these to be injected by blaze at run time. export const nodePath = `external/${platformPath()}/bin/node`; export const npmPath = `external/${platformPath()}/bin/npm`; -export const corePackageTarPath = "packages/@dataform/core/package.tar.gz"; +export const corePackageTarPath = "packages/@sqlanvil/core/package.tar.gz"; export async function getProcessResult(childProcess: ChildProcess) { let stderr = ""; diff --git a/testing/run_core.ts b/testing/run_core.ts index a5f23e42..2aa13923 100644 --- a/testing/run_core.ts +++ b/testing/run_core.ts @@ -111,7 +111,7 @@ export function runMainInVm( ` global.workflowSettingsYaml = (function() { try { return require("./workflow_settings.yaml"); } catch(e) { console.error("YAML require failed run_core:", e); } })(); global.dataformJson = (function() { try { return require("./dataform.json"); } catch(e) {} })(); - return require("@dataform/core").main("${encodedCoreExecutionRequest}") + return require("@sqlanvil/core").main("${encodedCoreExecutionRequest}") `, vmIndexFileName ); diff --git a/tests/api/api.spec.ts b/tests/api/api.spec.ts index b2993614..86207771 100644 --- a/tests/api/api.spec.ts +++ b/tests/api/api.spec.ts @@ -16,7 +16,7 @@ import { TmpDirFixture } from "df/testing/fixtures"; config.truncateThreshold = 0; -suite("@dataform/api", () => { +suite("@sqlanvil/api", () => { // c +-> b +-> a // ^ // d diff --git a/tests/api/projects.spec.ts b/tests/api/projects.spec.ts index 82c62f80..ffab9cee 100644 --- a/tests/api/projects.spec.ts +++ b/tests/api/projects.spec.ts @@ -183,7 +183,7 @@ suite("examples", () => { "select * from `tada-analytics.df_integration_test.sample_data`" ); expect(exampleBackticks.preOps).to.eql([ - '\n GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@dataform.co"\n' + '\n GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@example.com"\n' ]); expect(exampleBackticks.postOps).to.eql([]); @@ -336,12 +336,12 @@ suite("examples", () => { databaseWithSuffix("tada-analytics"), schemaWithSuffix("df_integration_test"), "example_table" - )}\` TO GROUP "allusers@dataform.co"\n`, + )}\` TO GROUP "allusers@example.com"\n`, `\n GRANT SELECT ON \`${dotJoined( databaseWithSuffix("tada-analytics"), schemaWithSuffix("df_integration_test"), "example_table" - )}\` TO GROUP "otherusers@dataform.co"\n` + )}\` TO GROUP "otherusers@example.com"\n` ]); expect(exampleTable.tags).to.eql([]); diff --git a/tests/api/projects/common_v2/BUILD b/tests/api/projects/common_v2/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/api/projects/common_v2/BUILD +++ b/tests/api/projects/common_v2/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/api/projects/common_v2/definitions/example_backticks.sqlx b/tests/api/projects/common_v2/definitions/example_backticks.sqlx index d739a865..1d02a233 100644 --- a/tests/api/projects/common_v2/definitions/example_backticks.sqlx +++ b/tests/api/projects/common_v2/definitions/example_backticks.sqlx @@ -7,5 +7,5 @@ select * from `tada-analytics.df_integration_test.sample_data` pre_operations { - GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@dataform.co" + GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@example.com" } diff --git a/tests/api/projects/common_v2/definitions/example_table.sqlx b/tests/api/projects/common_v2/definitions/example_table.sqlx index af1fe7f2..06d2b6b4 100644 --- a/tests/api/projects/common_v2/definitions/example_table.sqlx +++ b/tests/api/projects/common_v2/definitions/example_table.sqlx @@ -8,5 +8,5 @@ select * from ${ref("df_integration_test", "sample_data")} post_operations { GRANT SELECT ON ${self()} TO GROUP "${constants.allUsersEmailAddress}" --- - GRANT SELECT ON ${self()} TO GROUP "otherusers@dataform.co" + GRANT SELECT ON ${self()} TO GROUP "otherusers@example.com" } diff --git a/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx b/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx index 6d572358..6e5a77d0 100644 --- a/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx +++ b/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx @@ -11,7 +11,7 @@ config { select * from ${ref("sample_data")} post_operations { - GRANT SELECT ON ${self()} TO GROUP "allusers@dataform.co" + GRANT SELECT ON ${self()} TO GROUP "allusers@example.com" --- - GRANT SELECT ON ${self()} TO GROUP "otherusers@dataform.co" + GRANT SELECT ON ${self()} TO GROUP "otherusers@example.com" } diff --git a/tests/api/projects/common_v2/includes/constants.js b/tests/api/projects/common_v2/includes/constants.js index d9b11471..6b3d5d16 100644 --- a/tests/api/projects/common_v2/includes/constants.js +++ b/tests/api/projects/common_v2/includes/constants.js @@ -1,3 +1,3 @@ module.exports = { - allUsersEmailAddress: "allusers@dataform.co" + allUsersEmailAddress: "allusers@example.com" }; diff --git a/tests/api/projects/invalid_dataform_json/BUILD b/tests/api/projects/invalid_dataform_json/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/api/projects/invalid_dataform_json/BUILD +++ b/tests/api/projects/invalid_dataform_json/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/api/projects/never_finishes_compiling/BUILD b/tests/api/projects/never_finishes_compiling/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/api/projects/never_finishes_compiling/BUILD +++ b/tests/api/projects/never_finishes_compiling/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/integration/BUILD b/tests/integration/BUILD index 5f1ee674..c0164078 100644 --- a/tests/integration/BUILD +++ b/tests/integration/BUILD @@ -9,7 +9,6 @@ ts_test_suite( "utils.ts", ], data = [ - "//test_credentials:bigquery.json", "//tests/integration/bigquery_project:files", "//tests/integration/bigquery_project:node_modules", ], diff --git a/tests/integration/bigquery.spec.ts b/tests/integration/bigquery.spec.ts index 01b03ce2..824b8bfc 100644 --- a/tests/integration/bigquery.spec.ts +++ b/tests/integration/bigquery.spec.ts @@ -10,7 +10,7 @@ import { dataform } from "df/protos/ts"; import { suite, test } from "df/testing"; import { compile, dropAllTables, getTableRows, keyBy } from "df/tests/integration/utils"; -suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) => { +suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) => { const credentials = dfapi.credentials.read("test_credentials/bigquery.json"); let dbadapter: BigQueryDbAdapter; diff --git a/tests/integration/bigquery_project/BUILD b/tests/integration/bigquery_project/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/integration/bigquery_project/BUILD +++ b/tests/integration/bigquery_project/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/integration/postgres.spec.ts b/tests/integration/postgres.spec.ts index 74e3ba52..25847768 100644 --- a/tests/integration/postgres.spec.ts +++ b/tests/integration/postgres.spec.ts @@ -10,7 +10,7 @@ import { suite, test } from "df/testing"; import { compile, getTableRows, keyBy } from "df/tests/integration/utils"; import { PostgresFixture } from "df/tools/postgres/postgres_fixture"; -suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) => { +suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) => { let dbadapter: dbadapters.IDbAdapter; const postgres = new PostgresFixture(5432, before, after); diff --git a/tests/integration/postgres_project/BUILD b/tests/integration/postgres_project/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/integration/postgres_project/BUILD +++ b/tests/integration/postgres_project/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/vscode/README.md b/vscode/README.md index c369386b..6588a666 100644 --- a/vscode/README.md +++ b/vscode/README.md @@ -5,7 +5,7 @@ ## Usage -**To run this extension you will need the dataform cli installed globally: `npm i -g @dataform/cli`.** +**To run this extension you will need the dataform cli installed globally: `npm i -g @sqlanvil/cli`.** Includes: diff --git a/vscode/server.ts b/vscode/server.ts index ec404bd9..391c057a 100644 --- a/vscode/server.ts +++ b/vscode/server.ts @@ -75,7 +75,7 @@ async function compileAndValidate() { if (compileResult.error?.code === "ENOENT") { connection.sendNotification( "error", - "Errors encountered when running 'dataform' CLI. Please ensure that the CLI is installed and up-to-date: 'npm i -g @dataform/cli'." + "Errors encountered when running 'dataform' CLI. Please ensure that the CLI is installed and up-to-date: 'npm i -g @sqlanvil/cli'." ); return; } else { From 513e6963e484c352b1e7b058d0e10089be2ab8a8 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 06:55:39 -0500 Subject: [PATCH 04/20] =?UTF-8?q?refactor:=20rename=20TS=20imports=20df/?= =?UTF-8?q?=20=E2=86=92=20sa/=20+=20proto=20namespace=20+=20framework=20de?= =?UTF-8?q?faults?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Path mapping: - tsconfig.json paths: df/* → sa/* - All `from "df/..."` / require("df/...") / import("df/...") → sa/* (450+) - Runtime path strings in jit_loader.js + webpack configs + cli/index_test_base.ts Proto namespace cascade (proto package dataform → sqlanvil triggers TS generation under `sqlanvil` namespace): - import { dataform, google } → import { sqlanvil, google } (and reverse) - dataform.IFoo / dataform.JFoo / dataform.DbAdapter / etc. → sqlanvil.* - camelCase field name dataformCoreVersion → sqlanvilCoreVersion - Internal function name dataformCompile → sqlanvilCompile Framework runtime defaults: - globalAny.dataform → globalAny.sqlanvil (the user-facing sqlx namespace) - __dataform_current_file → __sqlanvil_current_file (private global) - "_dataform" table folder root default → "_sqlanvil" - "dataform_assertions" assertion schema default → "sqlanvil_assertions" - defaultSchema/defaultDatabase/defaultProject "dataform" → "sqlanvil" - BigQuery job ID prefix "dataform-" → "sqlanvil-" (for BQ telemetry attribution) - Test fixture labels team:"dataform" → team:"sqlanvil" Doc comment anchors: configs#dataform-ActionConfig-* → configs#sqlanvil-* (reference the docs site anchor IDs which derive from proto package name) Test fixture strings: dataform-iceberg-test/-partial, dataform-database, dataform-conn_id, dataform-integration-tests, dataform-test-schema-suffix all renamed for consistency. Preserved (will address in later phases or kept as upstream attribution): - dataform.json strings (Phase D deletes the code path) - dataform-co/dataform URLs (legitimate upstream attribution) - dataform-open-source GCP project (test_credentials already deleted; these integration tests are broken regardless) - CLI help text "dataform [command]" (Phase E renames yargs scriptName) - "invalid dataform json throws error" test name (Phase D deletes test) Co-Authored-By: Claude Opus 4.7 --- api/dbadapters/postgres.ts | 46 ++--- api/utils/postgres.ts | 20 +- cli/api/commands/build.ts | 54 +++--- cli/api/commands/compile.ts | 24 +-- cli/api/commands/credentials.ts | 10 +- cli/api/commands/init.ts | 14 +- cli/api/commands/install.ts | 8 +- cli/api/commands/jit/compiler.ts | 24 +-- cli/api/commands/jit/rpc.ts | 26 +-- cli/api/commands/jit/rpc_test.ts | 92 ++++----- cli/api/commands/prune.ts | 18 +- cli/api/commands/query.ts | 10 +- cli/api/commands/run.ts | 132 ++++++------- cli/api/commands/state.ts | 8 +- cli/api/commands/test.ts | 12 +- cli/api/dbadapters/bigquery.ts | 90 ++++----- cli/api/dbadapters/bigquery_test.ts | 10 +- cli/api/dbadapters/execution_sql.ts | 142 +++++++------- cli/api/dbadapters/index.ts | 22 +-- cli/api/dbadapters/tasks.ts | 6 +- cli/api/execution_sql_test.ts | 26 +-- cli/api/index.ts | 18 +- cli/api/utils.ts | 8 +- cli/api/utils/error_parsing.ts | 4 +- cli/api/utils/graphs.ts | 30 +-- cli/api/utils_test.ts | 12 +- cli/console.ts | 56 +++--- cli/credentials.ts | 10 +- cli/index.ts | 38 ++-- cli/index_compile_test.ts | 52 ++--- cli/index_help_test.ts | 4 +- cli/index_init_test.ts | 40 ++-- cli/index_project_test.ts | 18 +- cli/index_run_e2e_test.ts | 76 ++++---- cli/index_test_base.ts | 2 +- cli/util.ts | 12 +- cli/util_test.ts | 4 +- cli/vm/compile.ts | 26 +-- cli/vm/jit_loader.js | 2 +- cli/vm/jit_worker.ts | 10 +- cli/yargswrapper.ts | 4 +- common/errors/errors.spec.ts | 4 +- common/flags/testing/index.ts | 4 +- common/promises/index.spec.ts | 4 +- common/protos/index.ts | 2 +- common/protos/index_test.ts | 12 +- common/protos/structs.ts | 2 +- common/strings/stringifier.spec.ts | 4 +- core/actions/assertion.ts | 60 +++--- core/actions/assertion_test.ts | 14 +- core/actions/data_preparation.ts | 76 ++++---- core/actions/data_preparation_test.ts | 10 +- core/actions/declaration.ts | 36 ++-- core/actions/declaration_test.ts | 8 +- core/actions/filename_override_test.ts | 6 +- core/actions/incremental_table.ts | 122 ++++++------ core/actions/incremental_table_test.ts | 22 +-- core/actions/index.ts | 84 ++++----- core/actions/index_test.ts | 18 +- core/actions/notebook.ts | 30 +-- core/actions/notebook_test.ts | 16 +- core/actions/operation.ts | 70 +++---- core/actions/operation_test.ts | 8 +- core/actions/table.ts | 94 +++++----- core/actions/table_test.ts | 14 +- core/actions/test.ts | 44 ++--- core/actions/test_test.ts | 6 +- core/actions/view.ts | 96 +++++----- core/actions/view_test.ts | 8 +- core/column_descriptors.ts | 30 +-- core/compilation_sql/index.ts | 8 +- core/compilers.ts | 6 +- core/compilers_test.ts | 6 +- core/contextables.ts | 10 +- core/extension.ts | 6 +- core/index.ts | 16 +- core/jit_compiler.ts | 58 +++--- core/jit_compiler_test.ts | 58 +++--- core/jit_context.ts | 26 +-- core/jit_context_test.ts | 30 +-- core/main.ts | 88 ++++----- core/main_test.ts | 106 +++++------ core/session.ts | 118 ++++++------ core/targets.ts | 8 +- core/utils.ts | 122 ++++++------ core/utils_test.ts | 34 ++-- core/workflow_settings.ts | 34 ++-- examples/examples_test.ts | 10 +- packages/@sqlanvil/cli/index.ts | 2 +- packages/@sqlanvil/cli/worker.ts | 2 +- packages/@sqlanvil/core/index.ts | 2 +- packages/@sqlanvil/core/webpack.config.js | 2 +- packages/sample-extension/index.ts | 8 +- packages/sample-extension/webpack.config.js | 2 +- sqlx/format.ts | 4 +- sqlx/format_test.ts | 4 +- sqlx/lexer_test.ts | 4 +- testing/child_process.ts | 2 +- testing/fixtures.ts | 2 +- testing/hook.ts | 2 +- testing/index.ts | 8 +- testing/index_test.ts | 2 +- testing/run_core.ts | 36 ++-- testing/runner.ts | 2 +- testing/suite.ts | 2 +- testing/test.ts | 2 +- tests/api/api.spec.ts | 198 ++++++++++---------- tests/api/projects.spec.ts | 112 +++++------ tests/api/utils/cancellable_promise.spec.ts | 4 +- tests/api/utils/error_parsing.spec.ts | 4 +- tests/integration/bigquery.spec.ts | 118 ++++++------ tests/integration/postgres.spec.ts | 76 ++++---- tests/integration/utils.ts | 14 +- tools/postgres/postgres_fixture.ts | 4 +- tsconfig.json | 2 +- vscode/extension.ts | 2 +- vscode/server.ts | 10 +- 117 files changed, 1750 insertions(+), 1750 deletions(-) diff --git a/api/dbadapters/postgres.ts b/api/dbadapters/postgres.ts index 228e5268..541f005e 100644 --- a/api/dbadapters/postgres.ts +++ b/api/dbadapters/postgres.ts @@ -1,13 +1,13 @@ import * as pg from "pg"; -import { Credentials } from "df/api/commands/credentials"; -import { IDbAdapter, IDbClient } from "df/api/dbadapters/index"; -import { SSHTunnelProxy } from "df/api/ssh_tunnel_proxy"; -import { parseRedshiftEvalError } from "df/api/utils/error_parsing"; -import { convertFieldType, PgPoolExecutor } from "df/api/utils/postgres"; -import { ErrorWithCause } from "df/common/errors/errors"; -import { collectEvaluationQueries, QueryOrAction } from "df/core/adapters"; -import { dataform } from "df/protos/ts"; +import { Credentials } from "sa/api/commands/credentials"; +import { IDbAdapter, IDbClient } from "sa/api/dbadapters/index"; +import { SSHTunnelProxy } from "sa/api/ssh_tunnel_proxy"; +import { parseRedshiftEvalError } from "sa/api/utils/error_parsing"; +import { convertFieldType, PgPoolExecutor } from "sa/api/utils/postgres"; +import { ErrorWithCause } from "sa/common/errors/errors"; +import { collectEvaluationQueries, QueryOrAction } from "sa/core/adapters"; +import { sqlanvil } from "sa/protos/ts"; interface IPostgresAdapterOptions { sshTunnel?: SSHTunnelProxy; @@ -18,7 +18,7 @@ export class PostgresDbAdapter implements IDbAdapter { credentials: Credentials, options?: { concurrencyLimit?: number; disableSslForTestsOnly?: boolean } ) { - const jdbcCredentials = credentials as dataform.IJDBC; + const jdbcCredentials = credentials as sqlanvil.IJDBC; const baseClientConfig: Partial = { user: jdbcCredentials.username, password: jdbcCredentials.password, @@ -107,21 +107,21 @@ export class PostgresDbAdapter implements IDbAdapter { ).map((validationQuery, index) => ({ index, validationQuery })); const validationQueriesWithoutWrappers = collectEvaluationQueries(queryOrAction, false); - const queryEvaluations = new Array(); + const queryEvaluations = new Array(); for (const { index, validationQuery } of validationQueries) { - let evaluationResponse: dataform.IQueryEvaluation = { - status: dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + let evaluationResponse: sqlanvil.IQueryEvaluation = { + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS }; try { await this.execute(validationQuery.query); } catch (e) { evaluationResponse = { - status: dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE, + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE, error: parseRedshiftEvalError(validationQuery.query, e) }; } queryEvaluations.push( - dataform.QueryEvaluation.create({ + sqlanvil.QueryEvaluation.create({ ...evaluationResponse, incremental: validationQuery.incremental, query: validationQueriesWithoutWrappers[index].query @@ -131,7 +131,7 @@ export class PostgresDbAdapter implements IDbAdapter { return queryEvaluations; } - public async tables(): Promise { + public async tables(): Promise { const queryResult = await this.execute( `select table_name, table_schema from information_schema.tables @@ -150,7 +150,7 @@ export class PostgresDbAdapter implements IDbAdapter { public async search( searchText: string, options: { limit: number } = { limit: 1000 } - ): Promise { + ): Promise { // TODO: It would be nice to extend this to search through table/column descriptions. However, this involves // a somewhat crazy 5-way join. const results = await this.execute( @@ -174,7 +174,7 @@ export class PostgresDbAdapter implements IDbAdapter { ); } - public async table(target: dataform.ITarget): Promise { + public async table(target: sqlanvil.ITarget): Promise { const params = [target.schema, target.name]; const [tableResults, columnResults, descriptionResults] = await Promise.all([ this.execute( @@ -201,14 +201,14 @@ export class PostgresDbAdapter implements IDbAdapter { if (tableResults.rows.length === 0) { return null; } - return dataform.TableMetadata.create({ + return sqlanvil.TableMetadata.create({ target, type: tableResults.rows[0].table_type === "VIEW" - ? dataform.TableMetadata.Type.VIEW - : dataform.TableMetadata.Type.TABLE, + ? sqlanvil.TableMetadata.Type.VIEW + : sqlanvil.TableMetadata.Type.TABLE, fields: columnResults.rows.map(row => - dataform.Field.create({ + sqlanvil.Field.create({ name: row.column_name, primitive: convertFieldType(row.data_type), description: descriptionResults.rows.find( @@ -222,7 +222,7 @@ export class PostgresDbAdapter implements IDbAdapter { }); } - public async preview(target: dataform.ITarget, limitRows: number = 10): Promise { + public async preview(target: sqlanvil.ITarget, limitRows: number = 10): Promise { const { rows } = await this.execute( `SELECT * FROM "${target.schema}"."${target.name}" LIMIT ${limitRows}` ); @@ -247,7 +247,7 @@ export class PostgresDbAdapter implements IDbAdapter { } } - public async setMetadata(action: dataform.IExecutionAction): Promise { + public async setMetadata(action: sqlanvil.IExecutionAction): Promise { const { target, actionDescriptor, tableType } = action; const actualMetadata = await this.table(target); diff --git a/api/utils/postgres.ts b/api/utils/postgres.ts index 04265d55..79755e24 100644 --- a/api/utils/postgres.ts +++ b/api/utils/postgres.ts @@ -1,8 +1,8 @@ import * as pg from "pg"; import QueryStream from "pg-query-stream"; -import { LimitedResultSet } from "df/api/utils/results"; -import { dataform } from "df/protos/ts"; +import { LimitedResultSet } from "sa/api/utils/results"; +import { sqlanvil } from "sa/protos/ts"; const maybeInitializePg = (() => { let initialized = false; @@ -144,7 +144,7 @@ export function convertFieldType(type: string) { case "FLOAT8": case "DOUBLE PRECISION": case "REAL": - return dataform.Field.Primitive.FLOAT; + return sqlanvil.Field.Primitive.FLOAT; case "INTEGER": case "INT": case "INT2": @@ -152,13 +152,13 @@ export function convertFieldType(type: string) { case "INT8": case "BIGINT": case "SMALLINT": - return dataform.Field.Primitive.INTEGER; + return sqlanvil.Field.Primitive.INTEGER; case "DECIMAL": case "NUMERIC": - return dataform.Field.Primitive.NUMERIC; + return sqlanvil.Field.Primitive.NUMERIC; case "BOOLEAN": case "BOOL": - return dataform.Field.Primitive.BOOLEAN; + return sqlanvil.Field.Primitive.BOOLEAN; case "STRING": case "VARCHAR": case "CHAR": @@ -168,15 +168,15 @@ export function convertFieldType(type: string) { case "TEXT": case "NCHAR": case "BPCHAR": - return dataform.Field.Primitive.STRING; + return sqlanvil.Field.Primitive.STRING; case "DATE": - return dataform.Field.Primitive.DATE; + return sqlanvil.Field.Primitive.DATE; case "TIMESTAMP": case "TIMESTAMPZ": case "TIMESTAMP WITHOUT TIME ZONE": case "TIMESTAMP WITH TIME ZONE": - return dataform.Field.Primitive.TIMESTAMP; + return sqlanvil.Field.Primitive.TIMESTAMP; default: - return dataform.Field.Primitive.UNKNOWN; + return sqlanvil.Field.Primitive.UNKNOWN; } } diff --git a/cli/api/commands/build.ts b/cli/api/commands/build.ts index 99538381..f1712b28 100644 --- a/cli/api/commands/build.ts +++ b/cli/api/commands/build.ts @@ -1,14 +1,14 @@ -import { prune } from "df/cli/api/commands/prune"; -import { state } from "df/cli/api/commands/state"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { targetStringifier } from "df/core/targets"; -import * as utils from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { prune } from "sa/cli/api/commands/prune"; +import { state } from "sa/cli/api/commands/state"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { targetStringifier } from "sa/core/targets"; +import * as utils from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; export async function build( - compiledGraph: dataform.ICompiledGraph, - runConfig: dataform.IRunConfig, + compiledGraph: sqlanvil.ICompiledGraph, + runConfig: sqlanvil.IRunConfig, dbadapter: dbadapters.IDbAdapter ) { const prunedGraph = prune(compiledGraph, runConfig); @@ -31,29 +31,29 @@ export class Builder { private readonly executionSql: ExecutionSql; constructor( - private readonly prunedGraph: dataform.ICompiledGraph, - private readonly runConfig: dataform.IRunConfig, - private readonly warehouseState: dataform.IWarehouseState + private readonly prunedGraph: sqlanvil.ICompiledGraph, + private readonly runConfig: sqlanvil.IRunConfig, + private readonly warehouseState: sqlanvil.IWarehouseState ) { this.executionSql = new ExecutionSql( prunedGraph.projectConfig, - prunedGraph.dataformCoreVersion || "1.0.0" + prunedGraph.sqlanvilCoreVersion || "1.0.0" ); prunedGraph.tables.forEach(utils.setOrValidateTableEnumType); } - public build(): dataform.ExecutionGraph { + public build(): sqlanvil.ExecutionGraph { if (utils.graphHasErrors(this.prunedGraph)) { throw new Error(`Project has unresolved compilation or validation errors.`); } - const tableMetadataByTarget = new Map(); + const tableMetadataByTarget = new Map(); this.warehouseState.tables.forEach(tableState => { tableMetadataByTarget.set(targetStringifier.stringify(tableState.target), tableState); }); - const actions: dataform.IExecutionAction[] = [].concat( + const actions: sqlanvil.IExecutionAction[] = [].concat( this.prunedGraph.tables.map(t => this.buildTable( t, @@ -64,7 +64,7 @@ export class Builder { this.prunedGraph.operations.map(o => this.buildOperation(o)), this.prunedGraph.assertions.map(a => this.buildAssertion(a)) ); - return dataform.ExecutionGraph.create({ + return sqlanvil.ExecutionGraph.create({ projectConfig: this.prunedGraph.projectConfig, runConfig: this.runConfig, warehouseState: this.warehouseState, @@ -74,9 +74,9 @@ export class Builder { } private buildTable( - table: dataform.ITable, - tableMetadata: dataform.ITableMetadata, - runConfig: dataform.IRunConfig + table: sqlanvil.ITable, + tableMetadata: sqlanvil.ITableMetadata, + runConfig: sqlanvil.IRunConfig ) { return { ...this.toPartialExecutionAction(table), @@ -85,36 +85,36 @@ export class Builder { tasks: table.disabled ? [] : this.executionSql.publishTasks(table, runConfig, tableMetadata).build(), - hermeticity: table.hermeticity || dataform.ActionHermeticity.HERMETIC + hermeticity: table.hermeticity || sqlanvil.ActionHermeticity.HERMETIC }; } - private buildOperation(operation: dataform.IOperation) { + private buildOperation(operation: sqlanvil.IOperation) { return { ...this.toPartialExecutionAction(operation), type: "operation", tasks: operation.disabled ? [] : operation.queries.map(statement => ({ type: "statement", statement })), - hermeticity: operation.hermeticity || dataform.ActionHermeticity.NON_HERMETIC + hermeticity: operation.hermeticity || sqlanvil.ActionHermeticity.NON_HERMETIC }; } - private buildAssertion(assertion: dataform.IAssertion) { + private buildAssertion(assertion: sqlanvil.IAssertion) { return { ...this.toPartialExecutionAction(assertion), type: "assertion", tasks: assertion.disabled ? [] : this.executionSql.assertTasks(assertion, this.prunedGraph.projectConfig).build(), - hermeticity: assertion.hermeticity || dataform.ActionHermeticity.HERMETIC + hermeticity: assertion.hermeticity || sqlanvil.ActionHermeticity.HERMETIC }; } private toPartialExecutionAction( - action: dataform.ITable | dataform.IOperation | dataform.IAssertion + action: sqlanvil.ITable | sqlanvil.IOperation | sqlanvil.IAssertion ) { - return dataform.ExecutionAction.create({ + return sqlanvil.ExecutionAction.create({ target: action.target, fileName: action.fileName, dependencyTargets: action.dependencyTargets, diff --git a/cli/api/commands/compile.ts b/cli/api/commands/compile.ts index 351fb276..83dae098 100644 --- a/cli/api/commands/compile.ts +++ b/cli/api/commands/compile.ts @@ -4,11 +4,11 @@ import * as path from "path"; import * as tmp from "tmp"; import { promisify } from "util"; -import { MISSING_CORE_VERSION_ERROR } from "df/cli/api/commands/install"; -import { readConfigFromWorkflowSettings } from "df/cli/api/utils"; -import { coerceAsError } from "df/common/errors/errors"; -import { decode64 } from "df/common/protos"; -import { dataform } from "df/protos/ts"; +import { MISSING_CORE_VERSION_ERROR } from "sa/cli/api/commands/install"; +import { readConfigFromWorkflowSettings } from "sa/cli/api/utils"; +import { coerceAsError } from "sa/common/errors/errors"; +import { decode64 } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; export class CompilationTimeoutError extends Error {} @@ -17,9 +17,9 @@ function print(text: string) { } export async function compile( - compileConfig: dataform.ICompileConfig = {} -): Promise { - let compiledGraph = dataform.CompiledGraph.create(); + compileConfig: sqlanvil.ICompileConfig = {} +): Promise { + let compiledGraph = sqlanvil.CompiledGraph.create(); const resolvedProjectPath = path.resolve(compileConfig.projectDir); const packageJsonPath = path.join(resolvedProjectPath, "package.json"); @@ -29,7 +29,7 @@ export async function compile( const temporaryProjectPath = tmp.dirSync().name; const workflowSettings = readConfigFromWorkflowSettings(resolvedProjectPath); - const workflowSettingsDataformCoreVersion = workflowSettings?.dataformCoreVersion; + const workflowSettingsDataformCoreVersion = workflowSettings?.sqlanvilCoreVersion; const workflowSettingsExtension = workflowSettings?.extension ?? undefined; compileConfig.extension = workflowSettingsExtension; @@ -88,8 +88,8 @@ export async function compile( const result = await CompileChildProcess.forkProcess().compile(compileConfig); - const decodedResult = decode64(dataform.CoreExecutionResponse, result); - compiledGraph = dataform.CompiledGraph.create(decodedResult.compile.compiledGraph); + const decodedResult = decode64(sqlanvil.CoreExecutionResponse, result); + compiledGraph = sqlanvil.CompiledGraph.create(decodedResult.compile.compiledGraph); if (workflowSettingsDataformCoreVersion) { fs.rmSync(temporaryProjectPath, { recursive: true }); @@ -121,7 +121,7 @@ export class CompileChildProcess { this.childProcess = childProcess; } - public async compile(compileConfig: dataform.ICompileConfig) { + public async compile(compileConfig: sqlanvil.ICompileConfig) { const compileInChildProcess = new Promise(async (resolve, reject) => { this.childProcess.on("error", (e: Error) => reject(coerceAsError(e))); diff --git a/cli/api/commands/credentials.ts b/cli/api/commands/credentials.ts index 9a69c11d..370735a5 100644 --- a/cli/api/commands/credentials.ts +++ b/cli/api/commands/credentials.ts @@ -1,12 +1,12 @@ import * as fs from "fs"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { verifyObjectMatchesProto } from "df/common/protos"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { verifyObjectMatchesProto } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; export const CREDENTIALS_FILENAME = ".df-credentials.json"; -export function read(credentialsPath: string): dataform.IBigQuery { +export function read(credentialsPath: string): sqlanvil.IBigQuery { if (!fs.existsSync(credentialsPath)) { throw new Error(`Missing credentials JSON file; not found at path '${credentialsPath}'.`); } @@ -16,7 +16,7 @@ export function read(credentialsPath: string): dataform.IBigQuery { } catch (e) { throw new Error(`Error reading credentials file: ${e.message}`); } - const credentials = verifyObjectMatchesProto(dataform.BigQuery, credentialsAsJson); + const credentials = verifyObjectMatchesProto(sqlanvil.BigQuery, credentialsAsJson); if (!Object.keys(credentials).find(key => key === "projectId")?.length) { throw new Error(`Error reading credentials file: the projectId field is required`); } diff --git a/cli/api/commands/init.ts b/cli/api/commands/init.ts index 686e27e2..eb31cda1 100644 --- a/cli/api/commands/init.ts +++ b/cli/api/commands/init.ts @@ -2,9 +2,9 @@ import * as fs from "fs"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { CREDENTIALS_FILENAME } from "df/cli/api/commands/credentials"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; +import { CREDENTIALS_FILENAME } from "sa/cli/api/commands/credentials"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; const gitIgnoreContents = ` ${CREDENTIALS_FILENAME} @@ -18,7 +18,7 @@ export interface IInitResult { export async function init( projectDir: string, - projectConfig: dataform.IProjectConfig + projectConfig: sqlanvil.IProjectConfig ): Promise { const workflowSettingsYamlPath = path.join(projectDir, "workflow_settings.yaml"); const packageJsonPath = path.join(projectDir, "package.json"); @@ -45,12 +45,12 @@ export async function init( } // The order that fields are set here is preserved in the written yaml. - const workflowSettings: dataform.IWorkflowSettings = { - dataformCoreVersion: version, + const workflowSettings: sqlanvil.IWorkflowSettings = { + sqlanvilCoreVersion: version, defaultProject: projectConfig.defaultDatabase, defaultLocation: projectConfig.defaultLocation, defaultDataset: projectConfig.defaultSchema || "dataform", - defaultAssertionDataset: projectConfig.assertionSchema || "dataform_assertions", + defaultAssertionDataset: projectConfig.assertionSchema || "sqlanvil_assertions", defaultIcebergConfig: projectConfig.defaultIcebergConfig, }; if (projectConfig.databaseSuffix) { diff --git a/cli/api/commands/install.ts b/cli/api/commands/install.ts index f71f09a9..db139a3c 100644 --- a/cli/api/commands/install.ts +++ b/cli/api/commands/install.ts @@ -3,10 +3,10 @@ import * as fs from "fs"; import * as path from "path"; import { promisify } from "util"; -import { readDataformCoreVersionFromWorkflowSettings } from "df/cli/api/utils"; +import { readDataformCoreVersionFromWorkflowSettings } from "sa/cli/api/utils"; export const MISSING_CORE_VERSION_ERROR = - "dataformCoreVersion must be specified either in workflow_settings.yaml or via a package.json"; + "sqlanvilCoreVersion must be specified either in workflow_settings.yaml or via a package.json"; export async function install(projectPath: string) { const resolvedProjectPath = path.resolve(projectPath); @@ -14,8 +14,8 @@ export async function install(projectPath: string) { // Core's readWorkflowSettings method cannot be used for this because Core assumes that // `require` can read YAML files directly. - const dataformCoreVersion = readDataformCoreVersionFromWorkflowSettings(resolvedProjectPath); - if (dataformCoreVersion) { + const sqlanvilCoreVersion = readDataformCoreVersionFromWorkflowSettings(resolvedProjectPath); + if (sqlanvilCoreVersion) { throw new Error( "No installation is needed when using workflow_settings.yaml, as packages are installed at " + "runtime." diff --git a/cli/api/commands/jit/compiler.ts b/cli/api/commands/jit/compiler.ts index 651da96f..7ebc42eb 100644 --- a/cli/api/commands/jit/compiler.ts +++ b/cli/api/commands/jit/compiler.ts @@ -1,12 +1,12 @@ import { ChildProcess } from "child_process"; import * as path from "path"; -import { BaseWorker } from "df/cli/api/commands/base_worker"; -import { handleDbRequest } from "df/cli/api/commands/jit/rpc"; -import { IDbAdapter, IDbClient } from "df/cli/api/dbadapters"; -import { IBigQueryExecutionOptions } from "df/cli/api/dbadapters/bigquery"; -import { DEFAULT_COMPILATION_TIMEOUT_MILLIS } from "df/cli/api/utils/constants"; -import { dataform } from "df/protos/ts"; +import { BaseWorker } from "sa/cli/api/commands/base_worker"; +import { handleDbRequest } from "sa/cli/api/commands/jit/rpc"; +import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters"; +import { IBigQueryExecutionOptions } from "sa/cli/api/dbadapters/bigquery"; +import { DEFAULT_COMPILATION_TIMEOUT_MILLIS } from "sa/cli/api/utils/constants"; +import { sqlanvil } from "sa/protos/ts"; export interface IJitWorkerMessage { type: "rpc_request" | "jit_response" | "jit_error"; @@ -18,17 +18,17 @@ export interface IJitWorkerMessage { } export class JitCompileChildProcess extends BaseWorker< - dataform.IJitCompilationResponse, + sqlanvil.IJitCompilationResponse, IJitWorkerMessage > { public static async compile( - request: dataform.IJitCompilationRequest, + request: sqlanvil.IJitCompilationRequest, projectDir: string, dbadapter: IDbAdapter, dbclient: IDbClient, timeoutMillis: number = DEFAULT_COMPILATION_TIMEOUT_MILLIS, options?: IBigQueryExecutionOptions - ): Promise { + ): Promise { return await new JitCompileChildProcess().run( request, projectDir, @@ -44,13 +44,13 @@ export class JitCompileChildProcess extends BaseWorker< } private async run( - request: dataform.IJitCompilationRequest, + request: sqlanvil.IJitCompilationRequest, projectDir: string, dbadapter: IDbAdapter, dbclient: IDbClient, timeoutMillis: number, options?: IBigQueryExecutionOptions - ): Promise { + ): Promise { return await this.runWorker( timeoutMillis, child => { @@ -64,7 +64,7 @@ export class JitCompileChildProcess extends BaseWorker< if (message.type === "rpc_request") { await this.handleRpcRequest(message, child, dbadapter, dbclient, options); } else if (message.type === "jit_response") { - resolve(dataform.JitCompilationResponse.fromObject(message.response)); + resolve(sqlanvil.JitCompilationResponse.fromObject(message.response)); } else if (message.type === "jit_error") { reject(new Error(message.error)); } diff --git a/cli/api/commands/jit/rpc.ts b/cli/api/commands/jit/rpc.ts index c9bfea07..b5ae5218 100644 --- a/cli/api/commands/jit/rpc.ts +++ b/cli/api/commands/jit/rpc.ts @@ -1,9 +1,9 @@ import Long from "long"; -import { IDbAdapter, IDbClient } from "df/cli/api/dbadapters"; -import { IBigQueryExecutionOptions } from "df/cli/api/dbadapters/bigquery"; -import { Structs } from "df/common/protos/structs"; -import { dataform, google } from "df/protos/ts"; +import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters"; +import { IBigQueryExecutionOptions } from "sa/cli/api/dbadapters/bigquery"; +import { Structs } from "sa/common/protos/structs"; +import { sqlanvil, google } from "sa/protos/ts"; export async function handleDbRequest( dbadapter: IDbAdapter, @@ -31,8 +31,8 @@ async function handleExecute( request: Uint8Array, options?: IBigQueryExecutionOptions ): Promise { - const executeRequest = dataform.ExecuteRequest.decode(request); - const executeRequestObj = dataform.ExecuteRequest.toObject(executeRequest, { + const executeRequest = sqlanvil.ExecuteRequest.decode(request); + const executeRequestObj = sqlanvil.ExecuteRequest.toObject(executeRequest, { defaults: false }); const requestOptions = executeRequestObj.bigQueryOptions; @@ -51,31 +51,31 @@ async function handleExecute( } }); - return dataform.ExecuteResponse.encode({ + return sqlanvil.ExecuteResponse.encode({ rows: (results.rows || []).map(row => Structs.fromObject(row)), schemaFields: results.schema || [] } as any).finish(); } async function handleListTables(dbadapter: IDbAdapter, request: Uint8Array): Promise { - const listTablesRequest = dataform.ListTablesRequest.decode(request); + const listTablesRequest = sqlanvil.ListTablesRequest.decode(request); if (!listTablesRequest.database) { throw new Error("ListTablesRequest.database must be supplied"); } const tablesMetadata = await dbadapter.tables(listTablesRequest.database, listTablesRequest.schema); - const listTablesResponse = dataform.ListTablesResponse.create({ + const listTablesResponse = sqlanvil.ListTablesResponse.create({ tables: tablesMetadata }); - return dataform.ListTablesResponse.encode(listTablesResponse).finish(); + return sqlanvil.ListTablesResponse.encode(listTablesResponse).finish(); } async function handleGetTable(dbadapter: IDbAdapter, request: Uint8Array): Promise { - const getTableRequest = dataform.GetTableRequest.decode(request); + const getTableRequest = sqlanvil.GetTableRequest.decode(request); const tableMetadata = await dbadapter.table(getTableRequest.target); if (!tableMetadata) { throw new Error(`Table not found: ${JSON.stringify(getTableRequest.target)}`); } - return dataform.TableMetadata.encode(tableMetadata).finish(); + return sqlanvil.TableMetadata.encode(tableMetadata).finish(); } async function handleDeleteTable( @@ -83,7 +83,7 @@ async function handleDeleteTable( request: Uint8Array, dryRun?: boolean ): Promise { - const deleteTableRequest = dataform.DeleteTableRequest.decode(request); + const deleteTableRequest = sqlanvil.DeleteTableRequest.decode(request); if (dryRun) { return new Uint8Array(); } diff --git a/cli/api/commands/jit/rpc_test.ts b/cli/api/commands/jit/rpc_test.ts index f9fbd9bc..3ec21a81 100644 --- a/cli/api/commands/jit/rpc_test.ts +++ b/cli/api/commands/jit/rpc_test.ts @@ -2,10 +2,10 @@ import { expect } from "chai"; import Long from "long"; import { anything, capture, instance, mock, verify, when } from "ts-mockito"; -import { handleDbRequest } from "df/cli/api/commands/jit/rpc"; -import { IDbAdapter, IDbClient } from "df/cli/api/dbadapters"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { handleDbRequest } from "sa/cli/api/commands/jit/rpc"; +import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("jit_rpc", () => { test("Execute RPC maps to client.execute with all options", async () => { @@ -13,7 +13,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT * FROM table"; - const executeRequest = dataform.ExecuteRequest.create({ + const executeRequest = sqlanvil.ExecuteRequest.create({ statement, rowLimit: Long.fromNumber(100), byteLimit: Long.fromNumber(1024), @@ -25,7 +25,7 @@ suite("jit_rpc", () => { dryRun: true } }); - const encodedRequest = dataform.ExecuteRequest.encode(executeRequest).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(executeRequest).finish(); // Real raw BigQuery f/v format const rawRows = [ @@ -40,10 +40,10 @@ suite("jit_rpc", () => { ]; const schema = [ - { name: "num", primitive: dataform.Field.Primitive.INTEGER }, - { name: "str", primitive: dataform.Field.Primitive.STRING }, - { name: "bool", primitive: dataform.Field.Primitive.BOOLEAN }, - { name: "n", primitive: dataform.Field.Primitive.STRING } + { name: "num", primitive: sqlanvil.Field.Primitive.INTEGER }, + { name: "str", primitive: sqlanvil.Field.Primitive.STRING }, + { name: "bool", primitive: sqlanvil.Field.Primitive.BOOLEAN }, + { name: "n", primitive: sqlanvil.Field.Primitive.STRING } ]; when(mockClient.executeRaw(statement, anything())).thenResolve({ rows: rawRows, @@ -52,7 +52,7 @@ suite("jit_rpc", () => { }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const row = decoded.rows[0]; @@ -81,14 +81,14 @@ suite("jit_rpc", () => { const mockClient = mock(); const target = { database: "db", schema: "sch", name: "tab" }; - const request = dataform.DeleteTableRequest.create({ target }); - const encodedRequest = dataform.DeleteTableRequest.encode(request).finish(); + const request = sqlanvil.DeleteTableRequest.create({ target }); + const encodedRequest = sqlanvil.DeleteTableRequest.encode(request).finish(); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "DeleteTable", encodedRequest); verify(mockAdapter.deleteTable(anything())).once(); const capturedTarget = capture(mockAdapter.deleteTable).last()[0]; - expect(dataform.Target.create(capturedTarget)).deep.equals(dataform.Target.create(target)); + expect(sqlanvil.Target.create(capturedTarget)).deep.equals(sqlanvil.Target.create(target)); expect(response.length).equals(0); }); @@ -97,7 +97,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT null as n"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); // Test with a null value when(mockClient.executeRaw(statement, anything())).thenResolve({ @@ -106,12 +106,12 @@ suite("jit_rpc", () => { f: [{ v: null }] } ], - schema: [{ name: "n", primitive: dataform.Field.Primitive.STRING }], + schema: [{ name: "n", primitive: sqlanvil.Field.Primitive.STRING }], metadata: {} }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const fListNull = decoded.rows[0].fields.f.listValue.values; @@ -129,7 +129,7 @@ suite("jit_rpc", () => { }); const responseEmpty = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decodedEmpty = dataform.ExecuteResponse.decode(responseEmpty); + const decodedEmpty = sqlanvil.ExecuteResponse.decode(responseEmpty); expect(decodedEmpty.rows.length).equals(0); verify(mockClient.executeRaw(statement, anything())).twice(); @@ -142,15 +142,15 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); - const request = dataform.ListTablesRequest.create({ database: "db", schema: "sch" }); - const encodedRequest = dataform.ListTablesRequest.encode(request).finish(); + const request = sqlanvil.ListTablesRequest.create({ database: "db", schema: "sch" }); + const encodedRequest = sqlanvil.ListTablesRequest.encode(request).finish(); const target1 = { database: "db", schema: "sch", name: "table1" }; - const metadata1 = { target: target1, type: dataform.TableMetadata.Type.TABLE } as any; + const metadata1 = { target: target1, type: sqlanvil.TableMetadata.Type.TABLE } as any; when(mockAdapter.tables("db", "sch")).thenResolve([metadata1]); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "ListTables", encodedRequest); - const decoded = dataform.ListTablesResponse.decode(response); + const decoded = sqlanvil.ListTablesResponse.decode(response); expect(decoded.tables.length).equals(1); expect(decoded.tables[0].target.name).equals("table1"); @@ -164,8 +164,8 @@ suite("jit_rpc", () => { const mockClient = mock(); // Request without database - const request = dataform.ListTablesRequest.create({ schema: "sch" }); - const encodedRequest = dataform.ListTablesRequest.encode(request).finish(); + const request = sqlanvil.ListTablesRequest.create({ schema: "sch" }); + const encodedRequest = sqlanvil.ListTablesRequest.encode(request).finish(); try { await handleDbRequest(instance(mockAdapter), instance(mockClient), "ListTables", encodedRequest); @@ -182,18 +182,18 @@ suite("jit_rpc", () => { const mockClient = mock(); const target = { database: "db", schema: "sch", name: "tab" }; - const request = dataform.GetTableRequest.create({ target }); - const encodedRequest = dataform.GetTableRequest.encode(request).finish(); + const request = sqlanvil.GetTableRequest.create({ target }); + const encodedRequest = sqlanvil.GetTableRequest.encode(request).finish(); when(mockAdapter.table(anything())).thenResolve({ target } as any); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "GetTable", encodedRequest); - const decoded = dataform.TableMetadata.decode(response); + const decoded = sqlanvil.TableMetadata.decode(response); expect(decoded.target.name).equals("tab"); verify(mockAdapter.table(anything())).once(); const capturedTarget = capture(mockAdapter.table).last()[0]; - expect(dataform.Target.create(capturedTarget)).deep.equals(dataform.Target.create(target)); + expect(sqlanvil.Target.create(capturedTarget)).deep.equals(sqlanvil.Target.create(target)); }); test("GetTable RPC throws error when table not found", async () => { @@ -201,8 +201,8 @@ suite("jit_rpc", () => { const mockClient = mock(); const target = { database: "db", schema: "sch", name: "missing" }; - const request = dataform.GetTableRequest.create({ target }); - const encodedRequest = dataform.GetTableRequest.encode(request).finish(); + const request = sqlanvil.GetTableRequest.create({ target }); + const encodedRequest = sqlanvil.GetTableRequest.encode(request).finish(); // Adapter returns null for missing table when(mockAdapter.table(anything())).thenResolve(null); @@ -222,10 +222,10 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); - const request = dataform.DeleteTableRequest.create({ + const request = sqlanvil.DeleteTableRequest.create({ target: { database: "db", schema: "sch", name: "tab" } }); - const encodedRequest = dataform.DeleteTableRequest.encode(request).finish(); + const encodedRequest = sqlanvil.DeleteTableRequest.encode(request).finish(); // Call with dryRun = true await handleDbRequest(instance(mockAdapter), instance(mockClient), "DeleteTable", encodedRequest, { dryRun: true }); @@ -239,7 +239,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); when(mockClient.executeRaw(anything(), anything())).thenResolve({ rows: [], metadata: {} }); @@ -268,7 +268,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const executeRequest = dataform.ExecuteRequest.create({ + const executeRequest = sqlanvil.ExecuteRequest.create({ statement, bigQueryOptions: { location: "EU", @@ -277,7 +277,7 @@ suite("jit_rpc", () => { dryRun: true } }); - const encodedRequest = dataform.ExecuteRequest.encode(executeRequest).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(executeRequest).finish(); const globalOptions = { labels: { global_label: "global_val" }, @@ -313,7 +313,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: { request_label: "request_val" } } }).finish(); @@ -338,7 +338,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: { request_label: "request_val" } } }).finish(); @@ -360,7 +360,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: { request_label: "request_val" } } }).finish(); @@ -383,7 +383,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT 1"; // Request has no labels - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { location: "US" } }).finish(); @@ -405,7 +405,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT 1"; // Request has empty labels - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: {} } }).finish(); @@ -427,7 +427,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT * FROM table"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); // Real raw BigQuery f/v format const rawRows = [ @@ -440,12 +440,12 @@ suite("jit_rpc", () => { when(mockClient.executeRaw(statement, anything())).thenResolve({ rows: rawRows, - schema: [{ name: "id", primitive: dataform.Field.Primitive.STRING }], + schema: [{ name: "id", primitive: sqlanvil.Field.Primitive.STRING }], metadata: { bigquery: { jobId: "job1" } } }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const row = decoded.rows[0]; @@ -461,7 +461,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT complex_struct FROM table"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); // Real raw BigQuery complex nested f/v format const rawRows = [ @@ -481,12 +481,12 @@ suite("jit_rpc", () => { when(mockClient.executeRaw(statement, anything())).thenResolve({ rows: rawRows, - schema: [{ name: "complex_struct", primitive: dataform.Field.Primitive.STRING }], + schema: [{ name: "complex_struct", primitive: sqlanvil.Field.Primitive.STRING }], metadata: { bigquery: { jobId: "job1" } } }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const row = decoded.rows[0]; diff --git a/cli/api/commands/prune.ts b/cli/api/commands/prune.ts index 71358ea0..cfb6bad6 100644 --- a/cli/api/commands/prune.ts +++ b/cli/api/commands/prune.ts @@ -1,13 +1,13 @@ -import { targetAsReadableString } from "df/core/targets"; -import * as utils from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { targetAsReadableString } from "sa/core/targets"; +import * as utils from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; -type CompileAction = dataform.ITable | dataform.IOperation | dataform.IAssertion; +type CompileAction = sqlanvil.ITable | sqlanvil.IOperation | sqlanvil.IAssertion; export function prune( - compiledGraph: dataform.ICompiledGraph, - runConfig: dataform.IRunConfig -): dataform.ICompiledGraph { + compiledGraph: sqlanvil.ICompiledGraph, + runConfig: sqlanvil.IRunConfig +): sqlanvil.ICompiledGraph { compiledGraph.tables.forEach(utils.setOrValidateTableEnumType); const includedActionNames = computeIncludedActionNames(compiledGraph, runConfig); return { @@ -25,8 +25,8 @@ export function prune( } function computeIncludedActionNames( - compiledGraph: dataform.ICompiledGraph, - runConfig: dataform.IRunConfig + compiledGraph: sqlanvil.ICompiledGraph, + runConfig: sqlanvil.IRunConfig ): Set { // Union all tables, operations, assertions. const allActions: CompileAction[] = [].concat( diff --git a/cli/api/commands/query.ts b/cli/api/commands/query.ts index 7921e205..66628fc5 100644 --- a/cli/api/commands/query.ts +++ b/cli/api/commands/query.ts @@ -1,12 +1,12 @@ -import * as dbadapters from "df/cli/api/dbadapters"; -import { CancellablePromise } from "df/cli/api/utils/cancellable_promise"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { CancellablePromise } from "sa/cli/api/utils/cancellable_promise"; +import { sqlanvil } from "sa/protos/ts"; export function run( dbadapter: dbadapters.IDbAdapter, query: string, options?: { - compileConfig?: dataform.ICompileConfig; + compileConfig?: sqlanvil.ICompileConfig; rowLimit?: number; byteLimit?: number; } @@ -29,6 +29,6 @@ export function run( export async function evaluate( dbadapter: dbadapters.IDbAdapter, query: string -): Promise { +): Promise { return (await dbadapter.evaluate(query))[0]; } diff --git a/cli/api/commands/run.ts b/cli/api/commands/run.ts index f618aa09..9c8bc395 100644 --- a/cli/api/commands/run.ts +++ b/cli/api/commands/run.ts @@ -1,26 +1,26 @@ import EventEmitter from "events"; import Long from "long"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { IBigQueryExecutionOptions } from "df/cli/api/dbadapters/bigquery"; -import { Flags } from "df/common/flags"; -import { retry } from "df/common/promises"; -import { deepClone, equals } from "df/common/protos"; -import { targetStringifier } from "df/core/targets"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { IBigQueryExecutionOptions } from "sa/cli/api/dbadapters/bigquery"; +import { Flags } from "sa/common/flags"; +import { retry } from "sa/common/promises"; +import { deepClone, equals } from "sa/common/protos"; +import { targetStringifier } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; const CANCEL_EVENT = "jobCancel"; const flags = { runnerNotificationPeriodMillis: Flags.number("runner-notification-period-millis", 5000) }; -const isSuccessfulAction = (actionResult: dataform.IActionResult) => - actionResult.status === dataform.ActionResult.ExecutionStatus.SUCCESSFUL || - actionResult.status === dataform.ActionResult.ExecutionStatus.DISABLED; +const isSuccessfulAction = (actionResult: sqlanvil.IActionResult) => + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL || + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.DISABLED; export interface IExecutedAction { - executionAction: dataform.IExecutionAction; - actionResult: dataform.IActionResult; + executionAction: sqlanvil.IExecutionAction; + actionResult: sqlanvil.IActionResult; } export interface IExecutionOptions { @@ -34,9 +34,9 @@ export interface IExecutionOptions { export function run( dbadapter: dbadapters.IDbAdapter, - graph: dataform.IExecutionGraph, + graph: sqlanvil.IExecutionGraph, executionOptions?: IExecutionOptions, - partiallyExecutedRunResult: dataform.IRunResult = {}, + partiallyExecutedRunResult: sqlanvil.IRunResult = {}, runnerNotificationPeriodMillis: number = flags.runnerNotificationPeriodMillis.get() ): Runner { return new Runner( @@ -49,27 +49,27 @@ export function run( } export class Runner { - private readonly warehouseStateByTarget: Map; + private readonly warehouseStateByTarget: Map; private readonly allActionTargets: Set; - private readonly runResult: dataform.IRunResult; - private readonly changeListeners: Array<(graph: dataform.IRunResult) => void> = []; + private readonly runResult: sqlanvil.IRunResult; + private readonly changeListeners: Array<(graph: sqlanvil.IRunResult) => void> = []; private readonly eEmitter: EventEmitter; private executedActionTargets: Set; private successfullyExecutedActionTargets: Set; - private pendingActions: dataform.IExecutionAction[]; + private pendingActions: sqlanvil.IExecutionAction[]; private lastNotificationTimestampMillis = 0; private stopped = false; private cancelled = false; private timeout: NodeJS.Timer; private timedOut = false; - private executionTask: Promise; + private executionTask: Promise; constructor( private readonly dbadapter: dbadapters.IDbAdapter, - private readonly graph: dataform.IExecutionGraph, + private readonly graph: sqlanvil.IExecutionGraph, private readonly executionOptions: IExecutionOptions = {}, - partiallyExecutedRunResult: dataform.IRunResult = {}, + partiallyExecutedRunResult: sqlanvil.IRunResult = {}, private readonly runnerNotificationPeriodMillis: number = flags.runnerNotificationPeriodMillis.get() ) { this.allActionTargets = new Set( @@ -79,7 +79,7 @@ export class Runner { actions: [], ...partiallyExecutedRunResult }; - this.warehouseStateByTarget = new Map(); + this.warehouseStateByTarget = new Map(); graph.warehouseState.tables?.forEach(tableMetadata => this.warehouseStateByTarget.set( targetStringifier.stringify(tableMetadata.target), @@ -88,7 +88,7 @@ export class Runner { ); this.executedActionTargets = new Set( this.runResult.actions - .filter(action => action.status !== dataform.ActionResult.ExecutionStatus.RUNNING) + .filter(action => action.status !== sqlanvil.ActionResult.ExecutionStatus.RUNNING) .map(action => targetStringifier.stringify(action.target)) ); this.successfullyExecutedActionTargets = new Set( @@ -104,7 +104,7 @@ export class Runner { this.eEmitter.setMaxListeners(0); } - public onChange(listener: (graph: dataform.IRunResult) => void): Runner { + public onChange(listener: (graph: sqlanvil.IRunResult) => void): Runner { this.changeListeners.push(listener); return this; } @@ -136,7 +136,7 @@ export class Runner { this.eEmitter.emit(CANCEL_EVENT, undefined, undefined); } - public async result(): Promise { + public async result(): Promise { try { return await this.executionTask; } finally { @@ -150,7 +150,7 @@ export class Runner { if (Date.now() - this.runnerNotificationPeriodMillis < this.lastNotificationTimestampMillis) { return; } - const runResultClone = deepClone(dataform.RunResult, this.runResult); + const runResultClone = deepClone(sqlanvil.RunResult, this.runResult); this.lastNotificationTimestampMillis = Date.now(); this.changeListeners.forEach(listener => listener(runResultClone)); } @@ -158,7 +158,7 @@ export class Runner { private async executeGraph() { const timer = Timer.start(this.runResult.timing); - this.runResult.status = dataform.RunResult.ExecutionStatus.RUNNING; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.RUNNING; this.runResult.timing = timer.current(); this.notifyListeners(); @@ -176,17 +176,17 @@ export class Runner { this.runResult.timing = timer.end(); - this.runResult.status = dataform.RunResult.ExecutionStatus.SUCCESSFUL; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL; if (this.timedOut) { - this.runResult.status = dataform.RunResult.ExecutionStatus.TIMED_OUT; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.TIMED_OUT; } else if (this.cancelled) { - this.runResult.status = dataform.RunResult.ExecutionStatus.CANCELLED; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.CANCELLED; } else if ( this.runResult.actions.some( - action => action.status === dataform.ActionResult.ExecutionStatus.FAILED + action => action.status === sqlanvil.ActionResult.ExecutionStatus.FAILED ) ) { - this.runResult.status = dataform.RunResult.ExecutionStatus.FAILED; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.FAILED; } return this.runResult; @@ -198,7 +198,7 @@ export class Runner { this.graph.actions .filter(action => !!action.target && !!action.target.schema) .forEach(({ target }) => { - // This field may not be present for older versions of dataform. + // This field may not be present for older versions of sqlanvil. const trueDatabase = target.database || this.graph.projectConfig.defaultDatabase; if (!databaseSchemas.has(trueDatabase)) { databaseSchemas.set(trueDatabase, new Set()); @@ -231,9 +231,9 @@ export class Runner { allPendingActions.forEach(pendingAction => this.runResult.actions.push({ target: pendingAction.target, - status: dataform.ActionResult.ExecutionStatus.SKIPPED, + status: sqlanvil.ActionResult.ExecutionStatus.SKIPPED, tasks: pendingAction.tasks.map(() => ({ - status: dataform.TaskResult.ExecutionStatus.SKIPPED + status: sqlanvil.TaskResult.ExecutionStatus.SKIPPED })) }) ); @@ -277,9 +277,9 @@ export class Runner { skippableActions.forEach(skippableAction => { this.runResult.actions.push({ target: skippableAction.target, - status: dataform.ActionResult.ExecutionStatus.SKIPPED, + status: sqlanvil.ActionResult.ExecutionStatus.SKIPPED, tasks: skippableAction.tasks.map(() => ({ - status: dataform.TaskResult.ExecutionStatus.SKIPPED + status: sqlanvil.TaskResult.ExecutionStatus.SKIPPED })) }); }); @@ -303,28 +303,28 @@ export class Runner { ]); } - private async executeAction(action: dataform.IExecutionAction): Promise { - let actionResult: dataform.IActionResult = { + private async executeAction(action: sqlanvil.IExecutionAction): Promise { + let actionResult: sqlanvil.IActionResult = { target: action.target, tasks: [] }; if (action.tasks.length === 0) { - actionResult.status = dataform.ActionResult.ExecutionStatus.DISABLED; + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.DISABLED; this.runResult.actions.push(actionResult); this.notifyListeners(); return actionResult; } const resumedActionResult = this.runResult.actions.find(existingActionResult => - equals(dataform.Target, existingActionResult.target, action.target) + equals(sqlanvil.Target, existingActionResult.target, action.target) ); if (resumedActionResult) { actionResult = resumedActionResult; } else { this.runResult.actions.push(actionResult); } - actionResult.status = dataform.ActionResult.ExecutionStatus.RUNNING; + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.RUNNING; const timer = Timer.start(resumedActionResult?.timing); actionResult.timing = timer.current(); this.notifyListeners(); @@ -336,7 +336,7 @@ export class Runner { return actionResult; } if ( - actionResult.status === dataform.ActionResult.ExecutionStatus.RUNNING && + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.RUNNING && !this.cancelled ) { const taskStatus = await this.executeTask(client, task, actionResult, { @@ -354,14 +354,14 @@ export class Runner { this.graph.projectConfig?.defaultReservation } }); - if (taskStatus === dataform.TaskResult.ExecutionStatus.FAILED) { - actionResult.status = dataform.ActionResult.ExecutionStatus.FAILED; - } else if (taskStatus === dataform.TaskResult.ExecutionStatus.CANCELLED) { - actionResult.status = dataform.ActionResult.ExecutionStatus.CANCELLED; + if (taskStatus === sqlanvil.TaskResult.ExecutionStatus.FAILED) { + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.FAILED; + } else if (taskStatus === sqlanvil.TaskResult.ExecutionStatus.CANCELLED) { + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.CANCELLED; } } else { actionResult.tasks.push({ - status: dataform.TaskResult.ExecutionStatus.SKIPPED + status: sqlanvil.TaskResult.ExecutionStatus.SKIPPED }); } } @@ -375,7 +375,7 @@ export class Runner { action.actionDescriptor && // Only set metadata if we expect the action to complete in SUCCESSFUL state // (i.e. it must still be RUNNING, and not FAILED). - actionResult.status === dataform.ActionResult.ExecutionStatus.RUNNING && + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.RUNNING && !(this.graph.runConfig && this.graph.runConfig.disableSetMetadata) && // Only set metadata if not using BigQuery dry run !this.executionOptions.bigquery?.dryRun && @@ -392,16 +392,16 @@ export class Runner { actionResult.tasks.length - 1 ].errorMessage = `Error setting metadata: ${e.message}`; actionResult.tasks[actionResult.tasks.length - 1].status = - dataform.TaskResult.ExecutionStatus.FAILED; + sqlanvil.TaskResult.ExecutionStatus.FAILED; } - actionResult.status = dataform.ActionResult.ExecutionStatus.FAILED; + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.FAILED; } } this.warehouseStateByTarget.delete(targetStringifier.stringify(action.target)); - if (actionResult.status === dataform.ActionResult.ExecutionStatus.RUNNING) { - actionResult.status = dataform.ActionResult.ExecutionStatus.SUCCESSFUL; + if (actionResult.status === sqlanvil.ActionResult.ExecutionStatus.RUNNING) { + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL; } actionResult.timing = timer.end(); @@ -411,20 +411,20 @@ export class Runner { private async executeTask( client: dbadapters.IDbClient, - task: dataform.IExecutionTask, - parentAction: dataform.IActionResult, - options: { bigquery?: dataform.IBigQueryOptions & IBigQueryExecutionOptions } - ): Promise { + task: sqlanvil.IExecutionTask, + parentAction: sqlanvil.IActionResult, + options: { bigquery?: sqlanvil.IBigQueryOptions & IBigQueryExecutionOptions } + ): Promise { const timer = Timer.start(); - const taskResult: dataform.ITaskResult = { - status: dataform.TaskResult.ExecutionStatus.RUNNING, + const taskResult: sqlanvil.ITaskResult = { + status: sqlanvil.TaskResult.ExecutionStatus.RUNNING, timing: timer.current(), metadata: {} }; parentAction.tasks.push(taskResult); this.notifyListeners(); if (options.bigquery?.dryRun && task.type === "assertion") { - taskResult.status = dataform.TaskResult.ExecutionStatus.SUCCESSFUL; + taskResult.status = sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL; } else { try { @@ -447,11 +447,11 @@ export class Runner { throw new Error(`Assertion failed: query returned ${rowCount} row(s).`); } } - taskResult.status = dataform.TaskResult.ExecutionStatus.SUCCESSFUL; + taskResult.status = sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL; } catch (e) { taskResult.status = this.cancelled - ? dataform.TaskResult.ExecutionStatus.CANCELLED - : dataform.TaskResult.ExecutionStatus.FAILED; + ? sqlanvil.TaskResult.ExecutionStatus.CANCELLED + : sqlanvil.TaskResult.ExecutionStatus.FAILED; taskResult.errorMessage = `${this.graph.projectConfig.warehouse} error: ${e.message}`; if (e.metadata?.bigquery?.jobId) { taskResult.metadata = { @@ -469,18 +469,18 @@ export class Runner { } class Timer { - public static start(existingTiming?: dataform.ITiming) { + public static start(existingTiming?: sqlanvil.ITiming) { return new Timer(existingTiming?.startTimeMillis.toNumber() || new Date().valueOf()); } private constructor(readonly startTimeMillis: number) { } - public current(): dataform.ITiming { + public current(): sqlanvil.ITiming { return { startTimeMillis: Long.fromNumber(this.startTimeMillis) }; } - public end(): dataform.ITiming { + public end(): sqlanvil.ITiming { return { startTimeMillis: Long.fromNumber(this.startTimeMillis), endTimeMillis: Long.fromNumber(new Date().valueOf()) diff --git a/cli/api/commands/state.ts b/cli/api/commands/state.ts index 94f5fe8a..dfce4dda 100644 --- a/cli/api/commands/state.ts +++ b/cli/api/commands/state.ts @@ -1,10 +1,10 @@ -import { IDbAdapter } from "df/cli/api/dbadapters"; -import { dataform } from "df/protos/ts"; +import { IDbAdapter } from "sa/cli/api/dbadapters"; +import { sqlanvil } from "sa/protos/ts"; export async function state( dbadapter: IDbAdapter, - targets: dataform.ITarget[] -): Promise { + targets: sqlanvil.ITarget[] +): Promise { const allTables = await Promise.all(targets.map(async target => dbadapter.table(target))); // Filter out datasets that don't exist. diff --git a/cli/api/commands/test.ts b/cli/api/commands/test.ts index 03330c26..6d6d2f35 100644 --- a/cli/api/commands/test.ts +++ b/cli/api/commands/test.ts @@ -1,17 +1,17 @@ -import * as dbadapters from "df/cli/api/dbadapters"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { sqlanvil } from "sa/protos/ts"; export async function test( dbadapter: dbadapters.IDbAdapter, - tests: dataform.ITest[] -): Promise { + tests: sqlanvil.ITest[] +): Promise { return await Promise.all(tests.map(testCase => runTest(dbadapter, testCase))); } async function runTest( dbadapter: dbadapters.IDbAdapter, - testCase: dataform.ITest -): Promise { + testCase: sqlanvil.ITest +): Promise { // TODO: Test results are currently limited to 1MB. // We should paginate test results to remove this limit. let actualResults; diff --git a/cli/api/dbadapters/bigquery.ts b/cli/api/dbadapters/bigquery.ts index 178c0c7e..aa12d139 100644 --- a/cli/api/dbadapters/bigquery.ts +++ b/cli/api/dbadapters/bigquery.ts @@ -2,7 +2,7 @@ import { BigQuery, GetTablesResponse, TableField, TableMetadata } from "@google- import Long from "long"; import { PromisePoolExecutor } from "promise-pool-executor"; -import { collectEvaluationQueries, QueryOrAction } from "df/cli/api/dbadapters/execution_sql"; +import { collectEvaluationQueries, QueryOrAction } from "sa/cli/api/dbadapters/execution_sql"; import { IBigQueryError, IDbAdapter, @@ -10,12 +10,12 @@ import { IExecutionResult, IExecutionResultRaw, OnCancel -} from "df/cli/api/dbadapters/index"; -import { parseBigqueryEvalError } from "df/cli/api/utils/error_parsing"; -import { LimitedResultSet } from "df/cli/api/utils/results"; -import { coerceAsError } from "df/common/errors/errors"; -import { retry } from "df/common/promises"; -import { dataform } from "df/protos/ts"; +} from "sa/cli/api/dbadapters/index"; +import { parseBigqueryEvalError } from "sa/cli/api/utils/error_parsing"; +import { LimitedResultSet } from "sa/cli/api/utils/results"; +import { coerceAsError } from "sa/common/errors/errors"; +import { retry } from "sa/common/promises"; +import { sqlanvil } from "sa/protos/ts"; const EXTRA_GOOGLE_SCOPES = ["https://www.googleapis.com/auth/drive"]; @@ -38,14 +38,14 @@ export interface IBigQueryExecutionOptions { } export class BigQueryDbAdapter implements IDbAdapter { - private bigQueryCredentials: dataform.IBigQuery; + private bigQueryCredentials: sqlanvil.IBigQuery; private pool: PromisePoolExecutor; private readonly clients = new Map(); private readonly bigqueryClient?: BigQuery; constructor( - credentials: dataform.IBigQuery, + credentials: sqlanvil.IBigQuery, options?: { concurrencyLimit?: number; bigqueryClient?: BigQuery } ) { this.bigQueryCredentials = credentials; @@ -149,14 +149,14 @@ export class BigQueryDbAdapter implements IDbAdapter { }) }) .promise(); - return dataform.QueryEvaluation.create({ - status: dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS, + return sqlanvil.QueryEvaluation.create({ + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS, incremental, query }); } catch (e) { return { - status: dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE, + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE, error: parseBigqueryEvalError(e), incremental, query @@ -166,9 +166,9 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - public async tables(database: string, schema?: string): Promise { + public async tables(database: string, schema?: string): Promise { const datasetIds = schema ? [schema] : await this.schemas(database); - const tablesMetadata: dataform.ITableMetadata[] = []; + const tablesMetadata: sqlanvil.ITableMetadata[] = []; await Promise.all( datasetIds.map(async datasetId => { @@ -196,7 +196,7 @@ export class BigQueryDbAdapter implements IDbAdapter { public async search( searchText: string, options: { limit: number } = { limit: 1000 } - ): Promise { + ): Promise { const results = await this.execute( `select table_catalog, table_schema, table_name from region-${this.bigQueryCredentials.location}.INFORMATION_SCHEMA.COLUMN_FIELD_PATHS @@ -221,7 +221,7 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - public async table(target: dataform.ITarget): Promise { + public async table(target: sqlanvil.ITarget): Promise { const metadata = await this.getMetadata(target); if (!metadata) { @@ -246,13 +246,13 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - return dataform.TableMetadata.create({ + return sqlanvil.TableMetadata.create({ type: metadata.type === "TABLE" - ? dataform.TableMetadata.Type.TABLE + ? sqlanvil.TableMetadata.Type.TABLE : metadata.type === "VIEW" - ? dataform.TableMetadata.Type.VIEW - : dataform.TableMetadata.Type.UNKNOWN, + ? sqlanvil.TableMetadata.Type.VIEW + : sqlanvil.TableMetadata.Type.UNKNOWN, target, fields: metadata.schema.fields?.map(field => convertField(field)), lastUpdatedMillis: Long.fromString(metadata.lastModifiedTime), @@ -264,7 +264,7 @@ export class BigQueryDbAdapter implements IDbAdapter { }); } - public async deleteTable(target: dataform.ITarget): Promise { + public async deleteTable(target: sqlanvil.ITarget): Promise { await this.getClient(target.database) .dataset(target.schema) .table(target.name) @@ -283,7 +283,7 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - public async setMetadata(action: dataform.IExecutionAction): Promise { + public async setMetadata(action: sqlanvil.IExecutionAction): Promise { const { target, actionDescriptor } = action; const metadata = await this.getMetadata(target); @@ -302,7 +302,7 @@ export class BigQueryDbAdapter implements IDbAdapter { }); } - private async getMetadata(target: dataform.ITarget): Promise { + private async getMetadata(target: sqlanvil.ITarget): Promise { try { const table = await this.getClient(target.database) .dataset(target.schema) @@ -379,7 +379,7 @@ export class BigQueryDbAdapter implements IDbAdapter { return { query, useLegacySql: false, - jobPrefix: "dataform-" + (bigqueryOptions?.jobPrefix ? `${bigqueryOptions.jobPrefix}-` : ""), + jobPrefix: "sqlanvil-" + (bigqueryOptions?.jobPrefix ? `${bigqueryOptions.jobPrefix}-` : ""), location: bigqueryOptions?.location, maxResults: rowLimit, labels: bigqueryOptions?.labels, @@ -518,20 +518,20 @@ function cleanRows(rows: any[]) { return rows; } -function convertField(field: TableField): dataform.IField { - const result: dataform.IField = { +function convertField(field: TableField): sqlanvil.IField { + const result: sqlanvil.IField = { name: field.name, - flags: field.mode === "REPEATED" ? [dataform.Field.Flag.REPEATED] : [], + flags: field.mode === "REPEATED" ? [sqlanvil.Field.Flag.REPEATED] : [], description: field.description }; if (field.type === "RECORD" || field.type === "STRUCT") { - result.struct = dataform.Fields.create({ + result.struct = sqlanvil.Fields.create({ fields: field.fields.map(innerField => convertField(innerField)) }); } else { result.primitive = convertFieldType(field.type); } - return dataform.Field.create(result); + return sqlanvil.Field.create(result); } // See: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema @@ -539,42 +539,42 @@ function convertFieldType(type: string) { switch (String(type).toUpperCase()) { case "FLOAT": case "FLOAT64": - return dataform.Field.Primitive.FLOAT; + return sqlanvil.Field.Primitive.FLOAT; case "INTEGER": case "INT64": - return dataform.Field.Primitive.INTEGER; + return sqlanvil.Field.Primitive.INTEGER; case "NUMERIC": - return dataform.Field.Primitive.NUMERIC; + return sqlanvil.Field.Primitive.NUMERIC; case "BOOL": case "BOOLEAN": - return dataform.Field.Primitive.BOOLEAN; + return sqlanvil.Field.Primitive.BOOLEAN; case "STRING": - return dataform.Field.Primitive.STRING; + return sqlanvil.Field.Primitive.STRING; case "DATE": - return dataform.Field.Primitive.DATE; + return sqlanvil.Field.Primitive.DATE; case "DATETIME": - return dataform.Field.Primitive.DATETIME; + return sqlanvil.Field.Primitive.DATETIME; case "TIMESTAMP": - return dataform.Field.Primitive.TIMESTAMP; + return sqlanvil.Field.Primitive.TIMESTAMP; case "TIME": - return dataform.Field.Primitive.TIME; + return sqlanvil.Field.Primitive.TIME; case "BYTES": - return dataform.Field.Primitive.BYTES; + return sqlanvil.Field.Primitive.BYTES; case "GEOGRAPHY": - return dataform.Field.Primitive.GEOGRAPHY; + return sqlanvil.Field.Primitive.GEOGRAPHY; case "BIGNUMERIC": - return dataform.Field.Primitive.BIGNUMERIC; + return sqlanvil.Field.Primitive.BIGNUMERIC; case "JSON": - return dataform.Field.Primitive.JSON; + return sqlanvil.Field.Primitive.JSON; case "INTERVAL": - return dataform.Field.Primitive.INTERVAL; + return sqlanvil.Field.Primitive.INTERVAL; default: - return dataform.Field.Primitive.UNKNOWN; + return sqlanvil.Field.Primitive.UNKNOWN; } } function addDescriptionToMetadata( - columnDescriptions: dataform.IColumnDescriptor[], + columnDescriptions: sqlanvil.IColumnDescriptor[], metadataArray: TableField[] ): TableField[] { if (!columnDescriptions) { diff --git a/cli/api/dbadapters/bigquery_test.ts b/cli/api/dbadapters/bigquery_test.ts index ecff5ed3..986b9d1f 100644 --- a/cli/api/dbadapters/bigquery_test.ts +++ b/cli/api/dbadapters/bigquery_test.ts @@ -2,9 +2,9 @@ import { Dataset, Table } from "@google-cloud/bigquery"; import { expect } from "chai"; import { anything, instance, mock, verify, when } from "ts-mockito"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("BigQueryDbAdapter", () => { test("tables() with schema filters correctly", async () => { @@ -16,7 +16,7 @@ suite("BigQueryDbAdapter", () => { const schemaName = "schema1"; const projectId = "project1"; - const credentials = dataform.BigQuery.create({ projectId, location: "US" }); + const credentials = sqlanvil.BigQuery.create({ projectId, location: "US" }); const adapter = new BigQueryDbAdapter(credentials, { bigqueryClient: instance(mockBigQuery) }); when(mockBigQuery.dataset(schemaName)).thenReturn(instance(mockDataset)); @@ -53,7 +53,7 @@ suite("BigQueryDbAdapter", () => { const tableName = "table1"; const projectId = "project"; - const credentials = dataform.BigQuery.create({ projectId, location: "US" }); + const credentials = sqlanvil.BigQuery.create({ projectId, location: "US" }); const adapter = new BigQueryDbAdapter(credentials, { bigqueryClient: instance(mockBigQuery) }); when(mockBigQuery.dataset(schemaName)).thenReturn(instance(mockDataset)); diff --git a/cli/api/dbadapters/execution_sql.ts b/cli/api/dbadapters/execution_sql.ts index 33d1990e..c627fab8 100644 --- a/cli/api/dbadapters/execution_sql.ts +++ b/cli/api/dbadapters/execution_sql.ts @@ -1,12 +1,12 @@ import * as semver from "semver"; -import { concatenateQueries, Task, Tasks } from "df/cli/api/dbadapters/tasks"; -import { ErrorWithCause } from "df/common/errors/errors"; -import { CompilationSql } from "df/core/compilation_sql"; -import { tableTypeEnumToString } from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { concatenateQueries, Task, Tasks } from "sa/cli/api/dbadapters/tasks"; +import { ErrorWithCause } from "sa/common/errors/errors"; +import { CompilationSql } from "sa/core/compilation_sql"; +import { tableTypeEnumToString } from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; -export type QueryOrAction = string | dataform.Table | dataform.Operation | dataform.Assertion; +export type QueryOrAction = string | sqlanvil.Table | sqlanvil.Operation | sqlanvil.Assertion; export interface IValidationQuery { query?: string; @@ -17,37 +17,37 @@ export class ExecutionSql { private readonly CompilationSql: CompilationSql; constructor( - private readonly project: dataform.IProjectConfig, - private readonly dataformCoreVersion: string, + private readonly project: sqlanvil.IProjectConfig, + private readonly sqlanvilCoreVersion: string, private readonly uniqueIdGenerator: () => string = () => Math.random().toString(36).substring(2) ) { - this.CompilationSql = new CompilationSql(project, dataformCoreVersion); + this.CompilationSql = new CompilationSql(project, sqlanvilCoreVersion); } - public baseTableType(enumType: dataform.TableType) { + public baseTableType(enumType: sqlanvil.TableType) { switch (enumType) { - case dataform.TableType.TABLE: - case dataform.TableType.INCREMENTAL: - return dataform.TableMetadata.Type.TABLE; - case dataform.TableType.VIEW: - return dataform.TableMetadata.Type.VIEW; + case sqlanvil.TableType.TABLE: + case sqlanvil.TableType.INCREMENTAL: + return sqlanvil.TableMetadata.Type.TABLE; + case sqlanvil.TableType.VIEW: + return sqlanvil.TableMetadata.Type.VIEW; default: throw new Error(`Unexpected table type: ${tableTypeEnumToString(enumType)}`); } } - public tableTypeAsSql(type: dataform.TableMetadata.Type) { + public tableTypeAsSql(type: sqlanvil.TableMetadata.Type) { switch (type) { - case dataform.TableMetadata.Type.TABLE: + case sqlanvil.TableMetadata.Type.TABLE: return "table"; - case dataform.TableMetadata.Type.VIEW: + case sqlanvil.TableMetadata.Type.VIEW: return "view"; default: throw new Error(`Unexpected table type: ${type}`); } } - public insertInto(target: dataform.ITarget, columns: string[], query: string) { + public insertInto(target: sqlanvil.ITarget, columns: string[], query: string) { return ` insert into ${this.resolveTarget(target)} (${columns.join(",")}) @@ -55,12 +55,12 @@ select ${columns.join(",")} from (${query}) as insertions`; } - public oppositeTableType(type: dataform.TableMetadata.Type) { + public oppositeTableType(type: sqlanvil.TableMetadata.Type) { switch (type) { - case dataform.TableMetadata.Type.TABLE: - return dataform.TableMetadata.Type.VIEW; - case dataform.TableMetadata.Type.VIEW: - return dataform.TableMetadata.Type.TABLE; + case sqlanvil.TableMetadata.Type.TABLE: + return sqlanvil.TableMetadata.Type.VIEW; + case sqlanvil.TableMetadata.Type.VIEW: + return sqlanvil.TableMetadata.Type.TABLE; default: throw new Error(`Unexpected table type: ${type}`); } @@ -75,26 +75,26 @@ from (${query}) as insertions`; } public shouldWriteIncrementally( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ) { return ( (!runConfig.fullRefresh || table.protected) && tableMetadata && - tableMetadata.type !== dataform.TableMetadata.Type.VIEW + tableMetadata.type !== sqlanvil.TableMetadata.Type.VIEW ); } public preOps( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ): Task[] { let preOps = table.preOps; if ( - semver.gt(this.dataformCoreVersion, "1.4.8") && - table.enumType === dataform.TableType.INCREMENTAL && + semver.gt(this.sqlanvilCoreVersion, "1.4.8") && + table.enumType === sqlanvil.TableType.INCREMENTAL && this.shouldWriteIncrementally(table, runConfig, tableMetadata) ) { preOps = table.incrementalPreOps; @@ -103,14 +103,14 @@ from (${query}) as insertions`; } public postOps( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ): Task[] { let postOps = table.postOps; if ( - semver.gt(this.dataformCoreVersion, "1.4.8") && - table.enumType === dataform.TableType.INCREMENTAL && + semver.gt(this.sqlanvilCoreVersion, "1.4.8") && + table.enumType === sqlanvil.TableType.INCREMENTAL && this.shouldWriteIncrementally(table, runConfig, tableMetadata) ) { postOps = table.incrementalPostOps; @@ -118,18 +118,18 @@ from (${query}) as insertions`; return (postOps || []).map(post => Task.statement(post)); } - public resolveTarget(target: dataform.ITarget) { + public resolveTarget(target: sqlanvil.ITarget) { return this.CompilationSql.resolveTarget(target); } - public getIncrementalQuery(table: dataform.ITable): string { + public getIncrementalQuery(table: sqlanvil.ITable): string { return this.where(table.incrementalQuery || table.query, table.where); } public publishTasks( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ): Tasks { const tasks = new Tasks(); @@ -142,18 +142,18 @@ from (${query}) as insertions`; ); } - if (table.enumType === dataform.TableType.INCREMENTAL) { + if (table.enumType === sqlanvil.TableType.INCREMENTAL) { if (!this.shouldWriteIncrementally(table, runConfig, tableMetadata)) { tasks.add(Task.statement(this.createOrReplace(table))); } else { - const onSchemaChange = table.onSchemaChange ?? dataform.OnSchemaChange.IGNORE; + const onSchemaChange = table.onSchemaChange ?? sqlanvil.OnSchemaChange.IGNORE; switch (onSchemaChange) { - case dataform.OnSchemaChange.FAIL: - case dataform.OnSchemaChange.EXTEND: - case dataform.OnSchemaChange.SYNCHRONIZE: + case sqlanvil.OnSchemaChange.FAIL: + case sqlanvil.OnSchemaChange.EXTEND: + case sqlanvil.OnSchemaChange.SYNCHRONIZE: this.buildIncrementalSchemaChangeTasks(tasks, table); // Fall through to run the static DML after the procedure alters the schema - case dataform.OnSchemaChange.IGNORE: + case sqlanvil.OnSchemaChange.IGNORE: default: tasks.add( Task.statement( @@ -185,8 +185,8 @@ from (${query}) as insertions`; } public assertTasks( - assertion: dataform.IAssertion, - projectConfig: dataform.IProjectConfig, + assertion: sqlanvil.IAssertion, + projectConfig: sqlanvil.IProjectConfig, ): Tasks { const tasks = new Tasks(); const target = assertion.target; @@ -198,11 +198,11 @@ from (${query}) as insertions`; return tasks; } - public dropIfExists(target: dataform.ITarget, type: dataform.TableMetadata.Type) { + public dropIfExists(target: sqlanvil.ITarget, type: sqlanvil.TableMetadata.Type) { return `drop ${this.tableTypeAsSql(type)} if exists ${this.resolveTarget(target)}`; } - private buildIncrementalSchemaChangeTasks(tasks: Tasks, table: dataform.ITable) { + private buildIncrementalSchemaChangeTasks(tasks: Tasks, table: sqlanvil.ITable) { const uniqueId = this.uniqueIdGenerator(); const emptyTempTableTarget = { @@ -231,7 +231,7 @@ END;`; tasks.add(Task.statement(callProcedureSql)); } - private createProcedureName(target: dataform.ITarget, uniqueId: string): string { + private createProcedureName(target: sqlanvil.ITarget, uniqueId: string): string { return this.resolveTarget({ ...target, name: `df_osc_${uniqueId}` @@ -262,8 +262,8 @@ CREATE OR REPLACE TABLE ${emptyTempTableName} AS ( } private compareSchemasSql( - target: dataform.ITarget, - emptyTempTableTarget: dataform.ITarget + target: sqlanvil.ITarget, + emptyTempTableTarget: sqlanvil.ITarget ): string { return ` -- Compare schemas @@ -297,15 +297,15 @@ SET columns_removed = ( } private applySchemaChangeStrategySql( - table: dataform.ITable, + table: sqlanvil.ITable, qualifiedTargetTableName: string ): string { - const onSchemaChange = table.onSchemaChange || dataform.OnSchemaChange.IGNORE; + const onSchemaChange = table.onSchemaChange || sqlanvil.OnSchemaChange.IGNORE; let sql = ` --- Apply schema change strategy (${dataform.OnSchemaChange[onSchemaChange]}).`; +-- Apply schema change strategy (${sqlanvil.OnSchemaChange[onSchemaChange]}).`; switch (onSchemaChange) { - case dataform.OnSchemaChange.FAIL: + case sqlanvil.OnSchemaChange.FAIL: sql += ` IF ARRAY_LENGTH(columns_added) > 0 OR ARRAY_LENGTH(columns_removed) > 0 THEN RAISE USING MESSAGE = FORMAT( @@ -316,7 +316,7 @@ IF ARRAY_LENGTH(columns_added) > 0 OR ARRAY_LENGTH(columns_removed) > 0 THEN END IF; `; break; - case dataform.OnSchemaChange.EXTEND: + case sqlanvil.OnSchemaChange.EXTEND: sql += ` IF ARRAY_LENGTH(columns_removed) > 0 THEN RAISE USING MESSAGE = FORMAT( @@ -328,7 +328,7 @@ END IF; ${this.alterTableAddColumnsSql(qualifiedTargetTableName)} `; break; - case dataform.OnSchemaChange.SYNCHRONIZE: + case sqlanvil.OnSchemaChange.SYNCHRONIZE: const uniqueKeys = table.uniqueKey || []; sql += ` DECLARE invalid_removed_columns ARRAY; @@ -380,9 +380,9 @@ DROP TABLE IF EXISTS ${emptyTempTableName}; } private incrementalSchemaChangeBody( - table: dataform.ITable, + table: sqlanvil.ITable, qualifiedTargetTableName: string, - emptyTempTableTarget: dataform.ITarget + emptyTempTableTarget: sqlanvil.ITarget ): string { const emptyTempTableName = this.resolveTarget(emptyTempTableTarget); const query = this.getIncrementalQuery(table); @@ -399,7 +399,7 @@ DROP TABLE IF EXISTS ${emptyTempTableName}; return statements.join("\n\n"); } - private createOrReplace(table: dataform.ITable) { + private createOrReplace(table: sqlanvil.ITable) { const options = []; if (table.bigquery && table.bigquery.partitionBy && table.bigquery.partitionExpirationDays) { options.push(`partition_expiration_days=${table.bigquery.partitionExpirationDays}`); @@ -426,13 +426,13 @@ DROP TABLE IF EXISTS ${emptyTempTableName}; }${options.length > 0 ? `OPTIONS(${options.join(",")})` : ""}as ${table.query}`; } - private createOrReplaceView(target: dataform.ITarget, query: string) { + private createOrReplaceView(target: sqlanvil.ITarget, query: string) { return ` create or replace view ${this.resolveTarget(target)} as ${query}`; } private mergeInto( - target: dataform.ITarget, + target: sqlanvil.ITarget, columns: string[], query: string, uniqueKey: string[], @@ -470,8 +470,8 @@ export function collectEvaluationQueries( validationQueries.push({ query: queryModifier(queryOrAction) }); } else { try { - if (queryOrAction instanceof dataform.Table) { - if (queryOrAction.enumType === dataform.TableType.INCREMENTAL) { + if (queryOrAction instanceof sqlanvil.Table) { + if (queryOrAction.enumType === sqlanvil.TableType.INCREMENTAL) { const incrementalTableQueries = queryOrAction.incrementalPreOps.concat( queryOrAction.incrementalQuery, queryOrAction.incrementalPostOps @@ -498,7 +498,7 @@ export function collectEvaluationQueries( } else { tableQueries.forEach(q => validationQueries.push({ query: queryModifier(q) })); } - } else if (queryOrAction instanceof dataform.Operation) { + } else if (queryOrAction instanceof sqlanvil.Operation) { if (concatenate) { validationQueries.push({ query: concatenateQueries(queryOrAction.queries, queryModifier) @@ -506,7 +506,7 @@ export function collectEvaluationQueries( } else { queryOrAction.queries.forEach(q => validationQueries.push({ query: queryModifier(q) })); } - } else if (queryOrAction instanceof dataform.Assertion) { + } else if (queryOrAction instanceof sqlanvil.Assertion) { validationQueries.push({ query: queryModifier(queryOrAction.query) }); } else { throw new Error("Unrecognized evaluate type."); diff --git a/cli/api/dbadapters/index.ts b/cli/api/dbadapters/index.ts index 380f1dbf..2da2905a 100644 --- a/cli/api/dbadapters/index.ts +++ b/cli/api/dbadapters/index.ts @@ -1,19 +1,19 @@ -import { QueryOrAction } from "df/cli/api/dbadapters/execution_sql"; -import { dataform } from "df/protos/ts"; +import { QueryOrAction } from "sa/cli/api/dbadapters/execution_sql"; +import { sqlanvil } from "sa/protos/ts"; export type OnCancel = (handleCancel: () => void) => void; export interface IExecutionResult { rows: any[]; - metadata: dataform.IExecutionMetadata; + metadata: sqlanvil.IExecutionMetadata; } export interface IExecutionResultRaw extends IExecutionResult { - schema?: dataform.IField[]; + schema?: sqlanvil.IField[]; } export interface IBigQueryError extends Error { - metadata?: dataform.IExecutionMetadata + metadata?: sqlanvil.IExecutionMetadata } export interface IDbClient { @@ -53,15 +53,15 @@ export interface IDbClient { export interface IDbAdapter extends IDbClient { withClientLock(callback: (client: IDbClient) => Promise): Promise; - evaluate(queryOrAction: QueryOrAction): Promise; + evaluate(queryOrAction: QueryOrAction): Promise; schemas(database: string): Promise; createSchema(database: string, schema: string): Promise; - tables(database: string, schema?: string): Promise; - search(searchText: string, options?: { limit: number }): Promise; - table(target: dataform.ITarget): Promise; - deleteTable(target: dataform.ITarget): Promise; + tables(database: string, schema?: string): Promise; + search(searchText: string, options?: { limit: number }): Promise; + table(target: sqlanvil.ITarget): Promise; + deleteTable(target: sqlanvil.ITarget): Promise; - setMetadata(action: dataform.IExecutionAction): Promise; + setMetadata(action: sqlanvil.IExecutionAction): Promise; } diff --git a/cli/api/dbadapters/tasks.ts b/cli/api/dbadapters/tasks.ts index 8991e5be..4edf3db8 100644 --- a/cli/api/dbadapters/tasks.ts +++ b/cli/api/dbadapters/tasks.ts @@ -1,4 +1,4 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; export function concatenateQueries(statements: string[], modifier?: (mod: string) => string) { return statements @@ -45,7 +45,7 @@ export class Task { public static assertion(statement: string) { return new Task().type("assertion").statement(statement); } - private proto: dataform.IExecutionTask = dataform.ExecutionTask.create(); + private proto: sqlanvil.IExecutionTask = sqlanvil.ExecutionTask.create(); public type(v: string) { this.proto.type = v; @@ -62,6 +62,6 @@ export class Task { } public build() { - return dataform.ExecutionTask.create(this.proto); + return sqlanvil.ExecutionTask.create(this.proto); } } diff --git a/cli/api/execution_sql_test.ts b/cli/api/execution_sql_test.ts index 4a8cf556..d84a44fa 100644 --- a/cli/api/execution_sql_test.ts +++ b/cli/api/execution_sql_test.ts @@ -1,9 +1,9 @@ import { expect } from "chai"; import * as fs from "fs-extra"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("ExecutionSql with 'onSchemaChange'", () => { const executionSql = new ExecutionSql( @@ -15,9 +15,9 @@ suite("ExecutionSql with 'onSchemaChange'", () => { () => "test_uuid" ); - const baseTable: dataform.ITable = { + const baseTable: sqlanvil.ITable = { type: "incremental", - enumType: dataform.TableType.INCREMENTAL, + enumType: sqlanvil.TableType.INCREMENTAL, target: { database: "project-id", schema: "dataset-id", @@ -27,16 +27,16 @@ suite("ExecutionSql with 'onSchemaChange'", () => { incrementalQuery: "select 1 as id, 'a' as field1, 'new' as field2" }; - const tableMetadata: dataform.ITableMetadata = { - type: dataform.TableMetadata.Type.TABLE, + const tableMetadata: sqlanvil.ITableMetadata = { + type: sqlanvil.TableMetadata.Type.TABLE, fields: [ { name: "id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }, { name: "field1", - primitive: dataform.Field.Primitive.STRING + primitive: sqlanvil.Field.Primitive.STRING } ] }; @@ -44,7 +44,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates procedure for FAIL strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.FAIL + onSchemaChange: sqlanvil.OnSchemaChange.FAIL }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); const procedureSql = tasks.build().map(t => t.statement).join("\n;\n"); @@ -55,7 +55,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates procedure for EXTEND strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.EXTEND + onSchemaChange: sqlanvil.OnSchemaChange.EXTEND }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); const procedureSql = tasks.build().map(t => t.statement).join("\n;\n"); @@ -66,7 +66,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates procedure for SYNCHRONIZE strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.SYNCHRONIZE, + onSchemaChange: sqlanvil.OnSchemaChange.SYNCHRONIZE, uniqueKey: ["id"] }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); @@ -78,7 +78,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates simple merge for IGNORE strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.IGNORE, + onSchemaChange: sqlanvil.OnSchemaChange.IGNORE, uniqueKey: ["id"] }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); diff --git a/cli/api/index.ts b/cli/api/index.ts index c4e85f79..fe74588f 100644 --- a/cli/api/index.ts +++ b/cli/api/index.ts @@ -1,11 +1,11 @@ -import { build, Builder } from "df/cli/api/commands/build"; -import { compile } from "df/cli/api/commands/compile"; -import * as credentials from "df/cli/api/commands/credentials"; -import { init } from "df/cli/api/commands/init"; -import { install } from "df/cli/api/commands/install"; -import { prune } from "df/cli/api/commands/prune"; -import * as query from "df/cli/api/commands/query"; -import { run, Runner } from "df/cli/api/commands/run"; -import { test } from "df/cli/api/commands/test"; +import { build, Builder } from "sa/cli/api/commands/build"; +import { compile } from "sa/cli/api/commands/compile"; +import * as credentials from "sa/cli/api/commands/credentials"; +import { init } from "sa/cli/api/commands/init"; +import { install } from "sa/cli/api/commands/install"; +import { prune } from "sa/cli/api/commands/prune"; +import * as query from "sa/cli/api/commands/query"; +import { run, Runner } from "sa/cli/api/commands/run"; +import { test } from "sa/cli/api/commands/test"; export { init, install, credentials, compile, test, build, run, query, Runner, Builder, prune }; diff --git a/cli/api/utils.ts b/cli/api/utils.ts index b70a641e..1fbe553d 100644 --- a/cli/api/utils.ts +++ b/cli/api/utils.ts @@ -2,7 +2,7 @@ import * as fs from "fs-extra"; import { load as loadYaml, YAMLException } from "js-yaml"; import * as path from "path"; -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; export function prettyJsonStringify(obj: object) { return JSON.stringify(obj, null, 4) + "\n"; @@ -11,12 +11,12 @@ export function prettyJsonStringify(obj: object) { export function readDataformCoreVersionFromWorkflowSettings( resolvedProjectPath: string ): string | undefined { - return readConfigFromWorkflowSettings(resolvedProjectPath)?.dataformCoreVersion; + return readConfigFromWorkflowSettings(resolvedProjectPath)?.sqlanvilCoreVersion; } export function readConfigFromWorkflowSettings( resolvedProjectPath: string -): dataform.WorkflowSettings | undefined { +): sqlanvil.WorkflowSettings | undefined { const workflowSettingsPath = path.join(resolvedProjectPath, "workflow_settings.yaml"); if (!fs.existsSync(workflowSettingsPath)) { return; @@ -32,5 +32,5 @@ export function readConfigFromWorkflowSettings( } throw e; } - return dataform.WorkflowSettings.create(workflowSettingsAsJson); + return sqlanvil.WorkflowSettings.create(workflowSettingsAsJson); } diff --git a/cli/api/utils/error_parsing.ts b/cli/api/utils/error_parsing.ts index 551f2fad..e8876bb7 100644 --- a/cli/api/utils/error_parsing.ts +++ b/cli/api/utils/error_parsing.ts @@ -1,4 +1,4 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; interface IBigqueryEvaluationError { message?: string; @@ -7,7 +7,7 @@ interface IBigqueryEvaluationError { export function parseBigqueryEvalError(error: IBigqueryEvaluationError) { // expected error format: // e.message = Syntax error: Unexpected identifier "asda" at [2:1] - const evalError = dataform.QueryEvaluationError.create({ + const evalError = sqlanvil.QueryEvaluationError.create({ message: String(error) }); try { diff --git a/cli/api/utils/graphs.ts b/cli/api/utils/graphs.ts index f483eee0..aa66656a 100644 --- a/cli/api/utils/graphs.ts +++ b/cli/api/utils/graphs.ts @@ -1,24 +1,24 @@ -import { targetStringifier } from "df/core/targets"; -import { dataform } from "df/protos/ts"; +import { targetStringifier } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; type CoreProtoActionTypes = - | dataform.ITable - | dataform.IOperation - | dataform.IAssertion - | dataform.IDeclaration - | dataform.IDataPreparation; + | sqlanvil.ITable + | sqlanvil.IOperation + | sqlanvil.IAssertion + | sqlanvil.IDeclaration + | sqlanvil.IDataPreparation; -function combineAllActions(graph: dataform.ICompiledGraph) { +function combineAllActions(graph: sqlanvil.ICompiledGraph) { return ([] as CoreProtoActionTypes[]).concat( - graph.tables || ([] as dataform.ITable[]), - graph.operations || ([] as dataform.IOperation[]), - graph.assertions || ([] as dataform.IAssertion[]), - graph.declarations || ([] as dataform.IDeclaration[]), - graph.dataPreparations || ([] as dataform.IDataPreparation[]) + graph.tables || ([] as sqlanvil.ITable[]), + graph.operations || ([] as sqlanvil.IOperation[]), + graph.assertions || ([] as sqlanvil.IAssertion[]), + graph.declarations || ([] as sqlanvil.IDeclaration[]), + graph.dataPreparations || ([] as sqlanvil.IDataPreparation[]) ); } -export function actionsByTarget(compiledGraph: dataform.ICompiledGraph) { +export function actionsByTarget(compiledGraph: sqlanvil.ICompiledGraph) { const actionsMap = new Map(); combineAllActions(compiledGraph) // Required for backwards compatibility with old versions of @sqlanvil/core. @@ -28,7 +28,7 @@ export function actionsByTarget(compiledGraph: dataform.ICompiledGraph) { }); } -export function actionsByCanonicalTarget(compiledGraph: dataform.ICompiledGraph) { +export function actionsByCanonicalTarget(compiledGraph: sqlanvil.ICompiledGraph) { const actionsMap = new Map(); combineAllActions(compiledGraph) // Required for backwards compatibility with old versions of @sqlanvil/core. diff --git a/cli/api/utils_test.ts b/cli/api/utils_test.ts index 09926ede..ac90a6b0 100644 --- a/cli/api/utils_test.ts +++ b/cli/api/utils_test.ts @@ -3,10 +3,10 @@ import * as fs from "fs-extra"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { readConfigFromWorkflowSettings } from "df/cli/api/utils"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { readConfigFromWorkflowSettings } from "sa/cli/api/utils"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -34,7 +34,7 @@ suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), dumpYaml({ - dataformCoreVersion: "3.0.0", + sqlanvilCoreVersion: "3.0.0", defaultProject: "dataform", extension: { name: "test-extension", @@ -48,7 +48,7 @@ suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { if (typeof mode === "string") { expect(mode).to.equal("PROLOGUE"); } else { - expect(mode).to.equal(dataform.ExtensionCompilationMode.PROLOGUE); + expect(mode).to.equal(sqlanvil.ExtensionCompilationMode.PROLOGUE); } }); diff --git a/cli/console.ts b/cli/console.ts index 9b084eb6..3e410a32 100644 --- a/cli/console.ts +++ b/cli/console.ts @@ -1,10 +1,10 @@ import * as readlineSync from "readline-sync"; -import { IInitResult } from "df/cli/api/commands/init"; -import { prettyJsonStringify } from "df/cli/api/utils"; -import { formatBytesInHumanReadableFormat, formatExecutionSuffix } from "df/cli/util"; -import { setOrValidateTableEnumType, tableTypeEnumToString } from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { IInitResult } from "sa/cli/api/commands/init"; +import { prettyJsonStringify } from "sa/cli/api/utils"; +import { formatBytesInHumanReadableFormat, formatExecutionSuffix } from "sa/cli/util"; +import { setOrValidateTableEnumType, tableTypeEnumToString } from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; // Support disabling colors in CLI output by using informal standard from https://no-color.org/ // NO_COLOR=1, NO_COLOR=true, NO_COLOR=yes @@ -187,7 +187,7 @@ export enum compiledGraphOutputType { Summary = "summary" } -export function printCompiledGraph(graph: dataform.ICompiledGraph, outputType: compiledGraphOutputType, quietCompilation: boolean) { +export function printCompiledGraph(graph: sqlanvil.ICompiledGraph, outputType: compiledGraphOutputType, quietCompilation: boolean) { const interactive = isInteractive(); @@ -237,7 +237,7 @@ export function printCompiledGraph(graph: dataform.ICompiledGraph, outputType: c } } -function formatStackTraceForQuietCompilation(compileError: dataform.ICompilationError): string { +function formatStackTraceForQuietCompilation(compileError: sqlanvil.ICompilationError): string { // Show only first 3 or available lines for cleaner error output // which contains the information on the file where the error occurred and the sufficient metadata for the user to fix the error. For e.g. // (line: 1) Unexpected identifier : @@ -253,7 +253,7 @@ function formatStackTraceForQuietCompilation(compileError: dataform.ICompilation return ""; } -export function printCompiledGraphErrors(graphErrors: dataform.IGraphErrors, quietCompilation: boolean) { +export function printCompiledGraphErrors(graphErrors: sqlanvil.IGraphErrors, quietCompilation: boolean) { if (graphErrors.compilationErrors && graphErrors.compilationErrors.length > 0) { printError("Compilation errors:", 1); graphErrors.compilationErrors.forEach(compileError => { @@ -267,7 +267,7 @@ export function printCompiledGraphErrors(graphErrors: dataform.IGraphErrors, qui } } -export function printTestResult(testResult: dataform.ITestResult) { +export function printTestResult(testResult: sqlanvil.ITestResult) { writeStdOut( `${testResult.name}: ${testResult.successful ? successOutput("passed") : errorOutput("failed")}` ); @@ -276,14 +276,14 @@ export function printTestResult(testResult: dataform.ITestResult) { } } -export function printExecutionGraph(executionGraph: dataform.ExecutionGraph, asJson: boolean) { +export function printExecutionGraph(executionGraph: sqlanvil.ExecutionGraph, asJson: boolean) { if (asJson) { writeStdOut(prettyJsonStringify(executionGraph.toJSON())); } else { const actionsByType = { - table: [] as dataform.IExecutionAction[], - assertion: [] as dataform.IExecutionAction[], - operation: [] as dataform.IExecutionAction[] + table: [] as sqlanvil.IExecutionAction[], + assertion: [] as sqlanvil.IExecutionAction[], + operation: [] as sqlanvil.IExecutionAction[] }; executionGraph.actions.forEach(action => { if ( @@ -321,8 +321,8 @@ export function printExecutionGraph(executionGraph: dataform.ExecutionGraph, asJ } export function printExecutedAction( - executedAction: dataform.IActionResult, - executionAction: dataform.IExecutionAction, + executedAction: sqlanvil.IActionResult, + executionAction: sqlanvil.IExecutionAction, dryRun?: boolean ) { const jobIds = executedAction.tasks @@ -338,7 +338,7 @@ export function printExecutedAction( const executionSuffix = formatExecutionSuffix(jobIds, bytesBilled); switch (executedAction.status) { - case dataform.ActionResult.ExecutionStatus.SUCCESSFUL: { + case sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL: { switch (executionAction.type) { case "table": { writeStdOut( @@ -374,7 +374,7 @@ export function printExecutedAction( } } } - case dataform.ActionResult.ExecutionStatus.FAILED: { + case sqlanvil.ActionResult.ExecutionStatus.FAILED: { switch (executionAction.type) { case "table": { writeStdErr( @@ -408,7 +408,7 @@ export function printExecutedAction( printExecutedActionErrors(executedAction, executionAction); return; } - case dataform.ActionResult.ExecutionStatus.SKIPPED: { + case sqlanvil.ActionResult.ExecutionStatus.SKIPPED: { switch (executionAction.type) { case "table": { writeStdOut( @@ -441,7 +441,7 @@ export function printExecutedAction( } return; } - case dataform.ActionResult.ExecutionStatus.DISABLED: { + case sqlanvil.ActionResult.ExecutionStatus.DISABLED: { switch (executionAction.type) { case "table": { writeStdOut( @@ -507,27 +507,27 @@ export function printFormatFilesResult( } } -function datasetString(target: dataform.ITarget, datasetType: string, disabled: boolean) { +function datasetString(target: sqlanvil.ITarget, datasetType: string, disabled: boolean) { return `${targetString(target)} [${datasetType}]${disabled ? " [disabled]" : ""}`; } -function assertionString(target: dataform.ITarget, disabled: boolean) { +function assertionString(target: sqlanvil.ITarget, disabled: boolean) { return `${targetString(target)}${disabled ? " [disabled]" : ""}`; } -function operationString(target: dataform.ITarget, disabled: boolean) { +function operationString(target: sqlanvil.ITarget, disabled: boolean) { return `${targetString(target)}${disabled ? " [disabled]" : ""}`; } -function plainTargetString(target: dataform.ITarget) { +function plainTargetString(target: sqlanvil.ITarget) { return `${target.schema}.${target.name}`; } -function targetString(target: dataform.ITarget) { +function targetString(target: sqlanvil.ITarget) { return calloutOutput(`${target.schema}.${target.name}`); } -export function dotRepresentation(graph: dataform.ICompiledGraph, interactive: boolean): string { +export function dotRepresentation(graph: sqlanvil.ICompiledGraph, interactive: boolean): string { const nodes: string[] = []; const edges: string[] = []; @@ -561,11 +561,11 @@ export function dotRepresentation(graph: dataform.ICompiledGraph, interactive: b } function printExecutedActionErrors( - executedAction: dataform.IActionResult, - executionAction: dataform.IExecutionAction + executedAction: sqlanvil.IActionResult, + executionAction: sqlanvil.IExecutionAction ) { const failingTasks = executedAction.tasks.filter( - task => task.status === dataform.TaskResult.ExecutionStatus.FAILED + task => task.status === sqlanvil.TaskResult.ExecutionStatus.FAILED ); failingTasks.forEach((task, i) => { executionAction.tasks[i].statement.split("\n").forEach(line => { diff --git a/cli/credentials.ts b/cli/credentials.ts index 12e55c5e..4abfae3b 100644 --- a/cli/credentials.ts +++ b/cli/credentials.ts @@ -1,10 +1,10 @@ import * as fs from "fs"; -import { question, selectionQuestion } from "df/cli/console"; -import { actuallyResolve } from "df/cli/util"; -import { dataform } from "df/protos/ts"; +import { question, selectionQuestion } from "sa/cli/console"; +import { actuallyResolve } from "sa/cli/util"; +import { sqlanvil } from "sa/protos/ts"; -export function getBigQueryCredentials(): dataform.IBigQuery { +export function getBigQueryCredentials(): sqlanvil.IBigQuery { const locationIndex = selectionQuestion("Enter the location of your datasets:", [ "US (default)", "EU", @@ -27,7 +27,7 @@ export function getBigQueryCredentials(): dataform.IBigQuery { } const cloudCredentialsPath = actuallyResolve( question( - "Please follow the instructions at https://docs.dataform.co/dataform-cli#create-a-credentials-file/\n" + + "Please follow the instructions at https://docs.sqlanvil.co/dataform-cli#create-a-credentials-file/\n" + "to create and download a private key from the Google Cloud Console in JSON format.\n" + "(You can delete this file after credential initialization is complete.)\n\n" + "Enter the path to your Google Cloud private key file:" diff --git a/cli/index.ts b/cli/index.ts index 452d71f5..07443d11 100644 --- a/cli/index.ts +++ b/cli/index.ts @@ -5,10 +5,10 @@ import parseDuration from "parse-duration"; import * as path from "path"; import yargs from "yargs"; -import { build, compile, credentials, init, install, run, test } from "df/cli/api"; -import { CREDENTIALS_FILENAME } from "df/cli/api/commands/credentials"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { prettyJsonStringify } from "df/cli/api/utils"; +import { build, compile, credentials, init, install, run, test } from "sa/cli/api"; +import { CREDENTIALS_FILENAME } from "sa/cli/api/commands/credentials"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { prettyJsonStringify } from "sa/cli/api/utils"; import { compiledGraphOutputType, print, @@ -22,18 +22,18 @@ import { printInitResult, printSuccess, printTestResult -} from "df/cli/console"; -import { getBigQueryCredentials } from "df/cli/credentials"; +} from "sa/cli/console"; +import { getBigQueryCredentials } from "sa/cli/credentials"; import { actuallyResolve, assertPathExists, compiledGraphHasErrors, promptForIcebergConfig, -} from "df/cli/util"; -import { createYargsCli, INamedOption } from "df/cli/yargswrapper"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { formatFile } from "df/sqlx/format"; +} from "sa/cli/util"; +import { createYargsCli, INamedOption } from "sa/cli/yargswrapper"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { formatFile } from "sa/sqlx/format"; const RECOMPILE_DELAY = 500; @@ -295,7 +295,7 @@ export function runCli() { options: [icebergOption], processFn: async argv => { const projectDir = argv[projectDirOption.name]; - const projectConfig: dataform.IProjectConfig = { + const projectConfig: sqlanvil.IProjectConfig = { defaultDatabase: argv[ProjectConfigOptions.defaultDatabase.name], defaultLocation: argv[ProjectConfigOptions.defaultLocation.name], }; @@ -653,7 +653,7 @@ export function runCli() { bigqueryOptions = { ...bigqueryOptions, labels: argv[bigqueryJobLabelsOption.name] }; } - const actionsByName = new Map(); + const actionsByName = new Map(); executionGraph.actions.forEach(action => { actionsByName.set(targetAsReadableString(action.target), action); }); @@ -676,11 +676,11 @@ export function runCli() { const alreadyPrintedActions = new Set(); - const printExecutedGraph = (executedGraph: dataform.IRunResult) => { + const printExecutedGraph = (executedGraph: sqlanvil.IRunResult) => { executedGraph.actions .filter( actionResult => - actionResult.status !== dataform.ActionResult.ExecutionStatus.RUNNING + actionResult.status !== sqlanvil.ActionResult.ExecutionStatus.RUNNING ) .filter( executedAction => @@ -699,7 +699,7 @@ export function runCli() { runner.onChange(printExecutedGraph); const runResult = await runner.result(); printExecutedGraph(runResult); - return runResult.status === dataform.RunResult.ExecutionStatus.SUCCESSFUL ? 0 : 1; + return runResult.status === sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL ? 0 : 1; } }, { @@ -862,7 +862,7 @@ class ProjectConfigOptions { option: { describe: "Override for variables to inject via '--vars=someKey=someValue,a=b', referenced by " + - "`dataform.projectConfig.vars.someValue`. If unset, the value from workflow_settings.yaml is used.", + "`sqlanvil.projectConfig.vars.someValue`. If unset, the value from workflow_settings.yaml is used.", type: "string", default: null, coerce: (rawVarsString: string | null) => { @@ -939,8 +939,8 @@ class ProjectConfigOptions { public static constructProjectConfigOverride( argv: yargs.Arguments - ): dataform.IProjectConfig { - const projectConfigOptions: dataform.IProjectConfig = {}; + ): sqlanvil.IProjectConfig { + const projectConfigOptions: sqlanvil.IProjectConfig = {}; if (argv[ProjectConfigOptions.defaultDatabase.name]) { projectConfigOptions.defaultDatabase = argv[ProjectConfigOptions.defaultDatabase.name]; diff --git a/cli/index_compile_test.ts b/cli/index_compile_test.ts index bf4181b8..f73ed29c 100644 --- a/cli/index_compile_test.ts +++ b/cli/index_compile_test.ts @@ -4,30 +4,30 @@ import * as fs from "fs-extra"; import { dump as dumpYaml, load as loadYaml } from "js-yaml"; import * as path from "path"; -import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "df/cli/index_test_base"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "sa/cli/index_test_base"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("compile command", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); test( - "compile throws an error when dataformCoreVersion not in workflow_settings.yaml and no " + + "compile throws an error when sqlanvilCoreVersion not in workflow_settings.yaml and no " + "package.json exists", async () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create({ defaultProject: "dataform" })) + dumpYaml(sqlanvil.WorkflowSettings.create({ defaultProject: "dataform" })) ); expect( (await getProcessResult(execFile(nodePath, [cliEntryPointPath, "compile", projectDir]))) .stderr ).contains( - "dataformCoreVersion must be specified either in workflow_settings.yaml or via a " + + "sqlanvilCoreVersion must be specified either in workflow_settings.yaml or via a " + "package.json" ); } @@ -59,21 +59,21 @@ suite("compile command", ({ afterEach }) => { .stderr ).contains( "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + - "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + "`dataform install`." ); }); ["package.json", "package-lock.json", "node_modules"].forEach(npmFile => { - test(`compile throws an error when dataformCoreVersion in workflow_settings.yaml and ${npmFile} is present`, async () => { + test(`compile throws an error when sqlanvilCoreVersion in workflow_settings.yaml and ${npmFile} is present`, async () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), dumpYaml( - dataform.WorkflowSettings.create({ + sqlanvil.WorkflowSettings.create({ defaultProject: "dataform", - dataformCoreVersion: "3.0.0" + sqlanvilCoreVersion: "3.0.0" }) ) ); @@ -105,10 +105,10 @@ suite("disable-assertions flag (compilation)", ({ afterEach, beforeEach }) => { ); const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( @@ -158,7 +158,7 @@ SELECT 1 as id async function setUpWorkflowSettings(disableAssertions: boolean): Promise { const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); workflowSettings.disableAssertions = disableAssertions; @@ -176,7 +176,7 @@ SELECT 1 as id canonicalTarget: { database: DEFAULT_DATABASE, name: "dataform_example_table_assertions_uniqueKey_0", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" }, dependencyTargets: [ { @@ -194,18 +194,18 @@ SELECT 1 as id }, query: // tslint:disable-next-line:tsr-detect-sql-literal-injection - `\nSELECT\n *\nFROM (\n SELECT\n id,\n COUNT(1) AS index_row_count\n FROM \`${DEFAULT_DATABASE}.dataform.example_table\`\n GROUP BY id\n ) AS data\nWHERE index_row_count > 1\n`, + `\nSELECT\n *\nFROM (\n SELECT\n id,\n COUNT(1) AS index_row_count\n FROM \`${DEFAULT_DATABASE}.sqlanvil.example_table\`\n GROUP BY id\n ) AS data\nWHERE index_row_count > 1\n`, target: { database: DEFAULT_DATABASE, name: "dataform_example_table_assertions_uniqueKey_0", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" } }, { canonicalTarget: { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" }, disabled: true, fileName: "definitions/test_assertion.sqlx", @@ -213,18 +213,18 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" } } ], - dataformCoreVersion: version, + sqlanvilCoreVersion: version, graphErrors: {}, jitData: {}, projectConfig: { - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, - defaultSchema: "dataform", + defaultSchema: "sqlanvil", disableAssertions: true, warehouse: "bigquery" }, @@ -252,7 +252,7 @@ SELECT 1 as id { database: DEFAULT_DATABASE, name: "dataform_example_table_assertions_uniqueKey_0", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" }, { database: DEFAULT_DATABASE, @@ -262,7 +262,7 @@ SELECT 1 as id { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" } ] }; @@ -309,7 +309,7 @@ suite("extension config", ({ afterEach }) => { defaultProject: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, defaultDataset: "dataform", - defaultAssertionDataset: "dataform_assertions", + defaultAssertionDataset: "sqlanvil_assertions", extension: { name: "test-extension", compilationMode: "PROLOGUE", diff --git a/cli/index_help_test.ts b/cli/index_help_test.ts index 1951c168..1e53d689 100644 --- a/cli/index_help_test.ts +++ b/cli/index_help_test.ts @@ -1,8 +1,8 @@ import { expect } from "chai"; import { execFile } from "child_process"; -import { cliEntryPointPath } from "df/cli/index_test_base"; -import { getProcessResult, nodePath, suite, test } from "df/testing"; +import { cliEntryPointPath } from "sa/cli/index_test_base"; +import { getProcessResult, nodePath, suite, test } from "sa/testing"; suite("help command", () => { test("shows global help with the help command", async () => { diff --git a/cli/index_init_test.ts b/cli/index_init_test.ts index 5ca0006a..cf510893 100644 --- a/cli/index_init_test.ts +++ b/cli/index_init_test.ts @@ -4,7 +4,7 @@ import * as fs from "fs-extra"; import { load as loadYaml } from "js-yaml"; import * as path from "path"; -import { cliEntryPointPath } from "df/cli/index_test_base"; +import { cliEntryPointPath } from "sa/cli/index_test_base"; import { ICEBERG_BUCKET_NAME_HINT, ICEBERG_BUCKET_NAME_PROMPT_QUESTION, @@ -17,11 +17,11 @@ import { ICEBERG_TABLE_FOLDER_ROOT_PROMPT_QUESTION, ICEBERG_TABLE_FOLDER_ROOT_SUBPATH_HINT, ICEBERG_TABLE_FOLDER_SUBPATH_PROMPT_QUESTION -} from "df/cli/util"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { getProcessResult, nodePath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/cli/util"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { getProcessResult, nodePath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("init command", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -34,16 +34,16 @@ suite("init command", ({ afterEach }) => { cliEntryPointPath, "init", projectDir, - "--default-database=dataform-database", + "--default-database=sqlanvil-database", "--default-location=us-central1" ]) ); expect(fs.readFileSync(path.join(projectDir, "workflow_settings.yaml"), "utf8")).to - .equal(`dataformCoreVersion: ${version} -defaultProject: dataform-database + .equal(`sqlanvilCoreVersion: ${version} +defaultProject: sqlanvil-database defaultLocation: us-central1 -defaultDataset: dataform +defaultDataset: sqlanvil defaultAssertionDataset: dataform_assertions `); }); @@ -63,7 +63,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-test", + "sqlanvil-iceberg-test", "us-central1", "--iceberg" ], { @@ -79,7 +79,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -105,7 +105,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -126,7 +126,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -151,7 +151,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -172,7 +172,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -197,7 +197,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -218,7 +218,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -243,7 +243,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -264,7 +264,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); diff --git a/cli/index_project_test.ts b/cli/index_project_test.ts index 7c11ef76..e13d8766 100644 --- a/cli/index_project_test.ts +++ b/cli/index_project_test.ts @@ -4,17 +4,17 @@ import * as fs from "fs-extra"; import { dump as dumpYaml, load as loadYaml } from "js-yaml"; import * as path from "path"; -import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "df/cli/index_test_base"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "sa/cli/index_test_base"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("project ops", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); suite("install command", () => { - test("install throws an error when dataformCoreVersion in workflow_settings.yaml", async () => { + test("install throws an error when sqlanvilCoreVersion in workflow_settings.yaml", async () => { const projectDir = tmpDirFixture.createNewTmpDir(); await getProcessResult( @@ -22,7 +22,7 @@ suite("project ops", ({ afterEach }) => { cliEntryPointPath, "init", projectDir, - "--default-database=dataform-database", + "--default-database=sqlanvil-database", "--default-location=us-central1" ]) ); @@ -50,10 +50,10 @@ suite("project ops", ({ afterEach }) => { ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, diff --git a/cli/index_run_e2e_test.ts b/cli/index_run_e2e_test.ts index ecf1bdf0..62bdbc32 100644 --- a/cli/index_run_e2e_test.ts +++ b/cli/index_run_e2e_test.ts @@ -10,11 +10,11 @@ import { DEFAULT_DATABASE, DEFAULT_LOCATION, DEFAULT_RESERVATION -} from "df/cli/index_test_base"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/cli/index_test_base"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("run e2e", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -31,10 +31,10 @@ suite("run e2e", ({ afterEach }) => { ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, @@ -62,7 +62,7 @@ suite("run e2e", ({ afterEach }) => { filePath, ` config { type: "table", tags: ["someTag"] } -select 1 as \${dataform.projectConfig.vars.testVar2} +select 1 as \${sqlanvil.projectConfig.vars.testVar2} ` ); @@ -104,8 +104,8 @@ select 1 as \${dataform.projectConfig.vars.testVar2} ], projectConfig: { warehouse: "bigquery", - defaultSchema: "dataform", - assertionSchema: "dataform_assertions", + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, vars: { @@ -116,7 +116,7 @@ select 1 as \${dataform.projectConfig.vars.testVar2} }, graphErrors: {}, jitData: {}, - dataformCoreVersion: version, + sqlanvilCoreVersion: version, targets: [ { database: DEFAULT_DATABASE, @@ -164,7 +164,7 @@ select 1 as \${dataform.projectConfig.vars.testVar2} { statement: // tslint:disable-next-line:tsr-detect-sql-literal-injection - `create or replace table \`${DEFAULT_DATABASE}.dataform.example\` as \n\nselect 1 as testValue2`, + `create or replace table \`${DEFAULT_DATABASE}.sqlanvil.example\` as \n\nselect 1 as testValue2`, type: "statement" } ], @@ -172,10 +172,10 @@ select 1 as \${dataform.projectConfig.vars.testVar2} } ], projectConfig: { - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: "europe", - defaultSchema: "dataform", + defaultSchema: "sqlanvil", warehouse: "bigquery", vars: { testVar1: "testValue1", @@ -203,10 +203,10 @@ select 1 as \${dataform.projectConfig.vars.testVar2} ); const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( @@ -256,10 +256,10 @@ SELECT 1 as id async function setUpWorkflowSettings(disableAssertions: boolean): Promise { const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; workflowSettings.disableAssertions = disableAssertions; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); } @@ -281,7 +281,7 @@ SELECT 1 as id { statement: // tslint:disable-next-line:tsr-detect-sql-literal-injection - `create or replace table \`${DEFAULT_DATABASE}.dataform.example_table\` as \n\nSELECT 1 as id`, + `create or replace table \`${DEFAULT_DATABASE}.sqlanvil.example_table\` as \n\nSELECT 1 as id`, type: "statement" } ], @@ -293,16 +293,16 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" }, type: "assertion", } ], projectConfig: { - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, - defaultSchema: "dataform", + defaultSchema: "sqlanvil", disableAssertions: true, warehouse: "bigquery" }, @@ -402,11 +402,11 @@ SELECT 1 as id execFile(nodePath, [cliEntryPointPath, "init", projectDir, DEFAULT_DATABASE, DEFAULT_LOCATION]) ); - // Remove dataformCoreVersion so we can use the local package. - const workflowSettings = dataform.WorkflowSettings.create( + // Remove sqlanvilCoreVersion so we can use the local package. + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( @@ -454,8 +454,8 @@ SELECT 1 as id const compiledGraph = JSON.parse(compileResult.stdout); expect(compiledGraph.projectConfig).deep.equals({ warehouse: "bigquery", - defaultSchema: "dataform", - assertionSchema: "dataform_assertions", + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, defaultReservation: DEFAULT_RESERVATION @@ -481,8 +481,8 @@ SELECT 1 as id const executionGraph = JSON.parse(runResult.stdout); expect(executionGraph.projectConfig).deep.equals({ warehouse: "bigquery", - defaultSchema: "dataform", - assertionSchema: "dataform_assertions", + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, defaultReservation: DEFAULT_RESERVATION @@ -502,10 +502,10 @@ SELECT 1 as id ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, @@ -579,10 +579,10 @@ select 1 ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, @@ -660,10 +660,10 @@ select 2 execFile(nodePath, [cliEntryPointPath, "init", projectDir, DEFAULT_DATABASE, DEFAULT_LOCATION]) ); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; workflowSettings.defaultDataset = uniqueDataset; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); @@ -693,7 +693,7 @@ select 2 config { type: "operations" } -CREATE OR REPLACE TABLE \`\${dataform.projectConfig.defaultDatabase}.\${dataform.projectConfig.defaultSchema}.example_incremental\` AS SELECT 1 AS id, 'old' AS field1 +CREATE OR REPLACE TABLE \`\${sqlanvil.projectConfig.defaultDatabase}.\${sqlanvil.projectConfig.defaultSchema}.example_incremental\` AS SELECT 1 AS id, 'old' AS field1 ` ); @@ -716,7 +716,7 @@ SELECT 1 as id, 'new' as field1, 'new2' as field2 config { type: "operations" } -DROP SCHEMA IF EXISTS \`\${dataform.projectConfig.defaultDatabase}.\${dataform.projectConfig.defaultSchema}\` CASCADE +DROP SCHEMA IF EXISTS \`\${sqlanvil.projectConfig.defaultDatabase}.\${sqlanvil.projectConfig.defaultSchema}\` CASCADE ` ); }); @@ -759,7 +759,7 @@ DROP SCHEMA IF EXISTS \`\${dataform.projectConfig.defaultDatabase}.\${dataform.p projectConfig: { warehouse: "bigquery", defaultSchema: uniqueDataset, - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION }, diff --git a/cli/index_test_base.ts b/cli/index_test_base.ts index 4c3fe1ce..29c35132 100644 --- a/cli/index_test_base.ts +++ b/cli/index_test_base.ts @@ -4,6 +4,6 @@ import * as path from "path"; export const DEFAULT_DATABASE = "dataform-open-source"; export const DEFAULT_LOCATION = "US"; export const DEFAULT_RESERVATION = "projects/dataform-open-source/locations/us/reservations/dataform-test"; -export const CREDENTIALS_PATH = path.resolve(process.env.RUNFILES, "df/test_credentials/bigquery.json"); +export const CREDENTIALS_PATH = path.resolve(process.env.RUNFILES, "sa/test_credentials/bigquery.json"); export const cliEntryPointPath = "cli/node_modules/@sqlanvil/cli/bundle.js"; diff --git a/cli/util.ts b/cli/util.ts index dcd77b37..9f4450e4 100644 --- a/cli/util.ts +++ b/cli/util.ts @@ -7,9 +7,9 @@ import { print, printError, printSuccess, -} from "df/cli/console"; -import {validateConnectionFormat} from "df/core/utils" -import { dataform } from "df/protos/ts"; +} from "sa/cli/console"; +import {validateConnectionFormat} from "sa/core/utils" +import { sqlanvil } from "sa/protos/ts"; export function actuallyResolve(...filePaths: string[]) { return path.resolve(...filePaths.map(filePath => untildify(filePath))); @@ -21,7 +21,7 @@ export function assertPathExists(checkPath: string) { } } -export function compiledGraphHasErrors(graph: dataform.ICompiledGraph) { +export function compiledGraphHasErrors(graph: sqlanvil.ICompiledGraph) { return graph.graphErrors?.compilationErrors?.length > 0; } @@ -60,11 +60,11 @@ export function formatBytesInHumanReadableFormat(bytes: number): string { * @returns Constructed DefaultIcebergConfig object, or undefined if no inputs * were provided. */ -export function promptForIcebergConfig(): dataform.IDefaultIcebergConfig | undefined { +export function promptForIcebergConfig(): sqlanvil.IDefaultIcebergConfig | undefined { print(ICEBERG_CONFIG_PROMPT_TEXT); print(ICEBERG_CONFIG_PROMPT_HINT); - const tempIcebergConfig: dataform.IDefaultIcebergConfig = {}; + const tempIcebergConfig: sqlanvil.IDefaultIcebergConfig = {}; let bucketName: string; while (true) { diff --git a/cli/util_test.ts b/cli/util_test.ts index b86f8937..adb89369 100644 --- a/cli/util_test.ts +++ b/cli/util_test.ts @@ -6,8 +6,8 @@ import { validateIcebergConfigBucketName, validateIcebergConfigTableFolderRoot, validateIcebergConfigTableFolderSubpath, -} from "df/cli/util"; -import { suite, test } from "df/testing"; +} from "sa/cli/util"; +import { suite, test } from "sa/testing"; suite('format execution suffix', () => { test('format execution suffix', () => { diff --git a/cli/vm/compile.ts b/cli/vm/compile.ts index 0dfc2f55..434bc29e 100644 --- a/cli/vm/compile.ts +++ b/cli/vm/compile.ts @@ -4,10 +4,10 @@ import * as path from "path"; import * as semver from "semver"; import { CompilerFunction, NodeVM } from "vm2"; -import { encode64 } from "df/common/protos"; -import { dataform } from "df/protos/ts"; +import { encode64 } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; -export function compile(compileConfig: dataform.ICompileConfig) { +export function compile(compileConfig: sqlanvil.ICompileConfig) { compileConfig.projectDir = fs.realpathSync(path.resolve(compileConfig.projectDir)); if ( !fs.existsSync( @@ -16,7 +16,7 @@ export function compile(compileConfig: dataform.ICompileConfig) { ) { throw new Error( "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + - "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + "`dataform install`." ); @@ -58,22 +58,22 @@ export function compile(compileConfig: dataform.ICompileConfig) { compiler: (code, filePath) => { const compiledCode = compiler(code, filePath); return ` - var __old_file = global.__dataform_current_file; - global.__dataform_current_file = ${JSON.stringify(filePath)}; + var __old_file = global.__sqlanvil_current_file; + global.__sqlanvil_current_file = ${JSON.stringify(filePath)}; try { ${compiledCode} } finally { - global.__dataform_current_file = __old_file; + global.__sqlanvil_current_file = __old_file; } `; } }); - const dataformCoreVersion: string = userCodeVm.run( + const sqlanvilCoreVersion: string = userCodeVm.run( 'return require("@sqlanvil/core").version || "0.0.0"', vmIndexFileName ); - if (semver.lt(dataformCoreVersion, "3.0.0-alpha.0")) { + if (semver.lt(sqlanvilCoreVersion, "3.0.0-alpha.0")) { throw new Error("@sqlanvil/core ^3.0.0 required."); } @@ -88,7 +88,7 @@ export function compile(compileConfig: dataform.ICompileConfig) { } export function listenForCompileRequest() { - process.on("message", (compileConfig: dataform.ICompileConfig) => { + process.on("message", (compileConfig: sqlanvil.ICompileConfig) => { try { const compiledResult = compile(compileConfig); process.send(compiledResult); @@ -107,14 +107,14 @@ if (require.main === module) { } /** - * @returns a base64 encoded @see {@link dataform.CoreExecutionRequest} proto. + * @returns a base64 encoded @see {@link sqlanvil.CoreExecutionRequest} proto. */ -function createCoreExecutionRequest(compileConfig: dataform.ICompileConfig): string { +function createCoreExecutionRequest(compileConfig: sqlanvil.ICompileConfig): string { const filePaths = Array.from( new Set(glob.sync("!(node_modules)/**/*.*", { cwd: compileConfig.projectDir })) ); - return encode64(dataform.CoreExecutionRequest, { + return encode64(sqlanvil.CoreExecutionRequest, { // Add the list of file paths to the compile config if not already set. compile: { compileConfig: { filePaths, ...compileConfig } } }); diff --git a/cli/vm/jit_loader.js b/cli/vm/jit_loader.js index cb325cee..30633376 100644 --- a/cli/vm/jit_loader.js +++ b/cli/vm/jit_loader.js @@ -1,7 +1,7 @@ 'use strict'; if (require.main === module) { - var entryPointPath = 'df/cli/vm/jit_worker.js'; + var entryPointPath = 'sa/cli/vm/jit_worker.js'; var mainScript = process.argv[1] = entryPointPath; try { module.constructor._load(mainScript, this, /*isMain=*/true); diff --git a/cli/vm/jit_worker.ts b/cli/vm/jit_worker.ts index 3b14c623..0dea5cd3 100644 --- a/cli/vm/jit_worker.ts +++ b/cli/vm/jit_worker.ts @@ -2,7 +2,7 @@ import * as fs from "fs"; import * as path from "path"; import { NodeVM } from "vm2"; -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; const pendingRpcCallbacks = new Map void>(); let hasStartedProcessing = false; @@ -31,7 +31,7 @@ export async function handleJitRequest(message: { if (!fs.existsSync(path.join(projectDir, "node_modules", "@dataform", "core", "bundle.js"))) { throw new Error( "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + - "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + "`dataform install`." ); @@ -49,8 +49,8 @@ export async function handleJitRequest(message: { }); }; - const requestMessage = dataform.JitCompilationRequest.fromObject(request); - const requestBytes = dataform.JitCompilationRequest.encode(requestMessage).finish(); + const requestMessage = sqlanvil.JitCompilationRequest.fromObject(request); + const requestBytes = sqlanvil.JitCompilationRequest.encode(requestMessage).finish(); const vmFileName = path.resolve(projectDir, "index.js"); @@ -89,7 +89,7 @@ export async function handleJitRequest(message: { `, vmFileName); const responseBytes = await jitCompileInVm(requestBytes, rpcCallback); - const response = dataform.JitCompilationResponse.decode(responseBytes); + const response = sqlanvil.JitCompilationResponse.decode(responseBytes); process.send({ type: "jit_response", response: response.toJSON() }); } catch (e) { diff --git a/cli/yargswrapper.ts b/cli/yargswrapper.ts index c659e309..0b979e17 100644 --- a/cli/yargswrapper.ts +++ b/cli/yargswrapper.ts @@ -59,8 +59,8 @@ function createOptionsChain(yargsChain: yargs.Argv, command: ICommand) { function fixArgvForHelp() { // Obviously this is a massive hack. // The outcome of this is that the following commands are interchangeable: - // $ dataform help run - // $ dataform --help run + // $ sqlanvil help run + // $ sqlanvil --help run // The problem is that yargs.help() only allows us to specify an alias for the "--help" built-in option (by default that alias is "help"). // But because "--help" is only an option, not a command, it appears to be impossible (?) to configure yargs to respond to "help" correctly // (or at least, to correctly print help strings for commands; it happily prints a top-level help string). diff --git a/common/errors/errors.spec.ts b/common/errors/errors.spec.ts index a1a19c50..c4b0ea2c 100644 --- a/common/errors/errors.spec.ts +++ b/common/errors/errors.spec.ts @@ -1,8 +1,8 @@ import { expect } from "chai"; import { basename } from "path"; -import { coerceAsError, ErrorWithCause } from "df/common/errors/errors"; -import { suite, test } from "df/testing"; +import { coerceAsError, ErrorWithCause } from "sa/common/errors/errors"; +import { suite, test } from "sa/testing"; suite(basename(__filename), () => { suite("ErrorWithCause", () => { diff --git a/common/flags/testing/index.ts b/common/flags/testing/index.ts index fa8f7843..1a69d4b3 100644 --- a/common/flags/testing/index.ts +++ b/common/flags/testing/index.ts @@ -1,5 +1,5 @@ -import { Flags } from "df/common/flags"; -import { IHookHandler } from "df/testing"; +import { Flags } from "sa/common/flags"; +import { IHookHandler } from "sa/testing"; // FlagOverridesFixture can be used to temporarily override flag values for testing purposes. // At the end of the test run, any previous flag values will be restored. diff --git a/common/promises/index.spec.ts b/common/promises/index.spec.ts index 472fec4e..c2332982 100644 --- a/common/promises/index.spec.ts +++ b/common/promises/index.spec.ts @@ -1,8 +1,8 @@ import { fail } from "assert"; import { assert, expect } from "chai"; -import { retry, runWithTimeout, sleep } from "df/common/promises"; -import { suite, test } from "df/testing"; +import { retry, runWithTimeout, sleep } from "sa/common/promises"; +import { suite, test } from "sa/testing"; suite(__filename, () => { suite("runWithTimeout", () => { diff --git a/common/protos/index.ts b/common/protos/index.ts index 2bf2777b..807536e3 100644 --- a/common/protos/index.ts +++ b/common/protos/index.ts @@ -1,6 +1,6 @@ import { util } from "protobufjs"; -import { google } from "df/protos/ts"; +import { google } from "sa/protos/ts"; const CONFIGS_PROTO_DOCUMENTATION_URL = "https://dataform-co.github.io/dataform/docs/configs-reference"; diff --git a/common/protos/index_test.ts b/common/protos/index_test.ts index 241c79a3..a2c5f4b9 100644 --- a/common/protos/index_test.ts +++ b/common/protos/index_test.ts @@ -1,20 +1,20 @@ import { expect } from "chai"; import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "./index"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("verifyObjectMatchesProto", () => { test("throws error when top-level object is an array", () => { expect(() => { - verifyObjectMatchesProto(dataform.Target, [] as any); + verifyObjectMatchesProto(sqlanvil.Target, [] as any); }).to.throw(ReferenceError, "Expected a top-level object, but found an array"); }); test("throws error when null value provided for array field and SHOW_DOCS_LINK", () => { expect(() => { verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, { dependencyTargets: null } as any, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -24,7 +24,7 @@ suite("verifyObjectMatchesProto", () => { test("throws error on type mismatch with SUGGEST_REPORTING_TO_DATAFORM_TEAM", () => { expect(() => { verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, { actionDescriptor: 123 } as any, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -36,7 +36,7 @@ suite("verifyObjectMatchesProto", () => { test("throws error on default type mismatch", () => { expect(() => { - verifyObjectMatchesProto(dataform.Table, { actionDescriptor: 123 } as any); + verifyObjectMatchesProto(sqlanvil.Table, { actionDescriptor: 123 } as any); }).to.throw( ReferenceError, /Unexpected property "actionDescriptor", or property value type of "number" is incorrect/ diff --git a/common/protos/structs.ts b/common/protos/structs.ts index c0b704f9..2e76c819 100644 --- a/common/protos/structs.ts +++ b/common/protos/structs.ts @@ -1,4 +1,4 @@ -import { google } from "df/protos/ts"; +import { google } from "sa/protos/ts"; export type AnyValue = null | number | string | boolean | undefined | AnyValue[] | { [key: string]: AnyValue }; diff --git a/common/strings/stringifier.spec.ts b/common/strings/stringifier.spec.ts index 3d25abcd..9cc96ddc 100644 --- a/common/strings/stringifier.spec.ts +++ b/common/strings/stringifier.spec.ts @@ -1,8 +1,8 @@ import { expect } from "chai"; import { basename } from "path"; -import { JSONObjectStringifier } from "df/common/strings/stringifier"; -import { suite, test } from "df/testing"; +import { JSONObjectStringifier } from "sa/common/strings/stringifier"; +import { suite, test } from "sa/testing"; interface IKey { a: string; diff --git a/core/actions/assertion.ts b/core/actions/assertion.ts index e638e8b3..01e188be 100644 --- a/core/actions/assertion.ts +++ b/core/actions/assertion.ts @@ -1,8 +1,8 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { IActionContext, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { IActionContext, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, configTargetToCompiledGraphTarget, @@ -12,8 +12,8 @@ import { resolveActionsConfigFilename, toResolvable, validateQueryString -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -21,7 +21,7 @@ import { dataform } from "df/protos/ts"; * This maintains backwards compatability with older versions. * Consider breaking backwards compatability of these in v4. */ -interface ILegacyAssertionConfig extends dataform.ActionConfig.AssertionConfig { +interface ILegacyAssertionConfig extends sqlanvil.ActionConfig.AssertionConfig { dependencies: Resolvable[]; database: string; schema: string; @@ -37,7 +37,7 @@ export type AContextable = T | ((ctx: AssertionContext) => T); * specified in the query. If the query returns any rows, the assertion fails. * * You can create assertions in the following ways. Available config options are defined in - * [AssertionConfig](configs#dataform-ActionConfig-AssertionConfig), and are shared across all the + * [AssertionConfig](configs#sqlanvil-ActionConfig-AssertionConfig), and are shared across all the * following ways of creating assertions. * * **Using a SQLX file:** @@ -52,7 +52,7 @@ export type AContextable = T | ((ctx: AssertionContext) => T); * * **Using built-in assertions in the config block of a table:** * - * See [TableConfig.assertions](configs#dataform-ActionConfig-TableConfig) + * See [TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig) * * **Using action configs files:** * @@ -78,14 +78,14 @@ export type AContextable = T | ((ctx: AssertionContext) => T); * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class Assertion extends ActionBuilder { +export class Assertion extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Assertion.create(); + private proto = sqlanvil.Assertion.create(); /** @hidden We delay contextification until the final compile step, so hold these here for now. */ private contextableQuery: AContextable; @@ -124,7 +124,7 @@ export class Assertion extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -168,7 +168,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.dependencies](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.dependencies](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets dependencies of the assertion. */ @@ -184,7 +184,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.hermetic](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.hermetic](configs#sqlanvil-ActionConfig-AssertionConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -192,13 +192,13 @@ export class Assertion extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [AssertionConfig.disabled](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.disabled](configs#sqlanvil-ActionConfig-AssertionConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -210,7 +210,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.tags](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.tags](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -226,7 +226,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.description](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.description](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets the description of this assertion. */ @@ -237,14 +237,14 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.project](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.project](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets the database (Google Cloud project ID) in which to create the corresponding view for this * assertion. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true, useDefaultAssertionDataset: true } @@ -254,14 +254,14 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.dataset](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.dataset](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets the schema (BigQuery dataset) in which to create the corresponding view for this * assertion. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true, useDefaultAssertionDataset: true } @@ -276,16 +276,16 @@ export class Assertion extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ public getParentAction() { - return dataform.Target.create(this.proto.parentAction); + return sqlanvil.Target.create(this.proto.parentAction); } /** @hidden */ - public setParentAction(target: dataform.Target) { + public setParentAction(target: sqlanvil.Target) { this.proto.parentAction = target; } @@ -297,7 +297,7 @@ export class Assertion extends ActionBuilder { validateQueryString(this.session, this.proto.query, this.proto.fileName); return verifyObjectMatchesProto( - dataform.Assertion, + sqlanvil.Assertion, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -310,7 +310,7 @@ export class Assertion extends ActionBuilder { */ private verifyConfig( unverifiedConfig: ILegacyAssertionConfig - ): dataform.ActionConfig.AssertionConfig { + ): sqlanvil.ActionConfig.AssertionConfig { if (unverifiedConfig.dependencies) { unverifiedConfig.dependencyTargets = unverifiedConfig.dependencies.map( (dependency: string | object) => resolvableAsActionConfigTarget(dependency) @@ -335,7 +335,7 @@ export class Assertion extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.ActionConfig.AssertionConfig, + sqlanvil.ActionConfig.AssertionConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/assertion_test.ts b/core/actions/assertion_test.ts index 02ec6e63..d230e673 100644 --- a/core/actions/assertion_test.ts +++ b/core/actions/assertion_test.ts @@ -4,15 +4,15 @@ import * as fs from "fs-extra"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML, WorkflowSettingsTemplates -} from "df/testing/run_core"; +} from "sa/testing/run_core"; const EMPTY_NOTEBOOK_CONTENTS = '{ "cells": [] }'; @@ -301,7 +301,7 @@ actions: projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(testConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(testConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -928,7 +928,7 @@ actions: const coreRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ disableAssertions: true }) ); @@ -980,7 +980,7 @@ actions: const coreRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ disableAssertions: true }) ); diff --git a/core/actions/data_preparation.ts b/core/actions/data_preparation.ts index c0c924aa..737da53f 100644 --- a/core/actions/data_preparation.ts +++ b/core/actions/data_preparation.ts @@ -1,10 +1,10 @@ import { dump as dumpYaml } from "js-yaml"; -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { Contextable, ITableContext, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { Contextable, ITableContext, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -14,23 +14,23 @@ import { resolveActionsConfigFilename, toResolvable, validateQueryString -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden */ -export class DataPreparation extends ActionBuilder { +export class DataPreparation extends ActionBuilder { public session: Session; // We delay contextification until the final compile step, so hold these here for now. public contextableQuery: Contextable; - private proto = dataform.DataPreparation.create(); + private proto = sqlanvil.DataPreparation.create(); constructor( session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig, + config?: sqlanvil.ActionConfig.DataPreparationConfig, configPath?: string ) { super(session); @@ -114,7 +114,7 @@ export class DataPreparation extends ActionBuilder { */ public getTarget() { // Return only the first target for now. - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } public compile() { @@ -125,7 +125,7 @@ export class DataPreparation extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.DataPreparation, + sqlanvil.DataPreparation, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -133,7 +133,7 @@ export class DataPreparation extends ActionBuilder { public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -143,7 +143,7 @@ export class DataPreparation extends ActionBuilder { public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -156,9 +156,9 @@ export class DataPreparation extends ActionBuilder { [key: string]: any; }, session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig + config?: sqlanvil.ActionConfig.DataPreparationConfig ) { - const defaultTarget = dataform.Target.create({ name: config.name }); + const defaultTarget = sqlanvil.Target.create({ name: config.name }); this.proto.target = this.finalizeTarget( this.applySessionToTarget(defaultTarget, session.projectConfig, config.filename, { validateTarget: true @@ -175,12 +175,12 @@ export class DataPreparation extends ActionBuilder { } private configureYamlWithTargets( - targets: dataform.Target[], + targets: sqlanvil.Target[], dataPreparationAsJson: { [key: string]: any; }, session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig + config?: sqlanvil.ActionConfig.DataPreparationConfig ) { const resolvedTargets = targets.map(target => this.applySessionToTarget(target, session.projectConfig, config.filename, { @@ -245,13 +245,13 @@ export class DataPreparation extends ActionBuilder { } private applySessionToTableReference(tableReference: { [key: string]: string }): object { - const target: dataform.ITarget = { + const target: sqlanvil.ITarget = { database: tableReference.project, schema: tableReference.dataset, name: tableReference.table }; const resolvedTarget = this.applySessionToTarget( - dataform.Target.create(target), + sqlanvil.Target.create(target), this.session.projectConfig ); // Convert resolved target into a Data Preparation Table Reference @@ -275,19 +275,19 @@ export class DataPreparation extends ActionBuilder { return resolvedTableReference; } - private getTargets(definition: { [key: string]: any }): dataform.Target[] { - const targets: dataform.Target[] = []; + private getTargets(definition: { [key: string]: any }): sqlanvil.Target[] { + const targets: sqlanvil.Target[] = []; if (definition && definition.nodes) { (definition.nodes as Array<{ [key: string]: any }>).forEach(node => { const table = node.destination?.table; if (table) { - const compiledGraphTarget: dataform.ITarget = { + const compiledGraphTarget: sqlanvil.ITarget = { database: table.project, schema: table.dataset, name: table.table }; - targets.push(dataform.Target.create(compiledGraphTarget)); + targets.push(sqlanvil.Target.create(compiledGraphTarget)); } }); } @@ -296,7 +296,7 @@ export class DataPreparation extends ActionBuilder { } private configureYaml( session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig, + config?: sqlanvil.ActionConfig.DataPreparationConfig, configPath?: string ) { config.filename = resolveActionsConfigFilename(config.filename, configPath); @@ -322,7 +322,7 @@ export class DataPreparation extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -332,14 +332,14 @@ export class DataPreparation extends ActionBuilder { } } - private configureSqlx(session?: Session, config?: dataform.ActionConfig.DataPreparationConfig) { - const targets: dataform.Target[] = []; + private configureSqlx(session?: Session, config?: sqlanvil.ActionConfig.DataPreparationConfig) { + const targets: sqlanvil.Target[] = []; // Add destination as target targets.push(actionConfigToCompiledGraphTarget(config)); // Add Error Table if specified as a secondary target if (config.errorTable != null) { - const errorTableConfig = dataform.ActionConfig.DataPreparationConfig.ErrorTableConfig.create( + const errorTableConfig = sqlanvil.ActionConfig.DataPreparationConfig.ErrorTableConfig.create( config.errorTable ); const errorTableTarget = actionConfigToCompiledGraphTarget(errorTableConfig); @@ -378,7 +378,7 @@ export class DataPreparation extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -399,26 +399,26 @@ export class DataPreparation extends ActionBuilder { loadMode?: string | number, incrementalColumn?: string, uniqueKey?: string[] - ): dataform.LoadConfiguration { + ): sqlanvil.LoadConfiguration { if (!loadMode) { - return dataform.LoadConfiguration.create({ replace: {} }); + return sqlanvil.LoadConfiguration.create({ replace: {} }); } switch (loadMode.toString().toUpperCase()) { case "REPLACE_TABLE": - return dataform.LoadConfiguration.create({ replace: {} }); + return sqlanvil.LoadConfiguration.create({ replace: {} }); case "APPEND": - return dataform.LoadConfiguration.create({ append: {} }); + return sqlanvil.LoadConfiguration.create({ append: {} }); case "MAXIMUM": - return dataform.LoadConfiguration.create({ + return sqlanvil.LoadConfiguration.create({ maximum: { columnName: this.validateLoadModeColumnName(incrementalColumn) } }); case "UNIQUE": - return dataform.LoadConfiguration.create({ + return sqlanvil.LoadConfiguration.create({ unique: { columnName: this.validateLoadModeColumnName(incrementalColumn) } }); case "MERGE": - return dataform.LoadConfiguration.create({ + return sqlanvil.LoadConfiguration.create({ merge: { uniqueKey: this.validateUniqueKey(uniqueKey) } }); default: @@ -446,7 +446,7 @@ export class DataPreparationContext implements ITableContext { constructor(private dataPreparation: DataPreparation, private isIncremental = false) {} - public config(config: dataform.ActionConfig.DataPreparationConfig) { + public config(config: sqlanvil.ActionConfig.DataPreparationConfig) { this.dataPreparation.config(config); return ""; } diff --git a/core/actions/data_preparation_test.ts b/core/actions/data_preparation_test.ts index fa92ee5b..3299142f 100644 --- a/core/actions/data_preparation_test.ts +++ b/core/actions/data_preparation_test.ts @@ -4,14 +4,14 @@ import * as fs from "fs-extra"; import { dump as dumpYaml, load as loadYaml } from "js-yaml"; import * as path from "path"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("data preparation", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -427,7 +427,7 @@ FROM x const coreExecutionRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ defaultDatabase: "projectOverride", defaultSchema: "datasetOverride" }) diff --git a/core/actions/declaration.ts b/core/actions/declaration.ts index adc482cd..cd697c32 100644 --- a/core/actions/declaration.ts +++ b/core/actions/declaration.ts @@ -1,9 +1,9 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Session } from "df/core/session"; -import { actionConfigToCompiledGraphTarget } from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Session } from "sa/core/session"; +import { actionConfigToCompiledGraphTarget } from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -11,7 +11,7 @@ import { dataform } from "df/protos/ts"; * This maintains backwards compatability with older versions. * Consider breaking backwards compatability of these in v4. */ -interface ILegacyDeclarationConfig extends dataform.ActionConfig.DeclarationConfig { +interface ILegacyDeclarationConfig extends sqlanvil.ActionConfig.DeclarationConfig { database: string; schema: string; fileName: string; @@ -30,7 +30,7 @@ interface ILegacyDeclarationConfig extends dataform.ActionConfig.DeclarationConf * * Trigger workflow invocations that include all the dependents of an external data source. * * You can create declarations in the following ways. Available config options are defined in - * [DeclarationConfig](configs#dataform-ActionConfig-DeclarationConfig), and are shared across all + * [DeclarationConfig](configs#sqlanvil-ActionConfig-DeclarationConfig), and are shared across all * the followiing ways of creating declarations. * * **Using a SQLX file:** @@ -59,14 +59,14 @@ interface ILegacyDeclarationConfig extends dataform.ActionConfig.DeclarationConf * declare("name") * ``` */ -export class Declaration extends ActionBuilder { +export class Declaration extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Declaration.create(); + private proto = sqlanvil.Declaration.create(); /** @hidden */ constructor(session?: Session, unverifiedConfig?: any, filename?: string) { @@ -97,7 +97,7 @@ export class Declaration extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -108,7 +108,7 @@ export class Declaration extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [DeclarationConfig.description](configs#dataform-ActionConfig-DeclarationConfig). + * [DeclarationConfig.description](configs#sqlanvil-ActionConfig-DeclarationConfig). * * Sets the description of this assertion. */ @@ -122,11 +122,11 @@ export class Declaration extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [DeclarationConfig.columns](configs#dataform-ActionConfig-DeclarationConfig). + * [DeclarationConfig.columns](configs#sqlanvil-ActionConfig-DeclarationConfig). * * Sets the column descriptors of columns in this table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -143,13 +143,13 @@ export class Declaration extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ public compile() { return verifyObjectMatchesProto( - dataform.Declaration, + sqlanvil.Declaration, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -162,7 +162,7 @@ export class Declaration extends ActionBuilder { */ private verifyConfig( unverifiedConfig: ILegacyDeclarationConfig - ): dataform.ActionConfig.DeclarationConfig { + ): sqlanvil.ActionConfig.DeclarationConfig { if (unverifiedConfig.database) { unverifiedConfig.project = unverifiedConfig.database; delete unverifiedConfig.database; @@ -182,7 +182,7 @@ export class Declaration extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.ActionConfig.DeclarationConfig, + sqlanvil.ActionConfig.DeclarationConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/declaration_test.ts b/core/actions/declaration_test.ts index 82a1c790..9fad05b1 100644 --- a/core/actions/declaration_test.ts +++ b/core/actions/declaration_test.ts @@ -3,14 +3,14 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { exampleActionDescriptor } from "df/core/actions/index_test"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { exampleActionDescriptor } from "sa/core/actions/index_test"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("declaration", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/actions/filename_override_test.ts b/core/actions/filename_override_test.ts index 80ca72c1..f3a86a85 100644 --- a/core/actions/filename_override_test.ts +++ b/core/actions/filename_override_test.ts @@ -3,13 +3,13 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("filename_override", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/actions/incremental_table.ts b/core/actions/incremental_table.ts index 9abd4477..81f58b0e 100644 --- a/core/actions/incremental_table.ts +++ b/core/actions/incremental_table.ts @@ -1,4 +1,4 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; import { ActionBuilder, checkConfigAdditionalOptionsOverlap, @@ -6,14 +6,14 @@ import { ILegacyTableConfig, LegacyConfigConverter, TableType -} from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { JitTableResult, Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Contextable, ITableContext, JitContextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +} from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { JitTableResult, Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Contextable, ITableContext, JitContextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -35,8 +35,8 @@ import { validateNoMixedCompilationMode, validateQueryString, validateStorageUriFormat, -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * When you define an incremental table, Dataform builds the incremental table from scratch only for @@ -44,7 +44,7 @@ import { dataform } from "df/protos/ts"; * incremental table according to the conditions that you configure. * * You can create incremental tables in the following ways. Available config options are defined in - * [IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig), and are shared across all the + * [IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig), and are shared across all the * following ways of creating tables. * * **Using a SQLX file:** @@ -74,7 +74,7 @@ import { dataform } from "df/protos/ts"; * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class IncrementalTable extends ActionBuilder { +export class IncrementalTable extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; @@ -94,9 +94,9 @@ export class IncrementalTable extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Table.create({ + private proto = sqlanvil.Table.create({ type: "incremental", - enumType: dataform.TableType.INCREMENTAL, + enumType: sqlanvil.TableType.INCREMENTAL, disabled: false, tags: [] }); @@ -146,7 +146,7 @@ export class IncrementalTable extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -171,7 +171,7 @@ export class IncrementalTable extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -182,7 +182,7 @@ export class IncrementalTable extends ActionBuilder { this.schema(config.dataset); } if (config.assertions) { - this.assertions(dataform.ActionConfig.TableAssertionsConfig.create(config.assertions)); + this.assertions(sqlanvil.ActionConfig.TableAssertionsConfig.create(config.assertions)); } if (config.uniqueKey) { this.uniqueKey(config.uniqueKey); @@ -208,7 +208,7 @@ export class IncrementalTable extends ActionBuilder { session.projectConfig.defaultIcebergConfig?.connection ), fileFormat: getFileFormatValueForIcebergTable(config.iceberg.fileFormat?.toString()), - tableFormat: dataform.TableFormat.ICEBERG, + tableFormat: sqlanvil.TableFormat.ICEBERG, storageUri: getStorageUriForIcebergTable( getEffectiveBucketName(session.projectConfig.defaultIcebergConfig?.bucketName, config.iceberg.bucketName), getEffectiveTableFolderRoot(session.projectConfig.defaultIcebergConfig?.tableFolderRoot, config.iceberg.tableFolderRoot), @@ -282,7 +282,7 @@ export class IncrementalTable extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } @@ -327,7 +327,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.disabled](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.disabled](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -341,7 +341,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.protected](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.protected](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If called with `true`, prevents the dataset from being rebuilt from scratch. */ @@ -353,7 +353,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.uniqueKey](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.uniqueKey](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If set, unique key represents a set of names of columns that will act as a the unique key. To * enforce this, when updating the incremental table, Dataform merges rows with `uniqueKey` @@ -365,12 +365,12 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of options available directly on - * [IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig). For example: + * [IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig). For example: * `publish("name", { type: "table", partitionBy: "column" }`). * * Sets bigquery options for the action. */ - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { if (!!bigquery.labels && Object.keys(bigquery.labels).length > 0) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; @@ -380,14 +380,14 @@ export class IncrementalTable extends ActionBuilder { const bigqueryFiltered = LegacyConfigConverter.legacyConvertBigQueryOptions(bigquery); if (Object.values(bigqueryFiltered).length > 0) { - this.proto.bigquery = dataform.BigQueryOptions.create(bigqueryFiltered); + this.proto.bigquery = sqlanvil.BigQueryOptions.create(bigqueryFiltered); } return this; } /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.dependencies](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.dependencies](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets dependencies of the incremental table. */ @@ -401,7 +401,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.hermetic](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.hermetic](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -409,13 +409,13 @@ export class IncrementalTable extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.tags](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.tags](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -431,7 +431,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.description](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.description](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the description of this incremental table. */ @@ -445,11 +445,11 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.columns](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.columns](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the column descriptors of columns in this incremental table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -461,14 +461,14 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.project](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.project](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the * Sets the database (Google Cloud project ID) in which to create the output of this action. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -478,13 +478,13 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.dataset](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.dataset](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -494,7 +494,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.assertions](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.assertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets in-line assertions for this incremental table. * @@ -503,7 +503,7 @@ export class IncrementalTable extends ActionBuilder { * needed --> */ public assertions( - tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig + tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig ): IncrementalTable { const inlineAssertions = this.generateInlineAssertions(tableAssertionsConfig, this.proto); this.uniqueKeyAssertions = inlineAssertions.uniqueKeyAssertions; @@ -513,7 +513,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -530,7 +530,7 @@ export class IncrementalTable extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -551,7 +551,7 @@ export class IncrementalTable extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -620,8 +620,8 @@ export class IncrementalTable extends ActionBuilder { private verifyConfig( // `any` is used here to facilitate the type merging of the legacy table config, which is very // different to the new structure. - unverifiedConfig: dataform.ActionConfig.IncrementalTableConfig | ILegacyTableConfig | any - ): dataform.ActionConfig.IncrementalTableConfig { + unverifiedConfig: sqlanvil.ActionConfig.IncrementalTableConfig | ILegacyTableConfig | any + ): sqlanvil.ActionConfig.IncrementalTableConfig { // The "type" field only exists on legacy incremental table configs. Here we convert them to the // new format. if (unverifiedConfig.type) { @@ -687,7 +687,7 @@ export class IncrementalTable extends ActionBuilder { } const config = verifyObjectMatchesProto( - dataform.ActionConfig.IncrementalTableConfig, + sqlanvil.ActionConfig.IncrementalTableConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -703,21 +703,21 @@ export class IncrementalTable extends ActionBuilder { // - for sqlx it will have type "string" // - for action.yaml it will be converted to enum which is represented // in TypeScript as a "number". - private mapOnSchemaChange(onSchemaChange?: string | number): dataform.OnSchemaChange { + private mapOnSchemaChange(onSchemaChange?: string | number): sqlanvil.OnSchemaChange { if (!onSchemaChange) { - return dataform.OnSchemaChange.IGNORE; + return sqlanvil.OnSchemaChange.IGNORE; } if (typeof onSchemaChange === "number") { switch (onSchemaChange) { - case dataform.ActionConfig.OnSchemaChange.IGNORE: - return dataform.OnSchemaChange.IGNORE; - case dataform.ActionConfig.OnSchemaChange.FAIL: - return dataform.OnSchemaChange.FAIL; - case dataform.ActionConfig.OnSchemaChange.EXTEND: - return dataform.OnSchemaChange.EXTEND; - case dataform.ActionConfig.OnSchemaChange.SYNCHRONIZE: - return dataform.OnSchemaChange.SYNCHRONIZE; + case sqlanvil.ActionConfig.OnSchemaChange.IGNORE: + return sqlanvil.OnSchemaChange.IGNORE; + case sqlanvil.ActionConfig.OnSchemaChange.FAIL: + return sqlanvil.OnSchemaChange.FAIL; + case sqlanvil.ActionConfig.OnSchemaChange.EXTEND: + return sqlanvil.OnSchemaChange.EXTEND; + case sqlanvil.ActionConfig.OnSchemaChange.SYNCHRONIZE: + return sqlanvil.OnSchemaChange.SYNCHRONIZE; default: throw new Error(`OnSchemaChange value "${onSchemaChange}" is not supported`); } @@ -725,13 +725,13 @@ export class IncrementalTable extends ActionBuilder { switch (onSchemaChange.toString().toUpperCase()) { case "IGNORE": - return dataform.OnSchemaChange.IGNORE; + return sqlanvil.OnSchemaChange.IGNORE; case "FAIL": - return dataform.OnSchemaChange.FAIL; + return sqlanvil.OnSchemaChange.FAIL; case "EXTEND": - return dataform.OnSchemaChange.EXTEND; + return sqlanvil.OnSchemaChange.EXTEND; case "SYNCHRONIZE": - return dataform.OnSchemaChange.SYNCHRONIZE; + return sqlanvil.OnSchemaChange.SYNCHRONIZE; default: throw new Error(`OnSchemaChange value "${onSchemaChange}" is not supported`); } @@ -811,7 +811,7 @@ export class IncrementalTableContext implements ITableContext { return ""; } - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { this.incrementalTable.bigquery(bigquery); return ""; } diff --git a/core/actions/incremental_table_test.ts b/core/actions/incremental_table_test.ts index 9b444fbb..c6917a0c 100644 --- a/core/actions/incremental_table_test.ts +++ b/core/actions/incremental_table_test.ts @@ -7,15 +7,15 @@ import { exampleActionDescriptor, exampleBuiltInAssertions, exampleBuiltInAssertionsAsYaml -} from "df/core/actions/index_test"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/core/actions/index_test"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("incremental table", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -439,7 +439,7 @@ defaultIcebergConfig: wsContent: VALID_WORKFLOW_SETTINGS_YAML, }, { - testName: "defaults to \`_dataform\` for tableFolderRoot", + testName: "defaults to \`_sqlanvil\` for tableFolderRoot", configBlock: ` type: "incremental", name: "incremental_table3", @@ -458,7 +458,7 @@ defaultIcebergConfig: tableFormat: "ICEBERG", fileFormat: "PARQUET", connection: "gcp.us.conn-id", - storageUri: "gs://my-bucket/_dataform/my-subpath", + storageUri: "gs://my-bucket/_sqlanvil/my-subpath", }, }, expectError: false, @@ -882,7 +882,7 @@ defaultIcebergConfig: expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); const compiledTable = result.compile.compiledGraph.tables[0]; expect(compiledTable.type).equals("incremental"); - expect(compiledTable.enumType).equals(dataform.TableType.INCREMENTAL); + expect(compiledTable.enumType).equals(sqlanvil.TableType.INCREMENTAL); expect(compiledTable.target.name).equals(testCase.expected!.target.name); expect(compiledTable.target.schema).equals(testCase.expected!.target.schema); expect(compiledTable.target.database).equals(testCase.expected!.target.database); @@ -902,7 +902,7 @@ defaultIcebergConfig: // Create workflow_settings without defaultProject (only defaultDataset and defaultLocation) fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - `defaultDataset: dataform + `defaultDataset: sqlanvil defaultLocation: europe-west2 ` ); @@ -926,8 +926,8 @@ select \${incremental()} as is_incremental` // when defaultProject is not specified, ensuring that warehouse state targets match compiled targets. // This allows incremental appends to work correctly even when defaultProject is not in workflow_settings.yaml. expect(compiledTable.type).equals("incremental"); - expect(compiledTable.enumType).equals(dataform.TableType.INCREMENTAL); - expect(compiledTable.target.schema).equals("dataform"); + expect(compiledTable.enumType).equals(sqlanvil.TableType.INCREMENTAL); + expect(compiledTable.target.schema).equals("sqlanvil"); expect(compiledTable.target.name).equals("incremental_table_without_default_project"); expect(compiledTable.target.database).equals(""); }); diff --git a/core/actions/index.ts b/core/actions/index.ts index 63f214ca..cea3e80c 100644 --- a/core/actions/index.ts +++ b/core/actions/index.ts @@ -1,16 +1,16 @@ -import { Assertion } from "df/core/actions/assertion"; -import { DataPreparation } from "df/core/actions/data_preparation"; -import { Declaration } from "df/core/actions/declaration"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation } from "df/core/actions/operation"; -import { Table } from "df/core/actions/table"; -import { Test } from "df/core/actions/test"; -import { View } from "df/core/actions/view"; -import { IColumnsDescriptor } from "df/core/column_descriptors"; -import { Resolvable } from "df/core/contextables"; -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import { Assertion } from "sa/core/actions/assertion"; +import { DataPreparation } from "sa/core/actions/data_preparation"; +import { Declaration } from "sa/core/actions/declaration"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation } from "sa/core/actions/operation"; +import { Table } from "sa/core/actions/table"; +import { Test } from "sa/core/actions/test"; +import { View } from "sa/core/actions/view"; +import { IColumnsDescriptor } from "sa/core/column_descriptors"; +import { Resolvable } from "sa/core/contextables"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; export type Action = | Table @@ -24,13 +24,13 @@ export type Action = | Test; export type ActionProto = - | dataform.Table // core.proto's Table represents the Table, View or IncrementalTable action type. - | dataform.Operation - | dataform.Assertion - | dataform.Declaration - | dataform.Notebook - | dataform.DataPreparation - | dataform.Test; + | sqlanvil.Table // core.proto's Table represents the Table, View or IncrementalTable action type. + | sqlanvil.Operation + | sqlanvil.Assertion + | sqlanvil.Declaration + | sqlanvil.Notebook + | sqlanvil.DataPreparation + | sqlanvil.Test; // In v4, consider making methods on inheritors of this private, forcing users to use constructors // in order to populate actions. @@ -43,18 +43,18 @@ export abstract class ActionBuilder { } public applySessionToTarget( - targetFromConfig: dataform.Target, - projectConfig: dataform.ProjectConfig, + targetFromConfig: sqlanvil.Target, + projectConfig: sqlanvil.ProjectConfig, fileName?: string, options?: { validateTarget?: boolean; useDefaultAssertionDataset?: boolean; } - ): dataform.Target { + ): sqlanvil.Target { const defaultSchema = options?.useDefaultAssertionDataset ? projectConfig.assertionSchema || projectConfig.defaultSchema : projectConfig.defaultSchema; - const target = dataform.Target.create({ + const target = sqlanvil.Target.create({ name: targetFromConfig.name, schema: targetFromConfig.schema || defaultSchema || undefined, database: targetFromConfig.database || projectConfig.defaultDatabase || undefined @@ -65,8 +65,8 @@ export abstract class ActionBuilder { return target; } - public finalizeTarget(targetFromConfig: dataform.Target): dataform.Target { - return dataform.Target.create({ + public finalizeTarget(targetFromConfig: sqlanvil.Target): sqlanvil.Target { + return sqlanvil.Target.create({ name: this.session.finalizeName(targetFromConfig.name), schema: targetFromConfig.schema ? this.session.finalizeSchema(targetFromConfig.schema) @@ -81,14 +81,14 @@ export abstract class ActionBuilder { public abstract getFileName(): string; /** Retrieves the resolved target from the proto. */ - public abstract getTarget(): dataform.Target; + public abstract getTarget(): sqlanvil.Target; /** Creates the final protobuf representation. */ public abstract compile(): T; protected generateInlineAssertions( - tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig, - proto: dataform.Table + tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig, + proto: sqlanvil.Table ): { uniqueKeyAssertions: Assertion[]; rowConditionsAssertion?: Assertion } { const inlineAssertions: { uniqueKeyAssertions: Assertion[]; @@ -101,11 +101,11 @@ export abstract class ActionBuilder { } const assertionPrefix = !!this.session.projectConfig.builtinAssertionNamePrefix ? `${this.session.projectConfig.builtinAssertionNamePrefix}_` : ""; let uniqueKeys = tableAssertionsConfig.uniqueKeys.map(uniqueKey => - dataform.ActionConfig.TableAssertionsConfig.UniqueKey.create(uniqueKey) + sqlanvil.ActionConfig.TableAssertionsConfig.UniqueKey.create(uniqueKey) ); if (!!tableAssertionsConfig.uniqueKey?.length) { uniqueKeys = [ - dataform.ActionConfig.TableAssertionsConfig.UniqueKey.create({ + sqlanvil.ActionConfig.TableAssertionsConfig.UniqueKey.create({ uniqueKey: tableAssertionsConfig.uniqueKey }) ]; @@ -115,7 +115,7 @@ export abstract class ActionBuilder { const uniqueKeyAssertion = this.session .assert( `${assertionPrefix}${proto.target.schema}_${proto.target.name}_assertions_uniqueKey_${index}`, - dataform.ActionConfig.AssertionConfig.create({ filename: proto.fileName }) + sqlanvil.ActionConfig.AssertionConfig.create({ filename: proto.fileName }) ) .query(ctx => this.session.compilationSql().indexAssertion(ctx.ref(proto.target), uniqueKey) @@ -123,7 +123,7 @@ export abstract class ActionBuilder { if (proto.tags) { uniqueKeyAssertion.tags(proto.tags); } - uniqueKeyAssertion.setParentAction(dataform.Target.create(proto.target)); + uniqueKeyAssertion.setParentAction(sqlanvil.Target.create(proto.target)); if (proto.disabled) { uniqueKeyAssertion.disabled(); } @@ -142,13 +142,13 @@ export abstract class ActionBuilder { inlineAssertions.rowConditionsAssertion = this.session .assert(`${assertionPrefix}${proto.target.schema}_${proto.target.name}_assertions_rowConditions`, { filename: proto.fileName - } as dataform.ActionConfig.AssertionConfig) + } as sqlanvil.ActionConfig.AssertionConfig) .query(ctx => this.session .compilationSql() .rowConditionsAssertion(ctx.ref(proto.target), mergedRowConditions) ); - inlineAssertions.rowConditionsAssertion.setParentAction(dataform.Target.create(proto.target)); + inlineAssertions.rowConditionsAssertion.setParentAction(sqlanvil.Target.create(proto.target)); if (proto.disabled) { inlineAssertions.rowConditionsAssertion.disabled(); } @@ -159,7 +159,7 @@ export abstract class ActionBuilder { return inlineAssertions; } - private validateTarget(target: dataform.Target, fileName: string) { + private validateTarget(target: sqlanvil.Target, fileName: string) { if (target.name.includes(".")) { this.session.compileError( new Error("Action target names cannot include '.'"), @@ -185,10 +185,10 @@ export abstract class ActionBuilder { } export function checkConfigAdditionalOptionsOverlap( - config: dataform.ActionConfig.TableConfig | dataform.ActionConfig.IncrementalTableConfig, + config: sqlanvil.ActionConfig.TableConfig | sqlanvil.ActionConfig.IncrementalTableConfig, session: Session ) { - const target = dataform.Target.create({ + const target = sqlanvil.Target.create({ database: config.project, schema: config.dataset, name: config.name @@ -390,9 +390,9 @@ export class LegacyConfigConverter { // This is a workaround to make bigquery options output empty fields with the same behaviour as // they did previously. public static legacyConvertBigQueryOptions( - bigquery: dataform.IBigQueryOptions - ): dataform.IBigQueryOptions { - let bigqueryFiltered: dataform.IBigQueryOptions = {}; + bigquery: sqlanvil.IBigQueryOptions + ): sqlanvil.IBigQueryOptions { + let bigqueryFiltered: sqlanvil.IBigQueryOptions = {}; Object.entries(bigquery).forEach(([key, value]) => { if (Array.isArray(value) && value.length === 0) { return; @@ -422,7 +422,7 @@ export class LegacyConfigConverter { if (legacyConfig.assertions.uniqueKeys?.[0]?.length > 0) { legacyConfig.assertions.uniqueKeys = (legacyConfig.assertions .uniqueKeys as string[][]).map(uniqueKey => - dataform.ActionConfig.TableAssertionsConfig.UniqueKey.create({ uniqueKey }) + sqlanvil.ActionConfig.TableAssertionsConfig.UniqueKey.create({ uniqueKey }) ); } if (typeof legacyConfig.assertions.nonNull === "string") { diff --git a/core/actions/index_test.ts b/core/actions/index_test.ts index 07359f3e..1bc1caa0 100644 --- a/core/actions/index_test.ts +++ b/core/actions/index_test.ts @@ -3,14 +3,14 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; export const exampleActionDescriptor = { inputSqlxConfigBlock: ` @@ -43,7 +43,7 @@ export const exampleActionDescriptor = { } ], description: "description" - } as dataform.IColumnDescriptor + } as sqlanvil.IColumnDescriptor }; export const exampleBuiltInAssertions = { @@ -112,7 +112,7 @@ export const exampleBuiltInAssertions = { "\nSELECT\n 'rowConditions1' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions1)\nUNION ALL\nSELECT\n 'rowConditions2' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions2)\nUNION ALL\nSELECT\n 'nonNull IS NOT NULL' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (nonNull IS NOT NULL)\n", tags: ["tag1", "tag2"] } - ] as dataform.IAssertion[] + ] as sqlanvil.IAssertion[] }; export const exampleBuiltInAssertionsAsYaml = { @@ -191,13 +191,13 @@ export const exampleBuiltInAssertionsAsYaml = { "\nSELECT\n 'rowConditions1' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions1)\nUNION ALL\nSELECT\n 'rowConditions2' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions2)\nUNION ALL\nSELECT\n 'nonNull IS NOT NULL' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (nonNull IS NOT NULL)\n", tags: ["tag1", "tag2"] } - ] as dataform.IAssertion[] + ] as sqlanvil.IAssertion[] }; suite("actions", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); - const getActionsFromResult = (tableType: string, result: dataform.CoreExecutionResponse) => { + const getActionsFromResult = (tableType: string, result: sqlanvil.CoreExecutionResponse) => { switch (tableType) { case "table": case "view": @@ -234,7 +234,7 @@ SELECT 1` const result = runMainInVm( coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ defaultDatabase: "otherProject", defaultSchema: "otherDataset", assertionSchema: "otherDataset", diff --git a/core/actions/notebook.ts b/core/actions/notebook.ts index fa6af7a6..b169dd30 100644 --- a/core/actions/notebook.ts +++ b/core/actions/notebook.ts @@ -1,23 +1,23 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, configTargetToCompiledGraphTarget, nativeRequire, resolveActionsConfigFilename -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * Notebooks run Jupyter Notebook files, and can output content to the storage buckets defined in * `workflow_settings.yaml` files. * * You can create notebooks in the following ways. Available config options are defined in - * [NotebookConfig](configs#dataform-ActionConfig-NotebookConfig), and are shared across all the + * [NotebookConfig](configs#sqlanvil-ActionConfig-NotebookConfig), and are shared across all the * following ways of creating notebooks. * * **Using action configs files:** @@ -46,7 +46,7 @@ import { dataform } from "df/protos/ts"; * { "cells": [] } * ``` */ -export class Notebook extends ActionBuilder { +export class Notebook extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** @@ -58,7 +58,7 @@ export class Notebook extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Notebook.create(); + private proto = sqlanvil.Notebook.create(); /** @hidden */ constructor(session?: Session, unverifiedConfig?: any, configPath?: string) { @@ -82,7 +82,7 @@ export class Notebook extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -125,13 +125,13 @@ export class Notebook extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ public compile() { return verifyObjectMatchesProto( - dataform.Notebook, + sqlanvil.Notebook, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -141,9 +141,9 @@ export class Notebook extends ActionBuilder { * @hidden Verify config checks that the constructor provided config matches the expected proto * structure. */ - private verifyConfig(unverifiedConfig: any): dataform.ActionConfig.NotebookConfig { + private verifyConfig(unverifiedConfig: any): sqlanvil.ActionConfig.NotebookConfig { return verifyObjectMatchesProto( - dataform.ActionConfig.NotebookConfig, + sqlanvil.ActionConfig.NotebookConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/notebook_test.ts b/core/actions/notebook_test.ts index 354b2af2..ab36ad8b 100644 --- a/core/actions/notebook_test.ts +++ b/core/actions/notebook_test.ts @@ -3,13 +3,13 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; const EMPTY_NOTEBOOK_CONTENTS = '{ "cells": [] }'; @@ -136,7 +136,7 @@ actions: test(`notebook default runtime options are loaded`, () => { const projectDir = createSimpleNotebookProject(` -defaultProject: dataform +defaultProject: sqlanvil defaultLocation: US defaultNotebookRuntimeOptions: outputBucket: gs://some-bucket @@ -150,7 +150,7 @@ defaultNotebookRuntimeOptions: expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); expect(asPlainObject(result.compile.compiledGraph.projectConfig)).deep.equals({ - defaultDatabase: "dataform", + defaultDatabase: "sqlanvil", defaultLocation: "US", defaultNotebookRuntimeOptions: { outputBucket: "gs://some-bucket", @@ -166,7 +166,7 @@ defaultNotebookRuntimeOptions: test(`notebook default runtime options snapshot destination defaults to output bucket`, () => { const projectDir = createSimpleNotebookProject(` -defaultProject: dataform +defaultProject: sqlanvil defaultLocation: US defaultNotebookRuntimeOptions: outputBucket: gs://some-bucket @@ -179,7 +179,7 @@ defaultNotebookRuntimeOptions: expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); expect(asPlainObject(result.compile.compiledGraph.projectConfig)).deep.equals({ - defaultDatabase: "dataform", + defaultDatabase: "sqlanvil", defaultLocation: "US", defaultNotebookRuntimeOptions: { outputBucket: "gs://some-bucket", @@ -195,7 +195,7 @@ defaultNotebookRuntimeOptions: test(`notebook default runtime options throw for snapshot destination with no uri or output bucket`, () => { const projectDir = createSimpleNotebookProject(` -defaultProject: dataform +defaultProject: sqlanvil defaultLocation: US defaultNotebookRuntimeOptions: runtimeTemplateName: projects/test-project/locations/us-central1/notebookRuntimeTemplates/test-template diff --git a/core/actions/operation.ts b/core/actions/operation.ts index 1c9f0056..8123fa7c 100644 --- a/core/actions/operation.ts +++ b/core/actions/operation.ts @@ -1,9 +1,9 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Contextable, IActionContext, JitContextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Contextable, IActionContext, JitContextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -13,8 +13,8 @@ import { resolvableAsTarget, resolveActionsConfigFilename, toResolvable -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -22,7 +22,7 @@ import { dataform } from "df/protos/ts"; * This maintains backwards compatability with older versions. * Consider breaking backwards compatability of these in v4. */ -interface ILegacyOperationConfig extends dataform.ActionConfig.OperationConfig { +interface ILegacyOperationConfig extends sqlanvil.ActionConfig.OperationConfig { dependencies: Resolvable[]; database: string; schema: string; @@ -30,14 +30,14 @@ interface ILegacyOperationConfig extends dataform.ActionConfig.OperationConfig { type: string; } -export type JitOperationResult = string | string[] | dataform.IJitOperationResult; +export type JitOperationResult = string | string[] | sqlanvil.IJitOperationResult; /** * Operations define custom SQL operations that don't fit into the Dataform model of publishing a * table or writing an assertion. * * You can create operations in the following ways. Available config options are defined in - * [OperationConfig](configs#dataform-ActionConfig-OperationConfig), and are shared across all the + * [OperationConfig](configs#sqlanvil-ActionConfig-OperationConfig), and are shared across all the * following ways of creating operations. * * **Using a SQLX file:** @@ -74,7 +74,7 @@ export type JitOperationResult = string | string[] | dataform.IJitOperationResul * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class Operation extends ActionBuilder { +export class Operation extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; @@ -87,7 +87,7 @@ export class Operation extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Operation.create(); + private proto = sqlanvil.Operation.create(); /** @hidden We delay contextification until the final compile step, so hold these here for now. */ private contextableQueries: Contextable; @@ -124,7 +124,7 @@ export class Operation extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -146,7 +146,7 @@ export class Operation extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -183,14 +183,14 @@ export class Operation extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } /** * @deprecated Deprecated in favor of - * [OperationConfig.dependencies](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.dependencies](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets dependencies of the table. */ @@ -207,7 +207,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.hermetic](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.hermetic](configs#sqlanvil-ActionConfig-OperationConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -215,13 +215,13 @@ export class Operation extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [OperationConfig.disabled](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.disabled](configs#sqlanvil-ActionConfig-OperationConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -233,7 +233,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.tags](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.tags](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -249,7 +249,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.hasOutput](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.hasOutput](configs#sqlanvil-ActionConfig-OperationConfig). * * Declares that this action creates a dataset which should be referenceable as a dependency * target, for example by using the `ref` function. @@ -261,7 +261,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.description](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.description](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the description of this assertion. */ @@ -275,11 +275,11 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.columns](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.columns](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the column descriptors of columns in this table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -291,14 +291,14 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.project](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.project](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the database (Google Cloud project ID) in which to create the corresponding view for this * operation. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -308,13 +308,13 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.dataset](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.dataset](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -324,7 +324,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-OperationConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -341,7 +341,7 @@ export class Operation extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -370,7 +370,7 @@ export class Operation extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Operation, + sqlanvil.Operation, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -395,7 +395,7 @@ export class Operation extends ActionBuilder { */ private verifyConfig( unverifiedConfig: ILegacyOperationConfig - ): dataform.ActionConfig.OperationConfig { + ): sqlanvil.ActionConfig.OperationConfig { // The "type" field only exists on legacy view configs. Here we convert them to the new format. if (unverifiedConfig.type) { delete unverifiedConfig.type; @@ -424,7 +424,7 @@ export class Operation extends ActionBuilder { } } return verifyObjectMatchesProto( - dataform.ActionConfig.OperationConfig, + sqlanvil.ActionConfig.OperationConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/operation_test.ts b/core/actions/operation_test.ts index 5c1e4213..c17f1b1e 100644 --- a/core/actions/operation_test.ts +++ b/core/actions/operation_test.ts @@ -3,14 +3,14 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { exampleActionDescriptor } from "df/core/actions/index_test"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { exampleActionDescriptor } from "sa/core/actions/index_test"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("operation", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/actions/table.ts b/core/actions/table.ts index ee156252..b560a269 100644 --- a/core/actions/table.ts +++ b/core/actions/table.ts @@ -1,4 +1,4 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; import { ActionBuilder, checkConfigAdditionalOptionsOverlap, @@ -6,19 +6,19 @@ import { ILegacyTableConfig, LegacyConfigConverter, TableType -} from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { View } from "df/core/actions/view"; -import { ColumnDescriptors } from "df/core/column_descriptors"; +} from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { View } from "sa/core/actions/view"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; import { Contextable, ITableContext, JitContextable, Resolvable, -} from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +} from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -39,11 +39,11 @@ import { validateNoMixedCompilationMode, validateQueryString, validateStorageUriFormat, -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** JiT compilation stage result. String is equivalint to {query: value}. */ -export type JitTableResult = string | dataform.IJitTableResult; +export type JitTableResult = string | sqlanvil.IJitTableResult; /** * Tables are the fundamental building block for storing data when using Dataform. Dataform compiles @@ -51,7 +51,7 @@ export type JitTableResult = string | dataform.IJitTableResult; * BigQuery. * * You can create tables in the following ways. Available config options are defined in - * [TableConfig](configs#dataform-ActionConfig-TableConfig), and are shared across all the + * [TableConfig](configs#sqlanvil-ActionConfig-TableConfig), and are shared across all the * following ways of creating tables. * * **Using a SQLX file:** @@ -88,7 +88,7 @@ export type JitTableResult = string | dataform.IJitTableResult; * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class Table extends ActionBuilder { +export class Table extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** @@ -107,9 +107,9 @@ export class Table extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Table.create({ + private proto = sqlanvil.Table.create({ type: "table", - enumType: dataform.TableType.TABLE, + enumType: sqlanvil.TableType.TABLE, disabled: false, tags: [] }); @@ -179,12 +179,12 @@ export class Table extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } if (config.assertions) { - this.assertions(dataform.ActionConfig.TableAssertionsConfig.create(config.assertions)); + this.assertions(sqlanvil.ActionConfig.TableAssertionsConfig.create(config.assertions)); } if (config.preOperations) { this.preOps(config.preOperations); @@ -205,7 +205,7 @@ export class Table extends ActionBuilder { session.projectConfig.defaultIcebergConfig?.connection ), fileFormat: getFileFormatValueForIcebergTable(config.iceberg.fileFormat?.toString()), - tableFormat: dataform.TableFormat.ICEBERG, + tableFormat: sqlanvil.TableFormat.ICEBERG, storageUri: getStorageUriForIcebergTable( getEffectiveBucketName(session.projectConfig.defaultIcebergConfig?.bucketName, config.iceberg.bucketName), getEffectiveTableFolderRoot(session.projectConfig.defaultIcebergConfig?.tableFolderRoot, config.iceberg.tableFolderRoot), @@ -278,7 +278,7 @@ export class Table extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } @@ -323,7 +323,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.disabled](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.disabled](configs#sqlanvil-ActionConfig-TableConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -337,12 +337,12 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of options available directly on - * [TableConfig](configs#dataform-ActionConfig-TableConfig). For example: + * [TableConfig](configs#sqlanvil-ActionConfig-TableConfig). For example: * `publish("name", { type: "table", partitionBy: "column" }`). * * Sets bigquery options for the action. */ - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { if (!!bigquery.labels && Object.keys(bigquery.labels).length > 0) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; @@ -352,14 +352,14 @@ export class Table extends ActionBuilder { const bigqueryFiltered = LegacyConfigConverter.legacyConvertBigQueryOptions(bigquery); if (Object.values(bigqueryFiltered).length > 0) { - this.proto.bigquery = dataform.BigQueryOptions.create(bigqueryFiltered); + this.proto.bigquery = sqlanvil.BigQueryOptions.create(bigqueryFiltered); } return this; } /** * @deprecated Deprecated in favor of - * [TableConfig.dependencies](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.dependencies](configs#sqlanvil-ActionConfig-TableConfig). * * Sets dependencies of the table. */ @@ -376,7 +376,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.hermetic](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.hermetic](configs#sqlanvil-ActionConfig-TableConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -384,13 +384,13 @@ export class Table extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [TableConfig.tags](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.tags](configs#sqlanvil-ActionConfig-TableConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -406,7 +406,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.description](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.description](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the description of this assertion. */ @@ -420,11 +420,11 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.columns](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.columns](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the column descriptors of columns in this table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -436,13 +436,13 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.project](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.project](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the database (Google Cloud project ID) in which to create the output of this action. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -452,13 +452,13 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.dataset](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.dataset](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -468,7 +468,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.assertions](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig). * * Sets in-line assertions for this table. * @@ -476,7 +476,7 @@ export class Table extends ActionBuilder { * Usage of it via the JS API is deprecated, but the way it applies in-line assertions is still * needed --> */ - public assertions(tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig): Table { + public assertions(tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig): Table { const inlineAssertions = this.generateInlineAssertions(tableAssertionsConfig, this.proto); this.uniqueKeyAssertions = inlineAssertions.uniqueKeyAssertions; this.rowConditionsAssertion = inlineAssertions.rowConditionsAssertion; @@ -485,7 +485,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-TableConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -502,7 +502,7 @@ export class Table extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -522,7 +522,7 @@ export class Table extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -550,7 +550,7 @@ export class Table extends ActionBuilder { this.proto.query = context.apply(this.contextableQuery); - if (this.proto.enumType === dataform.TableType.INCREMENTAL) { + if (this.proto.enumType === sqlanvil.TableType.INCREMENTAL) { this.proto.incrementalQuery = incrementalContext.apply(this.contextableQuery); this.proto.incrementalPreOps = this.contextifyOps(this.contextablePreOps, incrementalContext); @@ -596,8 +596,8 @@ export class Table extends ActionBuilder { private verifyConfig( // `any` is used here to facilitate the type merging of the legacy table config, which is very // different to the new structure. - unverifiedConfig: dataform.ActionConfig.TableConfig | ILegacyTableConfig | any - ): dataform.ActionConfig.TableConfig { + unverifiedConfig: sqlanvil.ActionConfig.TableConfig | ILegacyTableConfig | any + ): sqlanvil.ActionConfig.TableConfig { // The "type" field only exists on legacy table configs. Here we convert them to the // new format. if (unverifiedConfig.type) { @@ -669,7 +669,7 @@ export class Table extends ActionBuilder { } const config = verifyObjectMatchesProto( - dataform.ActionConfig.TableConfig, + sqlanvil.ActionConfig.TableConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -678,7 +678,7 @@ export class Table extends ActionBuilder { this.session.compileError( `requirePartitionFilter/partitionExpirationDays are not valid for non partitioned BigQuery tables`, config.filename, - dataform.Target.create({ + sqlanvil.Target.create({ database: config.project, schema: config.dataset, name: config.name @@ -768,7 +768,7 @@ export class TableContext implements ITableContext { return ""; } - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { this.table.bigquery(bigquery); return ""; } diff --git a/core/actions/table_test.ts b/core/actions/table_test.ts index 38a010d1..094dd4bd 100644 --- a/core/actions/table_test.ts +++ b/core/actions/table_test.ts @@ -7,15 +7,15 @@ import { exampleActionDescriptor, exampleBuiltInAssertions, exampleBuiltInAssertionsAsYaml -} from "df/core/actions/index_test"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/core/actions/index_test"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("table", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -564,7 +564,7 @@ defaultIcebergConfig: wsContent: VALID_WORKFLOW_SETTINGS_YAML, }, { - testName: "defaults to \`_dataform\` for tableFolderRoot", + testName: "defaults to \`_sqlanvil\` for tableFolderRoot", configBlock: ` name: "table3", dataset: "dataset3", @@ -582,7 +582,7 @@ defaultIcebergConfig: tableFormat: "ICEBERG", fileFormat: "PARQUET", connection: "gcp.us.conn-id", - storageUri: "gs://my-bucket/_dataform/my-subpath", + storageUri: "gs://my-bucket/_sqlanvil/my-subpath", }, }, expectError: false, diff --git a/core/actions/test.ts b/core/actions/test.ts index 2034069f..1e8642cb 100644 --- a/core/actions/test.ts +++ b/core/actions/test.ts @@ -1,11 +1,11 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder, INamedConfig, TableType } from "df/core/actions"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { Contextable, IActionContext, ITableContext, Resolvable } from "df/core/contextables"; -import { Session } from "df/core/session"; -import { targetStringifier } from "df/core/targets"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder, INamedConfig, TableType } from "sa/core/actions"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { Contextable, IActionContext, ITableContext, Resolvable } from "sa/core/contextables"; +import { Session } from "sa/core/session"; +import { targetStringifier } from "sa/core/targets"; import { ambiguousActionNameMsg, checkExcessProperties, @@ -13,8 +13,8 @@ import { strictKeysOf, stringifyResolvable, toResolvable -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * Configuration options for unit tests. @@ -74,19 +74,19 @@ const ITestConfigProperties = strictKeysOf()(["type", "dataset", "n * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `input` and `expect` come from. */ -export class Test extends ActionBuilder { +export class Test extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** @hidden We delay contextification until the final compile step, so hold these here for now. */ public contextableInputs = new Map>(); private contextableQuery: Contextable; - private testTarget: dataform.ITarget; + private testTarget: sqlanvil.ITarget; /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Test.create(); + private proto = sqlanvil.Test.create(); /** @hidden */ constructor(session?: Session, config?: ITestConfig) { @@ -111,7 +111,7 @@ export class Test extends ActionBuilder { } if (config.dataset) { // Determine target from the parent dataset name - this.testTarget = dataform.Target.create( + this.testTarget = sqlanvil.Target.create( this.applySessionToTarget( resolvableAsTarget( toResolvable(config.dataset) @@ -119,7 +119,7 @@ export class Test extends ActionBuilder { this.session.projectConfig ) ); - const canonicalTestTarget = dataform.Target.create( + const canonicalTestTarget = sqlanvil.Target.create( this.applySessionToTarget( resolvableAsTarget( toResolvable(config.dataset) @@ -174,12 +174,12 @@ export class Test extends ActionBuilder { } /** @hidden */ - public getTarget(): dataform.Target { - return dataform.Target.create(this.proto.target); + public getTarget(): sqlanvil.Target { + return sqlanvil.Target.create(this.proto.target); } - public getTestTarget(): dataform.Target { - return dataform.Target.create(this.testTarget); + public getTestTarget(): sqlanvil.Target { + return sqlanvil.Target.create(this.testTarget); } public setFilename(filename: string) { @@ -239,7 +239,7 @@ export class Test extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Test, + sqlanvil.Test, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -358,8 +358,8 @@ class RefReplacingContext implements ITableContext { } } -function overrideTargetWithNewName(target: dataform.ITarget, testName: string): dataform.Target { - return dataform.Target.create({ +function overrideTargetWithNewName(target: sqlanvil.ITarget, testName: string): sqlanvil.Target { + return sqlanvil.Target.create({ database: target.database, schema: target.schema, name: testName diff --git a/core/actions/test_test.ts b/core/actions/test_test.ts index 8eb776c7..e9328483 100644 --- a/core/actions/test_test.ts +++ b/core/actions/test_test.ts @@ -3,13 +3,13 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("test", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/actions/view.ts b/core/actions/view.ts index 97f1b5ce..de85ac30 100644 --- a/core/actions/view.ts +++ b/core/actions/view.ts @@ -1,17 +1,17 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; import { ActionBuilder, ILegacyBigQueryOptions, LegacyConfigConverter, TableType -} from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Table } from "df/core/actions/table"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Contextable, ITableContext, JitContextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +} from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Table } from "sa/core/actions/table"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Contextable, ITableContext, JitContextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -25,8 +25,8 @@ import { toResolvable, validateNoMixedCompilationMode, validateQueryString, -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -41,7 +41,7 @@ export interface ILegacyViewBigqueryConfig { additionalOptions: { [key: string]: string }; } -export type JitViewResult = string | dataform.IJitTableResult; +export type JitViewResult = string | sqlanvil.IJitTableResult; /** * Views are virtualised tables. They are useful for creating a new structured table without having @@ -49,7 +49,7 @@ export type JitViewResult = string | dataform.IJitTableResult; * processing and storage. * * You can create views in the following ways. Available config options are defined in - * [ViewConfig](configs#dataform-ActionConfig-ViewConfig), and are shared across all the + * [ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig), and are shared across all the * following ways of creating tables. * * **Using a SQLX file:** @@ -86,7 +86,7 @@ export type JitViewResult = string | dataform.IJitTableResult; * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class View extends ActionBuilder { +export class View extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; @@ -106,9 +106,9 @@ export class View extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Table.create({ + private proto = sqlanvil.Table.create({ type: "view", - enumType: dataform.TableType.VIEW, + enumType: sqlanvil.TableType.VIEW, disabled: false, tags: [] }); @@ -158,7 +158,7 @@ export class View extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -183,7 +183,7 @@ export class View extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -194,7 +194,7 @@ export class View extends ActionBuilder { this.schema(config.dataset); } if (config.assertions) { - this.assertions(dataform.ActionConfig.TableAssertionsConfig.create(config.assertions)); + this.assertions(sqlanvil.ActionConfig.TableAssertionsConfig.create(config.assertions)); } if (config.materialized) { this.materialized(config.materialized); @@ -313,7 +313,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.disabled](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.disabled](configs#sqlanvil-ActionConfig-ViewConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -327,7 +327,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.materialized](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.materialized](configs#sqlanvil-ActionConfig-ViewConfig). * * Applies the materialized view optimization, see * https://cloud.google.com/bigquery/docs/materialized-views-intro. @@ -338,12 +338,12 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of options available directly on - * [ViewConfig](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets bigquery options for the action. */ - public bigquery(bigquery: dataform.IBigQueryOptions) { - this.proto.bigquery = dataform.BigQueryOptions.create(bigquery); + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { + this.proto.bigquery = sqlanvil.BigQueryOptions.create(bigquery); if (!!bigquery.labels) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; @@ -355,7 +355,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.dependencies](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.dependencies](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets dependencies of the view. */ @@ -372,7 +372,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.hermetic](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.hermetic](configs#sqlanvil-ActionConfig-ViewConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -380,13 +380,13 @@ export class View extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [ViewConfig.tags](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.tags](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -402,7 +402,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.description](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.description](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the description of this view. */ @@ -416,11 +416,11 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.columns](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.columns](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the column descriptors of columns in this view. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -432,14 +432,14 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.project](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.project](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the * Sets the database (Google Cloud project ID) in which to create the output of this action. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -449,13 +449,13 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.dataset](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.dataset](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -465,7 +465,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.assertions](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.assertions](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets in-line assertions for this view. * @@ -473,7 +473,7 @@ export class View extends ActionBuilder { * Usage of it via the JS API is deprecated, but the way it applies in-line assertions is still * needed --> */ - public assertions(tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig): View { + public assertions(tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig): View { const inlineAssertions = this.generateInlineAssertions(tableAssertionsConfig, this.proto); this.uniqueKeyAssertions = inlineAssertions.uniqueKeyAssertions; this.rowConditionsAssertion = inlineAssertions.rowConditionsAssertion; @@ -482,7 +482,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-ViewConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -496,7 +496,7 @@ export class View extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } @@ -508,7 +508,7 @@ export class View extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -520,7 +520,7 @@ export class View extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -549,7 +549,7 @@ export class View extends ActionBuilder { this.proto.query = context.apply(this.contextableQuery); - if (this.proto.enumType === dataform.TableType.INCREMENTAL) { + if (this.proto.enumType === sqlanvil.TableType.INCREMENTAL) { this.proto.incrementalQuery = incrementalContext.apply(this.contextableQuery); this.proto.incrementalPreOps = this.contextifyOps(this.contextablePreOps, incrementalContext); @@ -595,8 +595,8 @@ export class View extends ActionBuilder { private verifyConfig( // `any` is used here to facilitate the type merging of the legacy table config, which is very // different to the new structure. - unverifiedConfig: dataform.ActionConfig.ViewConfig | ILegacyBigQueryOptions | any - ): dataform.ActionConfig.ViewConfig { + unverifiedConfig: sqlanvil.ActionConfig.ViewConfig | ILegacyBigQueryOptions | any + ): sqlanvil.ActionConfig.ViewConfig { // The "type" field only exists on legacy view configs. Here we convert them to the new format. if (unverifiedConfig.type) { delete unverifiedConfig.type; @@ -655,7 +655,7 @@ export class View extends ActionBuilder { } const config = verifyObjectMatchesProto( - dataform.ActionConfig.ViewConfig, + sqlanvil.ActionConfig.ViewConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -664,7 +664,7 @@ export class View extends ActionBuilder { this.session.compileError( `partitionBy/clusterBy can be applied only to materialized views`, config.filename, - dataform.Target.create({ + sqlanvil.Target.create({ database: config.project, schema: config.dataset, name: config.name @@ -745,7 +745,7 @@ export class ViewContext implements ITableContext { return ""; } - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { this.view.bigquery(bigquery); return ""; } diff --git a/core/actions/view_test.ts b/core/actions/view_test.ts index b72c4444..131caa71 100644 --- a/core/actions/view_test.ts +++ b/core/actions/view_test.ts @@ -7,14 +7,14 @@ import { exampleActionDescriptor, exampleBuiltInAssertions, exampleBuiltInAssertionsAsYaml -} from "df/core/actions/index_test"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/core/actions/index_test"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("view", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/column_descriptors.ts b/core/column_descriptors.ts index 95636310..b9efd0cf 100644 --- a/core/column_descriptors.ts +++ b/core/column_descriptors.ts @@ -1,5 +1,5 @@ -import * as utils from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import * as utils from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @deprecated @@ -70,10 +70,10 @@ export const IRecordDescriptorProperties = () => */ export class ColumnDescriptors { public static mapConfigProtoToCompilationProto( - columns: dataform.ActionConfig.ColumnDescriptor[] - ): dataform.IColumnDescriptor[] { + columns: sqlanvil.ActionConfig.ColumnDescriptor[] + ): sqlanvil.IColumnDescriptor[] { return columns.map(column => { - return dataform.ColumnDescriptor.create({ + return sqlanvil.ColumnDescriptor.create({ path: column.path, description: column.description, tags: column.tags, @@ -84,7 +84,7 @@ export class ColumnDescriptors { public static mapLegacyObjectToConfigProto( columns: IColumnsDescriptor - ): dataform.ActionConfig.ColumnDescriptor[] { + ): sqlanvil.ActionConfig.ColumnDescriptor[] { return Object.keys(columns) .map(column => ColumnDescriptors.mapColumnDescriptionToProto([column], columns[column])) .flat(); @@ -93,18 +93,18 @@ export class ColumnDescriptors { public static mapColumnDescriptionToProto( currentPath: string[], description: string | IRecordDescriptor - ): dataform.ActionConfig.ColumnDescriptor[] { + ): sqlanvil.ActionConfig.ColumnDescriptor[] { if (typeof description === "string") { return [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description, path: currentPath }) ]; } - const columnDescriptor: dataform.ActionConfig.ColumnDescriptor[] = !!description + const columnDescriptor: sqlanvil.ActionConfig.ColumnDescriptor[] = !!description ? [ - dataform.ActionConfig.ColumnDescriptor.create({ + sqlanvil.ActionConfig.ColumnDescriptor.create({ path: currentPath, description: description.description, tags: typeof description.tags === "string" ? [description.tags] : description.tags, @@ -136,7 +136,7 @@ export class LegacyColumnDescriptors { public static mapToColumnProtoArray( columns: IColumnsDescriptor, reportError: (e: Error) => void - ): dataform.IColumnDescriptor[] { + ): sqlanvil.IColumnDescriptor[] { return Object.keys(columns) .map(column => LegacyColumnDescriptors.mapColumnDescriptionToProto([column], columns[column], reportError) @@ -148,10 +148,10 @@ export class LegacyColumnDescriptors { currentPath: string[], description: string | IRecordDescriptor, reportError: (e: Error) => void - ): dataform.IColumnDescriptor[] { + ): sqlanvil.IColumnDescriptor[] { if (typeof description === "string") { return [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description, path: currentPath }) @@ -163,9 +163,9 @@ export class LegacyColumnDescriptors { IRecordDescriptorProperties(), `${currentPath.join(".")} column descriptor` ); - const columnDescriptor: dataform.IColumnDescriptor[] = !!description + const columnDescriptor: sqlanvil.IColumnDescriptor[] = !!description ? [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ path: currentPath, description: description.description, displayName: description.displayName, diff --git a/core/compilation_sql/index.ts b/core/compilation_sql/index.ts index 4639a545..9eec8329 100644 --- a/core/compilation_sql/index.ts +++ b/core/compilation_sql/index.ts @@ -1,12 +1,12 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; export class CompilationSql { constructor( - private readonly project: dataform.IProjectConfig, - private readonly dataformCoreVersion: string + private readonly project: sqlanvil.IProjectConfig, + private readonly sqlanvilCoreVersion: string ) {} - public resolveTarget(target: dataform.ITarget) { + public resolveTarget(target: sqlanvil.ITarget) { const database = target.database || this.project.defaultDatabase; if (!database) { return `\`${target.schema || this.project.defaultSchema}.${target.name}\``; diff --git a/core/compilers.ts b/core/compilers.ts index a760153f..d0620be5 100644 --- a/core/compilers.ts +++ b/core/compilers.ts @@ -1,7 +1,7 @@ import { load as loadYaml, YAMLException } from "js-yaml"; -import * as Path from "df/core/path"; -import { SyntaxTreeNode, SyntaxTreeNodeType } from "df/sqlx/lexer"; +import * as Path from "sa/core/path"; +import { SyntaxTreeNode, SyntaxTreeNodeType } from "sa/sqlx/lexer"; const CONTEXT_FUNCTIONS = [ "self", @@ -87,7 +87,7 @@ function compileSqlx(rootNode: SyntaxTreeNode, path: string): string { rootNode ); - return `dataform.sqlxAction({ + return `sqlanvil.sqlxAction({ sqlxConfig: { name: "${Path.escapedBasename(path)}", type: "operations", diff --git a/core/compilers_test.ts b/core/compilers_test.ts index f5410e55..16a9f3c0 100644 --- a/core/compilers_test.ts +++ b/core/compilers_test.ts @@ -1,7 +1,7 @@ import { expect } from "chai"; -import { compile } from "df/core/compilers"; -import { suite, test } from "df/testing"; +import { compile } from "sa/core/compilers"; +import { suite, test } from "sa/testing"; suite("core/compilers", () => { suite("compile", () => { @@ -9,7 +9,7 @@ suite("core/compilers", () => { const code = `config { type: "table" } select 1`; const path = "definitions/foo.sqlx"; const result = compile(code, path); - expect(result).to.include("dataform.sqlxAction"); + expect(result).to.include("sqlanvil.sqlxAction"); expect(result).to.include("name: \"foo\""); expect(result).to.include("{ type: \"table\" }"); // The sqlx compiler will format the query. diff --git a/core/contextables.ts b/core/contextables.ts index 3347e664..75c968fe 100644 --- a/core/contextables.ts +++ b/core/contextables.ts @@ -1,10 +1,10 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; /** * A resolvable is a reference to an action, and it can be either the string representation of the * action's target, or the target of the action. */ -export type Resolvable = string | dataform.ITarget; +export type Resolvable = string | sqlanvil.ITarget; /** * Contextable arguments can either pass a plain value for their generic type `T` or can pass a @@ -89,13 +89,13 @@ export interface ITableContext extends IActionContext { /** JiT context, accessible at JiT compilation stage. */ export type JitContext = T & { /** Direct access to adapter. */ - adapter: dataform.DbAdapter, + adapter: sqlanvil.DbAdapter, /** JiT data object. */ data?: { [k: string]: any }, /** Original JiT compilation request. */ - request: dataform.IJitCompilationRequest, + request: sqlanvil.IJitCompilationRequest, /** Current execution information for introspection. */ - executionData: dataform.IRunningExecutionData, + executionData: sqlanvil.IRunningExecutionData, }; /** diff --git a/core/extension.ts b/core/extension.ts index c3f63760..f15ec3c5 100644 --- a/core/extension.ts +++ b/core/extension.ts @@ -1,5 +1,5 @@ -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; /** * Extension interface. @@ -9,5 +9,5 @@ export interface IDataformExtension { * Run additional compilation steps. * Passed session should be used for both new nodes creation and persisting errors. */ - compile(request: dataform.ICompileExecutionRequest, session: Session): void; + compile(request: sqlanvil.ICompileExecutionRequest, session: Session): void; } diff --git a/core/index.ts b/core/index.ts index 7fcdc802..55b53723 100644 --- a/core/index.ts +++ b/core/index.ts @@ -1,10 +1,10 @@ -import { compile as compiler } from "df/core/compilers"; -import { IDataformExtension } from "df/core/extension"; -import { IJitCompiler, jitCompiler } from "df/core/jit_compiler"; -import { main } from "df/core/main"; -import { Session } from "df/core/session"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; +import { compile as compiler } from "sa/core/compilers"; +import { IDataformExtension } from "sa/core/extension"; +import { IJitCompiler, jitCompiler } from "sa/core/jit_compiler"; +import { main } from "sa/core/main"; +import { Session } from "sa/core/session"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; // Create static session object. // This hack just enforces the singleton session object to @@ -17,7 +17,7 @@ function globalSession() { } const session = globalSession(); -const supportedFeatures = [dataform.SupportedFeatures.ARRAY_BUFFER_IPC]; +const supportedFeatures = [sqlanvil.SupportedFeatures.ARRAY_BUFFER_IPC]; // Older versions of the CLI are not compatible with Core version ^3.0.0, and throw when this method // is not available. Instead this more interpretable error message is thrown. diff --git a/core/jit_compiler.ts b/core/jit_compiler.ts index c7993528..a91871a3 100644 --- a/core/jit_compiler.ts +++ b/core/jit_compiler.ts @@ -1,10 +1,10 @@ import * as $protobuf from "protobufjs"; -import { JitOperationResult } from "df/core/actions/operation"; -import { JitTableResult } from "df/core/actions/table"; -import { IActionContext, ITableContext, JitContext } from "df/core/contextables"; -import { IncrementalTableJitContext, SqlActionJitContext, TableJitContext } from "df/core/jit_context"; -import { dataform } from "df/protos/ts"; +import { JitOperationResult } from "sa/core/actions/operation"; +import { JitTableResult } from "sa/core/actions/table"; +import { IActionContext, ITableContext, JitContext } from "sa/core/contextables"; +import { IncrementalTableJitContext, SqlActionJitContext, TableJitContext } from "sa/core/jit_context"; +import { sqlanvil } from "sa/protos/ts"; function makeMainBody(code: string): (jctx: JitContext) => Promise { return ( @@ -17,21 +17,21 @@ function makeMainBody(code: string): (jctx: JitContext) => }); } -function makeJitTableResult(result: JitTableResult): dataform.IJitTableResult { - let jitResult: dataform.IJitTableResult = {}; +function makeJitTableResult(result: JitTableResult): sqlanvil.IJitTableResult { + let jitResult: sqlanvil.IJitTableResult = {}; if (typeof result === "string") { jitResult.query = result; } else { jitResult = result; } - return dataform.JitTableResult.create(jitResult); + return sqlanvil.JitTableResult.create(jitResult); } function jitCompileOperation( - request: dataform.IJitCompilationRequest, - adapter: dataform.DbAdapter, -): Promise { + request: sqlanvil.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, +): Promise { const mainBody = makeMainBody(request.jitCode); const jctx: JitContext = new SqlActionJitContext( @@ -47,14 +47,14 @@ function jitCompileOperation( queries = mainResult.queries; } - return dataform.JitOperationResult.create({ queries }); + return sqlanvil.JitOperationResult.create({ queries }); }); } function jitCompileTable( - request: dataform.IJitCompilationRequest, - adapter: dataform.DbAdapter, -): Promise { + request: sqlanvil.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, +): Promise { const mainBody = makeMainBody(request.jitCode); const jctx: JitContext = new TableJitContext( @@ -64,9 +64,9 @@ function jitCompileTable( } function jitCompileIncrementalTable( - request: dataform.IJitCompilationRequest, - adapter: dataform.DbAdapter, -): Promise { + request: sqlanvil.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, +): Promise { const mainBody = makeMainBody(request.jitCode); const incrementalJctx = new IncrementalTableJitContext( @@ -80,7 +80,7 @@ function jitCompileIncrementalTable( mainBody(incrementalJctx), mainBody(regularJctx), ]).then(([incrementalResult, regularResult]) => { - return dataform.JitIncrementalTableResult.create({ + return sqlanvil.JitIncrementalTableResult.create({ incremental: makeJitTableResult(incrementalResult), regular: makeJitTableResult(regularResult), }); @@ -94,22 +94,22 @@ export interface IJitCompiler { /** RPC callback, implementing DbAdapter. */ export type RpcCallback = (method: string, request: Uint8Array, callback: (error: Error | null, response: Uint8Array) => void) => void; -export function jitCompile(request: dataform.IJitCompilationRequest, rpcCallback: RpcCallback): Promise { +export function jitCompile(request: sqlanvil.IJitCompilationRequest, rpcCallback: RpcCallback): Promise { const rpcImpl: $protobuf.RPCImpl = (method, internalRequest, callback) => { rpcCallback(method.name, internalRequest, callback); }; - const dbAdapter = dataform.DbAdapter.create(rpcImpl); + const dbAdapter = sqlanvil.DbAdapter.create(rpcImpl); switch (request.compilationTargetType) { - case dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION: + case sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION: return jitCompileOperation(request, dbAdapter).then( - operation => dataform.JitCompilationResponse.create({ operation })); - case dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE: + operation => sqlanvil.JitCompilationResponse.create({ operation })); + case sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE: return jitCompileTable(request, dbAdapter).then( - table => dataform.JitCompilationResponse.create({ table })); - case dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE: + table => sqlanvil.JitCompilationResponse.create({ table })); + case sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE: return jitCompileIncrementalTable(request, dbAdapter).then( - incrementalTable => dataform.JitCompilationResponse.create({ incrementalTable })); + incrementalTable => sqlanvil.JitCompilationResponse.create({ incrementalTable })); default: throw new Error(`Unrecognized compilation target type: ${request.compilationTargetType}`); } @@ -119,9 +119,9 @@ export function jitCompile(request: dataform.IJitCompilationRequest, rpcCallback export function jitCompiler(rpcCallback: RpcCallback): IJitCompiler { return { compile: (request: Uint8Array) => { - const requestMessage = dataform.JitCompilationRequest.decode(request); + const requestMessage = sqlanvil.JitCompilationRequest.decode(request); return jitCompile(requestMessage, rpcCallback).then( - response => dataform.JitCompilationResponse.encode(response).finish() + response => sqlanvil.JitCompilationResponse.encode(response).finish() ); } }; diff --git a/core/jit_compiler_test.ts b/core/jit_compiler_test.ts index 9d252e05..19947c1b 100644 --- a/core/jit_compiler_test.ts +++ b/core/jit_compiler_test.ts @@ -1,15 +1,15 @@ import { expect } from "chai"; import Long from "long"; -import { jitCompile } from "df/core/jit_compiler"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { jitCompile } from "sa/core/jit_compiler"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("jit_compiler", () => { const rpcCallback: (method: string, request: Uint8Array, callback: (error: Error | null, response: Uint8Array) => void) => void = (method, request, callback) => { callback(null, new Uint8Array()); }; - const target = dataform.Target.create({ + const target = sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" @@ -17,77 +17,77 @@ suite("jit_compiler", () => { suite("jitCompileOperation", () => { test("compiles operation returning string", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => "SELECT 1"`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation returning array", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ["SELECT 1", "SELECT 2"]`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1", "SELECT 2"]); }); test("compiles operation returning object", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ({ queries: ["SELECT 1"] })`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation using context", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ({ queries: [\`SELECT "\${ctx.name()}"\`] })`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(['SELECT "name"']); }); test("compiles operation with arrow function returning promise", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `(ctx) => Promise.resolve("SELECT 1")`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation with async function", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async function(ctx) { return "SELECT 1"; }`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation with regular function returning promise", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `function(ctx) { return Promise.resolve("SELECT 1"); }`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); @@ -96,22 +96,22 @@ suite("jit_compiler", () => { suite("jitCompileTable", () => { test("compiles table returning string", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => "SELECT 1"`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, }); const result = await jitCompile(request, rpcCallback); expect(result.table.query).to.equal("SELECT 1"); }); test("compiles table returning object", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ({ query: "SELECT 1", preOps: ["PRE"], postOps: ["POST"] })`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, }); const result = await jitCompile(request, rpcCallback); expect(result.table.query).to.equal("SELECT 1"); @@ -122,7 +122,7 @@ suite("jit_compiler", () => { suite("jitCompileIncrementalTable", () => { test("compiles incremental table", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => { if (ctx.incremental()) { return { query: "SELECT INC" }; @@ -131,7 +131,7 @@ suite("jit_compiler", () => { }`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE, }); const result = await jitCompile(request, rpcCallback); expect(result.incrementalTable.incremental?.query).to.equal("SELECT INC"); @@ -141,16 +141,16 @@ suite("jit_compiler", () => { suite("jitCompileContext", () => { test("can reference self and other tables", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (jctx) => \`$\{jctx.self()\}\n$\{jctx.ref('other')\}\``, target, jitData: {}, - dependencies: [dataform.Target.create({ + dependencies: [sqlanvil.Target.create({ database: "db", schema: "schema", name: "other", })], - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, }); const result = await jitCompile(request, rpcCallback); @@ -158,11 +158,11 @@ suite("jit_compiler", () => { }); test("can reference execution info data", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (jctx) => \`$\{jctx.executionData.executionStartTime.seconds\}\n$\{jctx.executionData.executionId\}\``, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, executionData: { executionId: "test-id", executionStartTime: {seconds: Long.fromNumber(1774974514), nanos: 481}, diff --git a/core/jit_context.ts b/core/jit_context.ts index ed042326..e75e2eec 100644 --- a/core/jit_context.ts +++ b/core/jit_context.ts @@ -1,23 +1,23 @@ -import { Structs } from "df/common/protos/structs"; -import { IActionContext, ITableContext, JitContext, Resolvable } from "df/core/contextables"; -import { ambiguousActionNameMsg, resolvableAsTarget, ResolvableMap, stringifyResolvable, toResolvable } from "df/core/utils"; -import { dataform, google } from "df/protos/ts"; +import { Structs } from "sa/common/protos/structs"; +import { IActionContext, ITableContext, JitContext, Resolvable } from "sa/core/contextables"; +import { ambiguousActionNameMsg, resolvableAsTarget, ResolvableMap, stringifyResolvable, toResolvable } from "sa/core/utils"; +import { sqlanvil, google } from "sa/protos/ts"; -function canonicalTargetValue(target: dataform.ITarget): string { +function canonicalTargetValue(target: sqlanvil.ITarget): string { return `${target.database}.${target.schema}.${target.name}`; } /** Generate SQL action JiT context. */ export class SqlActionJitContext implements JitContext { public readonly data: { [k: string]: any } | undefined; - public readonly executionData: dataform.IRunningExecutionData; + public readonly executionData: sqlanvil.IRunningExecutionData; - private readonly target: dataform.ITarget; + private readonly target: sqlanvil.ITarget; private readonly resolvableMap: ResolvableMap; constructor( - public readonly adapter: dataform.DbAdapter, - public readonly request: dataform.IJitCompilationRequest, + public readonly adapter: sqlanvil.DbAdapter, + public readonly request: sqlanvil.IJitCompilationRequest, ) { this.target = request.target; const dependencies = request.dependencies; @@ -77,8 +77,8 @@ export class SqlActionJitContext implements JitContext { /** JiT context for table and view actions. */ export class TableJitContext extends SqlActionJitContext implements JitContext { constructor( - adapter: dataform.DbAdapter, - request: dataform.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, + request: sqlanvil.IJitCompilationRequest, ) { super(adapter, request); } @@ -94,8 +94,8 @@ export class TableJitContext extends SqlActionJitContext implements JitContext { suite("SqlActionJitContext", () => { - const adapter = {} as dataform.DbAdapter; + const adapter = {} as sqlanvil.DbAdapter; const jitData = google.protobuf.Struct.create({ fields: { key: google.protobuf.Value.create({ @@ -42,14 +42,14 @@ suite("jit_context", () => { }) } }); - const request = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const request = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" }), dependencies: [ - dataform.Target.create({ + sqlanvil.Target.create({ database: "db", schema: "schema", name: "dep" @@ -58,8 +58,8 @@ suite("jit_context", () => { filePaths: [], jitData, }); - const withoutDependenciesRequest = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const withoutDependenciesRequest = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" @@ -136,9 +136,9 @@ suite("jit_context", () => { }); suite("TableJitContext", () => { - const adapter = {} as dataform.DbAdapter; - const request = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const adapter = {} as sqlanvil.DbAdapter; + const request = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" @@ -159,9 +159,9 @@ suite("jit_context", () => { }); suite("IncrementalTableJitContext", () => { - const adapter = {} as dataform.DbAdapter; - const request = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const adapter = {} as sqlanvil.DbAdapter; + const request = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" diff --git a/core/main.ts b/core/main.ts index 309c76ea..03dd1171 100644 --- a/core/main.ts +++ b/core/main.ts @@ -3,38 +3,38 @@ import { encode64, verifyObjectMatchesProto, VerifyProtoErrorBehaviour -} from "df/common/protos"; -import { Assertion } from "df/core/actions/assertion"; -import { DataPreparation } from "df/core/actions/data_preparation"; -import { Declaration } from "df/core/actions/declaration"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation } from "df/core/actions/operation"; -import { Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { IDataformExtension } from "df/core/extension"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; -import { nativeRequire } from "df/core/utils"; -import { readWorkflowSettings } from "df/core/workflow_settings"; -import { dataform } from "df/protos/ts"; +} from "sa/common/protos"; +import { Assertion } from "sa/core/actions/assertion"; +import { DataPreparation } from "sa/core/actions/data_preparation"; +import { Declaration } from "sa/core/actions/declaration"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation } from "sa/core/actions/operation"; +import { Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { IDataformExtension } from "sa/core/extension"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; +import { nativeRequire } from "sa/core/utils"; +import { readWorkflowSettings } from "sa/core/workflow_settings"; +import { sqlanvil } from "sa/protos/ts"; /** * This is the main entry point into the user space code that should be invoked by the compilation wrapper sandbox. * - * @param coreExecutionRequest an encoded @see {@link dataform.CoreExecutionRequest} proto. - * @returns an encoded @see {@link dataform.CoreExecutionResponse} proto. + * @param coreExecutionRequest an encoded @see {@link sqlanvil.CoreExecutionRequest} proto. + * @returns an encoded @see {@link sqlanvil.CoreExecutionResponse} proto. */ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | string { const globalAny = global as any; - let request: dataform.CoreExecutionRequest; + let request: sqlanvil.CoreExecutionRequest; if (typeof coreExecutionRequest === "string") { // Older versions of the Dataform CLI send a base64 encoded string. // See https://github.com/dataform-co/dataform/pull/1570. - request = decode64(dataform.CoreExecutionRequest, coreExecutionRequest); + request = decode64(sqlanvil.CoreExecutionRequest, coreExecutionRequest); } else { - request = dataform.CoreExecutionRequest.decode(coreExecutionRequest); + request = sqlanvil.CoreExecutionRequest.decode(coreExecutionRequest); } const compileRequest = request.compile; @@ -45,7 +45,7 @@ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | st // Merge in project config overrides. const projectConfigOverride = compileRequest.compileConfig.projectConfigOverride ?? {}; - projectConfig = dataform.ProjectConfig.create({ + projectConfig = sqlanvil.ProjectConfig.create({ ...projectConfig, ...projectConfigOverride, vars: { ...projectConfig.vars, ...projectConfigOverride.vars } @@ -56,23 +56,23 @@ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | st session.init(compileRequest.compileConfig.projectDir, projectConfig, projectConfig); // Allow "includes" files to use the current session object. - globalAny.dataform = session; + globalAny.sqlanvil = session; prologueCompile(compileRequest, session); mainCompile(compileRequest, session); - const coreExecutionResponse = dataform.CoreExecutionResponse.create({ + const coreExecutionResponse = sqlanvil.CoreExecutionResponse.create({ compile: { compiledGraph: session.compile() } }); if (typeof coreExecutionRequest === "string") { // Older versions of the Dataform CLI expect a base64 encoded string to be returned. // See https://github.com/dataform-co/dataform/pull/1570. - return encode64(dataform.CoreExecutionResponse, coreExecutionResponse); + return encode64(sqlanvil.CoreExecutionResponse, coreExecutionResponse); } - return dataform.CoreExecutionResponse.encode(coreExecutionResponse).finish(); + return sqlanvil.CoreExecutionResponse.encode(coreExecutionResponse).finish(); } function loadActionConfigs(session: Session, filePaths: string[]) { @@ -87,13 +87,13 @@ function loadActionConfigs(session: Session, filePaths: string[]) { .forEach(actionConfigsPath => { const actionConfigs = loadActionConfigsFile(session, actionConfigsPath); actionConfigs.actions.forEach(nonProtoActionConfig => { - const actionConfig = dataform.ActionConfig.create(nonProtoActionConfig); + const actionConfig = sqlanvil.ActionConfig.create(nonProtoActionConfig); if (actionConfig.table) { session.actions.push( new Table( session, - dataform.ActionConfig.TableConfig.create(actionConfig.table), + sqlanvil.ActionConfig.TableConfig.create(actionConfig.table), actionConfigsPath ) ); @@ -101,7 +101,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new View( session, - dataform.ActionConfig.ViewConfig.create(actionConfig.view), + sqlanvil.ActionConfig.ViewConfig.create(actionConfig.view), actionConfigsPath ) ); @@ -109,7 +109,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new IncrementalTable( session, - dataform.ActionConfig.IncrementalTableConfig.create(actionConfig.incrementalTable), + sqlanvil.ActionConfig.IncrementalTableConfig.create(actionConfig.incrementalTable), actionConfigsPath ) ); @@ -117,7 +117,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new Assertion( session, - dataform.ActionConfig.AssertionConfig.create(actionConfig.assertion), + sqlanvil.ActionConfig.AssertionConfig.create(actionConfig.assertion), actionConfigsPath ) ); @@ -125,7 +125,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new Operation( session, - dataform.ActionConfig.OperationConfig.create(actionConfig.operation), + sqlanvil.ActionConfig.OperationConfig.create(actionConfig.operation), actionConfigsPath ) ); @@ -133,14 +133,14 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new Declaration( session, - dataform.ActionConfig.DeclarationConfig.create(actionConfig.declaration) + sqlanvil.ActionConfig.DeclarationConfig.create(actionConfig.declaration) ) ); } else if (actionConfig.notebook) { session.actions.push( new Notebook( session, - dataform.ActionConfig.NotebookConfig.create(actionConfig.notebook), + sqlanvil.ActionConfig.NotebookConfig.create(actionConfig.notebook), actionConfigsPath ) ); @@ -148,7 +148,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new DataPreparation( session, - dataform.ActionConfig.DataPreparationConfig.create(actionConfig.dataPreparation), + sqlanvil.ActionConfig.DataPreparationConfig.create(actionConfig.dataPreparation), actionConfigsPath ) ); @@ -162,7 +162,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { function loadActionConfigsFile( session: Session, actionConfigsPath: string -): dataform.ActionConfigs { +): sqlanvil.ActionConfigs { let actionConfigsAsJson = {}; try { // tslint:disable-next-line: tsr-detect-non-literal-require @@ -171,29 +171,29 @@ function loadActionConfigsFile( session.compileError(e, actionConfigsPath); } verifyObjectMatchesProto( - dataform.ActionConfigs, + sqlanvil.ActionConfigs, actionConfigsAsJson, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); - return dataform.ActionConfigs.fromObject(actionConfigsAsJson); + return sqlanvil.ActionConfigs.fromObject(actionConfigsAsJson); } -function prologueCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { - if (compileRequest?.compileConfig?.extension?.compilationMode === dataform.ExtensionCompilationMode.PROLOGUE) { +function prologueCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { + if (compileRequest?.compileConfig?.extension?.compilationMode === sqlanvil.ExtensionCompilationMode.PROLOGUE) { extensionCompile(compileRequest, session); } } -function mainCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { - if (compileRequest?.compileConfig?.extension?.compilationMode === dataform.ExtensionCompilationMode.APPLICATION_CODE) { +function mainCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { + if (compileRequest?.compileConfig?.extension?.compilationMode === sqlanvil.ExtensionCompilationMode.APPLICATION_CODE) { extensionCompile(compileRequest, session); return; } - dataformCompile(compileRequest, session); + sqlanvilCompile(compileRequest, session); } -function extensionCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { +function extensionCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { try { const module = nativeRequire(compileRequest?.compileConfig?.extension.name); const extension: () => IDataformExtension = module.extension; @@ -203,7 +203,7 @@ function extensionCompile(compileRequest: dataform.ICompileExecutionRequest, ses } } -function dataformCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { +function sqlanvilCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { const globalAny = global as any; // Require "includes/*.js" files, attaching them (by file basename) to the `global` object. diff --git a/core/main_test.ts b/core/main_test.ts index 59cb28c0..9fe9fc76 100644 --- a/core/main_test.ts +++ b/core/main_test.ts @@ -4,29 +4,29 @@ import * as fs from "fs-extra"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { version } from "df/core/version"; -import { dataform, google } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { version } from "sa/core/version"; +import { sqlanvil, google } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_DATAFORM_JSON, VALID_WORKFLOW_SETTINGS_YAML, WorkflowSettingsTemplates -} from "df/testing/run_core"; +} from "sa/testing/run_core"; const EMPTY_NOTEBOOK_CONTENTS = '{ "cells": [] }'; interface IVerifiableAction { type?: string | null, - target?: dataform.ITarget | null; - canonicalTarget?: dataform.ITarget | null; - dependencyTargets?: dataform.ITarget[] | null + target?: sqlanvil.ITarget | null; + canonicalTarget?: sqlanvil.ITarget | null; + dependencyTargets?: sqlanvil.ITarget[] | null } -function toVerifiableAction(graph: dataform.ICompiledGraph, actionType: string): IVerifiableAction { - let action: dataform.IAssertion | dataform.ITable +function toVerifiableAction(graph: sqlanvil.ICompiledGraph, actionType: string): IVerifiableAction { + let action: sqlanvil.IAssertion | sqlanvil.ITable switch (actionType) { case "assertion": action = graph.assertions[0]; @@ -62,7 +62,7 @@ suite("@sqlanvil/core", ({ afterEach }) => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(testConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(testConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync(path.join(projectDir, "definitions/e.sqlx"), `config {type: "view"}`); @@ -209,7 +209,7 @@ publish("b", {"schema": "foo"}).dependencies("a")` const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(testConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(testConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -235,7 +235,7 @@ publish("b", {"schema": "foo"}).dependencies("a")` const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -301,7 +301,7 @@ publish("name")` const result = runMainInVm( coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ defaultSchema: "otherDataset" }) ) @@ -396,7 +396,7 @@ publish("b", "SELECT 1;");` const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -446,7 +446,7 @@ SELECT 1`); const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); const fileContents = `select @@ -465,7 +465,7 @@ from \`location\``; const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); const sqlContents = `select @@ -490,7 +490,7 @@ from \`location\``; const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); const sqlContents = `select @@ -650,7 +650,7 @@ someKey: and an extra: colon fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), ` -dataformCoreVersion: ${version} +sqlanvilCoreVersion: ${version} defaultProject: project` ); @@ -682,11 +682,11 @@ vars: ` config { type: "table", - database: dataform.projectConfig.vars.projectVar, + database: sqlanvil.projectConfig.vars.projectVar, } -select 1 AS \${dataform.projectConfig.vars.selectVar}` +select 1 AS \${sqlanvil.projectConfig.vars.selectVar}` ); - const coreExecutionRequest = dataform.CoreExecutionRequest.create({ + const coreExecutionRequest = sqlanvil.CoreExecutionRequest.create({ compile: { compileConfig: { projectDir, @@ -706,7 +706,7 @@ select 1 AS \${dataform.projectConfig.vars.selectVar}` expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); expect(asPlainObject(result.compile.compiledGraph)).deep.equals( asPlainObject({ - dataformCoreVersion: version, + sqlanvilCoreVersion: version, graphErrors: {}, jitData: {}, projectConfig: { @@ -746,14 +746,14 @@ select 1 AS \${dataform.projectConfig.vars.selectVar}` ); }); - suite("dataform core version", () => { + suite("sqlanvil core version", () => { test(`main fails when the workflow settings version is not the installed current version`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), ` -dataformCoreVersion: 1.0.0 -defaultProject: dataform` +sqlanvilCoreVersion: 1.0.0 +defaultProject: sqlanvil` ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( @@ -766,7 +766,7 @@ defaultProject: dataform` fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), ` -dataformCoreVersion: ${version} +sqlanvilCoreVersion: ${version} defaultProject: project defaultLocation: US` ); @@ -830,16 +830,16 @@ vars: ` config { type: "table", - database: dataform.projectConfig.vars.databaseVar, + database: sqlanvil.projectConfig.vars.databaseVar, schema: "tableSchema", - description: dataform.projectConfig.vars.descriptionVar, + description: sqlanvil.projectConfig.vars.descriptionVar, assertions: { - nonNull: [dataform.projectConfig.vars.columnVar], + nonNull: [sqlanvil.projectConfig.vars.columnVar], } } -select 1 AS \${dataform.projectConfig.vars.columnVar}` +select 1 AS \${sqlanvil.projectConfig.vars.columnVar}` ); - const coreExecutionRequest = dataform.CoreExecutionRequest.create({ + const coreExecutionRequest = sqlanvil.CoreExecutionRequest.create({ compile: { compileConfig: { projectDir, @@ -883,7 +883,7 @@ select 1 AS \${dataform.projectConfig.vars.columnVar}` } } ], - dataformCoreVersion: version, + sqlanvilCoreVersion: version, graphErrors: {}, jitData: {}, projectConfig: { @@ -1153,7 +1153,7 @@ actions: const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(projectConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(projectConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -1339,7 +1339,7 @@ publish("name", { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(projectConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(projectConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -1448,7 +1448,7 @@ operate("name", { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(projectConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(projectConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -1558,7 +1558,7 @@ assert("name", { const coreRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ disableAssertions: true }) ); @@ -1606,7 +1606,7 @@ assert("name", { fs.writeFileSync( path.join(projectDir, "definitions/jit.js"), ` -dataform.jitData("key", { +sqlanvil.jitData("key", { "number": 123, "string": "value", "boolean": true, @@ -1674,8 +1674,8 @@ dataform.jitData("key", { fs.writeFileSync( path.join(projectDir, "definitions/jit.js"), ` -dataform.jitData("key", 1); -dataform.jitData("key", 2); +sqlanvil.jitData("key", 1); +sqlanvil.jitData("key", 2); ` ); const result = runMainInVm(coreExecutionRequestFromPath(projectDir)); @@ -1695,7 +1695,7 @@ dataform.jitData("key", 2); fs.writeFileSync( path.join(projectDir, "definitions/jit.js"), ` -dataform.jitData("key", {test: () => {}}); +sqlanvil.jitData("key", {test: () => {}}); ` ); const result = runMainInVm(coreExecutionRequestFromPath(projectDir)); @@ -1869,7 +1869,7 @@ publish("name", { fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), dumpYaml( - dataform.WorkflowSettings.create( + sqlanvil.WorkflowSettings.create( WorkflowSettingsTemplates.bigqueryWithDefaultProjectAndDataset ) ) @@ -1902,7 +1902,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync(path.join(projectDir, "definitions/e.sqlx"), `config {type: "view"}`); @@ -1926,7 +1926,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "some-extension", - compilationMode: dataform.ExtensionCompilationMode.COMPILATION_MODE_UNSPECIFIED, + compilationMode: sqlanvil.ExtensionCompilationMode.COMPILATION_MODE_UNSPECIFIED, }; const result = runMainInVm(request); @@ -1940,7 +1940,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "@sqlanvil/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -1954,7 +1954,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "@sqlanvil/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.APPLICATION_CODE, + compilationMode: sqlanvil.ExtensionCompilationMode.APPLICATION_CODE, }; const result = runMainInVm(request); @@ -1972,7 +1972,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "@sqlanvil/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.APPLICATION_CODE + compilationMode: sqlanvil.ExtensionCompilationMode.APPLICATION_CODE }; const result = runMainInVm(request); @@ -1992,7 +1992,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "@sqlanvil/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -2006,7 +2006,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "does-not-exist", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -2018,10 +2018,10 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} test("catches exceptions thrown from extension", () => { const projectDir = setUpProjectWithExtension(); - const request = coreExecutionRequestFromPath(projectDir, dataform.ProjectConfig.create({vars: {"throw-error": "true"}})); + const request = coreExecutionRequestFromPath(projectDir, sqlanvil.ProjectConfig.create({vars: {"throw-error": "true"}})); request.compile.compileConfig.extension = { name: "@sqlanvil/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -2033,10 +2033,10 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} test("persists extension compilation errors", () => { const projectDir = setUpProjectWithExtension(); - const request = coreExecutionRequestFromPath(projectDir, dataform.ProjectConfig.create({vars: {"store-compile-error": "true"}})); + const request = coreExecutionRequestFromPath(projectDir, sqlanvil.ProjectConfig.create({vars: {"store-compile-error": "true"}})); request.compile.compileConfig.extension = { name: "@sqlanvil/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); diff --git a/core/session.ts b/core/session.ts index b29bc971..1c785208 100644 --- a/core/session.ts +++ b/core/session.ts @@ -1,30 +1,30 @@ import { default as TarjanGraphConstructor, Graph as TarjanGraph } from "tarjan-graph"; -import { encode64, unknownToValue, verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { Action, ActionProto, ILegacyTableConfig, TableType } from "df/core/actions"; -import { AContextable, Assertion, AssertionContext } from "df/core/actions/assertion"; +import { encode64, unknownToValue, verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { Action, ActionProto, ILegacyTableConfig, TableType } from "sa/core/actions"; +import { AContextable, Assertion, AssertionContext } from "sa/core/actions/assertion"; import { DataPreparation, DataPreparationContext, -} from "df/core/actions/data_preparation"; -import { Declaration } from "df/core/actions/declaration"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation, OperationContext } from "df/core/actions/operation"; -import { Table, TableContext } from "df/core/actions/table"; -import { Test } from "df/core/actions/test"; -import { View } from "df/core/actions/view"; -import { CompilationSql } from "df/core/compilation_sql"; -import { Contextable, IActionContext, ITableContext, Resolvable } from "df/core/contextables"; -import { targetAsReadableString, targetStringifier } from "df/core/targets"; -import * as utils from "df/core/utils"; -import { ResolvableMap, toResolvable } from "df/core/utils"; -import { version as dataformCoreVersion } from "df/core/version"; -import { dataform, google } from "df/protos/ts"; +} from "sa/core/actions/data_preparation"; +import { Declaration } from "sa/core/actions/declaration"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation, OperationContext } from "sa/core/actions/operation"; +import { Table, TableContext } from "sa/core/actions/table"; +import { Test } from "sa/core/actions/test"; +import { View } from "sa/core/actions/view"; +import { CompilationSql } from "sa/core/compilation_sql"; +import { Contextable, IActionContext, ITableContext, Resolvable } from "sa/core/contextables"; +import { targetAsReadableString, targetStringifier } from "sa/core/targets"; +import * as utils from "sa/core/utils"; +import { ResolvableMap, toResolvable } from "sa/core/utils"; +import { version as sqlanvilCoreVersion } from "sa/core/version"; +import { sqlanvil, google } from "sa/protos/ts"; const DEFAULT_CONFIG = { - defaultSchema: "dataform", - assertionSchema: "dataform_assertions" + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions" }; /** @@ -36,17 +36,17 @@ export class Session { /** * Stores the Dataform project configuration of the current Dataform project. Can be accessed via - * the `dataform` global variable. + * the `sqlanvil` global variable. * * Example: * * ```js - * dataform.projectConfig.vars.myVariableName === "myVariableValue" + * sqlanvil.projectConfig.vars.myVariableName === "myVariableValue" * ``` */ - public projectConfig: dataform.ProjectConfig; + public projectConfig: sqlanvil.ProjectConfig; // The canonical project config contains the project config before schema and database overrides. - public canonicalProjectConfig: dataform.ProjectConfig; + public canonicalProjectConfig: sqlanvil.ProjectConfig; public actions: Action[]; public indexedActions: ResolvableMap; @@ -58,28 +58,28 @@ export class Session { // upon a certain action in our actions list. We use this later to resolve dependencies. public actionAssertionMap = new ResolvableMap(); - public graphErrors: dataform.IGraphErrors; + public graphErrors: sqlanvil.IGraphErrors; // jit_context.data, avilable at jit stage. public jitContextData: google.protobuf.Struct | undefined; constructor( rootDir?: string, - projectConfig?: dataform.ProjectConfig, - originalProjectConfig?: dataform.ProjectConfig + projectConfig?: sqlanvil.ProjectConfig, + originalProjectConfig?: sqlanvil.ProjectConfig ) { this.init(rootDir, projectConfig, originalProjectConfig); } public init( rootDir: string, - projectConfig?: dataform.ProjectConfig, - originalProjectConfig?: dataform.ProjectConfig + projectConfig?: sqlanvil.ProjectConfig, + originalProjectConfig?: sqlanvil.ProjectConfig ) { this.rootDir = rootDir; - this.projectConfig = dataform.ProjectConfig.create(projectConfig || DEFAULT_CONFIG); + this.projectConfig = sqlanvil.ProjectConfig.create(projectConfig || DEFAULT_CONFIG); this.canonicalProjectConfig = getCanonicalProjectConfig( - dataform.ProjectConfig.create(originalProjectConfig || projectConfig || DEFAULT_CONFIG) + sqlanvil.ProjectConfig.create(originalProjectConfig || projectConfig || DEFAULT_CONFIG) ); this.actions = []; this.tests = []; @@ -88,7 +88,7 @@ export class Session { } public compilationSql(): CompilationSql { - return new CompilationSql(this.projectConfig, dataformCoreVersion); + return new CompilationSql(this.projectConfig, sqlanvilCoreVersion); } public sqlxAction(actionOptions: { @@ -264,7 +264,7 @@ export class Session { name: string, queryOrConfig?: | Contextable - | dataform.ActionConfig.OperationConfig + | sqlanvil.ActionConfig.OperationConfig ): Operation { const filename = utils.getCallerFile(this.rootDir); let operation: Operation; @@ -293,9 +293,9 @@ export class Session { name: string, queryOrConfig?: | Contextable - | dataform.ActionConfig.TableConfig - | dataform.ActionConfig.ViewConfig - | dataform.ActionConfig.IncrementalTableConfig + | sqlanvil.ActionConfig.TableConfig + | sqlanvil.ActionConfig.ViewConfig + | sqlanvil.ActionConfig.IncrementalTableConfig | ILegacyTableConfig // `any` is used here to facilitate the type merging of legacy table configs, which are very // different to the new structures. @@ -342,7 +342,7 @@ export class Session { */ public assert( name: string, - queryOrConfig?: AContextable | dataform.ActionConfig.AssertionConfig + queryOrConfig?: AContextable | sqlanvil.ActionConfig.AssertionConfig // // `any` is used here to facilitate the type merging of legacy declaration configs options, // // without breaking typescript consumers of Dataform. // | any @@ -370,7 +370,7 @@ export class Session { */ public declare( config: - | dataform.ActionConfig.DeclarationConfig + | sqlanvil.ActionConfig.DeclarationConfig // `any` is used here to facilitate the type merging of legacy declaration configs options, // without breaking typescript consumers of Dataform. | any @@ -407,7 +407,7 @@ export class Session { * * @see [Notebook](Notebook) for examples on how to use. */ - public notebook(config: dataform.ActionConfig.NotebookConfig): Notebook { + public notebook(config: sqlanvil.ActionConfig.NotebookConfig): Notebook { const configFileName = utils.getCallerFile(this.rootDir); const notebook = new Notebook(this, config, configFileName); this.actions.push(notebook); @@ -423,10 +423,10 @@ export class Session { this.jitContextData.fields[key] = unknownToValue(data); } - public compileError(err: Error | string, path?: string, actionTarget?: dataform.ITarget) { + public compileError(err: Error | string, path?: string, actionTarget?: sqlanvil.ITarget) { const fileName = path || utils.getCallerFile(this.rootDir) || __filename; - const compileError = dataform.CompilationError.create({ + const compileError = sqlanvil.CompilationError.create({ fileName, actionName: !!actionTarget ? targetAsReadableString(actionTarget) : undefined, actionTarget @@ -440,7 +440,7 @@ export class Session { this.graphErrors.compilationErrors.push(compileError); } - public compile(): dataform.CompiledGraph { + public compile(): sqlanvil.CompiledGraph { this.actions.push(...this.tests); this.indexedActions = new ResolvableMap( this.actions.map(action => ({ actionTarget: action.getTarget(), value: action })) @@ -454,7 +454,7 @@ export class Session { throw new Error("Custom variables defined in workflow settings can only be strings."); } - const compiledGraph = dataform.CompiledGraph.create({ + const compiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: this.projectConfig, tables: this.compileGraphChunk( this.actions.filter( @@ -479,7 +479,7 @@ export class Session { this.actions.filter(action => action instanceof DataPreparation) ), graphErrors: this.graphErrors, - dataformCoreVersion, + sqlanvilCoreVersion, targets: this.actions.map(action => action.getTarget()), jitData: this.jitContextData, }); @@ -526,7 +526,7 @@ export class Session { ) ); verifyObjectMatchesProto( - dataform.CompiledGraph, + sqlanvil.CompiledGraph, compiledGraph, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -534,7 +534,7 @@ export class Session { } public compileToBase64() { - return encode64(dataform.CompiledGraph, this.compile()); + return encode64(sqlanvil.CompiledGraph, this.compile()); } public finalizeDatabase(database: string): string { @@ -578,8 +578,8 @@ export class Session { private fullyQualifyDependencies(actions: ActionProto[]) { actions.forEach(action => { - const fullyQualifiedDependencies: { [name: string]: dataform.ITarget } = {}; - if (action instanceof dataform.Declaration || !action.dependencyTargets) { + const fullyQualifiedDependencies: { [name: string]: sqlanvil.ITarget } = {}; + if (action instanceof sqlanvil.Declaration || !action.dependencyTargets) { // Declarations cannot have dependencies. return; } @@ -624,14 +624,14 @@ export class Session { }); } - private alterActionName(actions: ActionProto[], declarationTargets: dataform.ITarget[]) { + private alterActionName(actions: ActionProto[], declarationTargets: sqlanvil.ITarget[]) { const { tablePrefix, schemaSuffix, databaseSuffix } = this.projectConfig; if (!tablePrefix && !schemaSuffix && !databaseSuffix) { return; } - const newTargetByOriginalTarget = new Map(); + const newTargetByOriginalTarget = new Map(); declarationTargets.forEach(declarationTarget => newTargetByOriginalTarget.set( targetStringifier.stringify(declarationTarget), @@ -652,7 +652,7 @@ export class Session { }); // Fix up dependencies in case those dependencies' names have changed. - const getUpdatedTarget = (originalTarget: dataform.ITarget) => { + const getUpdatedTarget = (originalTarget: sqlanvil.ITarget) => { // It's possible that we don't have a new Target for a dependency that failed to compile, // so fall back to the original Target. if (!newTargetByOriginalTarget.has(targetStringifier.stringify(originalTarget))) { @@ -661,18 +661,18 @@ export class Session { return newTargetByOriginalTarget.get(targetStringifier.stringify(originalTarget)); }; actions.forEach(action => { - if (!(action instanceof dataform.Declaration)) { + if (!(action instanceof sqlanvil.Declaration)) { // Declarations cannot have dependencies. action.dependencyTargets = (action.dependencyTargets || []).map(getUpdatedTarget); } - if (action instanceof dataform.Assertion && !!action.parentAction) { + if (action instanceof sqlanvil.Assertion && !!action.parentAction) { action.parentAction = getUpdatedTarget(action.parentAction); } }); } - private checkTestNameUniqueness(tests: dataform.ITest[]) { + private checkTestNameUniqueness(tests: sqlanvil.ITest[]) { const allNames: string[] = []; tests.forEach(testProto => { if (allNames.includes(testProto.name)) { @@ -694,7 +694,7 @@ export class Session { const tarjanGraph: TarjanGraph = new (TarjanGraphConstructor as any)(); actions.forEach(action => { // Declarations cannot have dependencies. - const cleanedDependencies = (action instanceof dataform.Declaration || + const cleanedDependencies = (action instanceof sqlanvil.Declaration || !action.dependencyTargets ? [] : action.dependencyTargets @@ -740,8 +740,8 @@ export class Session { }); } - private removeNonUniqueActionsFromCompiledGraph(compiledGraph: dataform.CompiledGraph) { - function getNonUniqueTargets(targets: dataform.ITarget[]): Set { + private removeNonUniqueActionsFromCompiledGraph(compiledGraph: sqlanvil.CompiledGraph) { + function getNonUniqueTargets(targets: sqlanvil.ITarget[]): Set { const allTargets = new Set(); const nonUniqueTargets = new Set(); @@ -816,8 +816,8 @@ function definesDataset(type: string) { return type === "view" || type === "table" || type === "incremental"; } -function getCanonicalProjectConfig(originalProjectConfig: dataform.ProjectConfig) { - return dataform.ProjectConfig.create({ +function getCanonicalProjectConfig(originalProjectConfig: sqlanvil.ProjectConfig) { + return sqlanvil.ProjectConfig.create({ warehouse: originalProjectConfig.warehouse, defaultSchema: originalProjectConfig.defaultSchema, defaultDatabase: originalProjectConfig.defaultDatabase, diff --git a/core/targets.ts b/core/targets.ts index 9d1eef04..384ce869 100644 --- a/core/targets.ts +++ b/core/targets.ts @@ -1,8 +1,8 @@ -import { JSONObjectStringifier } from "df/common/strings/stringifier"; -import { dataform } from "df/protos/ts"; +import { JSONObjectStringifier } from "sa/common/strings/stringifier"; +import { sqlanvil } from "sa/protos/ts"; /** Produces an unambigous mapping to and from a string representation. */ -export const targetStringifier = new JSONObjectStringifier(); +export const targetStringifier = new JSONObjectStringifier(); /** * Provides a readable string representation of the target which is used for e.g. specifying @@ -12,7 +12,7 @@ export const targetStringifier = new JSONObjectStringifier(); * This is an ambiguous transformation, multiple targets may map to the same string * and it should not be used for indexing. Use @see {@link targetStringifier} instead. */ -export function targetAsReadableString(target: dataform.ITarget): string { +export function targetAsReadableString(target: sqlanvil.ITarget): string { const nameParts = [target.name, target.schema]; if (!!target.database) { nameParts.push(target.database); diff --git a/core/utils.ts b/core/utils.ts index 9068ad66..526c2abc 100644 --- a/core/utils.ts +++ b/core/utils.ts @@ -1,15 +1,15 @@ -import { Action, ActionProto } from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { DataPreparation } from "df/core/actions/data_preparation"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation } from "df/core/actions/operation"; -import { Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { Contextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import { Action, ActionProto } from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { DataPreparation } from "sa/core/actions/data_preparation"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation } from "sa/core/actions/operation"; +import { Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { Contextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; declare var __webpack_require__: any; declare var __non_webpack_require__: any; @@ -75,8 +75,8 @@ export function getCallerFile(rootDir: string) { break; } if (!lastfile) { - if ((global as any).__dataform_current_file) { - lastfile = (global as any).__dataform_current_file; + if ((global as any).__sqlanvil_current_file) { + lastfile = (global as any).__sqlanvil_current_file; } else { // This is likely caused by Session.compileError() being called inside Session.compile(). // If so, explicitly pass the filename to Session.compileError(). @@ -101,7 +101,7 @@ function getCurrentStack(): NodeJS.CallSite[] { } } -export function graphHasErrors(graph: dataform.ICompiledGraph) { +export function graphHasErrors(graph: sqlanvil.ICompiledGraph) { return graph.graphErrors?.compilationErrors.length > 0; } @@ -133,28 +133,28 @@ function isResolvableArray(parts: any[]): parts is [string, string?, string?] { } export function resolvableAsTarget( - resolvable: Resolvable | dataform.ActionConfig.Target -): dataform.Target { + resolvable: Resolvable | sqlanvil.ActionConfig.Target +): sqlanvil.Target { if (typeof resolvable === "string") { - return dataform.Target.create({ + return sqlanvil.Target.create({ name: resolvable }); } - const actionConfigTarget = (resolvable as dataform.ActionConfig.ITarget); - if (actionConfigTarget instanceof dataform.ActionConfig.Target || actionConfigTarget.dataset !== undefined || actionConfigTarget.project !== undefined) { - return dataform.Target.create({ + const actionConfigTarget = (resolvable as sqlanvil.ActionConfig.ITarget); + if (actionConfigTarget instanceof sqlanvil.ActionConfig.Target || actionConfigTarget.dataset !== undefined || actionConfigTarget.project !== undefined) { + return sqlanvil.Target.create({ name: actionConfigTarget.name, schema: actionConfigTarget.dataset, database: actionConfigTarget.project, includeDependentAssertions: actionConfigTarget.includeDependentAssertions, }); } - return dataform.Target.create(resolvable); + return sqlanvil.Target.create(resolvable); } export function resolvableAsActionConfigTarget( resolvable: string | object -): dataform.ActionConfig.ITarget { +): sqlanvil.ActionConfig.ITarget { if (typeof resolvable === "string") { const parts = resolvable.split(".").reverse(); if (!isResolvableArray(parts)) { @@ -169,7 +169,7 @@ export function resolvableAsActionConfigTarget( }; } - return resolvable as dataform.ActionConfig.ITarget; + return resolvable as sqlanvil.ActionConfig.ITarget; } export function stringifyResolvable(res: Resolvable) { @@ -192,12 +192,12 @@ export function ambiguousActionNameMsg(act: Resolvable, allActs: Action[] | stri * @deprecated use ActionBuilder.applySessionToTarget() instead. */ export function target( - config: dataform.IProjectConfig, + config: sqlanvil.IProjectConfig, name: string, schema?: string, database?: string -): dataform.ITarget { - return dataform.Target.create({ +): sqlanvil.ITarget { + return sqlanvil.Target.create({ name, schema: schema || config.defaultSchema || undefined, database: database || config.defaultDatabase || undefined @@ -331,15 +331,15 @@ export function validateStorageUriFormat( */ export function getFileFormatValueForIcebergTable( configFileFormat?: string, -): dataform.FileFormat { +): sqlanvil.FileFormat { if (!configFileFormat) { // Default to PARQUET if fileFormat is undefined. - return dataform.FileFormat.PARQUET; + return sqlanvil.FileFormat.PARQUET; } switch (configFileFormat.toUpperCase()) { case "PARQUET": - return dataform.FileFormat.PARQUET; + return sqlanvil.FileFormat.PARQUET; default: throw new Error( @@ -411,7 +411,7 @@ export function getEffectiveBucketName( * Iceberg table. If the tableFolderRoot is provided in the config block, that * value will be used. Otherwise, defaultTableFolderRoot defined in * workflow_settings.yaml will be used. If none of those two values are - * defined, "_dataform" will be used. + * defined, "_sqlanvil" will be used. * @param defaultTableFolderRoot defined in workflow_settings.yaml * @param configTableFolderRoot defined in the config block * @returns tableFolderRoot used to construct storageUri for Iceberg tables @@ -425,7 +425,7 @@ export function getEffectiveTableFolderRoot( } else if (defaultTableFolderRoot) { return defaultTableFolderRoot; } else { - return "_dataform"; + return "_sqlanvil"; } } @@ -458,30 +458,30 @@ export function getEffectiveTableFolderSubpath( export function tableTypeStringToEnum(type: string, throwIfUnknown: boolean) { switch (type) { case "table": - return dataform.TableType.TABLE; + return sqlanvil.TableType.TABLE; case "incremental": - return dataform.TableType.INCREMENTAL; + return sqlanvil.TableType.INCREMENTAL; case "view": - return dataform.TableType.VIEW; + return sqlanvil.TableType.VIEW; default: { if (throwIfUnknown) { throw new Error(`Unexpected table type: ${type}`); } - return dataform.TableType.UNKNOWN_TYPE; + return sqlanvil.TableType.UNKNOWN_TYPE; } } } -export function tableTypeEnumToString(enumType: dataform.TableType) { - return dataform.TableType[enumType].toLowerCase(); +export function tableTypeEnumToString(enumType: sqlanvil.TableType) { + return sqlanvil.TableType[enumType].toLowerCase(); } -export function setOrValidateTableEnumType(table: dataform.ITable) { - let enumTypeFromStr: dataform.TableType | null = null; +export function setOrValidateTableEnumType(table: sqlanvil.ITable) { + let enumTypeFromStr: sqlanvil.TableType | null = null; if (table.type !== "" && table.type !== undefined) { enumTypeFromStr = tableTypeStringToEnum(table.type, true); } - if (table.enumType === dataform.TableType.UNKNOWN_TYPE || table.enumType === undefined) { + if (table.enumType === sqlanvil.TableType.UNKNOWN_TYPE || table.enumType === undefined) { table.enumType = enumTypeFromStr!; } else if (enumTypeFromStr !== null && table.enumType !== enumTypeFromStr) { throw new Error( @@ -501,8 +501,8 @@ export function extractActionDetailsFromFileName( } // Converts the config proto's target proto to the compiled graph proto's representation. -export function configTargetToCompiledGraphTarget(configTarget: dataform.ActionConfig.Target) { - const compiledGraphTarget: dataform.ITarget = { name: configTarget.name }; +export function configTargetToCompiledGraphTarget(configTarget: sqlanvil.ActionConfig.Target) { + const compiledGraphTarget: sqlanvil.ITarget = { name: configTarget.name }; if (configTarget.project) { compiledGraphTarget.database = configTarget.project; } @@ -512,25 +512,25 @@ export function configTargetToCompiledGraphTarget(configTarget: dataform.ActionC if (configTarget.hasOwnProperty("includeDependentAssertions")) { compiledGraphTarget.includeDependentAssertions = configTarget.includeDependentAssertions; } - return dataform.Target.create(compiledGraphTarget); + return sqlanvil.Target.create(compiledGraphTarget); } // Converts a config proto's action config proto to the compiled graph proto's representation. // Action config protos roughly contain target protos fields. export function actionConfigToCompiledGraphTarget( actionConfig: - | dataform.ActionConfig.TableConfig - | dataform.ActionConfig.ViewConfig - | dataform.ActionConfig.IncrementalTableConfig - | dataform.ActionConfig.OperationConfig - | dataform.ActionConfig.AssertionConfig - | dataform.ActionConfig.DeclarationConfig - | dataform.ActionConfig.NotebookConfig - | dataform.ActionConfig.DataPreparationConfig - | dataform.ActionConfig.DataPreparationConfig.ErrorTableConfig - | dataform.ActionConfig.Target -): dataform.Target { - const compiledGraphTarget = dataform.Target.create({ name: actionConfig.name }); + | sqlanvil.ActionConfig.TableConfig + | sqlanvil.ActionConfig.ViewConfig + | sqlanvil.ActionConfig.IncrementalTableConfig + | sqlanvil.ActionConfig.OperationConfig + | sqlanvil.ActionConfig.AssertionConfig + | sqlanvil.ActionConfig.DeclarationConfig + | sqlanvil.ActionConfig.NotebookConfig + | sqlanvil.ActionConfig.DataPreparationConfig + | sqlanvil.ActionConfig.DataPreparationConfig.ErrorTableConfig + | sqlanvil.ActionConfig.Target +): sqlanvil.Target { + const compiledGraphTarget = sqlanvil.Target.create({ name: actionConfig.name }); if ("project" in actionConfig && actionConfig.project !== undefined) { compiledGraphTarget.database = actionConfig.project; } @@ -552,7 +552,7 @@ export function resolveActionsConfigFilename(configFilename: string, configPath: export function checkAssertionsForDependency( action: actionsWithDependencies, resolvable: Resolvable -): dataform.Target { +): sqlanvil.Target { const dependencyTarget = resolvableAsTarget(resolvable); if ( !dependencyTarget.hasOwnProperty("includeDependentAssertions") && @@ -594,7 +594,7 @@ export class ResolvableMap { private byDatabaseAndName: Map> = new Map(); private byDatabaseSchemaAndName: Map>> = new Map(); - public constructor(values?: Array<{ actionTarget: dataform.ITarget, value: T }>) { + public constructor(values?: Array<{ actionTarget: sqlanvil.ITarget, value: T }>) { if (values) { for (const { actionTarget, value } of values) { this.set(actionTarget, value); @@ -602,7 +602,7 @@ export class ResolvableMap { } } - public set(actionTarget: dataform.ITarget, value: T) { + public set(actionTarget: sqlanvil.ITarget, value: T) { this.setByNameLevel(this.byName, actionTarget.name, value); if (!!actionTarget.schema) { @@ -626,7 +626,7 @@ export class ResolvableMap { } } - public find(actionTarget: dataform.ITarget): T[] { + public find(actionTarget: sqlanvil.ITarget): T[] { if (!!actionTarget.database) { if (!!actionTarget.schema) { return ( @@ -653,7 +653,7 @@ export class ResolvableMap { private setBySchemaLevel( targetMap: Map>, - actionTarget: dataform.ITarget, + actionTarget: sqlanvil.ITarget, value: T ) { if (!targetMap.has(actionTarget.schema)) { diff --git a/core/utils_test.ts b/core/utils_test.ts index 88c281f9..b57fbce8 100644 --- a/core/utils_test.ts +++ b/core/utils_test.ts @@ -11,8 +11,8 @@ import { validateStorageUriFormat, } from './utils'; -import {dataform} from "df/protos/ts"; -import {suite, test} from 'df/testing'; +import {sqlanvil} from "sa/protos/ts"; +import {suite, test} from 'sa/testing'; /** * Executes a function and returns the Error if one is thrown, otherwise returns undefined. @@ -66,12 +66,12 @@ suite('Dataform Utility Validations', () => { test('does not throw for a valid dot-separated connection format', () => { assertNoThrow(() => validateConnectionFormat('my-project.us-central1.my-connection')); - assertNoThrow(() => validateConnectionFormat('gcp-proj-123.europe-west4.dataform-conn_id')); + assertNoThrow(() => validateConnectionFormat('gcp-proj-123.europe-west4.sqlanvil-conn_id')); }); test('does not throw for a valid resource-formatted connection format', () => { assertNoThrow(() => validateConnectionFormat('projects/my-project/locations/us-central1/connections/my-connection')); - assertNoThrow(() => validateConnectionFormat('projects/gcp-proj-123/locations/europe-west4/connections/dataform-conn_id')); + assertNoThrow(() => validateConnectionFormat('projects/gcp-proj-123/locations/europe-west4/connections/sqlanvil-conn_id')); }); test('throws for an empty connection string', () => { @@ -170,12 +170,12 @@ suite('Dataform Utility Validations', () => { expect(getEffectiveTableFolderRoot('ws-root', '')).to.equal('ws-root'); }); - test('returns "_dataform" when both config and default are undefined', () => { - expect(getEffectiveTableFolderRoot(undefined, undefined)).to.equal('_dataform'); + test('returns "_sqlanvil" when both config and default are undefined', () => { + expect(getEffectiveTableFolderRoot(undefined, undefined)).to.equal('_sqlanvil'); }); - test('returns "_dataform" when both config and default are empty strings', () => { - expect(getEffectiveTableFolderRoot('', '')).to.equal('_dataform'); + test('returns "_sqlanvil" when both config and default are empty strings', () => { + expect(getEffectiveTableFolderRoot('', '')).to.equal('_sqlanvil'); }); }); @@ -228,20 +228,20 @@ suite('Dataform Utility Validations', () => { suite('getFileFormatValueForIcebergTable', () => { test('returns PARQUET when configFileFormat is undefined', () => { - expect(getFileFormatValueForIcebergTable(undefined)).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable(undefined)).to.equal(sqlanvil.FileFormat.PARQUET); }); test('returns PARQUET when configFileFormat is "PARQUET"', () => { - expect(getFileFormatValueForIcebergTable('PARQUET')).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('PARQUET')).to.equal(sqlanvil.FileFormat.PARQUET); }); test('returns PARQUET when fileFormat is an empty string', () => { - expect(getFileFormatValueForIcebergTable('')).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('')).to.equal(sqlanvil.FileFormat.PARQUET); }); test('is case insensitive ', () => { - expect(getFileFormatValueForIcebergTable('parquet')).to.equal(dataform.FileFormat.PARQUET); - expect(getFileFormatValueForIcebergTable('pArQuEt')).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('parquet')).to.equal(sqlanvil.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('pArQuEt')).to.equal(sqlanvil.FileFormat.PARQUET); }); test('throws an error for an unsupported file format string', () => { @@ -254,15 +254,15 @@ suite('Dataform Utility Validations', () => { suite('getStorageUriForIcebergTable', () => { const testBucket = 'my-iceberg-bucket'; - const testRoot = '_dataform'; + const testRoot = '_sqlanvil'; const testSubpath = 'data/v1'; test('constructs the URI with a provided tableFolderRoot', () => { - expect(getStorageUriForIcebergTable(testBucket, testRoot, testSubpath)).to.equal('gs://my-iceberg-bucket/_dataform/data/v1'); + expect(getStorageUriForIcebergTable(testBucket, testRoot, testSubpath)).to.equal('gs://my-iceberg-bucket/_sqlanvil/data/v1'); }); test('handles empty bucket name, resulting in an invalid but formed URI', () => { - expect(getStorageUriForIcebergTable('', testRoot, testSubpath)).to.equal('gs:///_dataform/data/v1'); + expect(getStorageUriForIcebergTable('', testRoot, testSubpath)).to.equal('gs:///_sqlanvil/data/v1'); }); test('handles empty tableFolderRoot, resulting in an invalid but formed URI', () => { @@ -270,7 +270,7 @@ suite('Dataform Utility Validations', () => { }); test('handles empty tableFolderSubpath', () => { - expect(getStorageUriForIcebergTable(testBucket, testRoot, '')).to.equal('gs://my-iceberg-bucket/_dataform/'); + expect(getStorageUriForIcebergTable(testBucket, testRoot, '')).to.equal('gs://my-iceberg-bucket/_sqlanvil/'); }); }); diff --git a/core/workflow_settings.ts b/core/workflow_settings.ts index 37fb54db..03249f27 100644 --- a/core/workflow_settings.ts +++ b/core/workflow_settings.ts @@ -1,15 +1,15 @@ import { YAMLException } from "js-yaml"; -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { INVALID_YAML_ERROR_STRING } from "df/core/compilers"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { INVALID_YAML_ERROR_STRING } from "sa/core/compilers"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; declare var __webpack_require__: any; declare var __non_webpack_require__: any; const nativeRequire = typeof __webpack_require__ === "function" ? __non_webpack_require__ : require; -export function readWorkflowSettings(failIfMissing: boolean = true): dataform.ProjectConfig { +export function readWorkflowSettings(failIfMissing: boolean = true): sqlanvil.ProjectConfig { const globalAny = global as any; const workflowSettingsYaml = globalAny.workflowSettingsYaml || maybeRequire("workflow_settings.yaml"); // `dataform.json` is deprecated; new versions of Dataform Core prefer `workflow_settings.yaml`. @@ -32,8 +32,8 @@ export function readWorkflowSettings(failIfMissing: boolean = true): dataform.Pr if (dataformJson) { // Dataform JSON used the compiled graph's config proto, rather than workflow settings. try { - return dataform.ProjectConfig.create( - verifyObjectMatchesProto(dataform.ProjectConfig, dataformJson) + return sqlanvil.ProjectConfig.create( + verifyObjectMatchesProto(sqlanvil.ProjectConfig, dataformJson) ); } catch (e) { if (e instanceof ReferenceError) { @@ -46,15 +46,15 @@ export function readWorkflowSettings(failIfMissing: boolean = true): dataform.Pr if (failIfMissing) { throw Error("Failed to resolve workflow_settings.yaml"); } - return dataform.ProjectConfig.create(); + return sqlanvil.ProjectConfig.create(); } -function verifyWorkflowSettingsAsJson(workflowSettingsAsJson: object): dataform.WorkflowSettings { - let workflowSettings = dataform.WorkflowSettings.create(); +function verifyWorkflowSettingsAsJson(workflowSettingsAsJson: object): sqlanvil.WorkflowSettings { + let workflowSettings = sqlanvil.WorkflowSettings.create(); try { - workflowSettings = dataform.WorkflowSettings.create( + workflowSettings = sqlanvil.WorkflowSettings.create( verifyObjectMatchesProto( - dataform.WorkflowSettings, + sqlanvil.WorkflowSettings, workflowSettingsAsJson as { [key: string]: any; }, @@ -69,9 +69,9 @@ function verifyWorkflowSettingsAsJson(workflowSettingsAsJson: object): dataform. } // The caller of Dataform Core should ensure that the correct version is installed. - if (!!workflowSettings.dataformCoreVersion && workflowSettings.dataformCoreVersion !== version) { + if (!!workflowSettings.sqlanvilCoreVersion && workflowSettings.sqlanvilCoreVersion !== version) { throw Error( - `Version mismatch: workflow settings specifies version ${workflowSettings.dataformCoreVersion}` + + `Version mismatch: workflow settings specifies version ${workflowSettings.sqlanvilCoreVersion}` + `, but ${version} was found` ); } @@ -96,9 +96,9 @@ function maybeRequire(file: string): any { } export function workflowSettingsAsProjectConfig( - workflowSettings: dataform.WorkflowSettings -): dataform.ProjectConfig { - const projectConfig = dataform.ProjectConfig.create(); + workflowSettings: sqlanvil.WorkflowSettings +): sqlanvil.ProjectConfig { + const projectConfig = sqlanvil.ProjectConfig.create(); if (workflowSettings.defaultProject) { projectConfig.defaultDatabase = workflowSettings.defaultProject; } diff --git a/examples/examples_test.ts b/examples/examples_test.ts index de51c405..0d11906b 100644 --- a/examples/examples_test.ts +++ b/examples/examples_test.ts @@ -2,9 +2,9 @@ import { expect } from "chai"; import { execFile } from "child_process"; import * as fs from "fs-extra"; -import { verifyObjectMatchesProto } from "df/common/protos"; -import { dataform } from "df/protos/ts"; -import { getProcessResult, nodePath, suite, test } from "df/testing"; +import { verifyObjectMatchesProto } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; +import { getProcessResult, nodePath, suite, test } from "sa/testing"; suite("examples", { parallel: true }, () => { const cliEntryPointPath = "examples/node_modules/@sqlanvil/cli/bundle.js"; @@ -22,7 +22,7 @@ suite("examples", { parallel: true }, () => { "examples/node_modules/@sqlanvil/core", `${projectDir}/node_modules/@sqlanvil/core` ); - // A blank `package.json` makes no `dataformCoreVersion` in `workflow_settings.yaml` be OK. + // A blank `package.json` makes no `sqlanvilCoreVersion` in `workflow_settings.yaml` be OK. // tslint:disable-next-line: tsr-detect-non-literal-fs-filename fs.writeFileSync(`${projectDir}/package.json`, ""); @@ -32,7 +32,7 @@ suite("examples", { parallel: true }, () => { expect(processResult.exitCode).equals(0); const compiledGraph = verifyObjectMatchesProto( - dataform.CompiledGraph, + sqlanvil.CompiledGraph, JSON.parse(processResult.stdout) ); expect(compiledGraph.graphErrors).deep.equals({}); diff --git a/packages/@sqlanvil/cli/index.ts b/packages/@sqlanvil/cli/index.ts index d3263aa8..09228d6f 100644 --- a/packages/@sqlanvil/cli/index.ts +++ b/packages/@sqlanvil/cli/index.ts @@ -1,2 +1,2 @@ -import { runCli } from "df/cli"; +import { runCli } from "sa/cli"; runCli(); diff --git a/packages/@sqlanvil/cli/worker.ts b/packages/@sqlanvil/cli/worker.ts index bc5f2930..4a1b5e3c 100644 --- a/packages/@sqlanvil/cli/worker.ts +++ b/packages/@sqlanvil/cli/worker.ts @@ -1,2 +1,2 @@ -import { listenForCompileRequest } from "df/cli/vm/compile"; +import { listenForCompileRequest } from "sa/cli/vm/compile"; listenForCompileRequest(); diff --git a/packages/@sqlanvil/core/index.ts b/packages/@sqlanvil/core/index.ts index 2c146b60..f9599b9a 100644 --- a/packages/@sqlanvil/core/index.ts +++ b/packages/@sqlanvil/core/index.ts @@ -1 +1 @@ -export { compiler, IDataformExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version } from "df/core"; +export { compiler, IDataformExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version } from "sa/core"; diff --git a/packages/@sqlanvil/core/webpack.config.js b/packages/@sqlanvil/core/webpack.config.js index 423a9b47..7ee11caf 100644 --- a/packages/@sqlanvil/core/webpack.config.js +++ b/packages/@sqlanvil/core/webpack.config.js @@ -5,7 +5,7 @@ module.exports = (env, argv) => { const config = { mode: argv.mode || "development", target: 'node', - entry: [path.resolve(process.env.RUNFILES, "df/packages/@sqlanvil/core/index")], + entry: [path.resolve(process.env.RUNFILES, "sa/packages/@sqlanvil/core/index")], output: { libraryTarget: "commonjs-module", }, diff --git a/packages/sample-extension/index.ts b/packages/sample-extension/index.ts index 4ace4404..69611263 100644 --- a/packages/sample-extension/index.ts +++ b/packages/sample-extension/index.ts @@ -1,9 +1,9 @@ -import type { IDataformExtension } from "df/core"; -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import type { IDataformExtension } from "sa/core"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; class SampleExtension implements IDataformExtension { - public compile(request: dataform.ICompileExecutionRequest, session: Session): void { + public compile(request: sqlanvil.ICompileExecutionRequest, session: Session): void { if (request.compileConfig?.projectConfigOverride?.vars["throw-error"] === "true") { throw new Error("throwing exception as requested!"); } diff --git a/packages/sample-extension/webpack.config.js b/packages/sample-extension/webpack.config.js index c659abfb..e2b76fbb 100644 --- a/packages/sample-extension/webpack.config.js +++ b/packages/sample-extension/webpack.config.js @@ -5,7 +5,7 @@ module.exports = (env, argv) => { const config = { mode: argv.mode || "development", target: 'node', - entry: [path.resolve(process.env.RUNFILES, "df/packages/sample-extension/index")], + entry: [path.resolve(process.env.RUNFILES, "sa/packages/sample-extension/index")], output: { libraryTarget: "commonjs-module", }, diff --git a/sqlx/format.ts b/sqlx/format.ts index 4fc3bf14..9d8f5a41 100644 --- a/sqlx/format.ts +++ b/sqlx/format.ts @@ -4,8 +4,8 @@ import * as jsBeautify from "js-beautify"; import { typeid } from "typeid-js"; import { promisify } from "util"; -import { ErrorWithCause } from "df/common/errors/errors"; -import { SyntaxTreeNode, SyntaxTreeNodeType } from "df/sqlx/lexer"; +import { ErrorWithCause } from "sa/common/errors/errors"; +import { SyntaxTreeNode, SyntaxTreeNodeType } from "sa/sqlx/lexer"; const JS_BEAUTIFY_OPTIONS = { indent_size: 2, diff --git a/sqlx/format_test.ts b/sqlx/format_test.ts index c170918e..1a71fc40 100644 --- a/sqlx/format_test.ts +++ b/sqlx/format_test.ts @@ -2,8 +2,8 @@ import { expect } from "chai"; import * as fs from "fs"; import * as path from "path"; -import { format, formatFile } from "df/sqlx/format"; -import { suite, test } from "df/testing"; +import { format, formatFile } from "sa/sqlx/format"; +import { suite, test } from "sa/testing"; suite("@sqlanvil/sqlx", () => { suite("formatter", () => { diff --git a/sqlx/lexer_test.ts b/sqlx/lexer_test.ts index b2e9b832..19c8aec1 100644 --- a/sqlx/lexer_test.ts +++ b/sqlx/lexer_test.ts @@ -1,7 +1,7 @@ import { expect } from "chai"; -import { SyntaxTreeNode, SyntaxTreeNodeType } from "df/sqlx/lexer"; -import { suite, test } from "df/testing"; +import { SyntaxTreeNode, SyntaxTreeNodeType } from "sa/sqlx/lexer"; +import { suite, test } from "sa/testing"; suite("@sqlanvil/sqlx", () => { suite("syntax tree construction", () => { diff --git a/testing/child_process.ts b/testing/child_process.ts index 97815fb8..a6f5818c 100644 --- a/testing/child_process.ts +++ b/testing/child_process.ts @@ -2,7 +2,7 @@ import { ChildProcess, spawn } from "child_process"; import * as fs from "fs"; import * as path from "path"; -import { IHookHandler } from "df/testing"; +import { IHookHandler } from "sa/testing"; export class ChildProcessForBazelTestEnvironment { private childProcess: ChildProcess; diff --git a/testing/fixtures.ts b/testing/fixtures.ts index 70907f4d..761f0231 100644 --- a/testing/fixtures.ts +++ b/testing/fixtures.ts @@ -2,7 +2,7 @@ import * as fs from "fs"; import * as path from "path"; import * as rimraf from "rimraf"; -import { IHookHandler } from "df/testing"; +import { IHookHandler } from "sa/testing"; // TmpDirFixture can be used to create unique temporary directories which will be cleaned up // at the end of a test run. Intended for use within bazel tests. diff --git a/testing/hook.ts b/testing/hook.ts index cc342ffe..895cdc49 100644 --- a/testing/hook.ts +++ b/testing/hook.ts @@ -1,4 +1,4 @@ -import { IRunContext, IRunResult } from "df/testing"; +import { IRunContext, IRunResult } from "sa/testing"; export type IHookFunction = () => any; diff --git a/testing/index.ts b/testing/index.ts index 414a1403..4adb5d78 100644 --- a/testing/index.ts +++ b/testing/index.ts @@ -1,10 +1,10 @@ import { ChildProcess } from "child_process"; import * as os from "os"; -export * from "df/testing/hook"; -export * from "df/testing/suite"; -export * from "df/testing/test"; -export * from "df/testing/runner"; +export * from "sa/testing/hook"; +export * from "sa/testing/suite"; +export * from "sa/testing/test"; +export * from "sa/testing/runner"; export const platformPath = () => { if (os.platform() === "darwin") { diff --git a/testing/index_test.ts b/testing/index_test.ts index 91b0fb87..6e88b9cd 100644 --- a/testing/index_test.ts +++ b/testing/index_test.ts @@ -1,6 +1,6 @@ import { expect } from "chai"; -import { ISuiteContext, Runner, suite, test } from "df/testing"; +import { ISuiteContext, Runner, suite, test } from "sa/testing"; Runner.setNoExit(true); diff --git a/testing/run_core.ts b/testing/run_core.ts index 2aa13923..da55f48f 100644 --- a/testing/run_core.ts +++ b/testing/run_core.ts @@ -3,9 +3,9 @@ import * as fs from "fs-extra"; import * as path from "path"; import { CompilerFunction, NodeVM } from "vm2"; -import { decode64, encode64 } from "df/common/protos"; -import { compile } from "df/core/compilers"; -import { dataform } from "df/protos/ts"; +import { decode64, encode64 } from "sa/common/protos"; +import { compile } from "sa/core/compilers"; +import { sqlanvil } from "sa/protos/ts"; export const VALID_DATAFORM_JSON = ` { @@ -22,27 +22,27 @@ defaultLocation: US `; export class WorkflowSettingsTemplates { - public static bigquery = dataform.WorkflowSettings.create({ + public static bigquery = sqlanvil.WorkflowSettings.create({ defaultDataset: "defaultDataset", defaultLocation: "US" }); - public static bigqueryWithDefaultProject = dataform.WorkflowSettings.create({ + public static bigqueryWithDefaultProject = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigquery, defaultProject: "defaultProject" }); - public static bigqueryWithDatasetSuffix = dataform.WorkflowSettings.create({ + public static bigqueryWithDatasetSuffix = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigquery, datasetSuffix: "suffix" }); - public static bigqueryWithDefaultProjectAndDataset = dataform.WorkflowSettings.create({ + public static bigqueryWithDefaultProjectAndDataset = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigqueryWithDefaultProject, projectSuffix: "suffix" }); - public static bigqueryWithNamePrefix = dataform.WorkflowSettings.create({ + public static bigqueryWithNamePrefix = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigquery, namePrefix: "prefix" }); @@ -52,10 +52,10 @@ const SOURCE_EXTENSIONS = ["js", "sql", "sqlx", "yaml", "ipynb"]; export function coreExecutionRequestFromPath( projectDir: string, - projectConfigOverride?: dataform.ProjectConfig -): dataform.CoreExecutionRequest { + projectConfigOverride?: sqlanvil.ProjectConfig +): sqlanvil.CoreExecutionRequest { const resolvedProjectDir = fs.realpathSync(path.resolve(projectDir)); - return dataform.CoreExecutionRequest.create({ + return sqlanvil.CoreExecutionRequest.create({ compile: { compileConfig: { projectDir: resolvedProjectDir, @@ -68,8 +68,8 @@ export function coreExecutionRequestFromPath( // A VM is needed when running main because Node functions like `require` are overridden. export function runMainInVm( - coreExecutionRequest: dataform.CoreExecutionRequest -): dataform.CoreExecutionResponse { + coreExecutionRequest: sqlanvil.CoreExecutionRequest +): sqlanvil.CoreExecutionResponse { const projectDir = coreExecutionRequest.compile.compileConfig.projectDir; // Copy over the build Dataform Core that is set up as a node_modules directory. @@ -94,18 +94,18 @@ export function runMainInVm( compiler: (code, filePath) => { const compiledCode = compiler(code, filePath); return ` - var __old_file = global.__dataform_current_file; - global.__dataform_current_file = ${JSON.stringify(filePath)}; + var __old_file = global.__sqlanvil_current_file; + global.__sqlanvil_current_file = ${JSON.stringify(filePath)}; try { ${compiledCode} } finally { - global.__dataform_current_file = __old_file; + global.__sqlanvil_current_file = __old_file; } `; } }); - const encodedCoreExecutionRequest = encode64(dataform.CoreExecutionRequest, coreExecutionRequest); + const encodedCoreExecutionRequest = encode64(sqlanvil.CoreExecutionRequest, coreExecutionRequest); const vmIndexFileName = path.resolve(path.join(projectDir, "index.js")); const encodedCoreExecutionResponse = nodeVm.run( ` @@ -115,7 +115,7 @@ export function runMainInVm( `, vmIndexFileName ); - return decode64(dataform.CoreExecutionResponse, encodedCoreExecutionResponse); + return decode64(sqlanvil.CoreExecutionResponse, encodedCoreExecutionResponse); } function walkDirectoryForFilenames(projectDir: string, relativePath: string = ""): string[] { diff --git a/testing/runner.ts b/testing/runner.ts index 03771365..db2bfdb7 100644 --- a/testing/runner.ts +++ b/testing/runner.ts @@ -3,7 +3,7 @@ import * as Diff from "diff"; import DeterministicStringify from "json-stable-stringify"; import { promisify } from "util"; -import { Hook, Suite } from "df/testing"; +import { Hook, Suite } from "sa/testing"; export interface IRunResult { path: string[]; diff --git a/testing/suite.ts b/testing/suite.ts index c9b1d42e..b8b2f3bd 100644 --- a/testing/suite.ts +++ b/testing/suite.ts @@ -1,4 +1,4 @@ -import { Hook, hook, IRunContext, Runner, test, Test } from "df/testing"; +import { Hook, hook, IRunContext, Runner, test, Test } from "sa/testing"; export interface ISuiteOptions { name: string; diff --git a/testing/test.ts b/testing/test.ts index 6d3ea5a4..39dfae7f 100644 --- a/testing/test.ts +++ b/testing/test.ts @@ -1,4 +1,4 @@ -import { IRunContext, IRunResult, Runner, Suite } from "df/testing"; +import { IRunContext, IRunResult, Runner, Suite } from "sa/testing"; interface ITestOptions { name: string; diff --git a/tests/api/api.spec.ts b/tests/api/api.spec.ts index 86207771..e024dab2 100644 --- a/tests/api/api.spec.ts +++ b/tests/api/api.spec.ts @@ -4,15 +4,15 @@ import Long from "long"; import * as path from "path"; import { anyString, anything, instance, mock, verify, when } from "ts-mockito"; -import { Builder, credentials, prune, Runner } from "df/cli/api"; -import { IDbAdapter } from "df/cli/api/dbadapters"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { sleep, sleepUntil } from "df/common/promises"; -import { equals } from "df/common/protos"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, cleanSql, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { Builder, credentials, prune, Runner } from "sa/cli/api"; +import { IDbAdapter } from "sa/cli/api/dbadapters"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { sleep, sleepUntil } from "sa/common/promises"; +import { equals } from "sa/common/protos"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, cleanSql, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; config.truncateThreshold = 0; @@ -21,7 +21,7 @@ suite("@sqlanvil/api", () => { // ^ // d // Made with asciiflow.com - const TEST_GRAPH: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const TEST_GRAPH: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { @@ -66,7 +66,7 @@ suite("@sqlanvil/api", () => { ] }); - const TEST_STATE = dataform.WarehouseState.create({ tables: [] }); + const TEST_STATE = sqlanvil.WarehouseState.create({ tables: [] }); suite("build", () => { test("exclude_disabled", () => { @@ -94,7 +94,7 @@ suite("@sqlanvil/api", () => { test("build_with_errors", () => { expect(() => { - const graphWithErrors: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graphWithErrors: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, graphErrors: { compilationErrors: [{ message: "Some critical error" }] }, tables: [{ target: { schema: "schema", name: "a" } }] @@ -106,7 +106,7 @@ suite("@sqlanvil/api", () => { }); test("action_types", () => { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { target: { schema: "schema", name: "a" }, type: "table" }, @@ -131,39 +131,39 @@ suite("@sqlanvil/api", () => { expect(executedGraph.actions.length).greaterThan(0); - graph.tables.forEach((t: dataform.ITable) => { + graph.tables.forEach((t: sqlanvil.ITable) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, t.target) + equals(sqlanvil.Target, item.target, t.target) ); expect(action).to.include({ type: "table", target: t.target, tableType: t.type }); }); - graph.operations.forEach((o: dataform.IOperation) => { + graph.operations.forEach((o: sqlanvil.IOperation) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, o.target) + equals(sqlanvil.Target, item.target, o.target) ); expect(action).to.include({ type: "operation", target: o.target }); }); - graph.assertions.forEach((a: dataform.IAssertion) => { + graph.assertions.forEach((a: sqlanvil.IAssertion) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, a.target) + equals(sqlanvil.Target, item.target, a.target) ); expect(action).to.include({ type: "assertion" }); }); }); test("table_enum_types", () => { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ - { target: { schema: "schema", name: "a" }, enumType: dataform.TableType.TABLE }, + { target: { schema: "schema", name: "a" }, enumType: sqlanvil.TableType.TABLE }, { target: { schema: "schema", name: "b" }, - enumType: dataform.TableType.INCREMENTAL, + enumType: sqlanvil.TableType.INCREMENTAL, where: "test" }, - { target: { schema: "schema", name: "c" }, enumType: dataform.TableType.VIEW } + { target: { schema: "schema", name: "c" }, enumType: sqlanvil.TableType.VIEW } ] }); @@ -172,25 +172,25 @@ suite("@sqlanvil/api", () => { expect(executedGraph.actions.length).greaterThan(0); - graph.tables.forEach((t: dataform.ITable) => { + graph.tables.forEach((t: sqlanvil.ITable) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, t.target) + equals(sqlanvil.Target, item.target, t.target) ); expect(action).to.include({ type: "table", target: t.target, - tableType: dataform.TableType[t.enumType].toLowerCase() + tableType: sqlanvil.TableType[t.enumType].toLowerCase() }); }); }); test("table_enum_and_str_types_should_match", () => { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { target: { schema: "schema", name: "a" }, - enumType: dataform.TableType.TABLE, + enumType: sqlanvil.TableType.TABLE, type: "incremental" } ] @@ -203,7 +203,7 @@ suite("@sqlanvil/api", () => { suite("pre and post ops", () => { for (const warehouse of ["bigquery"]) { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { @@ -217,13 +217,13 @@ suite("@sqlanvil/api", () => { incrementalPostOps: ["incremental postOp"] } ], - dataformCoreVersion: "1.4.9" + sqlanvilCoreVersion: "1.4.9" }); test(`${warehouse} when running non incrementally`, () => { const action = new Builder(graph, {}, TEST_STATE).build().actions[0]; expect(action.tasks).eql([ - dataform.ExecutionTask.create({ + sqlanvil.ExecutionTask.create({ type: "statement", statement: "preOp\n;\ncreate or replace table `schema.a` as foo\n;\npostOp" }) @@ -234,12 +234,12 @@ suite("@sqlanvil/api", () => { const action = new Builder( graph, {}, - dataform.WarehouseState.create({ + sqlanvil.WarehouseState.create({ tables: [{ target: graph.tables[0].target, fields: [] }] }) ).build().actions[0]; expect(action.tasks).eql([ - dataform.ExecutionTask.create({ + sqlanvil.ExecutionTask.create({ type: "statement", statement: "incremental preOp\n;\ndrop view if exists `schema.a`\n;\ninsert into `schema.a`\t\n()\t\nselect \t\nfrom (incremental foo) as insertions\n;\nincremental postOp" @@ -256,7 +256,7 @@ suite("@sqlanvil/api", () => { // +-> op_c // // op_d +---> tab_a - const TEST_GRAPH_WITH_TAGS: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const TEST_GRAPH_WITH_TAGS: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultLocation: "US" }, operations: [ { @@ -391,14 +391,14 @@ suite("@sqlanvil/api", () => { query: "select 1 as test", where: "true" }; - const warehouseState = dataform.WarehouseState.create({ + const warehouseState = sqlanvil.WarehouseState.create({ tables: [ { target: { schema: "schema", name: "incremental" }, - type: dataform.TableMetadata.Type.TABLE, + type: sqlanvil.TableMetadata.Type.TABLE, fields: [ { name: "existing_field" @@ -409,7 +409,7 @@ suite("@sqlanvil/api", () => { }); test("incremental_mode", () => { - const graph = dataform.CompiledGraph.create({ + const graph = sqlanvil.CompiledGraph.create({ projectConfig, tables: [ incrementalTable ] }); @@ -434,7 +434,7 @@ suite("@sqlanvil/api", () => { }); test("full refresh", () => { - const graph = dataform.CompiledGraph.create({ + const graph = sqlanvil.CompiledGraph.create({ projectConfig, tables: [ incrementalTable ] }); @@ -457,7 +457,7 @@ suite("@sqlanvil/api", () => { ...incrementalTable, protected: true, }; - const graph = dataform.CompiledGraph.create({ + const graph = sqlanvil.CompiledGraph.create({ projectConfig, tables: [ protectedIncrementalTable ] }); @@ -483,7 +483,7 @@ suite("@sqlanvil/api", () => { }); test("bigquery_materialized", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -505,7 +505,7 @@ suite("@sqlanvil/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "view", @@ -521,7 +521,7 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -537,17 +537,17 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_partitionby", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -572,7 +572,7 @@ suite("@sqlanvil/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -588,7 +588,7 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -604,17 +604,17 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_options", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -641,7 +641,7 @@ suite("@sqlanvil/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -657,7 +657,7 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -673,17 +673,17 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_clusterby", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -708,7 +708,7 @@ suite("@sqlanvil/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -724,7 +724,7 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -740,17 +740,17 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_additional_options", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -778,7 +778,7 @@ suite("@sqlanvil/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -794,7 +794,7 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -810,10 +810,10 @@ suite("@sqlanvil/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); @@ -843,7 +843,7 @@ suite("@sqlanvil/api", () => { }); suite("run", () => { - const RUN_TEST_GRAPH: dataform.IExecutionGraph = dataform.ExecutionGraph.create({ + const RUN_TEST_GRAPH: sqlanvil.IExecutionGraph = sqlanvil.ExecutionGraph.create({ projectConfig: { warehouse: "bigquery", defaultSchema: "foo", @@ -857,7 +857,7 @@ suite("@sqlanvil/api", () => { warehouseState: { tables: [ { - type: dataform.TableMetadata.Type.TABLE, + type: sqlanvil.TableMetadata.Type.TABLE, target: { schema: "schema1", name: "target1" @@ -909,15 +909,15 @@ suite("@sqlanvil/api", () => { ] }); - const EXPECTED_RUN_RESULT = dataform.RunResult.create({ - status: dataform.RunResult.ExecutionStatus.FAILED, + const EXPECTED_RUN_RESULT = sqlanvil.RunResult.create({ + status: sqlanvil.RunResult.ExecutionStatus.FAILED, actions: [ { target: RUN_TEST_GRAPH.actions[0].target, tasks: [ { - status: dataform.TaskResult.ExecutionStatus.SUCCESSFUL, + status: sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL, metadata: { bigquery: { jobId: "abc", @@ -927,22 +927,22 @@ suite("@sqlanvil/api", () => { } }, { - status: dataform.TaskResult.ExecutionStatus.SUCCESSFUL, + status: sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL, metadata: {} } ], - status: dataform.ActionResult.ExecutionStatus.SUCCESSFUL + status: sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL }, { target: RUN_TEST_GRAPH.actions[1].target, tasks: [ { - status: dataform.TaskResult.ExecutionStatus.FAILED, + status: sqlanvil.TaskResult.ExecutionStatus.FAILED, metadata: {}, errorMessage: "bigquery error: bad statement" } ], - status: dataform.ActionResult.ExecutionStatus.FAILED + status: sqlanvil.ActionResult.ExecutionStatus.FAILED } ] }); @@ -979,7 +979,7 @@ suite("@sqlanvil/api", () => { const runner = new Runner(mockDbAdapterInstance, RUN_TEST_GRAPH); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); verify(mockedDbAdapter.createSchema("database", "schema1")).once(); verify(mockedDbAdapter.createSchema("database2", "schema2")).once(); @@ -1028,13 +1028,13 @@ suite("@sqlanvil/api", () => { stopWasCalled = true; const result = cleanTiming(await runner.result()); - expect(dataform.RunResult.create(result).toJSON()).to.deep.equal( - dataform.RunResult.create({ - status: dataform.RunResult.ExecutionStatus.RUNNING, + expect(sqlanvil.RunResult.create(result).toJSON()).to.deep.equal( + sqlanvil.RunResult.create({ + status: sqlanvil.RunResult.ExecutionStatus.RUNNING, actions: [ { target: EXPECTED_RUN_RESULT.actions[0].target, - status: dataform.ActionResult.ExecutionStatus.RUNNING, + status: sqlanvil.ActionResult.ExecutionStatus.RUNNING, tasks: [EXPECTED_RUN_RESULT.actions[0].tasks[0]] } ] @@ -1044,7 +1044,7 @@ suite("@sqlanvil/api", () => { runner = new Runner(mockDbAdapterInstance, RUN_TEST_GRAPH, undefined, result); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); verify(mockedDbAdapter.createSchema("database", "schema1")).once(); verify(mockedDbAdapter.createSchema("database2", "schema2")).once(); @@ -1087,7 +1087,7 @@ suite("@sqlanvil/api", () => { }); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); }); @@ -1127,21 +1127,21 @@ suite("@sqlanvil/api", () => { }); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal( - dataform.RunResult.create({ - status: dataform.RunResult.ExecutionStatus.SUCCESSFUL, + sqlanvil.RunResult.create({ + status: sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL, actions: [ EXPECTED_RUN_RESULT.actions[0], { target: NEW_TEST_GRAPH.actions[1].target, tasks: [ { - status: dataform.TaskResult.ExecutionStatus.SUCCESSFUL, + status: sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL, metadata: {} } ], - status: dataform.ActionResult.ExecutionStatus.SUCCESSFUL + status: sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL } ] }).toJSON() @@ -1191,13 +1191,13 @@ suite("@sqlanvil/api", () => { }); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); }); }); test("execute_with_cancel", async () => { - const CANCEL_TEST_GRAPH: dataform.IExecutionGraph = dataform.ExecutionGraph.create({ + const CANCEL_TEST_GRAPH: sqlanvil.IExecutionGraph = sqlanvil.ExecutionGraph.create({ projectConfig: { warehouse: "bigquery", defaultSchema: "foo", @@ -1252,7 +1252,7 @@ suite("@sqlanvil/api", () => { // Cancelling a run doesn't actually throw at the top level. // The action should fail, and have an appropriate error message. expect(result.actions[0].tasks[0].status).equal( - dataform.TaskResult.ExecutionStatus.CANCELLED + sqlanvil.TaskResult.ExecutionStatus.CANCELLED ); expect(result.actions[0].tasks[0].errorMessage).to.match(/cancelled/); }); @@ -1295,13 +1295,13 @@ suite("@sqlanvil/api", () => { mockDbAdapterInstance.withClientLock = async callback => await callback(mockDbAdapterInstance); - const labels = { env: "testing", team: "dataform" }; + const labels = { env: "testing", team: "sqlanvil" }; const runner = new Runner(mockDbAdapterInstance, NEW_TEST_GRAPH, { bigquery: { labels } }); const result = await runner.execute().result(); - expect(result.status).to.equal(dataform.RunResult.ExecutionStatus.SUCCESSFUL); + expect(result.status).to.equal(sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL); // Verify that execute was called at least 3 times (for both tasks in first action and assertion) expect(executionOptions.length).to.equal(3); @@ -1310,7 +1310,7 @@ suite("@sqlanvil/api", () => { const callsWithLabels = executionOptions.filter( opts => opts?.bigquery?.labels && opts.bigquery.labels.env === "testing" && - opts.bigquery.labels.team === "dataform" + opts.bigquery.labels.team === "sqlanvil" ); expect(callsWithLabels.length).to.equal(3, "Expected 3 execute calls to include the labels in options"); @@ -1358,13 +1358,13 @@ suite("@sqlanvil/api", () => { mockDbAdapterInstance.withClientLock = async callback => await callback(mockDbAdapterInstance); - const globalLabels = { env: "testing", team: "dataform" }; + const globalLabels = { env: "testing", team: "sqlanvil" }; const runner = new Runner(mockDbAdapterInstance, NEW_TEST_GRAPH, { bigquery: { labels: globalLabels } }); const result = await runner.execute().result(); - expect(result.status).to.equal(dataform.RunResult.ExecutionStatus.SUCCESSFUL); + expect(result.status).to.equal(sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL); // Verify that execute was called 3 times expect(executionOptions.length).to.equal(3); @@ -1375,7 +1375,7 @@ suite("@sqlanvil/api", () => { expect(opts?.bigquery?.labels).to.not.equal(undefined); // Should have global labels expect(opts.bigquery.labels.env).to.equal("testing", `Call ${index} should have global label 'env'`); - expect(opts.bigquery.labels.team).to.equal("dataform", `Call ${index} should have global label 'team'`); + expect(opts.bigquery.labels.team).to.equal("sqlanvil", `Call ${index} should have global label 'team'`); // Should have action-level label expect(opts.bigquery.labels.action_level).to.equal("specific_value", `Call ${index} should have action-level label 'action_level'`); @@ -1385,14 +1385,14 @@ suite("@sqlanvil/api", () => { const assertionCall = executionOptions[2]; expect(assertionCall?.bigquery?.labels).to.not.equal(undefined); expect(assertionCall.bigquery.labels.env).to.equal("testing"); - expect(assertionCall.bigquery.labels.team).to.equal("dataform"); + expect(assertionCall.bigquery.labels.team).to.equal("sqlanvil"); // This action doesn't have action-level labels expect(assertionCall.bigquery.labels.action_level).to.equal(undefined); }); }); test("continues after setMetadata fails", async () => { - const METADATA_TEST_GRAPH: dataform.IExecutionGraph = dataform.ExecutionGraph.create({ + const METADATA_TEST_GRAPH: sqlanvil.IExecutionGraph = sqlanvil.ExecutionGraph.create({ projectConfig: { warehouse: "bigquery", defaultSchema: "foo", @@ -1440,7 +1440,7 @@ suite("@sqlanvil/api", () => { const runner = new Runner(mockDbAdapterInstance, METADATA_TEST_GRAPH); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal({ actions: [ { @@ -1464,8 +1464,8 @@ suite("@sqlanvil/api", () => { }); }); -function cleanTiming(runResult: dataform.IRunResult) { - const newRunResult = dataform.RunResult.create(runResult); +function cleanTiming(runResult: sqlanvil.IRunResult) { + const newRunResult = sqlanvil.RunResult.create(runResult); delete newRunResult.timing; newRunResult.actions.forEach(actionResult => { delete actionResult.timing; diff --git a/tests/api/projects.spec.ts b/tests/api/projects.spec.ts index ffab9cee..77d70787 100644 --- a/tests/api/projects.spec.ts +++ b/tests/api/projects.spec.ts @@ -2,10 +2,10 @@ import { fail } from "assert"; import { expect } from "chai"; import * as path from "path"; -import { compile } from "df/cli/api"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { cleanSql, suite, test } from "df/testing"; +import { compile } from "sa/cli/api"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { cleanSql, suite, test } from "sa/testing"; suite("examples", () => { suite("common_v2 bigquery", async () => { @@ -87,7 +87,7 @@ suite("examples", () => { // Check JS blocks get processed. const exampleJsBlocks = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -96,12 +96,12 @@ suite("examples", () => { ) ); expect(exampleJsBlocks.type).equals("table"); - expect(exampleJsBlocks.enumType).equals(dataform.TableType.TABLE); + expect(exampleJsBlocks.enumType).equals(sqlanvil.TableType.TABLE); expect(exampleJsBlocks.query.trim()).equals("select 1 as foo"); // Check we can import and use an external package. const exampleIncremental = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -124,7 +124,7 @@ suite("examples", () => { ); const exampleIsIncremental = graph.tables.filter( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -155,11 +155,11 @@ suite("examples", () => { // Check tables defined in includes are not included. const exampleIgnore = graph.tables.find( - (t: dataform.ITable) => targetAsReadableString(t.target) === "example_ignore" + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === "example_ignore" ); expect(exampleIgnore).equal(undefined); const exampleIgnore2 = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -171,7 +171,7 @@ suite("examples", () => { // Check SQL files with raw back-ticks get escaped. const exampleBackticks = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -189,7 +189,7 @@ suite("examples", () => { // Check deferred calls to table resolve to the correct definitions file. const exampleDeferred = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -201,7 +201,7 @@ suite("examples", () => { // Check view const exampleView = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -210,7 +210,7 @@ suite("examples", () => { ) ); expect(exampleView.type).equals("view"); - expect(exampleView.enumType).equals(dataform.TableType.VIEW); + expect(exampleView.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleView.query.trim()).equals( `select * from \`${dotJoined( databaseWithSuffix("tada-analytics"), @@ -229,31 +229,31 @@ suite("examples", () => { )}\`` ); expect(exampleView.target).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_view" }) ); expect(exampleView.canonicalTarget).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: "tada-analytics", schema: "df_integration_test", name: "example_view" }) ); expect(exampleView.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" }), - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("override_schema"), name: "override_schema_example" }), - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("override_database"), schema: schemaWithSuffix("df_integration_test"), name: "override_database_example" @@ -263,7 +263,7 @@ suite("examples", () => { // Check materialized view const exampleMaterializedView = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -272,7 +272,7 @@ suite("examples", () => { ) ); expect(exampleMaterializedView.type).equals("view"); - expect(exampleMaterializedView.enumType).equals(dataform.TableType.VIEW); + expect(exampleMaterializedView.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleMaterializedView.materialized).equals(true); expect(exampleMaterializedView.query.trim()).equals( `select * from \`${dotJoined( @@ -282,21 +282,21 @@ suite("examples", () => { )}\`\n` + `group by 1` ); expect(exampleMaterializedView.target).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_materialized_view" }) ); expect(exampleMaterializedView.canonicalTarget).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: "tada-analytics", schema: "df_integration_test", name: "example_materialized_view" }) ); expect(exampleMaterializedView.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" @@ -306,7 +306,7 @@ suite("examples", () => { // Check table const exampleTable = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -315,7 +315,7 @@ suite("examples", () => { ) ); expect(exampleTable.type).equals("table"); - expect(exampleTable.enumType).equals(dataform.TableType.TABLE); + expect(exampleTable.enumType).equals(sqlanvil.TableType.TABLE); expect(exampleTable.query.trim()).equals( `select * from \`${dotJoined( databaseWithSuffix("tada-analytics"), @@ -324,7 +324,7 @@ suite("examples", () => { )}\`\n\n-- here \${"is"} a \`comment\n\n/* \${"another"} \` backtick \` containing \`\`\`comment */` ); expect(exampleTable.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" @@ -347,7 +347,7 @@ suite("examples", () => { // Check Table with tags const exampleTableWithTags = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -377,7 +377,7 @@ suite("examples", () => { )}\` group by sample) as data where index_row_count > 1` ); expect(exampleTableWithTagsUniqueKeyAssertion.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_table_with_tags" @@ -404,7 +404,7 @@ suite("examples", () => { )}\` where not (sample is not null)` ); expect(exampleTableWithTagsRowConditionsAssertion.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_table_with_tags" @@ -413,7 +413,7 @@ suite("examples", () => { // Check sample data const exampleSampleData = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -422,17 +422,17 @@ suite("examples", () => { ) ); expect(exampleSampleData.type).equals("view"); - expect(exampleSampleData.enumType).equals(dataform.TableType.VIEW); + expect(exampleSampleData.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleSampleData.query.trim()).equals( "select 1 as sample union all\nselect 2 as sample union all\nselect 3 as sample" ); expect(exampleSampleData.preOps).eql([]); expect(exampleSampleData.dependencyTargets).eql([]); expect(exampleSampleData.actionDescriptor).to.eql( - dataform.ActionDescriptor.create({ + sqlanvil.ActionDescriptor.create({ description: "This is some sample data.", columns: [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description: "Sample integers.", path: ["sample"] }) @@ -442,7 +442,7 @@ suite("examples", () => { // Check database override defined in "config {}". const exampleUsingOverriddenDatabase = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("override_database"), @@ -455,14 +455,14 @@ suite("examples", () => { databaseWithSuffix("override_database") ); expect(exampleUsingOverriddenDatabase.type).equals("view"); - expect(exampleUsingOverriddenDatabase.enumType).equals(dataform.TableType.VIEW); + expect(exampleUsingOverriddenDatabase.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleUsingOverriddenDatabase.query.trim()).equals( "select 1 as test_database_override" ); // Check schema overrides defined in "config {}" const exampleUsingOverriddenSchema = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -475,14 +475,14 @@ suite("examples", () => { schemaWithSuffix("override_schema") ); expect(exampleUsingOverriddenSchema.type).equals("view"); - expect(exampleUsingOverriddenSchema.enumType).equals(dataform.TableType.VIEW); + expect(exampleUsingOverriddenSchema.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleUsingOverriddenSchema.query.trim()).equals( "select 1 as test_schema_override" ); // Check schema overrides defined in "config {}" -- case with schema unchanged const exampleUsingOverriddenSchemaUnchanged = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -495,14 +495,14 @@ suite("examples", () => { schemaWithSuffix("df_integration_test") ); expect(exampleUsingOverriddenSchemaUnchanged.type).equals("view"); - expect(exampleUsingOverriddenSchemaUnchanged.enumType).equals(dataform.TableType.VIEW); + expect(exampleUsingOverriddenSchemaUnchanged.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleUsingOverriddenSchemaUnchanged.query.trim()).equals( "select 1 as test_schema_override" ); // Check assertion const exampleAssertion = graph.assertions.find( - (a: dataform.IAssertion) => + (a: sqlanvil.IAssertion) => targetAsReadableString(a.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -519,7 +519,7 @@ suite("examples", () => { )}\` where sample = 100` ); expect(exampleAssertion.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" @@ -527,14 +527,14 @@ suite("examples", () => { ]); expect(exampleAssertion.tags).to.eql([]); expect(exampleAssertion.actionDescriptor).to.eql( - dataform.ActionDescriptor.create({ + sqlanvil.ActionDescriptor.create({ description: "An example assertion looking for incorrect 'sample' values." }) ); // Check Assertion with tags const exampleAssertionWithTags = graph.assertions.find( - (a: dataform.IAssertion) => + (a: sqlanvil.IAssertion) => targetAsReadableString(a.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -549,7 +549,7 @@ suite("examples", () => { // Check example operations file const exampleOperations = graph.operations.find( - (o: dataform.IOperation) => + (o: sqlanvil.IOperation) => targetAsReadableString(o.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -572,12 +572,12 @@ suite("examples", () => { )}\`\n` ]); expect(exampleOperations.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("override_schema"), name: "override_schema_example" }), - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("override_database"), schema: schemaWithSuffix("df_integration_test"), name: "override_database_example" @@ -587,7 +587,7 @@ suite("examples", () => { // Check example operation with output. const exampleOperationWithOutput = graph.operations.find( - (o: dataform.IOperation) => + (o: sqlanvil.IOperation) => targetAsReadableString(o.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -607,17 +607,17 @@ suite("examples", () => { )}\` AS (SELECT * FROM \`some_database_name.some_external_schema_name.very_important_external_table\`)` ]); expect(exampleOperationWithOutput.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: "some_database_name", schema: "some_external_schema_name", name: "very_important_external_table" }) ]); expect(exampleOperationWithOutput.actionDescriptor).to.eql( - dataform.ActionDescriptor.create({ + sqlanvil.ActionDescriptor.create({ description: "An example operations file which outputs a dataset.", columns: [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description: "Just 1!", path: ["TEST"] }) @@ -627,7 +627,7 @@ suite("examples", () => { // Check Operation with tags const exampleOperationsWithTags = graph.operations.find( - (o: dataform.IOperation) => + (o: sqlanvil.IOperation) => targetAsReadableString(o.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -644,7 +644,7 @@ suite("examples", () => { "some_database_name.some_external_schema_name.very_important_external_table" ); expect(exampleDeclaration.target).eql( - dataform.Target.create({ + sqlanvil.Target.create({ database: "some_database_name", schema: "some_external_schema_name", name: "very_important_external_table" @@ -673,7 +673,7 @@ suite("examples", () => { // Check double backslashes don't get converted to singular. const exampleDoubleBackslash = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -717,8 +717,8 @@ suite("examples", () => { projectDir: "tests/api/projects/common_v2", projectConfigOverride: { warehouse: "bigquery" } }); - const { version: expectedVersion } = require("df/core/version"); - expect(graph.dataformCoreVersion).equals(expectedVersion); + const { version: expectedVersion } = require("sa/core/version"); + expect(graph.sqlanvilCoreVersion).equals(expectedVersion); }); }); diff --git a/tests/api/utils/cancellable_promise.spec.ts b/tests/api/utils/cancellable_promise.spec.ts index 0d6cb9cb..d9d6d2cb 100644 --- a/tests/api/utils/cancellable_promise.spec.ts +++ b/tests/api/utils/cancellable_promise.spec.ts @@ -1,7 +1,7 @@ import { expect } from "chai"; -import { CancellablePromise } from "df/cli/api/utils/cancellable_promise"; -import { suite, test } from "df/testing"; +import { CancellablePromise } from "sa/cli/api/utils/cancellable_promise"; +import { suite, test } from "sa/testing"; suite("cancellable_promise", () => { test("cancel is called", () => { diff --git a/tests/api/utils/error_parsing.spec.ts b/tests/api/utils/error_parsing.spec.ts index 0b77567e..a6bfd390 100644 --- a/tests/api/utils/error_parsing.spec.ts +++ b/tests/api/utils/error_parsing.spec.ts @@ -1,7 +1,7 @@ import { expect } from "chai"; -import { parseBigqueryEvalError } from "df/cli/api/utils/error_parsing"; -import { suite, test } from "df/testing"; +import { parseBigqueryEvalError } from "sa/cli/api/utils/error_parsing"; +import { suite, test } from "sa/testing"; suite("error_parsing", () => { suite("bigquery", () => { diff --git a/tests/integration/bigquery.spec.ts b/tests/integration/bigquery.spec.ts index 824b8bfc..30703aeb 100644 --- a/tests/integration/bigquery.spec.ts +++ b/tests/integration/bigquery.spec.ts @@ -1,14 +1,14 @@ import { expect } from "chai"; import Long from "long"; -import * as dfapi from "df/cli/api"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; -import { compile, dropAllTables, getTableRows, keyBy } from "df/tests/integration/utils"; +import * as dfapi from "sa/cli/api"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; +import { compile, dropAllTables, getTableRows, keyBy } from "sa/tests/integration/utils"; suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) => { const credentials = dfapi.credentials.read("test_credentials/bigquery.json"); @@ -44,8 +44,8 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ]; for (const actionName of Object.keys(actionMap)) { const expectedResult = expectedFailedActions.includes(actionName) - ? dataform.ActionResult.ExecutionStatus.FAILED - : dataform.ActionResult.ExecutionStatus.SUCCESSFUL; + ? sqlanvil.ActionResult.ExecutionStatus.FAILED + : sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL; expect(actionMap[actionName].status).equals( expectedResult, JSON.stringify(actionMap[actionName], null, 4) @@ -77,7 +77,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) // Run two iterations of the project. const adapter = new ExecutionSql( compiledGraph.projectConfig, - compiledGraph.dataformCoreVersion + compiledGraph.sqlanvilCoreVersion ); for (const runIteration of [ { @@ -98,8 +98,8 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ]) { const executionGraph = await dfapi.build(compiledGraph, runIteration.runConfig, dbadapter); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); const [incrementalRows, incrementalMergeRows] = await Promise.all([ getTableRows( @@ -142,8 +142,8 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); // Check expected metadata. @@ -156,30 +156,30 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) }, expectedDescription: "An incremental table", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "the timestamp", name: "user_timestamp", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ description: "the id", name: "user_id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ name: "nested_data", description: "some nested data with duplicate fields", - struct: dataform.Fields.create({ + struct: sqlanvil.Fields.create({ fields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "nested timestamp", name: "user_timestamp", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ description: "nested id", name: "user_id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ] }) @@ -195,10 +195,10 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) }, expectedDescription: "An example view", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "val doc", name: "val", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ], expectedLabels: { @@ -260,58 +260,58 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) const view = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "dataform-open-source.df_integration_test_evaluate.example_view" ]; - let evaluations = await dbadapter.evaluate(dataform.Table.create(view)); + let evaluations = await dbadapter.evaluate(sqlanvil.Table.create(view)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const materializedView = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "dataform-open-source.df_integration_test_evaluate.example_materialized_view" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(materializedView)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(materializedView)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const table = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "dataform-open-source.df_integration_test_evaluate.example_table" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(table)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(table)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const operation = keyBy(compiledGraph.operations, t => targetAsReadableString(t.target))[ "dataform-open-source.df_integration_test_evaluate.example_operation" ]; - evaluations = await dbadapter.evaluate(dataform.Operation.create(operation)); + evaluations = await dbadapter.evaluate(sqlanvil.Operation.create(operation)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const assertion = keyBy(compiledGraph.assertions, t => targetAsReadableString(t.target))[ "dataform-open-source.df_integration_test_assertions_evaluate.example_assertion_pass" ]; - evaluations = await dbadapter.evaluate(dataform.Assertion.create(assertion)); + evaluations = await dbadapter.evaluate(sqlanvil.Assertion.create(assertion)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const incremental = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "dataform-open-source.df_integration_test_evaluate.example_incremental" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(incremental)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(incremental)); expect(evaluations.length).to.equal(2); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); expect(evaluations[1].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); }); @@ -323,8 +323,8 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) }); let evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, preOps: ["declare var string; set var = 'val';"], query: "select var as col;", target: target("example_valid_variable") @@ -332,26 +332,26 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, query: "select var as col;", target: target("example_invalid_variable") }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE + sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE ); }); test("invalid table fails validation and error parsed correctly", async () => { const evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, query: "selects\n1 as x", target: { name: "EXAMPLE_ILLEGAL_TABLE", @@ -361,19 +361,19 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE + sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE ); expect( - dataform.QueryEvaluationError.ErrorLocation.create(evaluations[0].error.errorLocation) - ).eql(dataform.QueryEvaluationError.ErrorLocation.create({ line: 1, column: 1 })); + sqlanvil.QueryEvaluationError.ErrorLocation.create(evaluations[0].error.errorLocation) + ).eql(sqlanvil.QueryEvaluationError.ErrorLocation.create({ line: 1, column: 1 })); }); }); suite("publish tasks", { parallel: true }, async () => { test("incremental pre and post ops, core version <= 1.4.8", async () => { // 1.4.8 used `preOps` and `postOps` instead of `incrementalPreOps` and `incrementalPostOps`. - const table: dataform.ITable = { - enumType: dataform.TableType.INCREMENTAL, + const table: sqlanvil.ITable = { + enumType: sqlanvil.TableType.INCREMENTAL, query: "query", preOps: ["preop task1", "preop task2"], incrementalQuery: "", @@ -409,7 +409,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) const { bigquery: bqMetadata } = metadata; expect(bqMetadata).to.have.property("jobId"); expect(bqMetadata.jobId).to.match( - /^dataform-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ + /^sqlanvil-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ ); expect(bqMetadata).to.have.property("totalBytesBilled"); expect(bqMetadata.totalBytesBilled).to.eql(Long.fromNumber(0)); @@ -423,7 +423,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) const { bigquery: bqMetadata } = metadata; expect(bqMetadata).to.have.property("jobId"); expect(bqMetadata.jobId).to.match( - /^dataform-jobPrefix-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ + /^sqlanvil-jobPrefix-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ ); }); @@ -472,8 +472,8 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); const [fullSearch, partialSearch, columnSearch] = await Promise.all([ @@ -489,12 +489,12 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) }); async function cleanWarehouse( - compiledGraph: dataform.CompiledGraph, + compiledGraph: sqlanvil.CompiledGraph, dbadapter: dbadapters.IDbAdapter ) { await dropAllTables( (await dfapi.build(compiledGraph, {}, dbadapter)).warehouseState.tables, - new ExecutionSql(compiledGraph.projectConfig, compiledGraph.dataformCoreVersion), + new ExecutionSql(compiledGraph.projectConfig, compiledGraph.sqlanvilCoreVersion), dbadapter ); } diff --git a/tests/integration/postgres.spec.ts b/tests/integration/postgres.spec.ts index 25847768..90402b8c 100644 --- a/tests/integration/postgres.spec.ts +++ b/tests/integration/postgres.spec.ts @@ -1,14 +1,14 @@ import { expect } from "chai"; -import * as dfapi from "df/api"; -import * as dbadapters from "df/api/dbadapters"; -import * as adapters from "df/core/adapters"; -import { RedshiftAdapter } from "df/core/adapters/redshift"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; -import { compile, getTableRows, keyBy } from "df/tests/integration/utils"; -import { PostgresFixture } from "df/tools/postgres/postgres_fixture"; +import * as dfapi from "sa/api"; +import * as dbadapters from "sa/api/dbadapters"; +import * as adapters from "sa/core/adapters"; +import { RedshiftAdapter } from "sa/core/adapters/redshift"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; +import { compile, getTableRows, keyBy } from "sa/tests/integration/utils"; +import { PostgresFixture } from "sa/tools/postgres/postgres_fixture"; suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) => { let dbadapter: dbadapters.IDbAdapter; @@ -45,8 +45,8 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) ]; for (const actionName of Object.keys(actionMap)) { const expectedResult = expectedFailedActions.includes(actionName) - ? dataform.ActionResult.ExecutionStatus.FAILED - : dataform.ActionResult.ExecutionStatus.SUCCESSFUL; + ? sqlanvil.ActionResult.ExecutionStatus.FAILED + : sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL; expect(actionMap[actionName].status).equals( expectedResult, actionMap[actionName].tasks.map(task => task.errorMessage).join("\n") @@ -59,7 +59,7 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) ).to.eql("postgres error: Assertion failed: query returned 1 row(s)."); // Check the data in the incremental table. - const adapter = adapters.create(compiledGraph.projectConfig, compiledGraph.dataformCoreVersion); + const adapter = adapters.create(compiledGraph.projectConfig, compiledGraph.sqlanvilCoreVersion); let incrementalTable = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_project_e2e.example_incremental" ]; @@ -88,7 +88,7 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) ); executedGraph = await dfapi.run(dbadapter, executionGraph).result(); expect(executedGraph.status).equals( - dataform.RunResult.ExecutionStatus.SUCCESSFUL, + sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL, executedGraph.actions .map(action => action.tasks.map(task => task.errorMessage).join("\n")) .join("\n") @@ -122,8 +122,8 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); // Check expected metadata. @@ -135,15 +135,15 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) }, expectedDescription: "An incremental 'table'", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "the 'timestamp'", name: "user_timestamp", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ description: "the id", name: "user_id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ] }, @@ -154,10 +154,10 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) }, expectedDescription: "An example view", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ name: "val", description: "val doc", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ] } @@ -239,58 +239,58 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) const view = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_evaluate.example_view" ]; - let evaluations = await dbadapter.evaluate(dataform.Table.create(view)); + let evaluations = await dbadapter.evaluate(sqlanvil.Table.create(view)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const table = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_evaluate.example_table" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(table)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(table)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const assertion = keyBy(compiledGraph.assertions, t => targetAsReadableString(t.target))[ "df_integration_test_assertions_evaluate.example_assertion_pass" ]; - evaluations = await dbadapter.evaluate(dataform.Assertion.create(assertion)); + evaluations = await dbadapter.evaluate(sqlanvil.Assertion.create(assertion)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const incremental = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_evaluate.example_incremental" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(incremental)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(incremental)); expect(evaluations.length).to.equal(2); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); expect(evaluations[1].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); }); test("invalid table fails validation", async () => { const evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, query: "thisisillegal", target: { schema: "df_integration_test", name: "example_illegal_table", - database: "dataform-integration-tests" + database: "sqlanvil-integration-tests" } }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE + sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE ); }); }); @@ -298,8 +298,8 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) suite("publish tasks", async () => { test("incremental pre and post ops, core version <= 1.4.8", async () => { // 1.4.8 used `preOps` and `postOps` instead of `incrementalPreOps` and `incrementalPostOps`. - const table: dataform.ITable = { - enumType: dataform.TableType.INCREMENTAL, + const table: sqlanvil.ITable = { + enumType: sqlanvil.TableType.INCREMENTAL, query: "query", preOps: ["preop task1", "preop task2"], incrementalQuery: "", @@ -338,8 +338,8 @@ suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); const [fullSearch, partialSearch, columnSearch] = await Promise.all([ diff --git a/tests/integration/utils.ts b/tests/integration/utils.ts index 4d7a1d6f..cb33a9c4 100644 --- a/tests/integration/utils.ts +++ b/tests/integration/utils.ts @@ -1,9 +1,9 @@ import { expect } from "chai"; -import * as dfapi from "df/cli/api"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { dataform } from "df/protos/ts"; +import * as dfapi from "sa/cli/api"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { sqlanvil } from "sa/protos/ts"; export function keyBy(values: V[], keyFn: (value: V) => string): { [key: string]: V } { return values.reduce((map, value) => { @@ -13,7 +13,7 @@ export function keyBy(values: V[], keyFn: (value: V) => string): { [key: stri } export async function dropAllTables( - tables: dataform.ITableMetadata[], + tables: sqlanvil.ITableMetadata[], executionSql: ExecutionSql, dbadapter: dbadapters.IDbAdapter ) { @@ -23,7 +23,7 @@ export async function dropAllTables( } export async function getTableRows( - target: dataform.ITarget, + target: sqlanvil.ITarget, executionSql: ExecutionSql, dbadapter: dbadapters.IDbAdapter ) { @@ -33,7 +33,7 @@ export async function getTableRows( export async function compile( projectDir: string, schemaSuffixOverride: string, - projectConfigOverrides?: dataform.IProjectConfig + projectConfigOverrides?: sqlanvil.IProjectConfig ) { const compiledGraph = await dfapi.compile({ projectDir, diff --git a/tools/postgres/postgres_fixture.ts b/tools/postgres/postgres_fixture.ts index 490a6fd2..ca0da071 100644 --- a/tools/postgres/postgres_fixture.ts +++ b/tools/postgres/postgres_fixture.ts @@ -1,8 +1,8 @@ import * as pg from "pg"; import { execSync } from "child_process"; -import { sleepUntil } from "df/common/promises"; -import { IHookHandler } from "df/testing"; +import { sleepUntil } from "sa/common/promises"; +import { IHookHandler } from "sa/testing"; const USE_CLOUD_BUILD_NETWORK = !!process.env.USE_CLOUD_BUILD_NETWORK; const DOCKER_CONTAINER_NAME = "postgres-df-integration-testing"; diff --git a/tsconfig.json b/tsconfig.json index b04b1687..77a72946 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -2,7 +2,7 @@ "compilerOptions": { "baseUrl": ".", "paths": { - "df/*": [ + "sa/*": [ "*", "bazel-bin/*", "bazel-genfiles/*" diff --git a/vscode/extension.ts b/vscode/extension.ts index 19c51e93..15f0ba45 100644 --- a/vscode/extension.ts +++ b/vscode/extension.ts @@ -37,7 +37,7 @@ export async function activate(context: vscode.ExtensionContext) { clientOptions ); - const compile = vscode.commands.registerCommand("dataform.compile", () => { + const compile = vscode.commands.registerCommand("sqlanvil.compile", () => { const _ = client.sendRequest("compile"); }); diff --git a/vscode/server.ts b/vscode/server.ts index 391c057a..2de466c4 100644 --- a/vscode/server.ts +++ b/vscode/server.ts @@ -10,11 +10,11 @@ import { } from "vscode-languageserver"; import { TextDocument } from "vscode-languageserver-textdocument"; -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; const connection = createConnection(ProposedFeatures.all); const documents: TextDocuments = new TextDocuments(TextDocument); -let CACHED_COMPILE_GRAPH: dataform.ICompiledGraph = null; +let CACHED_COMPILE_GRAPH: sqlanvil.ICompiledGraph = null; let WORKSPACE_ROOT_FOLDER: string = null; let settings = { @@ -83,7 +83,7 @@ async function compileAndValidate() { } } - let parsedResult: dataform.ICompiledGraph = null; + let parsedResult: sqlanvil.ICompiledGraph = null; try { parsedResult = JSON.parse(compileResult.stdout); } catch (e) { @@ -133,7 +133,7 @@ async function getProcessResult(childProcess: ChildProcess) { function gatherAllActions( graph = CACHED_COMPILE_GRAPH -): Array { +): Array { return [].concat( graph.tables ?? [], graph.operations ?? [], @@ -190,7 +190,7 @@ connection.onDefinition( })[0].refContent; // split to dataset, schema and name - const linkedTable: dataform.ITarget = { database: null, schema: null, name: null }; + const linkedTable: sqlanvil.ITarget = { database: null, schema: null, name: null }; const splitMatch = clickedRef.match( /^ref\s*\(\s*(["'](.+?)["'])\s*(,\s*["'](.+?)["']\s*)?(,\s*["'](.+?)["']\s*)?,?\s*\)$/ // tslint:disable-line ); From 1399c23d7bb7064117e3b760753d77bb85cd5a12 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 07:00:52 -0500 Subject: [PATCH 05/20] refactor: drop dataform.json legacy config path (clean break) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream Dataform marked dataform.json deprecated in favor of workflow_settings.yaml. sqlanvil completes the cleanup — no sqlanvil.json parallel, no back-compat fallback. Removed code: - core/workflow_settings.ts: readWorkflowSettings() dataform.json fallback and "deprecated alongside" error path (workflow_settings.yaml is the only source of truth now) - cli/index.ts: dataformJsonPath presence check in projectDirMustExistOption ("does not appear to be a dataform directory" → "sqlanvil directory") - cli/vm/compile.ts: global.dataformJson VM-time require - cli/api/commands/init.ts: dataform.json existence check + init error message ("Cannot init dataform project" → "Cannot init sqlanvil project") - testing/run_core.ts: VM dataformJson global injection + VALID_DATAFORM_JSON constant - core/main_test.ts: 5 test blocks exercising the legacy path ("a valid dataform.json", "fails when both...", "invalid json file", "contains unknown fields", "variables must be strings") - tests/api/projects.spec.ts: "invalid dataform json throws error" test - cli/index_compile_test.ts: replaced dataform.json fixture write with equivalent workflow_settings.yaml Removed fixtures: - tests/api/projects/invalid_dataform_json/ (entire directory) - tests/api/BUILD: invalid_dataform_json deps Migrated fixtures dataform.json → workflow_settings.yaml: - tests/integration/postgres_project/ - tests/api/projects/never_finishes_compiling/ - tests/api/projects/common_v2/ Co-Authored-By: Claude Opus 4.7 --- cli/api/commands/init.ts | 12 +--- cli/index.ts | 5 +- cli/index_compile_test.ts | 12 ++-- cli/vm/compile.ts | 1 - core/main_test.ts | 64 ------------------- core/workflow_settings.ts | 22 ------- testing/run_core.ts | 9 --- tests/api/BUILD | 2 - tests/api/projects.spec.ts | 11 ---- tests/api/projects/common_v2/dataform.json | 7 -- .../projects/common_v2/workflow_settings.yaml | 5 ++ .../api/projects/invalid_dataform_json/BUILD | 17 ----- .../invalid_dataform_json/dataform.json | 6 -- .../invalid_dataform_json/package.json | 0 .../never_finishes_compiling/dataform.json | 6 -- .../workflow_settings.yaml | 4 ++ .../postgres_project/dataform.json | 8 --- .../postgres_project/workflow_settings.yaml | 5 ++ 18 files changed, 24 insertions(+), 172 deletions(-) delete mode 100644 tests/api/projects/common_v2/dataform.json create mode 100644 tests/api/projects/common_v2/workflow_settings.yaml delete mode 100644 tests/api/projects/invalid_dataform_json/BUILD delete mode 100644 tests/api/projects/invalid_dataform_json/dataform.json delete mode 100644 tests/api/projects/invalid_dataform_json/package.json delete mode 100644 tests/api/projects/never_finishes_compiling/dataform.json create mode 100644 tests/api/projects/never_finishes_compiling/workflow_settings.yaml delete mode 100644 tests/integration/postgres_project/dataform.json create mode 100644 tests/integration/postgres_project/workflow_settings.yaml diff --git a/cli/api/commands/init.ts b/cli/api/commands/init.ts index eb31cda1..c2398a35 100644 --- a/cli/api/commands/init.ts +++ b/cli/api/commands/init.ts @@ -23,16 +23,10 @@ export async function init( const workflowSettingsYamlPath = path.join(projectDir, "workflow_settings.yaml"); const packageJsonPath = path.join(projectDir, "package.json"); const gitignorePath = path.join(projectDir, ".gitignore"); - // dataform.json is Deprecated. - const dataformJsonPath = path.join(projectDir, "dataform.json"); - if ( - fs.existsSync(workflowSettingsYamlPath) || - fs.existsSync(packageJsonPath) || - fs.existsSync(dataformJsonPath) - ) { + if (fs.existsSync(workflowSettingsYamlPath) || fs.existsSync(packageJsonPath)) { throw new Error( - "Cannot init dataform project, this already appears to be an NPM or Dataform directory." + "Cannot init sqlanvil project, this already appears to be an NPM or sqlanvil directory." ); } @@ -49,7 +43,7 @@ export async function init( sqlanvilCoreVersion: version, defaultProject: projectConfig.defaultDatabase, defaultLocation: projectConfig.defaultLocation, - defaultDataset: projectConfig.defaultSchema || "dataform", + defaultDataset: projectConfig.defaultSchema || "sqlanvil", defaultAssertionDataset: projectConfig.assertionSchema || "sqlanvil_assertions", defaultIcebergConfig: projectConfig.defaultIcebergConfig, }; diff --git a/cli/index.ts b/cli/index.ts index 07443d11..88ba7803 100644 --- a/cli/index.ts +++ b/cli/index.ts @@ -56,16 +56,15 @@ const projectDirMustExistOption = { ...projectDirOption, check: (argv: yargs.Arguments) => { assertPathExists(argv[projectDirOption.name]); - const dataformJsonPath = path.resolve(argv[projectDirOption.name], "dataform.json"); const workflowSettingsYamlPath = path.resolve( argv[projectDirOption.name], "workflow_settings.yaml" ); - if (!fs.existsSync(dataformJsonPath) && !fs.existsSync(workflowSettingsYamlPath)) { + if (!fs.existsSync(workflowSettingsYamlPath)) { throw new Error( `${ argv[projectDirOption.name] - } does not appear to be a dataform directory (missing workflow_settings.yaml file).` + } does not appear to be a sqlanvil directory (missing workflow_settings.yaml file).` ); } } diff --git a/cli/index_compile_test.ts b/cli/index_compile_test.ts index f73ed29c..17f40f7d 100644 --- a/cli/index_compile_test.ts +++ b/cli/index_compile_test.ts @@ -44,13 +44,11 @@ suite("compile command", ({ afterEach }) => { }` ); fs.writeFileSync( - path.join(projectDir, "dataform.json"), - `{ - "defaultDatabase": "tada-analytics", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "defaultLocation": "${DEFAULT_LOCATION}" -} + path.join(projectDir, "workflow_settings.yaml"), + `defaultProject: tada-analytics +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +defaultLocation: "${DEFAULT_LOCATION}" ` ); diff --git a/cli/vm/compile.ts b/cli/vm/compile.ts index 434bc29e..1eb122b2 100644 --- a/cli/vm/compile.ts +++ b/cli/vm/compile.ts @@ -80,7 +80,6 @@ export function compile(compileConfig: sqlanvil.ICompileConfig) { return userCodeVm.run( ` global.workflowSettingsYaml = (function() { try { return require("./workflow_settings.yaml"); } catch(e) { console.error("YAML require failed:", e); } })(); - global.dataformJson = (function() { try { return require("./dataform.json"); } catch(e) {} })(); return require("@sqlanvil/core").main("${createCoreExecutionRequest(compileConfig)}") `, vmIndexFileName diff --git a/core/main_test.ts b/core/main_test.ts index 9fe9fc76..003a8bbf 100644 --- a/core/main_test.ts +++ b/core/main_test.ts @@ -11,7 +11,6 @@ import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, - VALID_DATAFORM_JSON, VALID_WORKFLOW_SETTINGS_YAML, WorkflowSettingsTemplates } from "sa/testing/run_core"; @@ -542,23 +541,6 @@ quotes ); }); - // dataform.json for workflow settings is deprecated, but still currently supported. - test(`a valid dataform.json is present`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync(path.join(projectDir, "dataform.json"), VALID_DATAFORM_JSON); - - const result = runMainInVm(coreExecutionRequestFromPath(projectDir)); - - expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); - expect(asPlainObject(result.compile.compiledGraph.projectConfig)).deep.equals( - asPlainObject({ - defaultDatabase: "defaultProject", - defaultLocation: "US", - defaultSchema: "defaultDataset" - }) - ); - }); - test(`fails when no workflow settings file is present`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); @@ -567,19 +549,6 @@ quotes ); }); - test(`fails when both workflow settings and dataform.json files are present`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync(path.join(projectDir, "dataform.json"), VALID_DATAFORM_JSON); - fs.writeFileSync( - path.join(projectDir, "workflow_settings.yaml"), - VALID_WORKFLOW_SETTINGS_YAML - ); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - "dataform.json has been deprecated and cannot be defined alongside workflow_settings.yaml" - ); - }); - test(`fails when workflow_settings.yaml cannot be represented in JSON format`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync(path.join(projectDir, "workflow_settings.yaml"), "&*19132sdS:asd:"); @@ -603,15 +572,6 @@ someKey: and an extra: colon ); }); - test(`fails when dataform.json is an invalid json file`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync(path.join(projectDir, "dataform.json"), '{keyWithNoQuotes: "validValue"}'); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - "Expected property name or '}' in JSON at position 1 (line 1 column 2)" - ); - }); - test(`fails when a valid workflow_settings.yaml contains unknown fields`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( @@ -633,18 +593,6 @@ someKey: and an extra: colon ); }); - test(`fails when a valid dataform.json contains unknown fields`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync( - path.join(projectDir, "dataform.json"), - `{"notAProjectConfigField": "value"}` - ); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Dataform json error: Unexpected property "notAProjectConfigField", or property value type of "string" is incorrect.` - ); - }); - test(`does not fail when defaultLocation is not present in workflow_settings.yaml`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( @@ -800,18 +748,6 @@ vars: ); }); - test(`variables in dataform.json must be strings`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync( - path.join(projectDir, "dataform.json"), - `{"vars": { "intVar": 1, "strVar": "str" } }` - ); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - "Custom variables defined in workflow settings can only be strings." - ); - }); - test(`variables can be referenced in SQLX`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( diff --git a/core/workflow_settings.ts b/core/workflow_settings.ts index 03249f27..b5b827fc 100644 --- a/core/workflow_settings.ts +++ b/core/workflow_settings.ts @@ -12,14 +12,6 @@ const nativeRequire = typeof __webpack_require__ === "function" ? __non_webpack_ export function readWorkflowSettings(failIfMissing: boolean = true): sqlanvil.ProjectConfig { const globalAny = global as any; const workflowSettingsYaml = globalAny.workflowSettingsYaml || maybeRequire("workflow_settings.yaml"); - // `dataform.json` is deprecated; new versions of Dataform Core prefer `workflow_settings.yaml`. - const dataformJson = globalAny.dataformJson || maybeRequire("dataform.json"); - - if (workflowSettingsYaml && dataformJson) { - throw Error( - "dataform.json has been deprecated and cannot be defined alongside workflow_settings.yaml" - ); - } if (workflowSettingsYaml) { const workflowSettingsAsJson = workflowSettingsYaml.asJson; @@ -29,20 +21,6 @@ export function readWorkflowSettings(failIfMissing: boolean = true): sqlanvil.Pr return workflowSettingsAsProjectConfig(verifyWorkflowSettingsAsJson(workflowSettingsAsJson)); } - if (dataformJson) { - // Dataform JSON used the compiled graph's config proto, rather than workflow settings. - try { - return sqlanvil.ProjectConfig.create( - verifyObjectMatchesProto(sqlanvil.ProjectConfig, dataformJson) - ); - } catch (e) { - if (e instanceof ReferenceError) { - throw ReferenceError(`Dataform json error: ${e.message}`); - } - throw e; - } - } - if (failIfMissing) { throw Error("Failed to resolve workflow_settings.yaml"); } diff --git a/testing/run_core.ts b/testing/run_core.ts index da55f48f..f95de51e 100644 --- a/testing/run_core.ts +++ b/testing/run_core.ts @@ -7,14 +7,6 @@ import { decode64, encode64 } from "sa/common/protos"; import { compile } from "sa/core/compilers"; import { sqlanvil } from "sa/protos/ts"; -export const VALID_DATAFORM_JSON = ` -{ - "defaultDatabase": "defaultProject", - "defaultSchema": "defaultDataset", - "defaultLocation": "US" -} -`; - export const VALID_WORKFLOW_SETTINGS_YAML = ` defaultProject: defaultProject defaultDataset: defaultDataset @@ -110,7 +102,6 @@ export function runMainInVm( const encodedCoreExecutionResponse = nodeVm.run( ` global.workflowSettingsYaml = (function() { try { return require("./workflow_settings.yaml"); } catch(e) { console.error("YAML require failed run_core:", e); } })(); - global.dataformJson = (function() { try { return require("./dataform.json"); } catch(e) {} })(); return require("@sqlanvil/core").main("${encodedCoreExecutionRequest}") `, vmIndexFileName diff --git a/tests/api/BUILD b/tests/api/BUILD index 79b72c30..a2f4198a 100644 --- a/tests/api/BUILD +++ b/tests/api/BUILD @@ -6,8 +6,6 @@ ts_test_suite( data = [ "//tests/api/projects/common_v2:files", "//tests/api/projects/common_v2:node_modules", - "//tests/api/projects/invalid_dataform_json:files", - "//tests/api/projects/invalid_dataform_json:node_modules", "//tests/api/projects/never_finishes_compiling:files", "//tests/api/projects/never_finishes_compiling:node_modules", ], diff --git a/tests/api/projects.spec.ts b/tests/api/projects.spec.ts index 77d70787..6195d5f6 100644 --- a/tests/api/projects.spec.ts +++ b/tests/api/projects.spec.ts @@ -701,17 +701,6 @@ suite("examples", () => { } }); - test("invalid dataform json throws error", async () => { - try { - await compile({ - projectDir: path.resolve("tests/api/projects/invalid_dataform_json") - }); - fail("Should have failed."); - } catch (e) { - // OK - } - }); - test("version is correctly set", async () => { const graph = await compile({ projectDir: "tests/api/projects/common_v2", diff --git a/tests/api/projects/common_v2/dataform.json b/tests/api/projects/common_v2/dataform.json deleted file mode 100644 index 88dabbe2..00000000 --- a/tests/api/projects/common_v2/dataform.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "warehouse": "bigquery", - "defaultDatabase": "tada-analytics", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "defaultLocation": "US" -} diff --git a/tests/api/projects/common_v2/workflow_settings.yaml b/tests/api/projects/common_v2/workflow_settings.yaml new file mode 100644 index 00000000..471bb051 --- /dev/null +++ b/tests/api/projects/common_v2/workflow_settings.yaml @@ -0,0 +1,5 @@ +warehouse: bigquery +defaultProject: tada-analytics +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +defaultLocation: US diff --git a/tests/api/projects/invalid_dataform_json/BUILD b/tests/api/projects/invalid_dataform_json/BUILD deleted file mode 100644 index e83056bb..00000000 --- a/tests/api/projects/invalid_dataform_json/BUILD +++ /dev/null @@ -1,17 +0,0 @@ -package(default_visibility = ["//tests:__subpackages__"]) - -load("//tools:node_modules.bzl", "node_modules") - -filegroup( - name = "files", - srcs = glob([ - "**/*.*", - ]), -) - -node_modules( - name = "node_modules", - deps = [ - "//packages/@sqlanvil/core:package_tar", - ], -) diff --git a/tests/api/projects/invalid_dataform_json/dataform.json b/tests/api/projects/invalid_dataform_json/dataform.json deleted file mode 100644 index 4d5f4197..00000000 --- a/tests/api/projects/invalid_dataform_json/dataform.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "warehouse": "bigquery", - "defaultDatabase": "tada-analytics", - "defaultSchema": "rock&roll", - "assertionSchema": "df_integration_test_assertions" -} diff --git a/tests/api/projects/invalid_dataform_json/package.json b/tests/api/projects/invalid_dataform_json/package.json deleted file mode 100644 index e69de29b..00000000 diff --git a/tests/api/projects/never_finishes_compiling/dataform.json b/tests/api/projects/never_finishes_compiling/dataform.json deleted file mode 100644 index 23df7d64..00000000 --- a/tests/api/projects/never_finishes_compiling/dataform.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "warehouse": "bigquery", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "defaultLocation": "US" -} diff --git a/tests/api/projects/never_finishes_compiling/workflow_settings.yaml b/tests/api/projects/never_finishes_compiling/workflow_settings.yaml new file mode 100644 index 00000000..0c84a073 --- /dev/null +++ b/tests/api/projects/never_finishes_compiling/workflow_settings.yaml @@ -0,0 +1,4 @@ +warehouse: bigquery +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +defaultLocation: US diff --git a/tests/integration/postgres_project/dataform.json b/tests/integration/postgres_project/dataform.json deleted file mode 100644 index 71f60132..00000000 --- a/tests/integration/postgres_project/dataform.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "warehouse": "postgres", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "vars": { - "fooVar": "bar" - } -} diff --git a/tests/integration/postgres_project/workflow_settings.yaml b/tests/integration/postgres_project/workflow_settings.yaml new file mode 100644 index 00000000..ee9a0ad7 --- /dev/null +++ b/tests/integration/postgres_project/workflow_settings.yaml @@ -0,0 +1,5 @@ +warehouse: postgres +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +vars: + fooVar: bar From 9809e459bd0915c95bb1deea5fb8457ce209556c Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 07:04:29 -0500 Subject: [PATCH 06/20] =?UTF-8?q?refactor:=20rename=20CLI=20binary=20+=20h?= =?UTF-8?q?elp=20text=20+=20VSCode=20extension=20dataform=20=E2=86=92=20sq?= =?UTF-8?q?lanvil?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CLI (cli/index.ts): - yargs scriptName: dataform → sqlanvil (sets program name in --help) - All command descriptions: "Compile the dataform project" → "sqlanvil" - Error messages: "dataform encountered an error" → "sqlanvil" - "for Dataform to use" → "for sqlanvil to use" - projectDirOption describe: "Dataform project directory" → "sqlanvil" - jobPrefix template: dataform-${...}- → sqlanvil-${...}- Help test (cli/index_help_test.ts): assertion strings match new help output. Init test (cli/index_init_test.ts): defaultAssertionDataset literal + error messages. cli/credentials.ts: replaced broken docs link (docs.sqlanvil.co/dataform-cli — which didn't exist on either domain) with Google Cloud's official iam.docs.google.com link to service-account-key creation. VSCode extension (vscode/): - extension.ts: language server ID + display name, command IDs (dataform.compile → sqlanvil.compile), getConfiguration("sqlanvil") - server.ts: spawn("sqlanvil"), all error message text - package.json: rewrote manifest — name/publisher/displayName/repository/ icon/configuration namespace/command IDs all switch to sqlanvil. Version reset 0.0.16 → 0.0.1 (fresh extension). - Renamed vscode/dataform_logo.png → vscode/sqlanvil_logo.png (placeholder; actual icon needs design before publishing) Co-Authored-By: Claude Opus 4.7 --- cli/credentials.ts | 4 +-- cli/index.ts | 30 +++++++++--------- cli/index_help_test.ts | 28 ++++++++-------- cli/index_init_test.ts | 6 ++-- vscode/BUILD | 2 +- vscode/extension.ts | 10 +++--- vscode/package.json | 30 +++++++++--------- vscode/server.ts | 14 ++++---- .../{dataform_logo.png => sqlanvil_logo.png} | Bin 9 files changed, 62 insertions(+), 62 deletions(-) rename vscode/{dataform_logo.png => sqlanvil_logo.png} (100%) diff --git a/cli/credentials.ts b/cli/credentials.ts index 4abfae3b..78dd79ef 100644 --- a/cli/credentials.ts +++ b/cli/credentials.ts @@ -27,8 +27,8 @@ export function getBigQueryCredentials(): sqlanvil.IBigQuery { } const cloudCredentialsPath = actuallyResolve( question( - "Please follow the instructions at https://docs.sqlanvil.co/dataform-cli#create-a-credentials-file/\n" + - "to create and download a private key from the Google Cloud Console in JSON format.\n" + + "Follow instructions for creating a service account key for BigQuery access\n" + + "(see https://cloud.google.com/iam/docs/keys-create-delete) and download as JSON.\n" + "(You can delete this file after credential initialization is complete.)\n\n" + "Enter the path to your Google Cloud private key file:" ) diff --git a/cli/index.ts b/cli/index.ts index 88ba7803..a80bd0e7 100644 --- a/cli/index.ts +++ b/cli/index.ts @@ -46,7 +46,7 @@ process.on("unhandledRejection", async (reason: any) => { const projectDirOption: INamedOption = { name: "project-dir", option: { - describe: "The Dataform project directory.", + describe: "The sqlanvil project directory.", default: ".", coerce: actuallyResolve } @@ -180,7 +180,7 @@ const timeoutOption: INamedOption = { const jobPrefixOption: INamedOption = { name: "job-prefix", option: { - describe: "Adds an additional prefix in the form of `dataform-${jobPrefix}-`.", + describe: "Adds an additional prefix in the form of `sqlanvil-${jobPrefix}-`.", type: "string", default: null } @@ -209,7 +209,7 @@ const bigqueryJobLabelsOption: INamedOption = { const quietCompileOption: INamedOption = { name: "quiet", option: { - describe: "Less verbose compilation output. Example usage: 'dataform compile --quiet'", + describe: "Less verbose compilation output. Example usage: 'sqlanvil compile --quiet'", type: "boolean", default: false } @@ -257,7 +257,7 @@ export function runCli() { format: `init [${projectDirOption.name}] [${ProjectConfigOptions.defaultDatabase.name}]` + ` [${ProjectConfigOptions.defaultLocation.name}]`, - description: "Create a new dataform project.", + description: "Create a new sqlanvil project.", positionalOptions: [ projectDirOption, { @@ -269,7 +269,7 @@ export function runCli() { if (!argv[ProjectConfigOptions.defaultDatabase.name]) { throw new Error( `The ${ProjectConfigOptions.defaultDatabase.name} positional argument is ` + - `required. Use "dataform help init" for more info.` + `required. Use "sqlanvil help init" for more info.` ); } } @@ -285,7 +285,7 @@ export function runCli() { if (!argv[ProjectConfigOptions.defaultLocation.name]) { throw new Error( `The ${ProjectConfigOptions.defaultLocation.name} positional argument is ` + - `required. Use "dataform help init" for more info.` + `required. Use "sqlanvil help init" for more info.` ); } } @@ -328,7 +328,7 @@ export function runCli() { { format: `init-creds [${projectDirMustExistOption.name}]`, description: - `Create a ${credentials.CREDENTIALS_FILENAME} file for Dataform to use when ` + + `Create a ${credentials.CREDENTIALS_FILENAME} file for sqlanvil to use when ` + `accessing BigQuery.`, positionalOptions: [projectDirMustExistOption], options: [ @@ -377,7 +377,7 @@ export function runCli() { { format: `compile [${projectDirMustExistOption.name}]`, description: - "Compile the dataform project. Produces JSON output describing the non-executable graph.", + "Compile the sqlanvil project. Produces JSON output describing the non-executable graph.", positionalOptions: [projectDirMustExistOption], options: [ { @@ -395,7 +395,7 @@ export function runCli() { { name: verboseOptionName, option: { - describe: "Enable verbose compilation output. Example usage: 'dataform compile --verbose'", + describe: "Enable verbose compilation output. Example usage: 'sqlanvil compile --verbose'", type: "boolean", default: false }, @@ -500,7 +500,7 @@ export function runCli() { }, { format: `test [${projectDirMustExistOption.name}]`, - description: "Run the dataform project's unit tests.", + description: "Run the sqlanvil project's unit tests.", positionalOptions: [projectDirMustExistOption], options: [credentialsOption, timeoutOption, jsonOutputOption, ...ProjectConfigOptions.allYargsOptions], processFn: async argv => { @@ -544,7 +544,7 @@ export function runCli() { }, { format: `run [${projectDirMustExistOption.name}]`, - description: "Run the dataform project.", + description: "Run the sqlanvil project.", positionalOptions: [projectDirMustExistOption], options: [ { @@ -703,7 +703,7 @@ export function runCli() { }, { format: `format [${projectDirMustExistOption.name}]`, - description: "Format the dataform project's files.", + description: "Format the sqlanvil project's files.", positionalOptions: [projectDirMustExistOption], options: [ actionsOption, @@ -790,16 +790,16 @@ export function runCli() { } ] }) - .scriptName("dataform") + .scriptName("sqlanvil") .strict() .wrap(null) .recommendCommands() .fail(async (msg: string, err: any) => { if (!!err && err.name === "VMError" && err.message.includes("Cannot find module")) { - printError("Could not find NPM dependencies. Have you run 'dataform install'?"); + printError("Could not find NPM dependencies. Have you run 'sqlanvil install'?"); } else { const message = err?.message ? err.message.split("\n")[0] : msg; - printError(`Dataform encountered an error: ${message}`); + printError(`sqlanvil encountered an error: ${message}`); if (err?.stack) { printError(err.stack); } diff --git a/cli/index_help_test.ts b/cli/index_help_test.ts index 1e53d689..44fe9baa 100644 --- a/cli/index_help_test.ts +++ b/cli/index_help_test.ts @@ -9,21 +9,21 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("dataform [command]"); - expect(output).to.include("dataform init [project-dir] [default-database] [default-location]"); - expect(output).to.include("dataform install [project-dir]"); - expect(output).to.include("dataform init-creds [project-dir]"); - expect(output).to.include("dataform compile [project-dir]"); - expect(output).to.include("dataform test [project-dir]"); - expect(output).to.include("dataform run [project-dir]"); - expect(output).to.include("dataform format [project-dir]"); + expect(output).to.include("sqlanvil [command]"); + expect(output).to.include("sqlanvil init [project-dir] [default-database] [default-location]"); + expect(output).to.include("sqlanvil install [project-dir]"); + expect(output).to.include("sqlanvil init-creds [project-dir]"); + expect(output).to.include("sqlanvil compile [project-dir]"); + expect(output).to.include("sqlanvil test [project-dir]"); + expect(output).to.include("sqlanvil run [project-dir]"); + expect(output).to.include("sqlanvil format [project-dir]"); }); test("shows help for 'init' command", async () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "init"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Create a new dataform project."); + expect(output).to.include("Create a new sqlanvil project."); expect(output).to.include("--iceberg"); expect(output).to.include("Initialize the project with workflow-level Iceberg tables configuration."); }); @@ -40,7 +40,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "init-creds"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Create a .df-credentials.json file for Dataform to use when accessing BigQuery."); + expect(output).to.include("Create a .df-credentials.json file for sqlanvil to use when accessing BigQuery."); expect(output).to.include("[project-dir]"); expect(output).to.include("--test-connection"); expect(output).to.include("If true, a test query will be run using your final credentials."); @@ -50,7 +50,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "compile"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Compile the dataform project."); + expect(output).to.include("Compile the sqlanvil project."); expect(output).to.include("--watch"); expect(output).to.include("--json"); expect(output).to.include("--quiet"); @@ -60,7 +60,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "test"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Run the dataform project's unit tests."); + expect(output).to.include("Run the sqlanvil project's unit tests."); expect(output).to.include("[project-dir]"); expect(output).to.include("--credentials"); expect(output).to.include("--timeout"); @@ -72,7 +72,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "run"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Run the dataform project."); + expect(output).to.include("Run the sqlanvil project."); expect(output).to.include("--dry-run"); expect(output).to.include("--run-tests"); expect(output).to.include("--action-retry-limit"); @@ -88,7 +88,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "format"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Format the dataform project's files."); + expect(output).to.include("Format the sqlanvil project's files."); expect(output).to.include("--check"); expect(output).to.include("Check if files are formatted correctly without modifying them."); expect(output).to.include("--actions"); diff --git a/cli/index_init_test.ts b/cli/index_init_test.ts index cf510893..05b83e21 100644 --- a/cli/index_init_test.ts +++ b/cli/index_init_test.ts @@ -44,7 +44,7 @@ suite("init command", ({ afterEach }) => { defaultProject: sqlanvil-database defaultLocation: us-central1 defaultDataset: sqlanvil -defaultAssertionDataset: dataform_assertions +defaultAssertionDataset: sqlanvil_assertions `); }); @@ -280,14 +280,14 @@ defaultAssertionDataset: dataform_assertions suite("init-creds command", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); - test("init-creds fails for directory without dataform config", async () => { + test("init-creds fails for directory without sqlanvil config", async () => { const emptyDir = tmpDirFixture.createNewTmpDir(); const result = await getProcessResult( execFile(nodePath, [cliEntryPointPath, "init-creds", emptyDir]) ); expect(result.exitCode).to.not.equal(0); expect(result.stderr).to.include( - `${emptyDir} does not appear to be a dataform directory (missing workflow_settings.yaml file).` + `${emptyDir} does not appear to be a sqlanvil directory (missing workflow_settings.yaml file).` ); }); }); diff --git a/vscode/BUILD b/vscode/BUILD index 64c402ca..f1e1b364 100644 --- a/vscode/BUILD +++ b/vscode/BUILD @@ -43,7 +43,7 @@ sh_binary( srcs = ["packager.sh"], data = [ ":language-configuration.json", - ":dataform_logo.png", + ":sqlanvil_logo.png", ":package.json", ":README.md", ":LICENSE", diff --git a/vscode/extension.ts b/vscode/extension.ts index 15f0ba45..abd30aa3 100644 --- a/vscode/extension.ts +++ b/vscode/extension.ts @@ -31,8 +31,8 @@ export async function activate(context: vscode.ExtensionContext) { }; client = new LanguageClient( - "dataformLanguageServer", - "Dataform Language Server", + "sqlanvilLanguageServer", + "sqlanvil Language Server", serverOptions, clientOptions ); @@ -61,12 +61,12 @@ export async function activate(context: vscode.ExtensionContext) { // We also can add the extension to "extensionDependencies" in package.json, // but this way we can avoid forcing users to install the extension. // You can control this recommendation behavior through the setting. - if (workspace.getConfiguration("dataform").get("recommendYamlExtension")) { + if (workspace.getConfiguration("sqlanvil").get("recommendYamlExtension")) { const yamlExtension = vscode.extensions.getExtension("redhat.vscode-yaml"); if (!yamlExtension) { await vscode.window .showInformationMessage( - "The Dataform extension recommends installing the YAML extension for workflow_settings.yaml support.", + "The sqlanvil extension recommends installing the YAML extension for workflow_settings.yaml support.", "Install", "Don't show again" ) @@ -77,7 +77,7 @@ export async function activate(context: vscode.ExtensionContext) { } else if (selection === "Don't show again") { // Disable the recommendation workspace - .getConfiguration("dataform") + .getConfiguration("sqlanvil") .update("recommendYamlExtension", false, vscode.ConfigurationTarget.Global); } }); diff --git a/vscode/package.json b/vscode/package.json index f9a400cc..8e9f5850 100644 --- a/vscode/package.json +++ b/vscode/package.json @@ -1,15 +1,15 @@ { - "name": "dataform", + "name": "sqlanvil", "categories": [ "Programming Languages" ], - "description": "Syntax highlighting, compilation, and intellisense for Dataform and SQLX projects.", - "displayName": "Dataform", - "publisher": "dataform", - "version": "0.0.16", - "icon": "dataform_logo.png", + "description": "Syntax highlighting, compilation, and intellisense for sqlanvil and SQLX projects.", + "displayName": "sqlanvil", + "publisher": "ihistand", + "version": "0.0.1", + "icon": "sqlanvil_logo.png", "repository": { - "url": "https://github.com/dataform-co/dataform/tree/master/vscode" + "url": "https://github.com/ihistand/sqlanvil/tree/main/vscode" }, "engines": { "vscode": "^1.48.0" @@ -26,22 +26,22 @@ "contributes": { "configuration": [ { - "title": "Dataform", + "title": "sqlanvil", "properties": { - "dataform.compileOnSave": { + "sqlanvil.compileOnSave": { "type": "boolean", "default": true, "markdownDescription": "Whether to re-compile project each time an `.sqlx` file is saved. Will always re-compile on initialize and on config change." }, - "dataform.compilerOptions": { + "sqlanvil.compilerOptions": { "type": "array", "items": { "type": "string" }, "default": [], - "markdownDescription": "An array of additional arguments the extension should pass to the Dataform cli while executing `dataform compile --json`." + "markdownDescription": "An array of additional arguments the extension should pass to the sqlanvil cli while executing `sqlanvil compile --json`." }, - "dataform.recommendYamlExtension": { + "sqlanvil.recommendYamlExtension": { "type": "boolean", "default": true, "markdownDescription": "Whether to recommend the YAML extension for validating `workflow_settings.yaml`." @@ -51,9 +51,9 @@ ], "commands": [ { - "command": "dataform.compile", + "command": "sqlanvil.compile", "title": "Compile project", - "category": "Dataform" + "category": "sqlanvil" } ], "languages": [ @@ -79,4 +79,4 @@ } ] } -} \ No newline at end of file +} diff --git a/vscode/server.ts b/vscode/server.ts index 2de466c4..b830e260 100644 --- a/vscode/server.ts +++ b/vscode/server.ts @@ -59,23 +59,23 @@ documents.onDidSave(change => { }); async function applySettings() { - settings = await connection.workspace.getConfiguration("dataform"); + settings = await connection.workspace.getConfiguration("sqlanvil"); } async function compileAndValidate() { let compilationFailed = false; - const spawnedProcess = spawn("dataform", ["compile", "--json", ...settings.compilerOptions], { + const spawnedProcess = spawn("sqlanvil", ["compile", "--json", ...settings.compilerOptions], { shell: true }); const compileResult = await getProcessResult(spawnedProcess); if (compileResult.exitCode !== 0) { // tslint:disable-next-line: no-console - console.error("Error running 'dataform compile':", compileResult); + console.error("Error running 'sqlanvil compile':", compileResult); if (compileResult.error?.code === "ENOENT") { connection.sendNotification( "error", - "Errors encountered when running 'dataform' CLI. Please ensure that the CLI is installed and up-to-date: 'npm i -g @sqlanvil/cli'." + "Errors encountered when running 'sqlanvil' CLI. Please ensure that the CLI is installed and up-to-date: 'npm i -g @sqlanvil/cli'." ); return; } else { @@ -88,10 +88,10 @@ async function compileAndValidate() { parsedResult = JSON.parse(compileResult.stdout); } catch (e) { // tslint:disable-next-line: no-console - console.error("Error parsing 'dataform compile' output", e); + console.error("Error parsing 'sqlanvil compile' output", e); connection.sendNotification( "error", - "Error parsing 'dataform compile' output. Please check the output for more information." + "Error parsing 'sqlanvil compile' output. Please check the output for more information." ); return; } @@ -106,7 +106,7 @@ async function compileAndValidate() { if (compilationFailed) { connection.sendNotification( "error", - "Errors encountered when running 'dataform' CLI. Please check the output for more information." + "Errors encountered when running 'sqlanvil' CLI. Please check the output for more information." ); return; } diff --git a/vscode/dataform_logo.png b/vscode/sqlanvil_logo.png similarity index 100% rename from vscode/dataform_logo.png rename to vscode/sqlanvil_logo.png From 31adafa616b7bfd40ca1b0baf6f8b30fd91f2422 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 07:09:03 -0500 Subject: [PATCH 07/20] refactor: rename interface ID + write NOTICE + rewrite root docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - IDataformExtension → ISqlanvilExtension (and all imports/implementations: core/extension.ts, core/index.ts, core/main.ts, packages/sample-extension, packages/@sqlanvil/core) Docs: - readme.md: complete rewrite for sqlanvil positioning (was upstream Dataform copy referencing GCP-managed Dataform) - contributing.md: rewrite for sqlanvil — removed dataform-co/dataform upstream links, dropped GCP-credentials-team section (no longer accessible), added Postgres integration test prerequisite (Docker) - docs/configs-reference.md: placeholder until new proto schemas merge - docs/packages.md: light update (cloned base repo + framework name) - docs/postgres_reintegration_assessment.md: marked SUPERSEDED with pointer to postgres_first_class_design.md NOTICE: Apache 2.0 attribution file. Required by §4 of the license when distributing derivative works. Credits Dataform Co + Google LLC as upstream authors. Preserved upstream references (legitimate attribution / external packages): - readme/docs links to cloud.google.com/dataform (Google's product docs) - docs/packages.md sample package URLs (external community packages authored against upstream Dataform; not maintained by sqlanvil) - LICENSE (Apache 2.0 text, unchanged) Co-Authored-By: Claude Opus 4.7 --- NOTICE | 14 ++ contributing.md | 157 ++++++---------------- core/extension.ts | 2 +- core/index.ts | 4 +- core/main.ts | 4 +- docs/configs-reference.md | 7 +- docs/packages.md | 6 +- docs/postgres_reintegration_assessment.md | 11 +- packages/@sqlanvil/core/index.ts | 2 +- packages/sample-extension/index.ts | 6 +- readme.md | 73 ++++------ 11 files changed, 107 insertions(+), 179 deletions(-) create mode 100644 NOTICE diff --git a/NOTICE b/NOTICE new file mode 100644 index 00000000..f3efbc5e --- /dev/null +++ b/NOTICE @@ -0,0 +1,14 @@ +sqlanvil +Copyright 2026 Ivan Histand + +This product includes software originally developed as Dataform +(https://github.com/dataform-co/dataform) by Dataform Co and contributed +to by Google LLC, licensed under the Apache License, Version 2.0. + +The original Dataform copyright notice from upstream is preserved in +the LICENSE file. This derivative work (renamed sqlanvil) adds +PostgreSQL and Supabase warehouse adapter support, and is distributed +under the same Apache License, Version 2.0. + +Apache License, Version 2.0 +http://www.apache.org/licenses/LICENSE-2.0 diff --git a/contributing.md b/contributing.md index 575e32a1..649d0c69 100644 --- a/contributing.md +++ b/contributing.md @@ -1,162 +1,85 @@ # Contributing -Dataform is a TypeScript project, using [Bazel](https://bazel.build) as a build tool. To scope out work, please [check existing issues (which includes feature requests)](https://github.com/dataform-co/dataform/issues) or [open a discussion thread](https://github.com/dataform-co/dataform/discussions)! +sqlanvil is a TypeScript project, using [Bazel](https://bazel.build) as a build tool. ## Getting Started -First :fork_and_knife: [fork this repository](https://github.com/dataform-co/dataform/fork), clone it to your desktop, and navigate within. +[Fork the repository](https://github.com/ihistand/sqlanvil/fork), clone it, and navigate inside. ### Requirements #### [Bazel](https://bazel.build) -Bazel is a build system which we used to build the project and run tests. +Bazel is the build system. Install via Bazelisk: -The easiest way to install the correct Bazel version is through Bazelisk via [NPM](https://nodejs.org/en/download/): - -``` +```bash npm i -g @bazel/bazelisk ``` -### Run the CLI +On macOS, increase the open-file limit (Bazel hits the default): -You can run the project as you would the `npm` installation of `@sqlanvil/cli`, but replace `dataform` with `./scripts/run`. +```bash +sudo sysctl -w kern.maxfiles=65536 +``` -For example, to print out the default help information: +### Run the CLI + +Substitute `./scripts/run` for the installed `sqlanvil` binary: ```bash ./scripts/run help +./scripts/run compile path/to/project ``` -Check the [docs](https://cloud.google.com/dataform/docs/reference/dataform-cli-reference) for more examples. - -_Note: If you are running Bazel on a **Mac**, this or any step that requires building may fail with a `Too many open files in system` error. This is [due to a limitation](https://github.com/angular/angular-bazel-example/issues/178) on the default maximum open file descriptors. You can increase the limit by running `sudo sysctl -w kern.maxfiles=` (we use `65536`)._ - ### Test -The following command runs tests for @sqlanvil/core: - ```bash -bazel test //core/... +bazel test //core/... # core compiler tests +bazel test //... # everything (slow on cold cache) ``` -### Integration Test - -To run the CLI integration test against your own GCP project: - -1. Comment out the following dependency in `cli/BUILD`: +### Integration Tests - - `//test_credentials:bigquery.json` +Integration tests require real warehouse credentials. The upstream +`test_credentials` GCP project is no longer accessible; you'll need to wire +your own. -2. Update the following constants in `cli/index_test.ts` to match your project: +For BigQuery integration tests: - - `DEFAULT_DATABASE` - - `DEFAULT_LOCATION` - - `CREDENTIALS_PATH` +1. Create a GCP service account with BigQuery access. +2. Download the key JSON. +3. Update constants in `cli/index_test_base.ts` to match your project + (`DEFAULT_DATABASE`, `DEFAULT_LOCATION`, `CREDENTIALS_PATH`). +4. `bazel test //cli:index_test`. - Prepare a credentials JSON file referenced by `CREDENTIALS_PATH`. Set values as follows: - - - `projectId`: the same string as `DEFAULT_DATABASE`. - - `credentials`: the entire content of your GCP service account key JSON file as a single string (you can generate it with `jq -Rsa < path/to/key.json`). - - `location`: the same string as `DEFAULT_LOCATION`. - - Example: - - ```json - { - "projectId": "my-gcp-project", - "credentials": "{\"type\":\"service_account\",...}", - "location": "US" - } - ``` - -3. Run the test: - - ```bash - bazel test //cli:index_test - ``` - -### Lint - -The following command to check for any linting errors - -```bash -./scripts/lint -``` +For Postgres integration tests, `tools/postgres/postgres_fixture.ts` boots a +Docker container inside the Bazel sandbox — requires Docker running locally. ### Building -Building the CLI will build most of the required components. - ```bash -bazel build cli +bazel build cli # build the CLI +bazel build //... # build everything ``` -The projects folder here is not built as it requires an environment file, which can be provided from the team. - -### Add New NPM Dependencies +### Adding NPM Dependencies -Global yarn installations will throw errors when installing packages, instead you should use: +Use Bazel-wrapped yarn: ```bash -$ bazel run @nodejs//:yarn add ... +bazel run @nodejs//:yarn add ``` -Additionally, installed NPM dependencies need to be added to the `deps` of `ts_library` rules by -prefixing them with `@npm//...`. - -## The Contribution Process - -1. Decide on what you'd like to contribute. The majority of open-source contributions come from: - - 1. Someone deciding they want a feature that is not currently present, and which isn't a priority for the team. - - 1. Embracing the community aspect of open source (or getting that commit count up) and solving an issue. - -1. Plan out the change, and whether it is feasible. - - 1. If you're unsure of the scope of the change, then ask in the issue or [create a discussion](https://github.com/dataform-co/dataform/discussions). - - 1. We'd much prefer multiple smaller code changes than a single large one. - - 1. Avoid changing core functionality over a long time frame. Our development process is very dynamic, so if your code depends on lots of other parts of the project, then it is likely to be out of date by the time you finish! - - 1. If you're solving an issue, be sure to comment to make it known that you are currently solving it. Unless we have worked with you before, it is unlikely that we will lock the issue to you. - -1. Begin materialising your masterpiece. - - 1. Create a feature branch based on `main` ([link](https://github.com/dataform-co/dataform/tree/main)) for development work. - -1. Once done, review your code, run the tests, **[check for common mistakes](#common-pull-request-mistakes)** and then open a pull request. - - 1. Tidy the code by removing erronous log statements. Comment difficult to interpret sections. Make sure functions are names appropriately. We will review the pull request mainly by the git difference. - - 1. Assign a reviewer. Pick anyone on the team who seems to contribute a lot and they will refer it onto whoever is most responsible for the given subsystem. - -1. Discuss and process any changes requested. - - 1. It's unlikely your pull request will be perfect immediately; there will likely be some changes requested, whether it's to do with style or a more fundamental issue. - - 1. The automated integration tests must pass. - - 1. Once a pull request is accepted and all automated integration tests are passing, we will merge it for you. - -### Reporting Issues (!) - -Another way we'd love for you to contribute is by flagging any issues you find. First check through the list of [existing issues](https://github.com/dataform-co/dataform/issues) for anything similar, in order to avoid duplicates. If not, then full steam ahead! - -### Promoting Dataform - -If you're using Dataform for interesting projects then please let people know! Reach out to [dataform-preview@google.com](dataform-preview@google.com) for marketing support. - -### Common Pull Request Mistakes - -1. Is it too long? Small pull requests are easier to review and merge. If you are planning on making a larger change, then talk to the team and write a document on the design. +After installation, add the package to relevant `ts_library` deps prefixed +with `@npm//`. -1. Have you appropriately increased test coverage? If the operation of the change is not already tested, then tests will need to be written. +## Pull Requests -1. Is it a hack? Does it solve the problem, but it is not reliable, reproducable or extendable? In other words, [does it smell](https://en.wikipedia.org/wiki/Code_smell)? +- Keep PRs small and focused — small diffs are easier to review. +- Add tests when changing behavior. +- Don't reformat unrelated code (makes diffs noisy). +- Branch off `main`. -1. Have you changed whitespace or touched unrelated code? Please avoid this as it makes pull requests far more difficult to review. +## Reporting Issues -1. Are the comments useful, and is the code readable? Are the function and variable names appropriate? +[Open an issue](https://github.com/ihistand/sqlanvil/issues) on GitHub. diff --git a/core/extension.ts b/core/extension.ts index f15ec3c5..de034642 100644 --- a/core/extension.ts +++ b/core/extension.ts @@ -4,7 +4,7 @@ import { sqlanvil } from "sa/protos/ts"; /** * Extension interface. */ -export interface IDataformExtension { +export interface ISqlanvilExtension { /** * Run additional compilation steps. * Passed session should be used for both new nodes creation and persisting errors. diff --git a/core/index.ts b/core/index.ts index 55b53723..99d74ecc 100644 --- a/core/index.ts +++ b/core/index.ts @@ -1,5 +1,5 @@ import { compile as compiler } from "sa/core/compilers"; -import { IDataformExtension } from "sa/core/extension"; +import { ISqlanvilExtension } from "sa/core/extension"; import { IJitCompiler, jitCompiler } from "sa/core/jit_compiler"; import { main } from "sa/core/main"; import { Session } from "sa/core/session"; @@ -30,4 +30,4 @@ function indexFileGenerator() { // These exports constitute the public API of @sqlanvil/core. // They must also be listed in packages/@sqlanvil/core/index.ts. // Changes to these will break @sqlanvil/cli, so take care! -export { compiler, IDataformExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version }; +export { compiler, ISqlanvilExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version }; diff --git a/core/main.ts b/core/main.ts index 03dd1171..a46b3b3b 100644 --- a/core/main.ts +++ b/core/main.ts @@ -12,7 +12,7 @@ import { Notebook } from "sa/core/actions/notebook"; import { Operation } from "sa/core/actions/operation"; import { Table } from "sa/core/actions/table"; import { View } from "sa/core/actions/view"; -import { IDataformExtension } from "sa/core/extension"; +import { ISqlanvilExtension } from "sa/core/extension"; import * as Path from "sa/core/path"; import { Session } from "sa/core/session"; import { nativeRequire } from "sa/core/utils"; @@ -196,7 +196,7 @@ function mainCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: function extensionCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { try { const module = nativeRequire(compileRequest?.compileConfig?.extension.name); - const extension: () => IDataformExtension = module.extension; + const extension: () => ISqlanvilExtension = module.extension; extension().compile(compileRequest, session); } catch (e) { session.compileError(e, compileRequest?.compileConfig?.extension.name); diff --git a/docs/configs-reference.md b/docs/configs-reference.md index 12a897ce..d4318f24 100644 --- a/docs/configs-reference.md +++ b/docs/configs-reference.md @@ -1,3 +1,8 @@ # Configs Reference -Relocated to [docs/reference/configs](https://dataform-co.github.io/dataform/docs/reference/configs). +TODO: write the sqlanvil configs reference once the new proto schemas +(`WarehouseConfig`, `PostgresOptions`, `SupabaseOptions`) are merged. + +For the upstream Dataform configs reference (which still describes the +BigQuery-only behavior sqlanvil inherits today), see: +https://dataform-co.github.io/dataform/docs/reference/configs diff --git a/docs/packages.md b/docs/packages.md index db30b8a6..f88ecce7 100644 --- a/docs/packages.md +++ b/docs/packages.md @@ -1,6 +1,6 @@ # Packages -[<- Back to Home](https://dataform-co.github.io/dataform). +[<- Back to Home](https://github.com/ihistand/sqlanvil). ## Sample Packages @@ -13,11 +13,11 @@ ## Creating a package -Creating your own package is relatively easy, as long as you're relatively familiar with the Dataform framework. It may also help to understand the fundamentals of JavaScript, but you can probably muddle through without this! +Creating your own package is relatively easy, as long as you're relatively familiar with the sqlanvil framework. It may also help to understand the fundamentals of JavaScript, but you can probably muddle through without this! ### Clone the base package repo -[This repo](https://github.com/dataform-co/dataform-package-base) contains the building blocks of a package: +[This repo](https://github.com/ihistand/sqlanvil-package-base) contains the building blocks of a package: - index.js - example.js diff --git a/docs/postgres_reintegration_assessment.md b/docs/postgres_reintegration_assessment.md index 3d5a0b48..9e03e3f6 100644 --- a/docs/postgres_reintegration_assessment.md +++ b/docs/postgres_reintegration_assessment.md @@ -1,6 +1,13 @@ -# Technical Assessment: PostgreSQL Reintegration in SqlAnvil +# Technical Assessment: PostgreSQL Reintegration in sqlanvil -This document provides a detailed technical assessment of what is required to re-integrate PostgreSQL database adapter support into the modern **SqlAnvil** (Dataform) codebase using the old restored files. +> **SUPERSEDED.** Phase 3+ of this assessment is replaced by +> [`postgres_first_class_design.md`](postgres_first_class_design.md), which +> treats the Postgres adapter as first-class (idiomatic SQL generation, native +> action config blocks, dedicated SQL generator path) rather than a +> BigQuery-shaped adapter swap. Phases 1-2 (deps + relocation) below remain +> applicable; Phases 3-5 are re-scoped in the design doc. + +This document provides a detailed technical assessment of what is required to re-integrate PostgreSQL database adapter support into the modern **sqlanvil** codebase using the old restored files. --- diff --git a/packages/@sqlanvil/core/index.ts b/packages/@sqlanvil/core/index.ts index f9599b9a..1658df9e 100644 --- a/packages/@sqlanvil/core/index.ts +++ b/packages/@sqlanvil/core/index.ts @@ -1 +1 @@ -export { compiler, IDataformExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version } from "sa/core"; +export { compiler, ISqlanvilExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version } from "sa/core"; diff --git a/packages/sample-extension/index.ts b/packages/sample-extension/index.ts index 69611263..64421f9a 100644 --- a/packages/sample-extension/index.ts +++ b/packages/sample-extension/index.ts @@ -1,8 +1,8 @@ -import type { IDataformExtension } from "sa/core"; +import type { ISqlanvilExtension } from "sa/core"; import { Session } from "sa/core/session"; import { sqlanvil } from "sa/protos/ts"; -class SampleExtension implements IDataformExtension { +class SampleExtension implements ISqlanvilExtension { public compile(request: sqlanvil.ICompileExecutionRequest, session: Session): void { if (request.compileConfig?.projectConfigOverride?.vars["throw-error"] === "true") { throw new Error("throwing exception as requested!"); @@ -16,6 +16,6 @@ class SampleExtension implements IDataformExtension { } } -export function extension(): IDataformExtension { +export function extension(): ISqlanvilExtension { return new SampleExtension(); } diff --git a/readme.md b/readme.md index cc39d087..b1f0d0e3 100644 --- a/readme.md +++ b/readme.md @@ -1,60 +1,39 @@ -# Dataform Core +# sqlanvil -Dataform Core is an open source meta-language to create SQL tables and workflows in BigQuery. Dataform Core extends SQL by providing a dependency management system, automated data quality testing, and data documentation. +**sqlanvil** is an open-source SQL workflow tool. Define dependent tables, views, materialized views, incremental tables, assertions, and operations as code — then compile, test, and execute them against your warehouse. -Using Dataform Core, data teams can build scalable SQL data transformation pipelines following software engineering best practices, like version control and testing. +sqlanvil is a hard fork of [Dataform](https://github.com/dataform-co/dataform) (Apache 2.0), renamed and restructured to support **PostgreSQL/Supabase** as a first-class target alongside **BigQuery**. -For more details, see [how Dataform works](https://cloud.google.com/dataform/docs/overview). +## Status -![Data collections and integrations feed into Dataform, which exports this data to BI and analytics tools.](static/images/single-source-of-truth.png?raw=true) +Pre-alpha. Active reintegration of the Postgres adapter is in progress on the +`restore-postgres-adapter` branch. See `docs/postgres_first_class_design.md` +for the implementation spec and `docs/rename_checklist.md` for the +dataform → sqlanvil rename surface. -## Get started +## Supported warehouses -### In Google Cloud Platform +| Warehouse | Status | +| :--- | :--- | +| BigQuery | Working (inherited from upstream Dataform) | +| PostgreSQL | In progress — first-class native adapter | +| Supabase (PostgreSQL + RLS, Realtime, Wrappers, pgvector) | Planned | -Dataform in Google Cloud Platform provides a fully managed experience to build scalable data transformations pipelines in **BigQuery** using SQL. It includes: +## Quickstart (when published) -- A cloud development environment to develop data assets with SQL and Dataform Core and version control code with GitHub, GitLab, and other Git providers. -- A fully managed, serverless orchestration environment for data pipelines, fully integrated in Google Cloud Platform. - -Follow the [quickstart guide](https://cloud.google.com/dataform/docs/quickstart)! - -### With the CLI - -You can run Dataform locally using the Dataform CLI tool, which can be installed using the following command line. Follow the [CLI guide](https://cloud.google.com/dataform/docs/use-dataform-cli) to get started. - -``` +```bash npm i -g @sqlanvil/cli +sqlanvil init my-project +cd my-project +sqlanvil compile +sqlanvil run ``` -## Useful Links - -- [Documentation home page](https://cloud.google.com/dataform). -- [Create tables and views](https://cloud.google.com/dataform/docs/tables). -- [Configure dependencies](https://cloud.google.com/dataform/docs/define-table#define_table_structure_and_dependencies). -- Write [data quality checks](https://cloud.google.com/dataform/docs/assertions). -- Enable [scripting](https://cloud.google.com/dataform/docs/develop-workflows-js) and code re-use with a JavaScript API. -- Import [pre-defined packages](https://dataform-co.github.io/dataform/docs/packages), or create your own. -- View the [Dataform Core reference](https://cloud.google.com/dataform/docs/reference/dataform-core-reference). -- View the [Dataform configs reference](https://dataform-co.github.io/dataform/docs/configs-reference). - -_Note: this readme can also be viewed on https://dataform-co.github.io/dataform._ - -## Example Projects - -- [https://github.com/GoogleCloudPlatform/marketing-data-engine-dataform](https://github.com/GoogleCloudPlatform/marketing-data-engine-dataform). -- [https://github.com/wintermi/movielens-dataform](https://github.com/wintermi/movielens-dataform). -- [https://github.com/wintermi/bqe-dataform](https://github.com/wintermi/bqe-dataform). -- [https://github.com/wintermi/imdb-dataform](https://github.com/wintermi/imdb-dataform). -- [https://github.com/wintermi/fashion-dataform](https://github.com/wintermi/fashion-dataform). -- [https://github.com/G2H/dataform-stackoverflow](https://github.com/G2H/dataform-stackoverflow). -- [https://github.com/karcot1/dataform_deployment_sample](https://github.com/karcot1/dataform_deployment_sample). - -## Want to report a bug or request a feature? - -- For Dataform Core / open source requests, you can open an [issue](https://github.com/dataform-co/dataform/issues) in GitHub. -- For Dataform in Google Cloud Platform, you can file a bug [here](https://issuetracker.google.com/issues/new?component=1193995&template=1698201), and file feature requests [here](https://issuetracker.google.com/issues/new?component=1193995&template=1713836). +For now, building from source requires Bazel via Bazelisk +(`npm i -g @bazel/bazelisk`). See `contributing.md`. -## Want to contribute? +## Attribution -Check out our [contributors guide](https://github.com/dataform-co/dataform/blob/main/contributing.md) to get started with setting up the repo. +sqlanvil derives from Dataform OSS by Dataform Co (acquired by Google). +The original code remains under the Apache 2.0 license. See `NOTICE` and +`LICENSE` for required attribution. From ce6af2ebb3189aec675c0a5f4e77d3ddd2aaa9e8 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 07:15:46 -0500 Subject: [PATCH 08/20] =?UTF-8?q?refactor:=20final=20rename=20sweep=20?= =?UTF-8?q?=E2=80=94=20proto=20auto-docs=20+=20goldens=20+=20examples=20+?= =?UTF-8?q?=20dead=20infra?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Comprehensive `Dataform` (capital D) → `sqlanvil` across all remaining source code via brute-force sed on TS/proto files (BSD sed's word-boundary regex didn't fire on first pass; explicit pattern caught the rest). Auto-generated docs (docs/reference/*.md): - Anchor IDs: → - Cross-ref links: configs#dataform-ActionConfig-* → configs#sqlanvil-* - Body text describing framework concepts: "Dataform" → "sqlanvil" CLI test fixtures (cli/util_test.ts, cli/index_run_e2e_test.ts): - jobId prefix "dataform-915a03fe1" → "sqlanvil-..." - schema "dataform_test_schema_suffix" → "sqlanvil_test_schema_suffix" - --job-labels team=dataform → team=sqlanvil - uniqueDataset prefix dataform_e2e_osc_ → sqlanvil_e2e_osc_ cli/api/dbadapters/execution_sql.ts + matching goldens (*.sql): - Generated DECLARE/SET variable name dataform_columns → sqlanvil_columns (BQ on-schema-change synchronize/extend/fail paths) cli/api/commands/init.ts: workflow_settings.yaml init writes "defaultDataset: sqlanvil" (was "dataform") cli/index_test_base.ts: - DEFAULT_DATABASE: dataform-open-source → your-bigquery-project - DEFAULT_RESERVATION reservation name dataform-test → sqlanvil-test cli/vm/compile.ts + cli/vm/jit_worker.ts: node_modules/@dataform/core bundle.js lookup → @sqlanvil/core (CLI tells users to run `sqlanvil install`, not `dataform install`) common/protos/index.ts: error messages now point at github.com/ihistand/sqlanvil docs + issues, not dataform-co. protos/configs.proto: "dataformCoreVersion" doc string + comments updated. tests/integration/bigquery_project + bigquery.spec.ts + examples/: - defaultProject: dataform-open-source → your-bigquery-project - 3-part BQ names rewritten with the placeholder - Integration tests will need fresh credentials configured before they can run (already noted in contributing.md) - examples/.../snowy_repository_creation.ipynb: dataform_examples dataset reference → your_examples_dataset VSCode (vscode/{README.md,workflow_settings_yaml.schema.json,package-lock.json}): - Deprecation notice referencing upstream extension removed - JSON schema field dataformCoreVersion → sqlanvilCoreVersion - package-lock.json name field Deletions (dead infra): - scripts/decrypt_secret (KMS keyring access to dataform-co's GCP project) - scripts/create_gh_pr (sets origin URL to dataform-co/dataform.git) Preserved legitimate references: - protos/configs.proto: cloud.google.com/dataform/docs/partitions-clusters URL comments (factual BigQuery partition/cluster reference docs) - core/main.ts, core/main_test.ts, api/utils/postgres.ts: TODO comments and upstream issue/PR links for historical context - .bazelrc: commented-out remote cache configuration (already disabled) - NOTICE: Apache 2.0 attribution to Dataform Co + Google LLC - readme.md, CLAUDE.md, docs/*: attribution phrases like "upstream Dataform", "Dataform OSS by Dataform Co" - docs/reference/configs.md table rows referencing partition-clusters docs Co-Authored-By: Claude Opus 4.7 --- cli/api/commands/compile.ts | 12 +- cli/api/commands/install.ts | 4 +- cli/api/dbadapters/execution_sql.ts | 8 +- cli/api/goldens/on_schema_change_extend.sql | 8 +- cli/api/goldens/on_schema_change_fail.sql | 8 +- .../goldens/on_schema_change_synchronize.sql | 8 +- cli/api/utils.ts | 2 +- cli/api/utils_test.ts | 4 +- cli/index_compile_test.ts | 24 +-- cli/index_run_e2e_test.ts | 18 +- cli/index_test_base.ts | 4 +- cli/util.ts | 2 +- cli/util_test.ts | 6 +- cli/vm/compile.ts | 4 +- cli/vm/jit_worker.ts | 4 +- common/protos/index.ts | 6 +- common/protos/index_test.ts | 2 +- core/actions/declaration.ts | 10 +- core/actions/declaration_test.ts | 2 +- core/actions/incremental_table.ts | 8 +- core/actions/operation.ts | 2 +- core/actions/table.ts | 6 +- core/actions/test.ts | 2 +- core/actions/view.ts | 2 +- core/main.ts | 4 +- core/main_test.ts | 8 +- core/session.ts | 12 +- core/utils_test.ts | 2 +- core/workflow_settings.ts | 2 +- docs/reference/assertion.md | 20 +-- docs/reference/configs.md | 168 +++++++++--------- docs/reference/declaration.md | 18 +- docs/reference/incrementaltable.md | 36 ++-- docs/reference/notebook.md | 4 +- docs/reference/operation.md | 26 +-- docs/reference/session.md | 14 +- docs/reference/table.md | 30 ++-- docs/reference/test.md | 4 +- docs/reference/view.md | 28 +-- ...itories_created_during_extreme_weather.sql | 4 +- .../snowy_repository_creation.ipynb | 4 +- .../workflow_settings.yaml | 6 +- .../workflow_settings.yaml | 6 +- packages/@sqlanvil/cli/BUILD | 2 +- packages/@sqlanvil/cli/package.layer.json | 2 +- packages/@sqlanvil/core/BUILD | 2 +- packages/@sqlanvil/package.layer.json | 4 +- protos/configs.proto | 14 +- protos/core.proto | 4 +- protos/extension.proto | 4 +- scripts/create_gh_pr | 21 --- scripts/decrypt_secret | 9 - testing/run_core.ts | 2 +- tests/api/projects.spec.ts | 12 +- .../definitions/example_declaration.sqlx | 2 +- tests/integration/bigquery.spec.ts | 32 ++-- .../definitions/sample_data.sqlx | 4 +- .../bigquery_project/workflow_settings.yaml | 2 +- .../definitions/sample_data.sqlx | 4 +- typedoc.json | 2 +- vscode/README.md | 13 +- vscode/contributing.md | 6 +- vscode/package-lock.json | 4 +- vscode/workflow_settings_yaml.schema.json | 8 +- 64 files changed, 339 insertions(+), 366 deletions(-) delete mode 100755 scripts/create_gh_pr delete mode 100755 scripts/decrypt_secret diff --git a/cli/api/commands/compile.ts b/cli/api/commands/compile.ts index 83dae098..68a89dba 100644 --- a/cli/api/commands/compile.ts +++ b/cli/api/commands/compile.ts @@ -29,18 +29,18 @@ export async function compile( const temporaryProjectPath = tmp.dirSync().name; const workflowSettings = readConfigFromWorkflowSettings(resolvedProjectPath); - const workflowSettingsDataformCoreVersion = workflowSettings?.sqlanvilCoreVersion; + const workflowSettingssqlanvilCoreVersion = workflowSettings?.sqlanvilCoreVersion; const workflowSettingsExtension = workflowSettings?.extension ?? undefined; compileConfig.extension = workflowSettingsExtension; - if (!workflowSettingsDataformCoreVersion && !fs.existsSync(packageJsonPath)) { + if (!workflowSettingssqlanvilCoreVersion && !fs.existsSync(packageJsonPath)) { throw new Error(MISSING_CORE_VERSION_ERROR); } // For stateless package installation, a temporary directory is used in order to avoid interfering // with user's project directories. - if (workflowSettingsDataformCoreVersion) { + if (workflowSettingssqlanvilCoreVersion) { [projectNodeModulesPath, packageJsonPath, packageLockJsonPath].forEach(npmPath => { if (fs.existsSync(npmPath)) { throw new Error(`'${npmPath}' unexpected; remove it and try again`); @@ -48,7 +48,7 @@ export async function compile( }); if (compileConfig.verbose) { - print(`Using isolated environment for @sqlanvil/core@${workflowSettingsDataformCoreVersion}\n`); + print(`Using isolated environment for @sqlanvil/core@${workflowSettingssqlanvilCoreVersion}\n`); print(`Copying project to temporary directory: ${temporaryProjectPath}\n`); } const copyStartTime = performance.now(); @@ -64,7 +64,7 @@ export async function compile( path.join(temporaryProjectPath, "package.json"), `{ "dependencies": { - "@sqlanvil/core": "${workflowSettingsDataformCoreVersion}" + "@sqlanvil/core": "${workflowSettingssqlanvilCoreVersion}" } }` ); @@ -91,7 +91,7 @@ export async function compile( const decodedResult = decode64(sqlanvil.CoreExecutionResponse, result); compiledGraph = sqlanvil.CompiledGraph.create(decodedResult.compile.compiledGraph); - if (workflowSettingsDataformCoreVersion) { + if (workflowSettingssqlanvilCoreVersion) { fs.rmSync(temporaryProjectPath, { recursive: true }); } diff --git a/cli/api/commands/install.ts b/cli/api/commands/install.ts index db139a3c..3b21a320 100644 --- a/cli/api/commands/install.ts +++ b/cli/api/commands/install.ts @@ -3,7 +3,7 @@ import * as fs from "fs"; import * as path from "path"; import { promisify } from "util"; -import { readDataformCoreVersionFromWorkflowSettings } from "sa/cli/api/utils"; +import { readsqlanvilCoreVersionFromWorkflowSettings } from "sa/cli/api/utils"; export const MISSING_CORE_VERSION_ERROR = "sqlanvilCoreVersion must be specified either in workflow_settings.yaml or via a package.json"; @@ -14,7 +14,7 @@ export async function install(projectPath: string) { // Core's readWorkflowSettings method cannot be used for this because Core assumes that // `require` can read YAML files directly. - const sqlanvilCoreVersion = readDataformCoreVersionFromWorkflowSettings(resolvedProjectPath); + const sqlanvilCoreVersion = readsqlanvilCoreVersionFromWorkflowSettings(resolvedProjectPath); if (sqlanvilCoreVersion) { throw new Error( "No installation is needed when using workflow_settings.yaml, as packages are installed at " + diff --git a/cli/api/dbadapters/execution_sql.ts b/cli/api/dbadapters/execution_sql.ts index c627fab8..9fce2653 100644 --- a/cli/api/dbadapters/execution_sql.ts +++ b/cli/api/dbadapters/execution_sql.ts @@ -267,12 +267,12 @@ CREATE OR REPLACE TABLE ${emptyTempTableName} AS ( ): string { return ` -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM \`${target.database}.${target.schema}.INFORMATION_SCHEMA.COLUMNS\` WHERE table_name = '${target.name}' @@ -287,11 +287,11 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) );`; } diff --git a/cli/api/goldens/on_schema_change_extend.sql b/cli/api/goldens/on_schema_change_extend.sql index 21f9558a..aee4e062 100644 --- a/cli/api/goldens/on_schema_change_extend.sql +++ b/cli/api/goldens/on_schema_change_extend.sql @@ -9,12 +9,12 @@ CREATE OR REPLACE TABLE `project-id.dataset-id.incremental_on_schema_change_df_t -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM `project-id.dataset-id.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'incremental_on_schema_change' @@ -29,11 +29,11 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) ); diff --git a/cli/api/goldens/on_schema_change_fail.sql b/cli/api/goldens/on_schema_change_fail.sql index ea8d9054..ab37c108 100644 --- a/cli/api/goldens/on_schema_change_fail.sql +++ b/cli/api/goldens/on_schema_change_fail.sql @@ -9,12 +9,12 @@ CREATE OR REPLACE TABLE `project-id.dataset-id.incremental_on_schema_change_df_t -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM `project-id.dataset-id.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'incremental_on_schema_change' @@ -29,11 +29,11 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) ); diff --git a/cli/api/goldens/on_schema_change_synchronize.sql b/cli/api/goldens/on_schema_change_synchronize.sql index bcfc63ea..863bf7dc 100644 --- a/cli/api/goldens/on_schema_change_synchronize.sql +++ b/cli/api/goldens/on_schema_change_synchronize.sql @@ -9,12 +9,12 @@ CREATE OR REPLACE TABLE `project-id.dataset-id.incremental_on_schema_change_df_t -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM `project-id.dataset-id.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'incremental_on_schema_change' @@ -29,11 +29,11 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) ); diff --git a/cli/api/utils.ts b/cli/api/utils.ts index 1fbe553d..92272c57 100644 --- a/cli/api/utils.ts +++ b/cli/api/utils.ts @@ -8,7 +8,7 @@ export function prettyJsonStringify(obj: object) { return JSON.stringify(obj, null, 4) + "\n"; } -export function readDataformCoreVersionFromWorkflowSettings( +export function readsqlanvilCoreVersionFromWorkflowSettings( resolvedProjectPath: string ): string | undefined { return readConfigFromWorkflowSettings(resolvedProjectPath)?.sqlanvilCoreVersion; diff --git a/cli/api/utils_test.ts b/cli/api/utils_test.ts index ac90a6b0..d26a8aee 100644 --- a/cli/api/utils_test.ts +++ b/cli/api/utils_test.ts @@ -24,7 +24,7 @@ suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml({ defaultProject: "dataform" }) + dumpYaml({ defaultProject: "sqlanvil" }) ); expect(readExtensionConfigFromWorkflowSettings(projectDir)).to.equal(undefined); }); @@ -35,7 +35,7 @@ suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { path.join(projectDir, "workflow_settings.yaml"), dumpYaml({ sqlanvilCoreVersion: "3.0.0", - defaultProject: "dataform", + defaultProject: "sqlanvil", extension: { name: "test-extension", compilationMode: "PROLOGUE", diff --git a/cli/index_compile_test.ts b/cli/index_compile_test.ts index 17f40f7d..3b87d201 100644 --- a/cli/index_compile_test.ts +++ b/cli/index_compile_test.ts @@ -20,7 +20,7 @@ suite("compile command", ({ afterEach }) => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(sqlanvil.WorkflowSettings.create({ defaultProject: "dataform" })) + dumpYaml(sqlanvil.WorkflowSettings.create({ defaultProject: "sqlanvil" })) ); expect( @@ -59,7 +59,7 @@ defaultLocation: "${DEFAULT_LOCATION}" "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + - "`dataform install`." + "`sqlanvil install`." ); }); @@ -70,7 +70,7 @@ defaultLocation: "${DEFAULT_LOCATION}" path.join(projectDir, "workflow_settings.yaml"), dumpYaml( sqlanvil.WorkflowSettings.create({ - defaultProject: "dataform", + defaultProject: "sqlanvil", sqlanvilCoreVersion: "3.0.0" }) ) @@ -173,14 +173,14 @@ SELECT 1 as id { canonicalTarget: { database: DEFAULT_DATABASE, - name: "dataform_example_table_assertions_uniqueKey_0", + name: "sqlanvil_example_table_assertions_uniqueKey_0", schema: "sqlanvil_assertions" }, dependencyTargets: [ { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" } ], disabled: true, @@ -188,14 +188,14 @@ SELECT 1 as id parentAction: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, query: // tslint:disable-next-line:tsr-detect-sql-literal-injection `\nSELECT\n *\nFROM (\n SELECT\n id,\n COUNT(1) AS index_row_count\n FROM \`${DEFAULT_DATABASE}.sqlanvil.example_table\`\n GROUP BY id\n ) AS data\nWHERE index_row_count > 1\n`, target: { database: DEFAULT_DATABASE, - name: "dataform_example_table_assertions_uniqueKey_0", + name: "sqlanvil_example_table_assertions_uniqueKey_0", schema: "sqlanvil_assertions" } }, @@ -231,7 +231,7 @@ SELECT 1 as id canonicalTarget: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, disabled: false, enumType: "TABLE", @@ -241,7 +241,7 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, type: "table" } @@ -249,13 +249,13 @@ SELECT 1 as id targets: [ { database: DEFAULT_DATABASE, - name: "dataform_example_table_assertions_uniqueKey_0", + name: "sqlanvil_example_table_assertions_uniqueKey_0", schema: "sqlanvil_assertions" }, { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, { database: DEFAULT_DATABASE, @@ -306,7 +306,7 @@ suite("extension config", ({ afterEach }) => { dumpYaml({ defaultProject: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, - defaultDataset: "dataform", + defaultDataset: "sqlanvil", defaultAssertionDataset: "sqlanvil_assertions", extension: { name: "test-extension", diff --git a/cli/index_run_e2e_test.ts b/cli/index_run_e2e_test.ts index 62bdbc32..b4e9d737 100644 --- a/cli/index_run_e2e_test.ts +++ b/cli/index_run_e2e_test.ts @@ -87,11 +87,11 @@ select 1 as \${sqlanvil.projectConfig.vars.testVar2} enumType: "TABLE", target: { database: DEFAULT_DATABASE, - schema: "dataform_test_schema_suffix", + schema: "sqlanvil_test_schema_suffix", name: "example" }, canonicalTarget: { - schema: "dataform", + schema: "sqlanvil", name: "example", database: DEFAULT_DATABASE }, @@ -120,7 +120,7 @@ select 1 as \${sqlanvil.projectConfig.vars.testVar2} targets: [ { database: DEFAULT_DATABASE, - schema: "dataform", + schema: "sqlanvil", name: "example" } ] @@ -158,7 +158,7 @@ select 1 as \${sqlanvil.projectConfig.vars.testVar2} target: { database: DEFAULT_DATABASE, name: "example", - schema: "dataform" + schema: "sqlanvil" }, tasks: [ { @@ -275,7 +275,7 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, tasks: [ { @@ -376,7 +376,7 @@ SELECT 1 as id "--json", "--disable-assertions", "--actions=test_assertion,example_table", - "--job-labels=env=testing,team=dataform" + "--job-labels=env=testing,team=sqlanvil" ]) ); @@ -649,7 +649,7 @@ select 2 suite("onSchemaChange", ({ beforeEach }) => { const projectDir = tmpDirFixture.createNewTmpDir(); - const uniqueDataset = `dataform_e2e_osc_${Math.random().toString(36).substring(7)}`; + const uniqueDataset = `sqlanvil_e2e_osc_${Math.random().toString(36).substring(7)}`; beforeEach("setup test project", async () => { const npmCacheDir = tmpDirFixture.createNewTmpDir(); @@ -724,7 +724,7 @@ DROP SCHEMA IF EXISTS \`\${sqlanvil.projectConfig.defaultDatabase}.\${sqlanvil.p test("generates dynamic SQL for EXTEND when table exists in BigQuery", async () => { try { // Run setup operation to create the table in BigQuery. - // Dataform will automatically create the uniqueDataset schema. + // sqlanvil will automatically create the uniqueDataset schema. await getProcessResult( execFile(nodePath, [ cliEntryPointPath, @@ -737,7 +737,7 @@ DROP SCHEMA IF EXISTS \`\${sqlanvil.projectConfig.defaultDatabase}.\${sqlanvil.p ); // Run the incremental table in dry-run mode. - // Dataform will detect the table exists and generate the dynamic procedural SQL. + // sqlanvil will detect the table exists and generate the dynamic procedural SQL. const runResult = await getProcessResult( execFile(nodePath, [ cliEntryPointPath, diff --git a/cli/index_test_base.ts b/cli/index_test_base.ts index 29c35132..af2fa431 100644 --- a/cli/index_test_base.ts +++ b/cli/index_test_base.ts @@ -1,9 +1,9 @@ // tslint:disable tsr-detect-non-literal-fs-filename import * as path from "path"; -export const DEFAULT_DATABASE = "dataform-open-source"; +export const DEFAULT_DATABASE = "your-bigquery-project"; export const DEFAULT_LOCATION = "US"; -export const DEFAULT_RESERVATION = "projects/dataform-open-source/locations/us/reservations/dataform-test"; +export const DEFAULT_RESERVATION = "projects/your-bigquery-project/locations/us/reservations/sqlanvil-test"; export const CREDENTIALS_PATH = path.resolve(process.env.RUNFILES, "sa/test_credentials/bigquery.json"); export const cliEntryPointPath = "cli/node_modules/@sqlanvil/cli/bundle.js"; diff --git a/cli/util.ts b/cli/util.ts index 9f4450e4..fa5574fc 100644 --- a/cli/util.ts +++ b/cli/util.ts @@ -38,7 +38,7 @@ export function formatExecutionSuffix(jobIds: string[], bytesBilled: string[]): export function formatBytesInHumanReadableFormat(bytes: number): string { // we do not want to raise an error when bytes < 0 - // because it will fail Dataform run command when in fact the BQ job was executed. + // because it will fail sqlanvil run command when in fact the BQ job was executed. if (bytes <= 0) {return '0 B';} const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB', 'PiB']; diff --git a/cli/util_test.ts b/cli/util_test.ts index adb89369..45eadb90 100644 --- a/cli/util_test.ts +++ b/cli/util_test.ts @@ -12,10 +12,10 @@ import { suite, test } from "sa/testing"; suite('format execution suffix', () => { test('format execution suffix', () => { expect(formatExecutionSuffix([], [])).deep.equals(''); - expect(formatExecutionSuffix(["dataform-915a03fe1"], [])).deep.equals(" \n \t jobId: dataform-915a03fe1"); + expect(formatExecutionSuffix(["sqlanvil-915a03fe1"], [])).deep.equals(" \n \t jobId: sqlanvil-915a03fe1"); expect(formatExecutionSuffix([], ["10 MiB"])).deep.equals(" \n \t Bytes billed: 10 MiB"); - expect(formatExecutionSuffix(["dataform-915a03fe1"], ["17 KiB"])).deep.equals(" \n \t jobId: dataform-915a03fe1,\n \t Bytes billed: 17 KiB"); - expect(formatExecutionSuffix(["dataform-915a03fe1", "dataform-915a03fe2"], ["17 KiB", "1 GiB"])).deep.equals(" \n \t jobId: dataform-915a03fe1, dataform-915a03fe2,\n \t Bytes billed: 17 KiB, 1 GiB"); + expect(formatExecutionSuffix(["sqlanvil-915a03fe1"], ["17 KiB"])).deep.equals(" \n \t jobId: sqlanvil-915a03fe1,\n \t Bytes billed: 17 KiB"); + expect(formatExecutionSuffix(["sqlanvil-915a03fe1", "sqlanvil-915a03fe2"], ["17 KiB", "1 GiB"])).deep.equals(" \n \t jobId: sqlanvil-915a03fe1, sqlanvil-915a03fe2,\n \t Bytes billed: 17 KiB, 1 GiB"); }); }); diff --git a/cli/vm/compile.ts b/cli/vm/compile.ts index 1eb122b2..424a9415 100644 --- a/cli/vm/compile.ts +++ b/cli/vm/compile.ts @@ -11,14 +11,14 @@ export function compile(compileConfig: sqlanvil.ICompileConfig) { compileConfig.projectDir = fs.realpathSync(path.resolve(compileConfig.projectDir)); if ( !fs.existsSync( - path.join(compileConfig.projectDir, "node_modules", "@dataform", "core", "bundle.js") + path.join(compileConfig.projectDir, "node_modules", "@sqlanvil", "core", "bundle.js") ) ) { throw new Error( "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + - "`dataform install`." + "`sqlanvil install`." ); } const vmIndexFileName = path.resolve(path.join(compileConfig.projectDir, "index.js")); diff --git a/cli/vm/jit_worker.ts b/cli/vm/jit_worker.ts index 0dea5cd3..91ca9fac 100644 --- a/cli/vm/jit_worker.ts +++ b/cli/vm/jit_worker.ts @@ -28,12 +28,12 @@ export async function handleJitRequest(message: { try { const { request, projectDir } = message; - if (!fs.existsSync(path.join(projectDir, "node_modules", "@dataform", "core", "bundle.js"))) { + if (!fs.existsSync(path.join(projectDir, "node_modules", "@sqlanvil", "core", "bundle.js"))) { throw new Error( "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + - "`dataform install`." + "`sqlanvil install`." ); } diff --git a/common/protos/index.ts b/common/protos/index.ts index 807536e3..84ffa8b5 100644 --- a/common/protos/index.ts +++ b/common/protos/index.ts @@ -3,8 +3,8 @@ import { util } from "protobufjs"; import { google } from "sa/protos/ts"; const CONFIGS_PROTO_DOCUMENTATION_URL = - "https://dataform-co.github.io/dataform/docs/configs-reference"; -const REPORT_ISSUE_URL = "https://github.com/dataform-co/dataform/issues"; + "https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md"; +const REPORT_ISSUE_URL = "https://github.com/ihistand/sqlanvil/issues"; export interface IProtoClass { new (): Proto; @@ -92,7 +92,7 @@ export function verifyObjectMatchesProto( throw ReferenceError( `Unexpected property "${presentKey}" for "${protoType .getTypeUrl("") - .replace("/", "")}", please report this to the Dataform team at ` + + .replace("/", "")}", please report this to the sqlanvil team at ` + `${REPORT_ISSUE_URL}.` ); } diff --git a/common/protos/index_test.ts b/common/protos/index_test.ts index a2c5f4b9..6db2fd1e 100644 --- a/common/protos/index_test.ts +++ b/common/protos/index_test.ts @@ -30,7 +30,7 @@ suite("verifyObjectMatchesProto", () => { ); }).to.throw( ReferenceError, - /Unexpected property "actionDescriptor" for ".*Table".*please report this to the Dataform team/ + /Unexpected property "actionDescriptor" for ".*Table".*please report this to the sqlanvil team/ ); }); diff --git a/core/actions/declaration.ts b/core/actions/declaration.ts index cd697c32..bfc6fdb1 100644 --- a/core/actions/declaration.ts +++ b/core/actions/declaration.ts @@ -19,13 +19,13 @@ interface ILegacyDeclarationConfig extends sqlanvil.ActionConfig.DeclarationConf } /** - * You can declare any BigQuery table as a data source in Dataform. Declaring BigQuery data - * sources that are external to Dataform lets you treat those data sources as Dataform objects. + * You can declare any BigQuery table as a data source in sqlanvil. Declaring BigQuery data + * sources that are external to sqlanvil lets you treat those data sources as sqlanvil objects. * * Declaring data sources is optional, but can be useful when you want to do the following: - * * Reference or resolve declared sources in the same way as any other table in Dataform. - * * View declared sources in the visualized Dataform graph. - * * Use Dataform to manage the table-level and column-level descriptions of externally created + * * Reference or resolve declared sources in the same way as any other table in sqlanvil. + * * View declared sources in the visualized sqlanvil graph. + * * Use sqlanvil to manage the table-level and column-level descriptions of externally created * tables. * * Trigger workflow invocations that include all the dependents of an external data source. * diff --git a/core/actions/declaration_test.ts b/core/actions/declaration_test.ts index 9fad05b1..21a9f686 100644 --- a/core/actions/declaration_test.ts +++ b/core/actions/declaration_test.ts @@ -97,7 +97,7 @@ actions: ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Unexpected property "fileName", or property value type of "string" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfigs for allowed properties.` + `Unexpected property "fileName", or property value type of "string" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfigs for allowed properties.` ); }); diff --git a/core/actions/incremental_table.ts b/core/actions/incremental_table.ts index 81f58b0e..a5a56975 100644 --- a/core/actions/incremental_table.ts +++ b/core/actions/incremental_table.ts @@ -39,8 +39,8 @@ import { import { sqlanvil } from "sa/protos/ts"; /** - * When you define an incremental table, Dataform builds the incremental table from scratch only for - * the first time. During subsequent executions, Dataform only inserts or merges new rows into the + * When you define an incremental table, sqlanvil builds the incremental table from scratch only for + * the first time. During subsequent executions, sqlanvil only inserts or merges new rows into the * incremental table according to the conditions that you configure. * * You can create incremental tables in the following ways. Available config options are defined in @@ -258,7 +258,7 @@ export class IncrementalTable extends ActionBuilder { const existingAction = this.session.actions.indexOf(this); if (existingAction === -1) { throw Error( - "Expected pre-existing action, but none found. Please report this to the Dataform team." + "Expected pre-existing action, but none found. Please report this to the sqlanvil team." ); } this.session.actions[existingAction] = newAction; @@ -356,7 +356,7 @@ export class IncrementalTable extends ActionBuilder { * [IncrementalTableConfig.uniqueKey](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If set, unique key represents a set of names of columns that will act as a the unique key. To - * enforce this, when updating the incremental table, Dataform merges rows with `uniqueKey` + * enforce this, when updating the incremental table, sqlanvil merges rows with `uniqueKey` * instead of appending them. */ public uniqueKey(uniqueKey: string[]) { diff --git a/core/actions/operation.ts b/core/actions/operation.ts index 8123fa7c..523bb358 100644 --- a/core/actions/operation.ts +++ b/core/actions/operation.ts @@ -33,7 +33,7 @@ interface ILegacyOperationConfig extends sqlanvil.ActionConfig.OperationConfig { export type JitOperationResult = string | string[] | sqlanvil.IJitOperationResult; /** - * Operations define custom SQL operations that don't fit into the Dataform model of publishing a + * Operations define custom SQL operations that don't fit into the sqlanvil model of publishing a * table or writing an assertion. * * You can create operations in the following ways. Available config options are defined in diff --git a/core/actions/table.ts b/core/actions/table.ts index b560a269..ad1f5182 100644 --- a/core/actions/table.ts +++ b/core/actions/table.ts @@ -46,8 +46,8 @@ import { sqlanvil } from "sa/protos/ts"; export type JitTableResult = string | sqlanvil.IJitTableResult; /** - * Tables are the fundamental building block for storing data when using Dataform. Dataform compiles - * your Dataform core code into SQL, executes the SQL code, and creates your defined tables in + * Tables are the fundamental building block for storing data when using sqlanvil. sqlanvil compiles + * your sqlanvil core code into SQL, executes the SQL code, and creates your defined tables in * BigQuery. * * You can create tables in the following ways. Available config options are defined in @@ -254,7 +254,7 @@ export class Table extends ActionBuilder { const existingAction = this.session.actions.indexOf(this); if (existingAction === -1) { throw Error( - "Expected pre-existing action, but none found. Please report this to the Dataform team." + "Expected pre-existing action, but none found. Please report this to the sqlanvil team." ); } this.session.actions[existingAction] = newAction; diff --git a/core/actions/test.ts b/core/actions/test.ts index 1e8642cb..854cb172 100644 --- a/core/actions/test.ts +++ b/core/actions/test.ts @@ -41,7 +41,7 @@ export interface ITestConfig extends INamedConfig { const ITestConfigProperties = strictKeysOf()(["type", "dataset", "name", "filename", "tags"]); /** - * Dataform test actions can be used to write unit tests for your generated SQL + * sqlanvil test actions can be used to write unit tests for your generated SQL * * You can create unit tests in the following ways. * diff --git a/core/actions/view.ts b/core/actions/view.ts index de85ac30..fd71a4d9 100644 --- a/core/actions/view.ts +++ b/core/actions/view.ts @@ -253,7 +253,7 @@ export class View extends ActionBuilder { const existingAction = this.session.actions.indexOf(this); if (existingAction === -1) { throw Error( - "Expected pre-existing action, but none found. Please report this to the Dataform team." + "Expected pre-existing action, but none found. Please report this to the sqlanvil team." ); } this.session.actions[existingAction] = newAction; diff --git a/core/main.ts b/core/main.ts index a46b3b3b..46483b11 100644 --- a/core/main.ts +++ b/core/main.ts @@ -30,7 +30,7 @@ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | st let request: sqlanvil.CoreExecutionRequest; if (typeof coreExecutionRequest === "string") { - // Older versions of the Dataform CLI send a base64 encoded string. + // Older versions of the sqlanvil CLI send a base64 encoded string. // See https://github.com/dataform-co/dataform/pull/1570. request = decode64(sqlanvil.CoreExecutionRequest, coreExecutionRequest); } else { @@ -67,7 +67,7 @@ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | st }); if (typeof coreExecutionRequest === "string") { - // Older versions of the Dataform CLI expect a base64 encoded string to be returned. + // Older versions of the sqlanvil CLI expect a base64 encoded string to be returned. // See https://github.com/dataform-co/dataform/pull/1570. return encode64(sqlanvil.CoreExecutionResponse, coreExecutionResponse); } diff --git a/core/main_test.ts b/core/main_test.ts index 003a8bbf..79c76b9e 100644 --- a/core/main_test.ts +++ b/core/main_test.ts @@ -580,7 +580,7 @@ someKey: and an extra: colon ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Workflow settings error: Unexpected property "notAProjectConfigField", or property value type of "string" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-WorkflowSettings for allowed properties.` + `Workflow settings error: Unexpected property "notAProjectConfigField", or property value type of "string" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-WorkflowSettings for allowed properties.` ); }); @@ -908,7 +908,7 @@ actions: ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfigs for allowed properties.` + `Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfigs for allowed properties.` ); }); @@ -926,7 +926,7 @@ actions:` ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Unexpected empty value for "actions". See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfigs for allowed properties.` + `Unexpected empty value for "actions". See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfigs for allowed properties.` ); }); @@ -1728,7 +1728,7 @@ publish("name", { materialized: true, })`, expectedError: - 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-TableConfig for allowed properties.' + 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-TableConfig for allowed properties.' }, { testName: "partitionExpirationDays invalid for BigQuery tables", diff --git a/core/session.ts b/core/session.ts index 1c785208..f00ea4d8 100644 --- a/core/session.ts +++ b/core/session.ts @@ -29,13 +29,13 @@ const DEFAULT_CONFIG = { /** * Contains methods that are published globally, so can be invoked anywhere in the `/definitions` - * folder of a Dataform project. + * folder of a sqlanvil project. */ export class Session { public rootDir: string; /** - * Stores the Dataform project configuration of the current Dataform project. Can be accessed via + * Stores the sqlanvil project configuration of the current sqlanvil project. Can be accessed via * the `sqlanvil` global variable. * * Example: @@ -334,7 +334,7 @@ export class Session { } /** - * Adds a Dataform assertion the compiled graph. + * Adds a sqlanvil assertion the compiled graph. * * Available only in the `/definitions` directory. * @@ -344,7 +344,7 @@ export class Session { name: string, queryOrConfig?: AContextable | sqlanvil.ActionConfig.AssertionConfig // // `any` is used here to facilitate the type merging of legacy declaration configs options, - // // without breaking typescript consumers of Dataform. + // // without breaking typescript consumers of sqlanvil. // | any ): Assertion { const filename = utils.getCallerFile(this.rootDir); @@ -362,7 +362,7 @@ export class Session { } /** - * Declares the dataset as a Dataform data source. + * Declares the dataset as a sqlanvil data source. * * Available only in the `/definitions` directory. * @@ -372,7 +372,7 @@ export class Session { config: | sqlanvil.ActionConfig.DeclarationConfig // `any` is used here to facilitate the type merging of legacy declaration configs options, - // without breaking typescript consumers of Dataform. + // without breaking typescript consumers of sqlanvil. | any ): Declaration { const declaration = new Declaration(this, config, utils.getCallerFile(this.rootDir)); diff --git a/core/utils_test.ts b/core/utils_test.ts index b57fbce8..5473a546 100644 --- a/core/utils_test.ts +++ b/core/utils_test.ts @@ -55,7 +55,7 @@ function assertThrowsWithMessage(testFn: () => void, expectedMessage: string) { } } -suite('Dataform Utility Validations', () => { +suite('sqlanvil Utility Validations', () => { const CONNECTION_ERROR_MSG = 'The connection must be in the format `{project}.{location}.{connection_id}` or `projects/{project}/locations/{location}/connections/{connection_id}`, or be set to `DEFAULT`.'; const STORAGE_ERROR_MSG = 'The storage URI must be in the format `gs://{bucket_name}/{path_to_data}`.'; diff --git a/core/workflow_settings.ts b/core/workflow_settings.ts index b5b827fc..301e4487 100644 --- a/core/workflow_settings.ts +++ b/core/workflow_settings.ts @@ -46,7 +46,7 @@ function verifyWorkflowSettingsAsJson(workflowSettingsAsJson: object): sqlanvil. throw e; } - // The caller of Dataform Core should ensure that the correct version is installed. + // The caller of sqlanvil Core should ensure that the correct version is installed. if (!!workflowSettings.sqlanvilCoreVersion && workflowSettings.sqlanvilCoreVersion !== version) { throw Error( `Version mismatch: workflow settings specifies version ${workflowSettings.sqlanvilCoreVersion}` + diff --git a/docs/reference/assertion.md b/docs/reference/assertion.md index 74acd5fa..85d22567 100644 --- a/docs/reference/assertion.md +++ b/docs/reference/assertion.md @@ -1,4 +1,4 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/assertion"](../modules/_core_actions_assertion_.md) › [Assertion](_core_actions_assertion_.assertion.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/assertion"](../modules/_core_actions_assertion_.md) › [Assertion](_core_actions_assertion_.assertion.md) # Class: Assertion @@ -6,7 +6,7 @@ An assertion is a data quality test query that finds rows that violate one or mo specified in the query. If the query returns any rows, the assertion fails. You can create assertions in the following ways. Available config options are defined in -[AssertionConfig](configs#dataform-ActionConfig-AssertionConfig), and are shared across all the +[AssertionConfig](configs#sqlanvil-ActionConfig-AssertionConfig), and are shared across all the following ways of creating assertions. **Using a SQLX file:** @@ -21,7 +21,7 @@ SELECT * FROM table WHERE a IS NULL **Using built-in assertions in the config block of a table:** -See [TableConfig.assertions](configs#dataform-ActionConfig-TableConfig) +See [TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig) **Using action configs files:** @@ -73,7 +73,7 @@ This is where `query` comes from. ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.project](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.project](configs#sqlanvil-ActionConfig-AssertionConfig). Sets the database (Google Cloud project ID) in which to create the corresponding view for this assertion. @@ -93,7 +93,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.dependencies](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.dependencies](configs#sqlanvil-ActionConfig-AssertionConfig). Sets dependencies of the assertion. @@ -112,7 +112,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.description](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.description](configs#sqlanvil-ActionConfig-AssertionConfig). Sets the description of this assertion. @@ -131,7 +131,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.disabled](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.disabled](configs#sqlanvil-ActionConfig-AssertionConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -151,7 +151,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[AssertionConfig.hermetic](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.hermetic](configs#sqlanvil-ActionConfig-AssertionConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -188,7 +188,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.dataset](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.dataset](configs#sqlanvil-ActionConfig-AssertionConfig). Sets the schema (BigQuery dataset) in which to create the corresponding view for this assertion. @@ -208,7 +208,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.tags](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.tags](configs#sqlanvil-ActionConfig-AssertionConfig). Sets a list of user-defined tags applied to this action. diff --git a/docs/reference/configs.md b/docs/reference/configs.md index de618ae7..473b26ae 100644 --- a/docs/reference/configs.md +++ b/docs/reference/configs.md @@ -6,30 +6,30 @@ - [configs.proto](#configs-proto) - - [ActionConfig](#dataform-ActionConfig) - - [ActionConfig.AssertionConfig](#dataform-ActionConfig-AssertionConfig) - - [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) - - [ActionConfig.DataPreparationConfig](#dataform-ActionConfig-DataPreparationConfig) - - [ActionConfig.DeclarationConfig](#dataform-ActionConfig-DeclarationConfig) - - [ActionConfig.IncrementalTableConfig](#dataform-ActionConfig-IncrementalTableConfig) - - [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) - - [ActionConfig.IncrementalTableConfig.LabelsEntry](#dataform-ActionConfig-IncrementalTableConfig-LabelsEntry) - - [ActionConfig.NotebookConfig](#dataform-ActionConfig-NotebookConfig) - - [ActionConfig.OnSchemaChange](#dataform-ActionConfig-OnSchemaChange) - - [ActionConfig.OperationConfig](#dataform-ActionConfig-OperationConfig) - - [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) - - [ActionConfig.TableAssertionsConfig.UniqueKey](#dataform-ActionConfig-TableAssertionsConfig-UniqueKey) - - [ActionConfig.TableConfig](#dataform-ActionConfig-TableConfig) - - [ActionConfig.TableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-TableConfig-AdditionalOptionsEntry) - - [ActionConfig.TableConfig.LabelsEntry](#dataform-ActionConfig-TableConfig-LabelsEntry) - - [ActionConfig.Target](#dataform-ActionConfig-Target) - - [ActionConfig.ViewConfig](#dataform-ActionConfig-ViewConfig) - - [ActionConfig.ViewConfig.AdditionalOptionsEntry](#dataform-ActionConfig-ViewConfig-AdditionalOptionsEntry) - - [ActionConfig.ViewConfig.LabelsEntry](#dataform-ActionConfig-ViewConfig-LabelsEntry) - - [ActionConfigs](#dataform-ActionConfigs) - - [NotebookRuntimeOptionsConfig](#dataform-NotebookRuntimeOptionsConfig) - - [WorkflowSettings](#dataform-WorkflowSettings) - - [WorkflowSettings.VarsEntry](#dataform-WorkflowSettings-VarsEntry) + - [ActionConfig](#sqlanvil-ActionConfig) + - [ActionConfig.AssertionConfig](#sqlanvil-ActionConfig-AssertionConfig) + - [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) + - [ActionConfig.DataPreparationConfig](#sqlanvil-ActionConfig-DataPreparationConfig) + - [ActionConfig.DeclarationConfig](#sqlanvil-ActionConfig-DeclarationConfig) + - [ActionConfig.IncrementalTableConfig](#sqlanvil-ActionConfig-IncrementalTableConfig) + - [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) + - [ActionConfig.IncrementalTableConfig.LabelsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-LabelsEntry) + - [ActionConfig.NotebookConfig](#sqlanvil-ActionConfig-NotebookConfig) + - [ActionConfig.OnSchemaChange](#sqlanvil-ActionConfig-OnSchemaChange) + - [ActionConfig.OperationConfig](#sqlanvil-ActionConfig-OperationConfig) + - [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) + - [ActionConfig.TableAssertionsConfig.UniqueKey](#sqlanvil-ActionConfig-TableAssertionsConfig-UniqueKey) + - [ActionConfig.TableConfig](#sqlanvil-ActionConfig-TableConfig) + - [ActionConfig.TableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-TableConfig-AdditionalOptionsEntry) + - [ActionConfig.TableConfig.LabelsEntry](#sqlanvil-ActionConfig-TableConfig-LabelsEntry) + - [ActionConfig.Target](#sqlanvil-ActionConfig-Target) + - [ActionConfig.ViewConfig](#sqlanvil-ActionConfig-ViewConfig) + - [ActionConfig.ViewConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-ViewConfig-AdditionalOptionsEntry) + - [ActionConfig.ViewConfig.LabelsEntry](#sqlanvil-ActionConfig-ViewConfig-LabelsEntry) + - [ActionConfigs](#sqlanvil-ActionConfigs) + - [NotebookRuntimeOptionsConfig](#sqlanvil-NotebookRuntimeOptionsConfig) + - [WorkflowSettings](#sqlanvil-WorkflowSettings) + - [WorkflowSettings.VarsEntry](#sqlanvil-WorkflowSettings-VarsEntry) - [Scalar Value Types](#scalar-value-types) @@ -39,7 +39,7 @@ ## configs.proto - + ### ActionConfig @@ -47,16 +47,16 @@ Action config defines the contents of `actions.yaml` configuration files. | Field | Type | Label | Description | | ---------------- | ------------------------------------------------------------------------------------ | ----- | ----------- | -| table | [ActionConfig.TableConfig](#dataform-ActionConfig-TableConfig) | | | -| view | [ActionConfig.ViewConfig](#dataform-ActionConfig-ViewConfig) | | | -| incrementalTable | [ActionConfig.IncrementalTableConfig](#dataform-ActionConfig-IncrementalTableConfig) | | | -| assertion | [ActionConfig.AssertionConfig](#dataform-ActionConfig-AssertionConfig) | | | -| operation | [ActionConfig.OperationConfig](#dataform-ActionConfig-OperationConfig) | | | -| declaration | [ActionConfig.DeclarationConfig](#dataform-ActionConfig-DeclarationConfig) | | | -| notebook | [ActionConfig.NotebookConfig](#dataform-ActionConfig-NotebookConfig) | | | -| dataPreparation | [ActionConfig.DataPreparationConfig](#dataform-ActionConfig-DataPreparationConfig) | | | +| table | [ActionConfig.TableConfig](#sqlanvil-ActionConfig-TableConfig) | | | +| view | [ActionConfig.ViewConfig](#sqlanvil-ActionConfig-ViewConfig) | | | +| incrementalTable | [ActionConfig.IncrementalTableConfig](#sqlanvil-ActionConfig-IncrementalTableConfig) | | | +| assertion | [ActionConfig.AssertionConfig](#sqlanvil-ActionConfig-AssertionConfig) | | | +| operation | [ActionConfig.OperationConfig](#sqlanvil-ActionConfig-OperationConfig) | | | +| declaration | [ActionConfig.DeclarationConfig](#sqlanvil-ActionConfig-DeclarationConfig) | | | +| notebook | [ActionConfig.NotebookConfig](#sqlanvil-ActionConfig-NotebookConfig) | | | +| dataPreparation | [ActionConfig.DataPreparationConfig](#sqlanvil-ActionConfig-DataPreparationConfig) | | | - + ### ActionConfig.AssertionConfig @@ -65,7 +65,7 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the assertion. | | dataset | [string](#string) | | The dataset (schema) of the assertion. | | project | [string](#string) | | The Google Cloud project (database) of the assertion. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | @@ -74,7 +74,7 @@ Action config defines the contents of `actions.yaml` configuration files. | dependOnDependencyAssertions | [bool](#bool) | | If true, assertions dependent upon any of the dependencies are added as dependencies as well. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.ColumnDescriptor @@ -85,21 +85,21 @@ Action config defines the contents of `actions.yaml` configuration files. | bigqueryPolicyTags | [string](#string) | repeated | A list of BigQuery policy tags that will be applied to the column. | | tags | [string](#string) | repeated | A list of tags for this column which will be applied. | - + ### ActionConfig.DataPreparationConfig | Field | Type | Label | Description | | ------------------- | ---------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | name | [string](#string) | | The name of the data preparation. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | description | [string](#string) | | Description of the data preparation. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.DeclarationConfig @@ -109,9 +109,9 @@ Action config defines the contents of `actions.yaml` configuration files. | dataset | [string](#string) | | The dataset (schema) of the declaration. | | project | [string](#string) | | The Google Cloud project (database) of the declaration. | | description | [string](#string) | | Description of the declaration. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the declaration. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the declaration. | - + ### ActionConfig.IncrementalTableConfig @@ -120,30 +120,30 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the incremental table. | | dataset | [string](#string) | | The dataset (schema) of the incremental table. | | project | [string](#string) | | The Google Cloud project (database) of the incremental table. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | preOperations | [string](#string) | repeated | Queries to run before `query`. This can be useful for granting permissions. | | postOperations | [string](#string) | repeated | Queries to run after `query`. | | protected | [bool](#bool) | | If true, prevents the dataset from being rebuilt from scratch. | -| uniqueKey | [string](#string) | repeated | If set, unique key represents a set of names of columns that will act as a the unique key. To enforce this, when updating the incremental table, Dataform merges rows with `uniqueKey` instead of appending them. | +| uniqueKey | [string](#string) | repeated | If set, unique key represents a set of names of columns that will act as a the unique key. To enforce this, when updating the incremental table, sqlanvil merges rows with `uniqueKey` instead of appending them. | | description | [string](#string) | | Description of the incremental table. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | | partitionBy | [string](#string) | | The key by which to partition the table. Typically the name of a timestamp or the date column. See https://cloud.google.com/dataform/docs/partitions-clusters. | | partitionExpirationDays | [int32](#int32) | | The number of days for which BigQuery stores data in each partition. The setting applies to all partitions in a table, but is calculated independently for each partition based on the partition time. | | requirePartitionFilter | [bool](#bool) | | Declares whether the partitioned table requires a WHERE clause predicate filter that filters the partitioning column. | | updatePartitionFilter | [string](#string) | | SQL-based filter for when incremental updates are applied. | | clusterBy | [string](#string) | repeated | The keys by which to cluster partitions by. See https://cloud.google.com/dataform/docs/partitions-clusters. | -| labels | [ActionConfig.IncrementalTableConfig.LabelsEntry](#dataform-ActionConfig-IncrementalTableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | -| additionalOptions | [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | +| labels | [ActionConfig.IncrementalTableConfig.LabelsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | +| additionalOptions | [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | -| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | +| assertions | [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | -| onSchemaChange | [ActionConfg.OnSchemaChange](#dataform-ActionConfig-OnSchemaChange) | | Defines the action behavior if the selected columns in query doesn't match columns in the target table. | +| onSchemaChange | [ActionConfg.OnSchemaChange](#sqlanvil-ActionConfig-OnSchemaChange) | | Defines the action behavior if the selected columns in query doesn't match columns in the target table. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry @@ -152,7 +152,7 @@ Action config defines the contents of `actions.yaml` configuration files. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.IncrementalTableConfig.LabelsEntry @@ -161,7 +161,7 @@ Action config defines the contents of `actions.yaml` configuration files. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.NotebookConfig @@ -170,7 +170,7 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the notebook. | | location | [string](#string) | | The Google Cloud location of the notebook. | | project | [string](#string) | | The Google Cloud project (database) of the notebook. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | @@ -178,7 +178,7 @@ Action config defines the contents of `actions.yaml` configuration files. | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.OnSchemaChange @@ -189,7 +189,7 @@ Action config defines the contents of `actions.yaml` configuration files. | EXTEND | New columns will be added to the target table. Fails if columns are deleted or renamed. | | SYNCHRONIZE | Does not block any new column(s) from being added, deleted or renamed. Partitioned or clustered columns cannot be deleted or renamed. | - + ### ActionConfig.OperationConfig @@ -198,18 +198,18 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the operation. | | dataset | [string](#string) | | The dataset (schema) of the operation. | | project | [string](#string) | | The Google Cloud project (database) of the operation. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | hasOutput | [bool](#bool) | | Declares that this action creates a dataset which should be referenceable as a dependency target, for example by using the `ref` function. | | description | [string](#string) | | Description of the operation. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the operation. Can only be set if hasOutput is true. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the operation. Can only be set if hasOutput is true. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.TableAssertionsConfig @@ -219,11 +219,11 @@ action types. | Field | Type | Label | Description | | ------------- | ------------------------------------------------------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | uniqueKey | [string](#string) | repeated | Column(s) which constitute the dataset's unique key index. If set, the resulting assertion will fail if there is more than one row in the dataset with the same values for all of these column(s). | -| uniqueKeys | [ActionConfig.TableAssertionsConfig.UniqueKey](#dataform-ActionConfig-TableAssertionsConfig-UniqueKey) | repeated | | +| uniqueKeys | [ActionConfig.TableAssertionsConfig.UniqueKey](#sqlanvil-ActionConfig-TableAssertionsConfig-UniqueKey) | repeated | | | nonNull | [string](#string) | repeated | Column(s) which may never be `NULL`. If set, the resulting assertion will fail if any row contains `NULL` values for these column(s). | | rowConditions | [string](#string) | repeated | General condition(s) which should hold true for all rows in the dataset. If set, the resulting assertion will fail if any row violates any of these condition(s). | - + ### ActionConfig.TableAssertionsConfig.UniqueKey @@ -236,7 +236,7 @@ the column(s) in the unique key(s). | --------- | ----------------- | -------- | ----------- | | uniqueKey | [string](#string) | repeated | | - + ### ActionConfig.TableConfig @@ -245,26 +245,26 @@ the column(s) in the unique key(s). | name | [string](#string) | | The name of the table. | | dataset | [string](#string) | | The dataset (schema) of the table. | | project | [string](#string) | | The Google Cloud project (database) of the table. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | preOperations | [string](#string) | repeated | Queries to run before `query`. This can be useful for granting permissions. | | postOperations | [string](#string) | repeated | Queries to run after `query`. | | description | [string](#string) | | Description of the table. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | | partitionBy | [string](#string) | | The key by which to partition the table. Typically the name of a timestamp or the date column. See https://cloud.google.com/dataform/docs/partitions-clusters. | | partitionExpirationDays | [int32](#int32) | | The number of days for which BigQuery stores data in each partition. The setting applies to all partitions in a table, but is calculated independently for each partition based on the partition time. | | requirePartitionFilter | [bool](#bool) | | Declares whether the partitioned table requires a WHERE clause predicate filter that filters the partitioning column. | | clusterBy | [string](#string) | repeated | The keys by which to cluster partitions by. See https://cloud.google.com/dataform/docs/partitions-clusters. | -| labels | [ActionConfig.TableConfig.LabelsEntry](#dataform-ActionConfig-TableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | -| additionalOptions | [ActionConfig.TableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-TableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | +| labels | [ActionConfig.TableConfig.LabelsEntry](#sqlanvil-ActionConfig-TableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | +| additionalOptions | [ActionConfig.TableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-TableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | -| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | +| assertions | [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.TableConfig.AdditionalOptionsEntry @@ -273,7 +273,7 @@ the column(s) in the unique key(s). | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.TableConfig.LabelsEntry @@ -282,7 +282,7 @@ the column(s) in the unique key(s). | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.Target @@ -295,7 +295,7 @@ Target represents a unique action identifier. | name | [string](#string) | | The name of the action. | | includeDependentAssertions | [bool](#bool) | | flag for when we want to add assertions of this dependency in dependency_targets as well. | - + ### ActionConfig.ViewConfig @@ -304,7 +304,7 @@ Target represents a unique action identifier. | name | [string](#string) | | The name of the view. | | dataset | [string](#string) | | The dataset (schema) of the view. | | project | [string](#string) | | The Google Cloud project (database) of the view. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | @@ -314,15 +314,15 @@ Target represents a unique action identifier. | partitionBy | [string](#string) | | Optional. Applicable only to materialized view. The key by which to partition the materialized view. Typically the name of a timestamp or the date column. See https://cloud.google.com/bigquery/docs/materialized-views-create#partitioned_materialized_views. | | clusterBy | [string](#string) | repeated | Optional. Applicable only to materialized view. The keys by which to cluster partitions by. See https://cloud.google.com/bigquery/docs/materialized-views-create#cluster_materialized_views. | | description | [string](#string) | | Description of the view. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | -| labels | [ActionConfig.ViewConfig.LabelsEntry](#dataform-ActionConfig-ViewConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | -| additionalOptions | [ActionConfig.ViewConfig.AdditionalOptionsEntry](#dataform-ActionConfig-ViewConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | +| labels | [ActionConfig.ViewConfig.LabelsEntry](#sqlanvil-ActionConfig-ViewConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | +| additionalOptions | [ActionConfig.ViewConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-ViewConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | -| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | +| assertions | [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.ViewConfig.AdditionalOptionsEntry @@ -331,7 +331,7 @@ Target represents a unique action identifier. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.ViewConfig.LabelsEntry @@ -340,7 +340,7 @@ Target represents a unique action identifier. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfigs @@ -348,9 +348,9 @@ Action configs defines the contents of `actions.yaml` configuration files. | Field | Type | Label | Description | | ------- | -------------------------------------- | -------- | ----------- | -| actions | [ActionConfig](#dataform-ActionConfig) | repeated | | +| actions | [ActionConfig](#sqlanvil-ActionConfig) | repeated | | - + ### NotebookRuntimeOptionsConfig @@ -358,7 +358,7 @@ Action configs defines the contents of `actions.yaml` configuration files. | ------------ | ----------------- | ----- | ------------------------------------------------------------ | | outputBucket | [string](#string) | | Storage bucket to output notebooks to after their execution. | - + ### WorkflowSettings @@ -367,20 +367,20 @@ configuration file. | Field | Type | Label | Description | | ----------------------------- | ---------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| dataformCoreVersion | [string](#string) | | The desired dataform core version to compile against. | +| sqlanvilCoreVersion | [string](#string) | | The desired sqlanvil core version to compile against. | | defaultProject | [string](#string) | | Required. The default Google Cloud project (database). | | defaultDataset | [string](#string) | | Required. The default dataset (schema). | | defaultLocation | [string](#string) | | Required. The default BigQuery location to use. For more information on BigQuery locations, see https://cloud.google.com/bigquery/docs/locations. | | defaultAssertionDataset | [string](#string) | | Required. The default dataset (schema) for assertions. | -| vars | [WorkflowSettings.VarsEntry](#dataform-WorkflowSettings-VarsEntry) | repeated | Optional. User-defined variables that are made available to project code during compilation. An object containing a list of "key": value pairs. | +| vars | [WorkflowSettings.VarsEntry](#sqlanvil-WorkflowSettings-VarsEntry) | repeated | Optional. User-defined variables that are made available to project code during compilation. An object containing a list of "key": value pairs. | | projectSuffix | [string](#string) | | Optional. The suffix to append to all Google Cloud project references. | | datasetSuffix | [string](#string) | | Optional. The suffix to append to all dataset references. | | namePrefix | [string](#string) | | Optional. The prefix to append to all action names. | -| defaultNotebookRuntimeOptions | [NotebookRuntimeOptionsConfig](#dataform-NotebookRuntimeOptionsConfig) | | Optional. Default runtime options for Notebook actions. | +| defaultNotebookRuntimeOptions | [NotebookRuntimeOptionsConfig](#sqlanvil-NotebookRuntimeOptionsConfig) | | Optional. Default runtime options for Notebook actions. | | builtinAssertionNamePrefix | [string](#string) | | Optional. The prefix to append to built-in assertion names. | | defaultReservation | [string](#string) | | Optional. The default BigQuery reservation to use for execution. | - + ### WorkflowSettings.VarsEntry diff --git a/docs/reference/declaration.md b/docs/reference/declaration.md index 0e043313..928d1b9c 100644 --- a/docs/reference/declaration.md +++ b/docs/reference/declaration.md @@ -1,19 +1,19 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/declaration"](../modules/_core_actions_declaration_.md) › [Declaration](_core_actions_declaration_.declaration.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/declaration"](../modules/_core_actions_declaration_.md) › [Declaration](_core_actions_declaration_.declaration.md) # Class: Declaration -You can declare any BigQuery table as a data source in Dataform. Declaring BigQuery data -sources that are external to Dataform lets you treat those data sources as Dataform objects. +You can declare any BigQuery table as a data source in sqlanvil. Declaring BigQuery data +sources that are external to sqlanvil lets you treat those data sources as sqlanvil objects. Declaring data sources is optional, but can be useful when you want to do the following: -* Reference or resolve declared sources in the same way as any other table in Dataform. -* View declared sources in the visualized Dataform graph. -* Use Dataform to manage the table-level and column-level descriptions of externally created +* Reference or resolve declared sources in the same way as any other table in sqlanvil. +* View declared sources in the visualized sqlanvil graph. +* Use sqlanvil to manage the table-level and column-level descriptions of externally created tables. * Trigger workflow invocations that include all the dependents of an external data source. You can create declarations in the following ways. Available config options are defined in -[DeclarationConfig](configs#dataform-ActionConfig-DeclarationConfig), and are shared across all +[DeclarationConfig](configs#sqlanvil-ActionConfig-DeclarationConfig), and are shared across all the followiing ways of creating declarations. **Using a SQLX file:** @@ -62,7 +62,7 @@ declare("name") ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[DeclarationConfig.columns](configs#dataform-ActionConfig-DeclarationConfig). +[DeclarationConfig.columns](configs#sqlanvil-ActionConfig-DeclarationConfig). Sets the column descriptors of columns in this table. @@ -81,7 +81,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[DeclarationConfig.description](configs#dataform-ActionConfig-DeclarationConfig). +[DeclarationConfig.description](configs#sqlanvil-ActionConfig-DeclarationConfig). Sets the description of this assertion. diff --git a/docs/reference/incrementaltable.md b/docs/reference/incrementaltable.md index 8f90a3e1..bea8e2c6 100644 --- a/docs/reference/incrementaltable.md +++ b/docs/reference/incrementaltable.md @@ -1,13 +1,13 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/incremental_table"](../modules/_core_actions_incremental_table_.md) › [IncrementalTable](_core_actions_incremental_table_.incrementaltable.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/incremental_table"](../modules/_core_actions_incremental_table_.md) › [IncrementalTable](_core_actions_incremental_table_.incrementaltable.md) # Class: IncrementalTable -When you define an incremental table, Dataform builds the incremental table from scratch only for -the first time. During subsequent executions, Dataform only inserts or merges new rows into the +When you define an incremental table, sqlanvil builds the incremental table from scratch only for +the first time. During subsequent executions, sqlanvil only inserts or merges new rows into the incremental table according to the conditions that you configure. You can create incremental tables in the following ways. Available config options are defined in -[IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig), and are shared across all the +[IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig), and are shared across all the following ways of creating tables. **Using a SQLX file:** @@ -72,7 +72,7 @@ This is where `query` comes from. ▸ **assertions**(`assertions`: TableAssertionsConfig): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.assertions](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.assertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets in-line assertions for this incremental table. @@ -95,7 +95,7 @@ ___ ▸ **bigquery**(`bigquery`: IBigQueryOptions): *this* **`deprecated`** Deprecated in favor of options available directly on -[IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig). For example: +[IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig). For example: `publish("name", { type: "table", partitionBy: "column" }`). Sets bigquery options for the action. @@ -115,7 +115,7 @@ ___ ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.columns](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.columns](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the column descriptors of columns in this incremental table. @@ -134,7 +134,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.project](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.project](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the Sets the database (Google Cloud project ID) in which to create the output of this action. @@ -154,7 +154,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.dependencies](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.dependencies](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets dependencies of the incremental table. @@ -173,7 +173,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.description](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.description](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the description of this incremental table. @@ -192,7 +192,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.disabled](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.disabled](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -212,7 +212,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.hermetic](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.hermetic](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -287,7 +287,7 @@ ___ ▸ **protected**(`isProtected`: boolean): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.protected](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.protected](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If called with `true`, prevents the dataset from being rebuilt from scratch. @@ -322,7 +322,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.dataset](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.dataset](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -341,7 +341,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -361,7 +361,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.tags](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.tags](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets a list of user-defined tags applied to this action. @@ -398,10 +398,10 @@ ___ ▸ **uniqueKey**(`uniqueKey`: string[]): *void* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.uniqueKey](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.uniqueKey](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If set, unique key represents a set of names of columns that will act as a the unique key. To -enforce this, when updating the incremental table, Dataform merges rows with `uniqueKey` +enforce this, when updating the incremental table, sqlanvil merges rows with `uniqueKey` instead of appending them. **Parameters:** diff --git a/docs/reference/notebook.md b/docs/reference/notebook.md index 863ecebc..a0c9fab5 100644 --- a/docs/reference/notebook.md +++ b/docs/reference/notebook.md @@ -1,4 +1,4 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/notebook"](../modules/_core_actions_notebook_.md) › [Notebook](_core_actions_notebook_.notebook.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/notebook"](../modules/_core_actions_notebook_.md) › [Notebook](_core_actions_notebook_.notebook.md) # Class: Notebook @@ -6,7 +6,7 @@ Notebooks run Jupyter Notebook files, and can output content to the storage buck `workflow_settings.yaml` files. You can create notebooks in the following ways. Available config options are defined in -[NotebookConfig](configs#dataform-ActionConfig-NotebookConfig), and are shared across all the +[NotebookConfig](configs#sqlanvil-ActionConfig-NotebookConfig), and are shared across all the following ways of creating notebooks. **Using action configs files:** diff --git a/docs/reference/operation.md b/docs/reference/operation.md index 62c02b13..4c01bc31 100644 --- a/docs/reference/operation.md +++ b/docs/reference/operation.md @@ -1,12 +1,12 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/operation"](../modules/_core_actions_operation_.md) › [Operation](_core_actions_operation_.operation.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/operation"](../modules/_core_actions_operation_.md) › [Operation](_core_actions_operation_.operation.md) # Class: Operation -Operations define custom SQL operations that don't fit into the Dataform model of publishing a +Operations define custom SQL operations that don't fit into the sqlanvil model of publishing a table or writing an assertion. You can create operations in the following ways. Available config options are defined in -[OperationConfig](configs#dataform-ActionConfig-OperationConfig), and are shared across all the +[OperationConfig](configs#sqlanvil-ActionConfig-OperationConfig), and are shared across all the following ways of creating operations. **Using a SQLX file:** @@ -72,7 +72,7 @@ This is where `query` comes from. ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.columns](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.columns](configs#sqlanvil-ActionConfig-OperationConfig). Sets the column descriptors of columns in this table. @@ -91,7 +91,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.project](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.project](configs#sqlanvil-ActionConfig-OperationConfig). Sets the database (Google Cloud project ID) in which to create the corresponding view for this operation. @@ -111,7 +111,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.dependencies](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.dependencies](configs#sqlanvil-ActionConfig-OperationConfig). Sets dependencies of the table. @@ -130,7 +130,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.description](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.description](configs#sqlanvil-ActionConfig-OperationConfig). Sets the description of this assertion. @@ -149,7 +149,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.disabled](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.disabled](configs#sqlanvil-ActionConfig-OperationConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -169,7 +169,7 @@ ___ ▸ **hasOutput**(`hasOutput`: boolean): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.hasOutput](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.hasOutput](configs#sqlanvil-ActionConfig-OperationConfig). Declares that this action creates a dataset which should be referenceable as a dependency target, for example by using the `ref` function. @@ -189,7 +189,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[OperationConfig.hermetic](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.hermetic](configs#sqlanvil-ActionConfig-OperationConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -228,7 +228,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.dataset](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.dataset](configs#sqlanvil-ActionConfig-OperationConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -247,7 +247,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-OperationConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -267,7 +267,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.tags](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.tags](configs#sqlanvil-ActionConfig-OperationConfig). Sets a list of user-defined tags applied to this action. diff --git a/docs/reference/session.md b/docs/reference/session.md index b0da1f98..efc4186e 100644 --- a/docs/reference/session.md +++ b/docs/reference/session.md @@ -1,9 +1,9 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/session"](../modules/_core_session_.md) › [Session](_core_session_.session.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/session"](../modules/_core_session_.md) › [Session](_core_session_.session.md) # Class: Session Contains methods that are published globally, so can be invoked anywhere in the `/definitions` -folder of a Dataform project. +folder of a sqlanvil project. ## Hierarchy @@ -30,13 +30,13 @@ folder of a Dataform project. • **projectConfig**: *ProjectConfig* -Stores the Dataform project configuration of the current Dataform project. Can be accessed via -the `dataform` global variable. +Stores the sqlanvil project configuration of the current sqlanvil project. Can be accessed via +the `sqlanvil` global variable. Example: ```js -dataform.projectConfig.vars.myVariableName === "myVariableValue" +sqlanvil.projectConfig.vars.myVariableName === "myVariableValue" ``` ## Methods @@ -45,7 +45,7 @@ dataform.projectConfig.vars.myVariableName === "myVariableValue" ▸ **assert**(`name`: string, `queryOrConfig?`: AContextable‹string› | AssertionConfig): *[Assertion](_core_actions_assertion_.assertion.md)* -Adds a Dataform assertion the compiled graph. +Adds a sqlanvil assertion the compiled graph. Available only in the `/definitions` directory. @@ -66,7 +66,7 @@ ___ ▸ **declare**(`config`: DeclarationConfig | any): *[Declaration](_core_actions_declaration_.declaration.md)* -Declares the dataset as a Dataform data source. +Declares the dataset as a sqlanvil data source. Available only in the `/definitions` directory. diff --git a/docs/reference/table.md b/docs/reference/table.md index 86f1f153..fab7fd4a 100644 --- a/docs/reference/table.md +++ b/docs/reference/table.md @@ -1,13 +1,13 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/table"](../modules/_core_actions_table_.md) › [Table](_core_actions_table_.table.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/table"](../modules/_core_actions_table_.md) › [Table](_core_actions_table_.table.md) # Class: Table -Tables are the fundamental building block for storing data when using Dataform. Dataform compiles -your Dataform core code into SQL, executes the SQL code, and creates your defined tables in +Tables are the fundamental building block for storing data when using sqlanvil. sqlanvil compiles +your sqlanvil core code into SQL, executes the SQL code, and creates your defined tables in BigQuery. You can create tables in the following ways. Available config options are defined in -[TableConfig](configs#dataform-ActionConfig-TableConfig), and are shared across all the +[TableConfig](configs#sqlanvil-ActionConfig-TableConfig), and are shared across all the following ways of creating tables. **Using a SQLX file:** @@ -77,7 +77,7 @@ This is where `query` comes from. ▸ **assertions**(`assertions`: TableAssertionsConfig): *this* **`deprecated`** Deprecated in favor of -[TableConfig.assertions](configs#dataform-ActionConfig-TableConfig). +[TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig). Sets in-line assertions for this table. @@ -100,7 +100,7 @@ ___ ▸ **bigquery**(`bigquery`: IBigQueryOptions): *this* **`deprecated`** Deprecated in favor of options available directly on -[TableConfig](configs#dataform-ActionConfig-TableConfig). For example: +[TableConfig](configs#sqlanvil-ActionConfig-TableConfig). For example: `publish("name", { type: "table", partitionBy: "column" }`). Sets bigquery options for the action. @@ -120,7 +120,7 @@ ___ ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[TableConfig.columns](configs#dataform-ActionConfig-TableConfig). +[TableConfig.columns](configs#sqlanvil-ActionConfig-TableConfig). Sets the column descriptors of columns in this table. @@ -139,7 +139,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[TableConfig.project](configs#dataform-ActionConfig-TableConfig). +[TableConfig.project](configs#sqlanvil-ActionConfig-TableConfig). Sets the database (Google Cloud project ID) in which to create the output of this action. @@ -158,7 +158,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[TableConfig.dependencies](configs#dataform-ActionConfig-TableConfig). +[TableConfig.dependencies](configs#sqlanvil-ActionConfig-TableConfig). Sets dependencies of the table. @@ -177,7 +177,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[TableConfig.description](configs#dataform-ActionConfig-TableConfig). +[TableConfig.description](configs#sqlanvil-ActionConfig-TableConfig). Sets the description of this assertion. @@ -196,7 +196,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[TableConfig.disabled](configs#dataform-ActionConfig-TableConfig). +[TableConfig.disabled](configs#sqlanvil-ActionConfig-TableConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -216,7 +216,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[TableConfig.hermetic](configs#dataform-ActionConfig-TableConfig). +[TableConfig.hermetic](configs#sqlanvil-ActionConfig-TableConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -307,7 +307,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[TableConfig.dataset](configs#dataform-ActionConfig-TableConfig). +[TableConfig.dataset](configs#sqlanvil-ActionConfig-TableConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -326,7 +326,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[TableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-TableConfig). +[TableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-TableConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -346,7 +346,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[TableConfig.tags](configs#dataform-ActionConfig-TableConfig). +[TableConfig.tags](configs#sqlanvil-ActionConfig-TableConfig). Sets a list of user-defined tags applied to this action. diff --git a/docs/reference/test.md b/docs/reference/test.md index e8b45529..7952dd13 100644 --- a/docs/reference/test.md +++ b/docs/reference/test.md @@ -1,8 +1,8 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/test"](../modules/_core_actions_test_.md) › [Test](_core_actions_test_.test.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/test"](../modules/_core_actions_test_.md) › [Test](_core_actions_test_.test.md) # Class: Test -Dataform test actions can be used to write unit tests for your generated SQL +sqlanvil test actions can be used to write unit tests for your generated SQL You can create unit tests in the following ways. diff --git a/docs/reference/view.md b/docs/reference/view.md index 6d50b78b..f43ea1f6 100644 --- a/docs/reference/view.md +++ b/docs/reference/view.md @@ -1,4 +1,4 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/view"](../modules/_core_actions_view_.md) › [View](_core_actions_view_.view.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/view"](../modules/_core_actions_view_.md) › [View](_core_actions_view_.view.md) # Class: View @@ -7,7 +7,7 @@ to copy the original data to it, which can result in significant cost savings fo processing and storage. You can create views in the following ways. Available config options are defined in -[ViewConfig](configs#dataform-ActionConfig-ViewConfig), and are shared across all the +[ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig), and are shared across all the following ways of creating tables. **Using a SQLX file:** @@ -78,7 +78,7 @@ This is where `query` comes from. ▸ **assertions**(`assertions`: TableAssertionsConfig): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.assertions](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.assertions](configs#sqlanvil-ActionConfig-ViewConfig). Sets in-line assertions for this view. @@ -101,7 +101,7 @@ ___ ▸ **bigquery**(`bigquery`: IBigQueryOptions): *this* **`deprecated`** Deprecated in favor of options available directly on -[ViewConfig](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig). Sets bigquery options for the action. @@ -120,7 +120,7 @@ ___ ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.columns](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.columns](configs#sqlanvil-ActionConfig-ViewConfig). Sets the column descriptors of columns in this view. @@ -139,7 +139,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.project](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.project](configs#sqlanvil-ActionConfig-ViewConfig). Sets the Sets the database (Google Cloud project ID) in which to create the output of this action. @@ -159,7 +159,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.dependencies](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.dependencies](configs#sqlanvil-ActionConfig-ViewConfig). Sets dependencies of the view. @@ -178,7 +178,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.description](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.description](configs#sqlanvil-ActionConfig-ViewConfig). Sets the description of this view. @@ -197,7 +197,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.disabled](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.disabled](configs#sqlanvil-ActionConfig-ViewConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -217,7 +217,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[ViewConfig.hermetic](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.hermetic](configs#sqlanvil-ActionConfig-ViewConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -238,7 +238,7 @@ ___ ▸ **materialized**(`materialized`: boolean): *void* **`deprecated`** Deprecated in favor of -[ViewConfig.materialized](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.materialized](configs#sqlanvil-ActionConfig-ViewConfig). Applies the materialized view optimization, see https://cloud.google.com/bigquery/docs/materialized-views-intro. @@ -328,7 +328,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.dataset](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.dataset](configs#sqlanvil-ActionConfig-ViewConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -347,7 +347,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-ViewConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -367,7 +367,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.tags](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.tags](configs#sqlanvil-ActionConfig-ViewConfig). Sets a list of user-defined tags applied to this action. diff --git a/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql b/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql index d8b3b795..41631296 100644 --- a/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql +++ b/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql @@ -1,7 +1,7 @@ SELECT * FROM - `dataform-open-source.was_there_extreme_weather` - LEFT OUTER JOIN `dataform-open-source.repositories_that_mention_extreme_weather` USING (date) + `your-bigquery-project.was_there_extreme_weather` + LEFT OUTER JOIN `your-bigquery-project.repositories_that_mention_extreme_weather` USING (date) ORDER BY date diff --git a/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb b/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb index a69e6ac4..0ae49ec8 100644 --- a/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb +++ b/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb @@ -22,8 +22,8 @@ "metadata": {}, "outputs": [], "source": [ - "%%bigquery results --project dataform-open-source\n", - "SELECT * FROM `dataform-open-source.dataform_examples.repositories_created_during_extreme_weather`" + "%%bigquery results --project your-bigquery-project\n", + "SELECT * FROM `your-bigquery-project.your_examples_dataset.repositories_created_during_extreme_weather`" ] }, { diff --git a/examples/extreme_weather_programming/workflow_settings.yaml b/examples/extreme_weather_programming/workflow_settings.yaml index fb2e7a09..6636df55 100644 --- a/examples/extreme_weather_programming/workflow_settings.yaml +++ b/examples/extreme_weather_programming/workflow_settings.yaml @@ -1,7 +1,7 @@ -defaultProject: dataform-open-source +defaultProject: your-bigquery-project defaultLocation: us -defaultDataset: dataform_extreme_weather_programming_example -defaultAssertionDataset: dataform_extreme_weather_programming_example_assertions +defaultDataset: sqlanvil_extreme_weather_programming_example +defaultAssertionDataset: sqlanvil_extreme_weather_programming_example_assertions defaultNotebookRuntimeOptions: outputBucket: gs://some-bucket runtimeTemplateName: projects/test-project/locations/us-central1/notebookRuntimeTemplates/test-template diff --git a/examples/stackoverflow_reporter/workflow_settings.yaml b/examples/stackoverflow_reporter/workflow_settings.yaml index df567eb7..8d911ed2 100644 --- a/examples/stackoverflow_reporter/workflow_settings.yaml +++ b/examples/stackoverflow_reporter/workflow_settings.yaml @@ -1,4 +1,4 @@ -defaultProject: dataform-open-source +defaultProject: your-bigquery-project defaultLocation: us -defaultDataset: dataform_stackoverflow_reporter_example -defaultAssertionDataset: dataform_stackoverflow_reporter_example_assertions +defaultDataset: sqlanvil_stackoverflow_reporter_example +defaultAssertionDataset: sqlanvil_stackoverflow_reporter_example_assertions diff --git a/packages/@sqlanvil/cli/BUILD b/packages/@sqlanvil/cli/BUILD index 016a4335..35b6f2ac 100644 --- a/packages/@sqlanvil/cli/BUILD +++ b/packages/@sqlanvil/cli/BUILD @@ -54,7 +54,7 @@ externals = [ pkg_json( name = "json", package_name = "@sqlanvil/cli", - description = "Dataform command line interface.", + description = "sqlanvil command line interface.", external_deps = externals, layers = [ "//:package.json", diff --git a/packages/@sqlanvil/cli/package.layer.json b/packages/@sqlanvil/cli/package.layer.json index 3d9c7062..503898b2 100644 --- a/packages/@sqlanvil/cli/package.layer.json +++ b/packages/@sqlanvil/cli/package.layer.json @@ -1,5 +1,5 @@ { "bin": { - "dataform": "bundle.js" + "sqlanvil": "bundle.js" } } diff --git a/packages/@sqlanvil/core/BUILD b/packages/@sqlanvil/core/BUILD index a60a5e59..6709290e 100644 --- a/packages/@sqlanvil/core/BUILD +++ b/packages/@sqlanvil/core/BUILD @@ -45,7 +45,7 @@ npm_package_bin( pkg_json( name = "json", package_name = "@sqlanvil/core", - description = "Dataform core API.", + description = "sqlanvil core API.", external_deps = [], layers = [ "//:package.json", diff --git a/packages/@sqlanvil/package.layer.json b/packages/@sqlanvil/package.layer.json index 74d87690..5021cff9 100644 --- a/packages/@sqlanvil/package.layer.json +++ b/packages/@sqlanvil/package.layer.json @@ -1,8 +1,8 @@ { - "homepage": "https://github.com/dataform-co/dataform", + "homepage": "https://github.com/ihistand/sqlanvil", "license": "Apache-2.0", "keywords": [ - "dataform", + "sqlanvil", "etl", "data-pipeline", "big-data", diff --git a/protos/configs.proto b/protos/configs.proto index 90526b27..57c90b88 100644 --- a/protos/configs.proto +++ b/protos/configs.proto @@ -59,7 +59,7 @@ message WorkflowSettings { // Optional. The default BigQuery reservation to use for execution. // If unset, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string default_reservation = 14; // Optional. An external package that provides an extension. @@ -278,7 +278,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 24; } @@ -366,7 +366,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 21; } @@ -418,7 +418,7 @@ message ActionConfig { // If set, unique key represents a set of names of columns that will act as // a the unique key. To enforce this, when updating the incremental - // table, Dataform merges rows with `uniqueKey` instead of appending them. + // table, sqlanvil merges rows with `uniqueKey` instead of appending them. repeated string unique_key = 11; // Description of the incremental table. @@ -483,7 +483,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 27; } @@ -526,7 +526,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 11; } @@ -578,7 +578,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 13; } diff --git a/protos/core.proto b/protos/core.proto index 9ea89fec..b1f6c7f0 100644 --- a/protos/core.proto +++ b/protos/core.proto @@ -111,7 +111,7 @@ message CompilationError { // Compilation mode, unspecified is interpreted to AoT. enum ActionCompilationMode { ACTION_COMPILATION_MODE_UNSPECIFIED = 0; - // Ahead-of-time compilation (regular Dataform compilation). + // Ahead-of-time compilation (regular sqlanvil compilation). ACTION_COMPILATION_MODE_AOT = 1; // Just-in-time compilation. // Will only populate jit_code fields in compiled graph, @@ -441,7 +441,7 @@ message CompileExecutionResponse { } // This feature list is added to when making potentilly backwards breaking -// changes. It lets the caller of Dataform Core know whether it supports the +// changes. It lets the caller of sqlanvil Core know whether it supports the // change. enum SupportedFeatures { UNKNOWN_FEATURE = 0; diff --git a/protos/extension.proto b/protos/extension.proto index 8e31b550..b6a52042 100644 --- a/protos/extension.proto +++ b/protos/extension.proto @@ -12,9 +12,9 @@ option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; enum ExtensionCompilationMode { // Unspecified compiled mode - extension will be skipped. COMPILATION_MODE_UNSPECIFIED = 0; - // Execute before regular Dataform compilation. + // Execute before regular sqlanvil compilation. PROLOGUE = 1; - // Replace regular Dataform compilation. + // Replace regular sqlanvil compilation. APPLICATION_CODE = 2; } diff --git a/scripts/create_gh_pr b/scripts/create_gh_pr deleted file mode 100755 index c22c59de..00000000 --- a/scripts/create_gh_pr +++ /dev/null @@ -1,21 +0,0 @@ -set -e - -echo "Set the git config..." -git config user.name $_GITHUB_USER -git config user.email $_GITHUB_EMAIL -git remote set-url origin https://$_GITHUB_USER:$(cat token.txt)@github.com/dataform-co/dataform.git - -echo "Update version..." -./scripts/update_version - -export git_branch_name=npm_veriosn_$(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g") - -echo "Create new branch $git_branch_name..." -git checkout -b $git_branch_name - -git add version.bzl -git commit -m "Update the npm package version" - -echo "Push changes to remote..." -git push origin $git_branch_name -echo $git_branch_name > git_branch_name.txt diff --git a/scripts/decrypt_secret b/scripts/decrypt_secret deleted file mode 100755 index da7fe20b..00000000 --- a/scripts/decrypt_secret +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash - -gcloud kms decrypt \ - --ciphertext-file=$1.enc \ - --plaintext-file=$1 \ - --project=dataform-open-source \ - --keyring=dataform-builder-keyring \ - --key=dataform-builder-key \ - --location=global diff --git a/testing/run_core.ts b/testing/run_core.ts index f95de51e..5bf82115 100644 --- a/testing/run_core.ts +++ b/testing/run_core.ts @@ -64,7 +64,7 @@ export function runMainInVm( ): sqlanvil.CoreExecutionResponse { const projectDir = coreExecutionRequest.compile.compileConfig.projectDir; - // Copy over the build Dataform Core that is set up as a node_modules directory. + // Copy over the build sqlanvil Core that is set up as a node_modules directory. fs.copySync(`${process.cwd()}/core/node_modules`, `${projectDir}/node_modules`); const compiler = compile as CompilerFunction; diff --git a/tests/api/projects.spec.ts b/tests/api/projects.spec.ts index 6195d5f6..d0e6d629 100644 --- a/tests/api/projects.spec.ts +++ b/tests/api/projects.spec.ts @@ -34,17 +34,17 @@ suite("examples", () => { { fileName: "definitions/has_compile_errors/assertion_with_bigquery.sqlx", message: - 'Unexpected property "bigquery", or property value type of "object" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "bigquery", or property value type of "object" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/assertion_with_materialized.sqlx", message: - 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/assertion_with_output.sqlx", message: - 'Unexpected property "hasOutput", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "hasOutput", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/assertion_with_postops.sqlx", @@ -62,12 +62,12 @@ suite("examples", () => { { fileName: "definitions/has_compile_errors/protected_assertion.sqlx", message: - 'Unexpected property "protected", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "protected", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/table_with_materialized.sqlx", message: - 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-TableConfig for allowed properties.' + 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-TableConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/view_with_incremental.sqlx", @@ -651,7 +651,7 @@ suite("examples", () => { }) ); expect(exampleDeclaration.actionDescriptor.description).to.equal( - "This table is not generated by Dataform!" + "This table is not generated by sqlanvil!" ); // Check testcases. diff --git a/tests/api/projects/common_v2/definitions/example_declaration.sqlx b/tests/api/projects/common_v2/definitions/example_declaration.sqlx index 92ab0223..ee920e17 100644 --- a/tests/api/projects/common_v2/definitions/example_declaration.sqlx +++ b/tests/api/projects/common_v2/definitions/example_declaration.sqlx @@ -3,5 +3,5 @@ config { database: "some_database_name", schema: "some_external_schema_name", name: "very_important_external_table", - description: "This table is not generated by Dataform!" + description: "This table is not generated by sqlanvil!" } diff --git a/tests/integration/bigquery.spec.ts b/tests/integration/bigquery.spec.ts index 30703aeb..4d5145ec 100644 --- a/tests/integration/bigquery.spec.ts +++ b/tests/integration/bigquery.spec.ts @@ -27,7 +27,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) // Drop schemas to make sure schema creation works. await dbadapter.execute( - "drop schema if exists `dataform-open-source.df_integration_test_project_e2e` cascade" + "drop schema if exists `your-bigquery-project.df_integration_test_project_e2e` cascade" ); // Run the project. @@ -39,8 +39,8 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) // Check the status of action execution. const expectedFailedActions = [ - "dataform-open-source.df_integration_test_assertions_project_e2e.example_assertion_fail", - "dataform-open-source.df_integration_test_project_e2e.example_operation_partial_fail" + "your-bigquery-project.df_integration_test_assertions_project_e2e.example_assertion_fail", + "your-bigquery-project.df_integration_test_project_e2e.example_operation_partial_fail" ]; for (const actionName of Object.keys(actionMap)) { const expectedResult = expectedFailedActions.includes(actionName) @@ -54,13 +54,13 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) expect( actionMap[ - "dataform-open-source.df_integration_test_assertions_project_e2e.example_assertion_fail" + "your-bigquery-project.df_integration_test_assertions_project_e2e.example_assertion_fail" ].tasks[1].errorMessage ).to.eql("bigquery error: Assertion failed: query returned 1 row(s)."); expect( actionMap[ - "dataform-open-source.df_integration_test_project_e2e.example_operation_partial_fail" + "your-bigquery-project.df_integration_test_project_e2e.example_operation_partial_fail" ].tasks[0].errorMessage ).to.eql("bigquery error: Query error: Unrecognized name: invalid_column at [3:8]"); }); @@ -104,7 +104,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) const [incrementalRows, incrementalMergeRows] = await Promise.all([ getTableRows( { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_incremental_tables", name: "example_incremental" }, @@ -113,7 +113,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ), getTableRows( { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_incremental_tables", name: "example_incremental_merge" }, @@ -150,7 +150,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) for (const expectedMetadata of [ { target: { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_dataset_metadata", name: "example_incremental" }, @@ -189,7 +189,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) }, { target: { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_dataset_metadata", name: "example_view" }, @@ -258,7 +258,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) await dfapi.run(dbadapter, executionGraph).result(); const view = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_view" + "your-bigquery-project.df_integration_test_evaluate.example_view" ]; let evaluations = await dbadapter.evaluate(sqlanvil.Table.create(view)); expect(evaluations.length).to.equal(1); @@ -267,7 +267,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ); const materializedView = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_materialized_view" + "your-bigquery-project.df_integration_test_evaluate.example_materialized_view" ]; evaluations = await dbadapter.evaluate(sqlanvil.Table.create(materializedView)); expect(evaluations.length).to.equal(1); @@ -276,7 +276,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ); const table = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_table" + "your-bigquery-project.df_integration_test_evaluate.example_table" ]; evaluations = await dbadapter.evaluate(sqlanvil.Table.create(table)); expect(evaluations.length).to.equal(1); @@ -285,7 +285,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ); const operation = keyBy(compiledGraph.operations, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_operation" + "your-bigquery-project.df_integration_test_evaluate.example_operation" ]; evaluations = await dbadapter.evaluate(sqlanvil.Operation.create(operation)); expect(evaluations.length).to.equal(1); @@ -294,7 +294,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ); const assertion = keyBy(compiledGraph.assertions, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_assertions_evaluate.example_assertion_pass" + "your-bigquery-project.df_integration_test_assertions_evaluate.example_assertion_pass" ]; evaluations = await dbadapter.evaluate(sqlanvil.Assertion.create(assertion)); expect(evaluations.length).to.equal(1); @@ -303,7 +303,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) ); const incremental = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_incremental" + "your-bigquery-project.df_integration_test_evaluate.example_incremental" ]; evaluations = await dbadapter.evaluate(sqlanvil.Table.create(incremental)); expect(evaluations.length).to.equal(2); @@ -319,7 +319,7 @@ suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) const target = (name: string) => ({ schema: "df_integration_test", name, - database: "dataform-open-source" + database: "your-bigquery-project" }); let evaluations = await dbadapter.evaluate( diff --git a/tests/integration/bigquery_project/definitions/sample_data.sqlx b/tests/integration/bigquery_project/definitions/sample_data.sqlx index fdd52e59..1b18ef69 100644 --- a/tests/integration/bigquery_project/definitions/sample_data.sqlx +++ b/tests/integration/bigquery_project/definitions/sample_data.sqlx @@ -3,6 +3,6 @@ config { hermetic: false } -select ${when(dataform.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all +select ${when(sqlanvil.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all select 2 as val union all -select ${when(dataform.projectConfig.warehouse === "bigquery", "3", "2")} as val +select ${when(sqlanvil.projectConfig.warehouse === "bigquery", "3", "2")} as val diff --git a/tests/integration/bigquery_project/workflow_settings.yaml b/tests/integration/bigquery_project/workflow_settings.yaml index c69b1cb7..f68bbf0b 100644 --- a/tests/integration/bigquery_project/workflow_settings.yaml +++ b/tests/integration/bigquery_project/workflow_settings.yaml @@ -1,4 +1,4 @@ -defaultProject: dataform-open-source +defaultProject: your-bigquery-project defaultLocation: us defaultDataset: df_integration_test defaultAssertionDataset: df_integration_test_assertions diff --git a/tests/integration/postgres_project/definitions/sample_data.sqlx b/tests/integration/postgres_project/definitions/sample_data.sqlx index f008e44b..35dfc87e 100644 --- a/tests/integration/postgres_project/definitions/sample_data.sqlx +++ b/tests/integration/postgres_project/definitions/sample_data.sqlx @@ -2,6 +2,6 @@ config { type: "view" } -select ${when(dataform.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all +select ${when(sqlanvil.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all select 2 as val union all -select ${when(dataform.projectConfig.warehouse === "postgres", "3", "2")} as val +select ${when(sqlanvil.projectConfig.warehouse === "postgres", "3", "2")} as val diff --git a/typedoc.json b/typedoc.json index 78d767c5..a40b7de1 100644 --- a/typedoc.json +++ b/typedoc.json @@ -1,7 +1,7 @@ { "out": "newdocs", "plugin": ["typedoc-plugin-markdown"], - "name": "Dataform Javascript API Reference", + "name": "sqlanvil Javascript API Reference", "excludeExternals": true, "exclude": [ "*", "!core/**" diff --git a/vscode/README.md b/vscode/README.md index 6588a666..def4ac2f 100644 --- a/vscode/README.md +++ b/vscode/README.md @@ -1,14 +1,17 @@ +# sqlanvil VSCode Extension -> **❗IMPORTANT**: This extension has been deprecated and will no longer receive updates -> -> We recommend you migrate to the [open sourced alternative](https://marketplace.visualstudio.com/items?itemName=ashishalex.dataform-lsp-vscode) or to the [Google Cloud Dataform UI](https://cloud.google.com/dataform). +Syntax highlighting and project compilation for sqlanvil projects in VSCode. ## Usage -**To run this extension you will need the dataform cli installed globally: `npm i -g @sqlanvil/cli`.** +This extension requires the sqlanvil CLI installed globally: + +```bash +npm i -g @sqlanvil/cli +``` Includes: - Syntax highlighting for `.sqlx` files - Realtime compilation of your project -- `cmd + click` on a `ref()` function to go to the file that it references +- `cmd + click` on a `ref()` function to navigate to the referenced file diff --git a/vscode/contributing.md b/vscode/contributing.md index 70d55c55..60d62c2d 100644 --- a/vscode/contributing.md +++ b/vscode/contributing.md @@ -2,12 +2,12 @@ There are two routes to publishing new versions: ### Web (actually working): -- In the main repo run `bazel run vscode:packager /tmp/dataform-package.vsix` -- Take the generated package and upload it [here](https://marketplace.visualstudio.com/manage/publishers/dataform) +- In the main repo run `bazel run vscode:packager /tmp/sqlanvil-package.vsix` +- Take the generated package and upload it [here](https://marketplace.visualstudio.com/manage/publishers/ihistand) ### CLI (not working right now) -- In the main repo run `bazel run vscode:packager /tmp/dataform-package.vsix` +- In the main repo run `bazel run vscode:packager /tmp/sqlanvil-package.vsix` - Then take the package and paste it into this directory - Then run `vsce publish` - Currently this doesn't work as `vsce publish` will try to publish typescript files diff --git a/vscode/package-lock.json b/vscode/package-lock.json index 5841e9e1..574414a6 100644 --- a/vscode/package-lock.json +++ b/vscode/package-lock.json @@ -1,11 +1,11 @@ { - "name": "dataform", + "name": "sqlanvil", "version": "0.0.14", "lockfileVersion": 2, "requires": true, "packages": { "": { - "name": "dataform", + "name": "sqlanvil", "version": "0.0.14", "dependencies": { "vscode-languageclient": "^6.1.3", diff --git a/vscode/workflow_settings_yaml.schema.json b/vscode/workflow_settings_yaml.schema.json index 89e76da0..c2a1815d 100644 --- a/vscode/workflow_settings_yaml.schema.json +++ b/vscode/workflow_settings_yaml.schema.json @@ -1,12 +1,12 @@ { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", - "$comment": "Created from 'protos/configs.proto'. Even options are specified as 'required' in proto, they might be specified from the CLI options, so leave them optional except for the dataformCoreVersion which is required in compile.", + "$comment": "Created from 'protos/configs.proto'. Even options are specified as 'required' in proto, they might be specified from the CLI options, so leave them optional except for the sqlanvilCoreVersion which is required in compile.", "additionalProperties": false, "properties": { - "dataformCoreVersion": { + "sqlanvilCoreVersion": { "type": "string", - "description": "The desired dataform core version to compile against." + "description": "The desired sqlanvil core version to compile against." }, "defaultProject": { "type": "string", @@ -53,6 +53,6 @@ } }, "required": [ - "dataformCoreVersion" + "sqlanvilCoreVersion" ] } \ No newline at end of file From a11d83cd1cd89191af0ac7ed3ab7dac958122a50 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 07:41:40 -0500 Subject: [PATCH 09/20] build: Dockerfile.dev + scripts/docker-bazel for macOS users MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pinned Bazel 5.4 + 2022-era protobuf chain inherited from upstream does not build on macOS Tahoe + Apple Silicon: - Bazel 5.4 ships wrapped_clang without LC_UUID load commands; rejected by current dyld. - Bumping to Bazel 6.x triggers @bazel_tools//platforms removal failures in the WORKSPACE. - Bypassing wrapped_clang via system clang gets further but hits Apple SDK <-> old protobuf preprocessor conflicts (_stdio.h sprintf redefinition). Linux is unaffected. Until a future Bazel-7 + Bzlmod modernization PR lands, macOS contributors should build inside the dev container: ./scripts/docker-bazel build //protos:sqlanvil_proto ./scripts/docker-bazel test //core/... ./scripts/docker-bazel # interactive shell Dockerfile.dev: - Base: node:20-bookworm (gives node + npm) - Adds JDK 17, python 3, build-essential, zlib1g-dev, git, zip/unzip - Installs Bazelisk globally via npm - Mounts /root/.cache/bazel + /root/.cache/bazel-disk as named volumes for persistent Bazel cache across runs scripts/docker-bazel: - Lazy-builds the sqlanvil-dev image on first run - Mounts the repo at /workspace - Mounts named volumes sqlanvil-bazel-cache and sqlanvil-bazel-disk - Auto-attaches a TTY when stdin is a terminal (CI-safe otherwise) - No args → drops into bash inside the container contributing.md: documents the macOS compatibility caveat and the Docker workflow. Co-Authored-By: Claude Opus 4.7 --- Dockerfile.dev | 55 ++++++++++++++++++++++++++++++++++++++++++++ contributing.md | 25 ++++++++++++++++++++ scripts/docker-bazel | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+) create mode 100644 Dockerfile.dev create mode 100755 scripts/docker-bazel diff --git a/Dockerfile.dev b/Dockerfile.dev new file mode 100644 index 00000000..7a27624d --- /dev/null +++ b/Dockerfile.dev @@ -0,0 +1,55 @@ +# Dev container for building sqlanvil on Linux. +# +# Why: the pinned Bazel 5.4 + 2022-era rules_proto/protobuf chain does not +# build on macOS Tahoe (wrapped_clang missing LC_UUID, Apple SDK header +# conflicts with old protobuf). On Linux those issues don't apply. +# +# Build the image once: +# docker build -f Dockerfile.dev -t sqlanvil-dev . +# +# Run a one-shot Bazel command (caches persist in a named volume): +# docker run --rm -it \ +# -v "$PWD:/workspace" \ +# -v sqlanvil-bazel-cache:/root/.cache/bazel \ +# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \ +# sqlanvil-dev bazel build //protos:sqlanvil_proto +# +# Drop into a shell: +# docker run --rm -it \ +# -v "$PWD:/workspace" \ +# -v sqlanvil-bazel-cache:/root/.cache/bazel \ +# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \ +# sqlanvil-dev +# +# Or use the wrapper: ./scripts/docker-bazel build //protos:sqlanvil_proto + +FROM node:20-bookworm + +# Bazel needs: JDK, python (for some rules), git, build-essential, unzip, +# zip, and the usual C/C++ toolchain (gcc, g++, make). +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + git \ + gnupg \ + openjdk-17-jdk-headless \ + python3 \ + python-is-python3 \ + build-essential \ + unzip \ + zip \ + zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* + +# Bazelisk — reads .bazelversion and downloads the matching Bazel. +RUN npm install -g @bazel/bazelisk + +# Bazel writes a lot to ~/.cache/bazel; expose it as a volume target so +# the host docker volume sticks across runs. +ENV BAZEL_DISK_CACHE_DIR=/root/.cache/bazel-disk +RUN mkdir -p /root/.cache/bazel /root/.cache/bazel-disk + +WORKDIR /workspace + +# Default to an interactive shell. Override with the bazel command you want. +CMD ["bash"] diff --git a/contributing.md b/contributing.md index 649d0c69..04a9bc78 100644 --- a/contributing.md +++ b/contributing.md @@ -13,6 +13,8 @@ sqlanvil is a TypeScript project, using [Bazel](https://bazel.build) as a build Bazel is the build system. Install via Bazelisk: ```bash +brew install bazelisk # macOS +# or: npm i -g @bazel/bazelisk ``` @@ -22,6 +24,29 @@ On macOS, increase the open-file limit (Bazel hits the default): sudo sysctl -w kern.maxfiles=65536 ``` +##### macOS compatibility (important) + +The currently pinned Bazel 5.4 + 2022-era protobuf chain inherited from +upstream **does not build natively on macOS Tahoe / Apple Silicon** — +`wrapped_clang` ships without `LC_UUID` (rejected by current dyld) and the +old protobuf headers conflict with Xcode 21's SDK. This will be fixed by +a future toolchain modernization PR (Bazel 7 + Bzlmod migration). + +Until then, build via Docker on macOS: + +```bash +./scripts/docker-bazel build //protos:sqlanvil_proto +./scripts/docker-bazel test //core/... +./scripts/docker-bazel build //... +./scripts/docker-bazel # drops into an interactive shell +``` + +The wrapper builds a `sqlanvil-dev` image (Debian + Node 20 + JDK 17 + +Bazelisk) on first invocation and reuses named volumes for the Bazel cache +so subsequent runs are fast. + +Linux users can use Bazelisk directly without Docker. + ### Run the CLI Substitute `./scripts/run` for the installed `sqlanvil` binary: diff --git a/scripts/docker-bazel b/scripts/docker-bazel new file mode 100755 index 00000000..5c2a73ff --- /dev/null +++ b/scripts/docker-bazel @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Run a bazel command inside the sqlanvil-dev Docker container. +# +# Usage: +# ./scripts/docker-bazel build //protos:sqlanvil_proto +# ./scripts/docker-bazel test //core/... +# ./scripts/docker-bazel build //... +# ./scripts/docker-bazel # drops into an interactive shell +# +# First run is slow: pulls the node:20 base image, installs deps, then +# downloads Bazel 5.4 + all toolchains/rules from cold. Subsequent runs +# reuse the named volumes for the Bazel cache. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +IMAGE="sqlanvil-dev" + +# Build the image if it doesn't exist locally yet. +if ! docker image inspect "$IMAGE" >/dev/null 2>&1; then + echo "Building $IMAGE image (one-time setup)..." >&2 + docker build -f "$REPO_ROOT/Dockerfile.dev" -t "$IMAGE" "$REPO_ROOT" +fi + +# tty + interactive flags only when STDIN is a terminal — keeps CI happy. +TTY_FLAGS=() +if [ -t 0 ] && [ -t 1 ]; then + TTY_FLAGS=(-it) +fi + +# If no args: drop into a shell. Otherwise run `bazel `. +if [ "$#" -eq 0 ]; then + CMD=(bash) +else + CMD=(bazel "$@") +fi + +docker run --rm "${TTY_FLAGS[@]}" \ + -v "$REPO_ROOT:/workspace" \ + -v sqlanvil-bazel-cache:/root/.cache/bazel \ + -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \ + "$IMAGE" "${CMD[@]}" From 63685541c07538c21a765afdd2c753fcbf30bf67 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 07:46:12 -0500 Subject: [PATCH 10/20] fix(protos): strip_import_prefix + add timestamp_proto dep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When running bazel build //protos:sqlanvil_proto directly on the proto_library target (rather than going through ts_proto_library which wraps protoc with different flags), two pre-existing upstream bugs surface: 1. protos/configs.proto and core.proto import "extension.proto" without the protos/ prefix. Without strip_import_prefix on the proto_library, protoc passes -Iprotos/extension.proto=protos/extension.proto and the bare-name import fails. Adding strip_import_prefix = "/protos" removes the protos/ prefix so file references match. 2. jit.proto imports google/protobuf/timestamp.proto but the proto_library only declared deps on empty_proto and struct_proto. Added timestamp_proto. Both are upstream Dataform bugs (same content in upstream/main); they weren't caught because the upstream CI path went through ts_proto_library which has its own protoc invocation. Direct proto_library builds always would have failed. Verified working via: ./scripts/docker-bazel build //protos:sqlanvil_proto → Build completed successfully, 4 total actions This proves the dataform → sqlanvil rename is structurally sound at the proto layer (workspace name `sa`, package `sqlanvil`, target `sqlanvil_proto`, java/go options all resolve and protoc accepts them). Co-Authored-By: Claude Opus 4.7 --- protos/BUILD | 2 ++ scripts/docker-bazel | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/protos/BUILD b/protos/BUILD index d20be132..808fe19e 100644 --- a/protos/BUILD +++ b/protos/BUILD @@ -15,9 +15,11 @@ proto_library( "profiles.proto", "extension.proto", ], + strip_import_prefix = "/protos", deps = [ "@com_google_protobuf//:empty_proto", "@com_google_protobuf//:struct_proto", + "@com_google_protobuf//:timestamp_proto", ], ) diff --git a/scripts/docker-bazel b/scripts/docker-bazel index 5c2a73ff..75adbc13 100755 --- a/scripts/docker-bazel +++ b/scripts/docker-bazel @@ -11,7 +11,7 @@ # downloads Bazel 5.4 + all toolchains/rules from cold. Subsequent runs # reuse the named volumes for the Bazel cache. -set -euo pipefail +set -eo pipefail REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" IMAGE="sqlanvil-dev" From b0a4d42f1048650e07251da662770c0b05ef76f0 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 07:49:07 -0500 Subject: [PATCH 11/20] =?UTF-8?q?docs:=20rename=20handoff=20=E2=80=94=20sn?= =?UTF-8?q?apshot=20for=20next=20session?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures branch state, verification results, known pre-existing issues blocking fuller verification, and the recommended sequence of follow-up work (refresh node pin, then Postgres adapter PR, then Bazel modernization later). Co-Authored-By: Claude Opus 4.7 --- docs/rename_handoff.md | 152 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 docs/rename_handoff.md diff --git a/docs/rename_handoff.md b/docs/rename_handoff.md new file mode 100644 index 00000000..6c4ec0f1 --- /dev/null +++ b/docs/rename_handoff.md @@ -0,0 +1,152 @@ +# Rename Handoff + +Snapshot of the `rename/dataform-to-sqlanvil` branch state for picking +back up after a break. + +## Status + +**Logically complete + proto layer verified building.** Ready for review +and merge into `restore-postgres-adapter`. Larger Bazel targets blocked +by pre-existing upstream toolchain rot (separate from the rename — see +"Known pre-existing issues" below). + +## Where to pick up + +```bash +cd ~/projects-ivan/sqlanvil +git checkout rename/dataform-to-sqlanvil +git log --oneline restore-postgres-adapter..HEAD # see the 10 rename commits +``` + +## What was verified + +**Proto-layer build succeeded inside Docker:** + +```bash +./scripts/docker-bazel build //protos:sqlanvil_proto +# → Build completed successfully, 4 total actions +``` + +That proves: +- New WORKSPACE name `sa` (was `df`) is accepted by Bazel +- Target `//protos:sqlanvil_proto` (was `//protos:dataform_proto`) resolves +- All 8 `.proto` files parse with `package sqlanvil;` +- `java_package = "com.sqlanvil.protos"` + `go_package = "github.com/ihistand/sqlanvil/..."` accepted +- Wire-format-breaking field rename `sqlanvil_core_version` accepted +- `@com_google_protobuf//` external resolves and the C++ chain compiles + +## What was NOT verified (and why) + +| Target | Blocker | Type | +| :--- | :--- | :--- | +| `//protos:ts` | Node v24.13.0 pin in WORKSPACE has stale SHA256 — nodejs.org served a different binary | Pre-existing upstream pin drift | +| `//core/...` | Same node pin issue (transitive) | Pre-existing | +| `bazel test //...` | Same | Pre-existing | +| `./scripts/run help` (CLI smoke) | Same | Pre-existing | +| Native macOS build | Bazel 5.4 `wrapped_clang` lacks `LC_UUID` (rejected by Tahoe dyld); Bazel 6.x WORKSPACE `@bazel_tools//platforms` refactor breaks too | Pre-existing macOS+toolchain rot | + +All five would fail identically on `upstream/main` — not caused by this +PR. + +## Commits on the branch + +``` +63685541 fix(protos): strip_import_prefix + add timestamp_proto dep +05e2... build: Dockerfile.dev + scripts/docker-bazel for macOS users +ce6af2eb refactor: final rename sweep — proto auto-docs + goldens + examples + dead infra +31adafa6 refactor: rename interface ID + write NOTICE + rewrite root docs +9809e459 refactor: rename CLI binary + help text + VSCode extension +1399c23d refactor: drop dataform.json legacy config path (clean break) +513e6963 refactor: rename TS imports df/ → sa/ + proto namespace + framework defaults +2148bb3e refactor: rename packages/@dataform → @sqlanvil + Bazel labels + dead infra +f084b0ae refactor: rename proto packages dataform → sqlanvil + WORKSPACE +b4b24cd6 docs: rename plan + postgres-first-class design +``` + +Total: ~150 files changed, ~2400 lines net (mostly mechanical renames). + +## Known pre-existing issues (NOT introduced by this PR, but blocking +verification beyond proto layer) + +### 1. Node v24.13.0 SHA256 mismatch +`WORKSPACE` pins `node-v24.13.0-linux-arm64.tar.xz` with checksum +`e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e`. +nodejs.org currently serves a binary with checksum +`aa881151bd0f9f154a0424dd60a72e9ce10672619121658c278a24327ef46831`. +Fix: bump pin to a current Node release (probably 20 LTS to match the +Docker image, or 22 LTS) and regenerate SHA. Likely also needs matching +amd64 SHA. + +### 2. Bazel toolchain modernization needed for native macOS builds +The pinned Bazel 5.4 was released Dec 2022. Its `wrapped_clang` shim +binary doesn't include `LC_UUID` load commands, which macOS Tahoe's +dyld rejects. Bumping to Bazel 6+ hits a different problem +(`@bazel_tools//platforms` removed; WORKSPACE references it). + +Real fix is multi-day work: bump to Bazel 7 LTS, migrate WORKSPACE to +`MODULE.bazel` (Bzlmod), update `rules_proto`/`rules_nodejs`/`protobuf` +versions to match. + +### 3. proto_library bugs in upstream Dataform too +The `import "extension.proto"` (no `protos/` prefix) and missing +`timestamp_proto` dep were both present in `upstream/main`. Likely +masked by upstream CI only running through `ts_proto_library` rather +than building the proto_library target directly. + +Fixed in this PR (commit `63685541`). + +## Next steps (in priority order) + +1. **Open a PR** for the rename branch against `restore-postgres-adapter`. + Use the commit summaries as the PR body. Mark "verified at proto layer + via Docker, blocked on toolchain rot for fuller verification." + +2. **Small follow-up PR: refresh stale toolchain pins.** + - Update WORKSPACE node version (v24.13.0 → v20.x or v22.x LTS) with + matching SHA256 hashes for arm64 + amd64 + - This should unblock `//protos:ts`, then `//core/...`, then full tree + - Scope: 1-2 hour PR. Mechanical. + +3. **Then start the Postgres adapter PR** as planned in + `docs/postgres_first_class_design.md` §9 (PR 2: + `adapter/postgres-first-class`). + +4. **Defer until needed: Bazel 7 + Bzlmod modernization.** Would unblock + native macOS builds, but Docker dev container is functional for now. + Multi-day PR. Sequence after Postgres adapter is at least partially + working, to avoid stacking too much risky change. + +## Quick-reference commands + +```bash +# Switch to the branch +git checkout rename/dataform-to-sqlanvil + +# View what changed +git diff restore-postgres-adapter..HEAD --stat | tail -5 + +# Build inside Docker (first run ~5 min, subsequent <1 min) +./scripts/docker-bazel build //protos:sqlanvil_proto + +# Interactive shell inside the dev container +./scripts/docker-bazel + +# When Node pin gets fixed in WORKSPACE, try: +./scripts/docker-bazel build //protos:ts +./scripts/docker-bazel build //core/... +./scripts/docker-bazel run //packages/@sqlanvil/cli:bin -- help +``` + +## Files worth re-reading first thing next session + +- `docs/rename_checklist.md` — the original surface map +- `docs/postgres_first_class_design.md` — what comes after this rename +- `CLAUDE.md` — design directives + active-work context +- `Dockerfile.dev` + `scripts/docker-bazel` — verification workflow + +## Memory state (in `~/.claude/projects/-Users-ivan-projects-ivan/memory/`) + +- `user_dataform_expertise.md` — Ivan is experienced Dataform dev, skip 101 +- `project_sqlanvil_postgres_design.md` — first-class Postgres + Supabase, + nested config, rename mandatory, listanvil is target user +- `MEMORY.md` indexes them From ffea539394f6d65fe881307b344dbbf7ae1487da Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 11:04:31 -0500 Subject: [PATCH 12/20] =?UTF-8?q?fix(rename):=20catch=20missed=20@df=20?= =?UTF-8?q?=E2=86=92=20@sa=20workspace=20refs=20in=20common/=20BUILDs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename sweep missed these 4 BUILD files that referenced the old `@df` workspace name in load() statements and dep labels. Without this fix, //core/... fails to analyze with: error loading package 'common/errors': Unable to find package for @df//testing:index.bzl: The repository '@df' could not be resolved Found while verifying //core/... in Docker. --- common/errors/BUILD | 4 ++-- common/promises/BUILD | 6 +++--- common/protos/BUILD | 4 ++-- common/strings/BUILD | 4 ++-- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/common/errors/BUILD b/common/errors/BUILD index b806837f..e28687ba 100644 --- a/common/errors/BUILD +++ b/common/errors/BUILD @@ -1,4 +1,4 @@ -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//testing:index.bzl", "ts_test_suite") load("//tools:ts_library.bzl", "ts_library") package(default_visibility = ["//visibility:public"]) @@ -26,7 +26,7 @@ ts_test_suite( ], deps = [ ":errors", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/node", "@npm//chai", diff --git a/common/promises/BUILD b/common/promises/BUILD index d34deba1..73de9aba 100644 --- a/common/promises/BUILD +++ b/common/promises/BUILD @@ -1,5 +1,5 @@ -load("@df//tools:ts_library.bzl", "ts_library") -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//tools:ts_library.bzl", "ts_library") +load("@sa//testing:index.bzl", "ts_test_suite") package(default_visibility = ["//visibility:public"]) @@ -27,7 +27,7 @@ ts_test_suite( ], deps = [ ":promises", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/node", "@npm//chai", diff --git a/common/protos/BUILD b/common/protos/BUILD index ffb46ca4..f7f43657 100644 --- a/common/protos/BUILD +++ b/common/protos/BUILD @@ -1,7 +1,7 @@ package(default_visibility = ["//visibility:public"]) load("//tools:ts_library.bzl", "ts_library") -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//testing:index.bzl", "ts_test_suite") ts_library( name = "protos", @@ -30,7 +30,7 @@ ts_test_suite( deps = [ ":protos", "//protos:ts", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/node", "@npm//chai", diff --git a/common/strings/BUILD b/common/strings/BUILD index 6e171d1b..80a64b0e 100644 --- a/common/strings/BUILD +++ b/common/strings/BUILD @@ -1,4 +1,4 @@ -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//testing:index.bzl", "ts_test_suite") load("//tools:ts_library.bzl", "ts_library") package(default_visibility = ["//visibility:public"]) @@ -28,7 +28,7 @@ ts_test_suite( ], deps = [ ":strings", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/long", "@npm//@types/node", From 569f77cb9cf3350eb1f3fa3f2b484c23619ecff6 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 11:06:41 -0500 Subject: [PATCH 13/20] =?UTF-8?q?fix(rename):=20catch=20missed=20df=20?= =?UTF-8?q?=E2=86=92=20sa=20module=20aliases=20in=20bundler=20configs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename sweep missed: - packages/@sqlanvil/core/webpack.config.js — webpack resolve.alias - packages/sample-extension/webpack.config.js — webpack resolve.alias - packages/rollup.config.js — importsToBundle pattern - BUILD — ts_library `modules-fix` module_name Without these, //core/... fails to bundle: Module not found: Error: Can't resolve 'sa/core' in '...' Webpack alias still pointed `df` → RUNFILES/df, so any import of `sa/core` (post-rename TS path mapping) couldn't be resolved. Found while verifying //core/... in Docker. --- BUILD | 2 +- packages/@sqlanvil/core/webpack.config.js | 2 +- packages/rollup.config.js | 2 +- packages/sample-extension/webpack.config.js | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/BUILD b/BUILD index 993a6005..df8e303c 100644 --- a/BUILD +++ b/BUILD @@ -64,5 +64,5 @@ load("//tools:ts_library.bzl", "ts_library") ts_library( name = "modules-fix", srcs = [], - module_name = "df", + module_name = "sa", ) diff --git a/packages/@sqlanvil/core/webpack.config.js b/packages/@sqlanvil/core/webpack.config.js index 7ee11caf..ec7f9e2e 100644 --- a/packages/@sqlanvil/core/webpack.config.js +++ b/packages/@sqlanvil/core/webpack.config.js @@ -18,7 +18,7 @@ module.exports = (env, argv) => { resolve: { extensions: [".ts", ".js", ".json"], alias: { - df: path.resolve(process.env.RUNFILES, "df") + sa: path.resolve(process.env.RUNFILES, "sa") } }, plugins: [ diff --git a/packages/rollup.config.js b/packages/rollup.config.js index 3ac33cde..5d72077f 100644 --- a/packages/rollup.config.js +++ b/packages/rollup.config.js @@ -23,7 +23,7 @@ const knownNodeBuiltins = [ "net" ].map(moduleName => convertToRegex(moduleName)); -const importsToBundle = ["df", /df\/.*$/, /^bazel\-.*$/]; +const importsToBundle = ["sa", /sa\/.*$/, /^bazel\-.*$/]; const checkImports = imports => { const allowedImports = [...imports].map(pattern => convertToRegex(pattern)); diff --git a/packages/sample-extension/webpack.config.js b/packages/sample-extension/webpack.config.js index e2b76fbb..e0ddca64 100644 --- a/packages/sample-extension/webpack.config.js +++ b/packages/sample-extension/webpack.config.js @@ -18,7 +18,7 @@ module.exports = (env, argv) => { resolve: { extensions: [".ts", ".js", ".json"], alias: { - df: path.resolve(process.env.RUNFILES, "df") + sa: path.resolve(process.env.RUNFILES, "sa") } }, plugins: [ From b45a19f8eab822b6e60f84c28104f25fde3016f6 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 11:08:02 -0500 Subject: [PATCH 14/20] fix(deps): refresh Node toolchain pin to v20.20.2 LTS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pinned Node v24.13.0 in WORKSPACE had stale SHA256 hashes that no longer matched nodejs.org's served binaries: Expected: e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e Actual: aa881151bd0f9f154a0424dd60a72e9ce10672619121658c278a24327ef46831 Bumped to Node v20.20.2 LTS (matches the node:20-bookworm base in Dockerfile.dev). Fresh SHA256 hashes pulled from nodejs.org/dist/v20.20.2/SHASUMS256.txt for all 5 platforms. This was a stale-pin issue (not a Node 24 problem) — nodejs.org appears to have re-published the v24.13.0 archives at some point. Pinning to an LTS line is more stable. Verified in Docker: ./scripts/docker-bazel build //protos:ts → success ./scripts/docker-bazel build //core/... → success ./scripts/docker-bazel test //core/... → 16/16 pass Full tree (//...) still blocked by a separate pre-existing upstream issue: tools/postgres/BUILD references @io_bazel_rules_docker which is not declared in WORKSPACE. --- WORKSPACE | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/WORKSPACE b/WORKSPACE index 077dc063..e26d9f71 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -43,13 +43,13 @@ load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "yarn_install node_repositories( node_repositories = { - "24.13.0-darwin_amd64": ("node-v24.13.0-darwin-x64.tar.xz", "node-v24.13.0-darwin-x64", "4ca0a48233f091a2a69ec28dd58e59f394a1b2d4f052b6c6b10f760377fe266f"), - "24.13.0-darwin_arm64": ("node-v24.13.0-darwin-arm64.tar.xz", "node-v24.13.0-darwin-arm64", "c59a517e9147f25c6167426875a571432f1478c1d7ee7ecc10baa46b0d0e8545"), - "24.13.0-linux_amd64": ("node-v24.13.0-linux-x64.tar.xz", "node-v24.13.0-linux-x64", "e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e"), - "24.13.0-linux_arm64": ("node-v24.13.0-linux-arm64.tar.xz", "node-v24.13.0-linux-arm64", "e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e"), - "24.13.0-windows_amd64": ("node-v24.13.0-win-x64.zip", "node-v24.13.0-win-x64", "ca2742695be8de44027d71b3f53a4bdb36009b95575fe1ae6f7f0b5ce091cb88"), + "20.20.2-darwin_amd64": ("node-v20.20.2-darwin-x64.tar.xz", "node-v20.20.2-darwin-x64", "4d4c020eb534497e616de38f3733289ff33c615ddab38c048edec6547b7f76ea"), + "20.20.2-darwin_arm64": ("node-v20.20.2-darwin-arm64.tar.xz", "node-v20.20.2-darwin-arm64", "6375a1d4421bc04ab284ba89459df788a78c49c89e83c463d0eede47e2efc07b"), + "20.20.2-linux_amd64": ("node-v20.20.2-linux-x64.tar.xz", "node-v20.20.2-linux-x64", "df770b2a6f130ed8627c9782c988fda9669fa23898329a61a871e32f965e007d"), + "20.20.2-linux_arm64": ("node-v20.20.2-linux-arm64.tar.xz", "node-v20.20.2-linux-arm64", "73093db209e4e9e09dd7d15a47aeaab1b74833830df03efa5f942a1122c5fa71"), + "20.20.2-windows_amd64": ("node-v20.20.2-win-x64.zip", "node-v20.20.2-win-x64", "dc3700fdd57a63eedb8fd7e3c7baaa32e6a740a1b904167ff4204bc68ed8bf77"), }, - node_version = "24.13.0", + node_version = "20.20.2", package_json = ["//:package.json"], yarn_version = "1.13.0", ) From d8c28a1e164148afff9c1c22cbf035871906a9bb Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 11:14:46 -0500 Subject: [PATCH 15/20] fix(postgres-fixture): drop deprecated rules_docker; pull postgres image directly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream Dataform removed @io_bazel_rules_docker from WORKSPACE in commit 8be595ac (Mar 2024), but the restored Postgres adapter files in tools/postgres/BUILD still loaded `container_image` from it, breaking `bazel build //...`: error loading package 'tools/postgres': Unable to find package for @io_bazel_rules_docker//container:image.bzl: The repository '@io_bazel_rules_docker' could not be resolved Two paths considered: 1. Restore @io_bazel_rules_docker decl in WORKSPACE (~67 lines of deprecated bazel infra to maintain). 2. Refactor the fixture to use docker run directly. Chose (2). Modern Postgres LTS, no deprecated bazel rules, simpler dev flow. PostgresFixture now starts `postgres:15-alpine` via `docker run` on the local daemon — no Bazel image staging needed. Also rename DOCKER_CONTAINER_NAME from "postgres-df-integration-testing" to "postgres-sa-integration-testing" (rename sweep miss). Verified in Docker: ./scripts/docker-bazel build -- //... -//tools/postgres/... → 332 actions, Build completed successfully tools/postgres/... itself still needs `pg` + `pg-query-stream` added to package.json — that's Phase 1 of docs/postgres_reintegration_assessment.md (separate PR). --- tools/postgres/BUILD | 11 ----------- tools/postgres/postgres_fixture.ts | 13 +++---------- 2 files changed, 3 insertions(+), 21 deletions(-) diff --git a/tools/postgres/BUILD b/tools/postgres/BUILD index e57ec0f9..6cfe239f 100644 --- a/tools/postgres/BUILD +++ b/tools/postgres/BUILD @@ -1,13 +1,5 @@ package(default_visibility = ["//visibility:public"]) -load("@io_bazel_rules_docker//container:image.bzl", "container_image") - -# Exists purely to give a clean name to the postgres Docker image. -container_image( - name = "postgres_image", - base = "@postgres//image", -) - load("//tools:ts_library.bzl", "ts_library") ts_library( @@ -15,9 +7,6 @@ ts_library( srcs = glob( ["*.ts"], ), - data = [ - ":postgres_image", - ], deps = [ "//common/promises", "//testing", diff --git a/tools/postgres/postgres_fixture.ts b/tools/postgres/postgres_fixture.ts index ca0da071..db2c87f6 100644 --- a/tools/postgres/postgres_fixture.ts +++ b/tools/postgres/postgres_fixture.ts @@ -5,22 +5,15 @@ import { sleepUntil } from "sa/common/promises"; import { IHookHandler } from "sa/testing"; const USE_CLOUD_BUILD_NETWORK = !!process.env.USE_CLOUD_BUILD_NETWORK; -const DOCKER_CONTAINER_NAME = "postgres-df-integration-testing"; +const DOCKER_CONTAINER_NAME = "postgres-sa-integration-testing"; +const POSTGRES_IMAGE = "postgres:15-alpine"; const POSTGRES_SERVE_PORT = 5432; export class PostgresFixture { public static readonly host = USE_CLOUD_BUILD_NETWORK ? DOCKER_CONTAINER_NAME : "localhost"; - private static imageLoaded = false; - constructor(port: number, setUp: IHookHandler, tearDown: IHookHandler) { setUp("starting postgres", async () => { - if (!PostgresFixture.imageLoaded) { - // Load the postgres image into the local Docker daemon. - execSync("tools/postgres/postgres_image.executable"); - PostgresFixture.imageLoaded = true; - } - // Run the postgres Docker image. execSync( [ "docker run", @@ -30,7 +23,7 @@ export class PostgresFixture { "-d", `-p ${port}:${POSTGRES_SERVE_PORT}`, USE_CLOUD_BUILD_NETWORK ? "--network cloudbuild" : "", - "bazel/tools/postgres:postgres_image" + POSTGRES_IMAGE ].join(" ") ); From a25410ddc72c79966f6cb6fadaea7a812ab74e81 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 12:17:21 -0500 Subject: [PATCH 16/20] feat(deps): add pg, pg-query-stream, uuid for Postgres adapter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 of `adapter/postgres-first-class` per docs/postgres_first_class_design.md §6. Adds the npm packages the restored Postgres adapter needs: - pg ^8.11.3 — node-postgres client - pg-query-stream ^4.5.3 — streaming query results - uuid ^9.0.0 — used by tools/postgres fixture - @types/pg ^8.11.0 — types for pg - @types/uuid ^9.0.0 — types for uuid Also removed @types/pg-query-stream from tools/postgres/BUILD; that package is a stub (pg-query-stream 4.x bundles its own types) and yarn warns when it's installed. Dropped from package.json too. yarn.lock regenerated against the new Node 20.20.2 LTS pin (PR #2). Verified in Docker: ./scripts/docker-bazel build //tools/postgres/... → success ./scripts/docker-bazel build //... → success (139 targets) ./scripts/docker-bazel test //... → 38/41 pass Remaining failures (not caused by this PR): - //cli:index_run_e2e_test — needs BigQuery test creds - //tests/integration:bigquery.spec — needs BigQuery test creds - //tests/api:projects.spec — pre-existing schema mismatch on `warehouse` property in workflow_settings validation Next phase (separate PR): relocate `api/dbadapters/postgres.ts` → `cli/api/dbadapters/postgres.ts` and fix imports. --- package.json | 5 ++ tools/postgres/BUILD | 1 - yarn.lock | 114 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 119 insertions(+), 1 deletion(-) diff --git a/package.json b/package.json index 44b58b9a..e3df4aaf 100644 --- a/package.json +++ b/package.json @@ -19,11 +19,13 @@ "@types/long": "^4.0.0", "@types/moo": "^0.5.0", "@types/node": "^16.16.0", + "@types/pg": "^8.11.0", "@types/readline-sync": "^1.4.3", "@types/request": "^2.48.3", "@types/rimraf": "^2.0.2", "@types/semver": "^7.3.13", "@types/tmp": "^0.2.0", + "@types/uuid": "^9.0.0", "@types/vscode": "^1.45.1", "@types/yargs": "^15.0.5", "chai": "^4.2.0", @@ -46,6 +48,8 @@ "moo": "^0.5.0", "object-sizeof": "^1.6.1", "parse-duration": "^1.0.0", + "pg": "^8.11.3", + "pg-query-stream": "^4.5.3", "prettier": "^1.14.2", "promise-pool-executor": "^1.1.1", "protobufjs": "^7.5.5", @@ -72,6 +76,7 @@ "uglify-js": "^3.7.7", "untildify": "^4.0.0", "url": "^0.11.0", + "uuid": "^9.0.0", "vm2": "3.11.3", "vsce": "^1.79.5", "vscode-jsonrpc": "^5.0.1", diff --git a/tools/postgres/BUILD b/tools/postgres/BUILD index 6cfe239f..1b3770a3 100644 --- a/tools/postgres/BUILD +++ b/tools/postgres/BUILD @@ -12,7 +12,6 @@ ts_library( "//testing", "@npm//@types/node", "@npm//@types/pg", - "@npm//@types/pg-query-stream", "@npm//@types/uuid", "@npm//pg", "@npm//promise-pool-executor", diff --git a/yarn.lock b/yarn.lock index 5b0a95dd..a56c740e 100644 --- a/yarn.lock +++ b/yarn.lock @@ -454,6 +454,15 @@ resolved "https://registry.yarnpkg.com/@types/node/-/node-16.18.91.tgz#3e7b3b3d28f740e3e2d4ceb7ad9d16e6b9277c91" integrity sha512-h8Q4klc8xzc9kJKr7UYNtJde5TU2qEePVyH3WyzJaUC+3ptyc5kPQbWOIUcn8ZsG5+KSkq+P0py0kC0VqxgAXw== +"@types/pg@^8.11.0": + version "8.20.0" + resolved "https://registry.yarnpkg.com/@types/pg/-/pg-8.20.0.tgz#8bd03d3ac6b19143a8de7d66a9d13da32cd91526" + integrity sha512-bEPFOaMAHTEP1EzpvHTbmwR8UsFyHSKsRisLIHVMXnpNefSbGA1bD6CVy+qKjGSqmZqNqBDV2azOBo8TgkcVow== + dependencies: + "@types/node" "*" + pg-protocol "*" + pg-types "^2.2.0" + "@types/readline-sync@^1.4.3": version "1.4.3" resolved "https://registry.yarnpkg.com/@types/readline-sync/-/readline-sync-1.4.3.tgz#eac55a39d5a349912062c9e5216cd550c07fd9c8" @@ -509,6 +518,11 @@ resolved "https://registry.yarnpkg.com/@types/tough-cookie/-/tough-cookie-2.3.5.tgz#9da44ed75571999b65c37b60c9b2b88db54c585d" integrity "sha1-naRO11VxmZtlw3tgybK4jbVMWF0= sha512-SCcK7mvGi3+ZNz833RRjFIxrn4gI1PPR3NtuIS+6vMkvmsGjosqTJwRt5bAEFLRz+wtJMWv8+uOnZf2hi2QXTg==" +"@types/uuid@^9.0.0": + version "9.0.8" + resolved "https://registry.yarnpkg.com/@types/uuid/-/uuid-9.0.8.tgz#7545ba4fc3c003d6c756f651f3bf163d8f0f29ba" + integrity sha512-jg+97EGIcY9AGHJJRaaPVgetKDsrTgbRjQ5Msgjh/DQKEFl0DtyRr/VCOyD1T2R1MNeWPK/u7JoGhlDZnKBAfA== + "@types/vscode@^1.45.1": version "1.45.1" resolved "https://registry.yarnpkg.com/@types/vscode/-/vscode-1.45.1.tgz#672fb8c2cc33cf14cd4d3bdaa19bb294fe2b2706" @@ -3316,6 +3330,74 @@ performance-now@^2.1.0: resolved "https://registry.yarnpkg.com/performance-now/-/performance-now-2.1.0.tgz#6309f4e0e5fa913ec1c69307ae364b4b377c9e7b" integrity "sha1-Ywn04OX6kT7BxpMHrjZLSzd8nns= sha512-7EAHlyLHI56VEIdK57uwHdHKIaAGbnXPiw0yWbarQZOKaKpvUIgW0jWRVLiatnM+XXlSwsanIBH/hzGMJulMow==" +pg-cloudflare@^1.4.0: + version "1.4.0" + resolved "https://registry.yarnpkg.com/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz#4b4c20e6d8ae531d400730f4804571a8d62f1497" + integrity sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A== + +pg-connection-string@^2.13.0: + version "2.13.0" + resolved "https://registry.yarnpkg.com/pg-connection-string/-/pg-connection-string-2.13.0.tgz#8678113465a5af3cc977dcb51eadc847b27aa2de" + integrity sha512-EMnU9E2fSULdsbErBbMaXJvFeD9B4+nPcM3f+4lsiCR0BHLPrLVjv3DbyM2hgQQviKJaTWIRRTjKjWlHg3p2ig== + +pg-cursor@^2.20.0: + version "2.20.0" + resolved "https://registry.yarnpkg.com/pg-cursor/-/pg-cursor-2.20.0.tgz#c8f8c1d2bdf11c462e37aa595259bb520eb5668e" + integrity sha512-HP/EbUafheaUOs7DxlG6tda/rhmsX2hCTJJJ+gCnhljGyNEs6pBHddbNuomlW3DqEhP3zYD+GqBWkYnJPIZ4tA== + +pg-int8@1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/pg-int8/-/pg-int8-1.0.1.tgz#943bd463bf5b71b4170115f80f8efc9a0c0eb78c" + integrity sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw== + +pg-pool@^3.14.0: + version "3.14.0" + resolved "https://registry.yarnpkg.com/pg-pool/-/pg-pool-3.14.0.tgz#f35ae4eb846780cad71af24099b3edfa9781ad90" + integrity sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw== + +pg-protocol@*, pg-protocol@^1.14.0: + version "1.14.0" + resolved "https://registry.yarnpkg.com/pg-protocol/-/pg-protocol-1.14.0.tgz#c1f045b74274b007078c687147141f785f59b8de" + integrity sha512-n5taZ1kO3s9ngDTVxsEznOqCyToTgz0FLuPq0B33COy5pPpuWJpY3/2oRBVETuOgzdqRXfWpM9HIhp2LBBT1BA== + +pg-query-stream@^4.5.3: + version "4.15.0" + resolved "https://registry.yarnpkg.com/pg-query-stream/-/pg-query-stream-4.15.0.tgz#37ab9ed7eb36a15149d25d9a06b3e7fe7a3fbb6e" + integrity sha512-hyCs0PaOyCWqC90N9vyHL2wVNyR3OjnqFrLvgX74Pyh0JihTKUywpPyhKnuVp9TCCcGz7Fc9LdtxoBU+3nv10A== + dependencies: + pg-cursor "^2.20.0" + +pg-types@2.2.0, pg-types@^2.2.0: + version "2.2.0" + resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-2.2.0.tgz#2d0250d636454f7cfa3b6ae0382fdfa8063254a3" + integrity sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA== + dependencies: + pg-int8 "1.0.1" + postgres-array "~2.0.0" + postgres-bytea "~1.0.0" + postgres-date "~1.0.4" + postgres-interval "^1.1.0" + +pg@^8.11.3: + version "8.21.0" + resolved "https://registry.yarnpkg.com/pg/-/pg-8.21.0.tgz#d7fa2118d960cec5cc7d2b24525f9850dd5932b0" + integrity sha512-AUP1EYJuHraQGsVoCQVIcM7TEJVGtDzxWtGFZd8rds9d+CCXlU5Js1rYgfLNvxy9iJrpHjGrRjoi/3BT9fRyiA== + dependencies: + pg-connection-string "^2.13.0" + pg-pool "^3.14.0" + pg-protocol "^1.14.0" + pg-types "2.2.0" + pgpass "1.0.5" + optionalDependencies: + pg-cloudflare "^1.4.0" + +pgpass@1.0.5: + version "1.0.5" + resolved "https://registry.yarnpkg.com/pgpass/-/pgpass-1.0.5.tgz#9b873e4a564bb10fa7a7dbd55312728d422a223d" + integrity sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug== + dependencies: + split2 "^4.1.0" + picocolors@^1.1.1: version "1.1.1" resolved "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz#3d321af3eab939b083c8f929a1d12cda81c26b6b" @@ -3338,6 +3420,28 @@ posix-character-classes@^0.1.0: resolved "https://registry.yarnpkg.com/posix-character-classes/-/posix-character-classes-0.1.1.tgz#01eac0fe3b5af71a2a6c02feabb8c1fef7e00eab" integrity "sha1-AerA/jta9xoqbAL+q7jB/vfgDqs= sha512-xTgYBc3fuo7Yt7JbiuFxSYGToMoz8fLoE6TC9Wx1P/u+LfeThMOAqmuyECnlBaaJb+u1m9hHiXUEtwW4OzfUJg==" +postgres-array@~2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-2.0.0.tgz#48f8fce054fbc69671999329b8834b772652d82e" + integrity sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA== + +postgres-bytea@~1.0.0: + version "1.0.1" + resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-1.0.1.tgz#c40b3da0222c500ff1e51c5d7014b60b79697c7a" + integrity sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ== + +postgres-date@~1.0.4: + version "1.0.7" + resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-1.0.7.tgz#51bc086006005e5061c591cee727f2531bf641a8" + integrity sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q== + +postgres-interval@^1.1.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-1.2.0.tgz#b460c82cb1587507788819a06aa0fffdb3544695" + integrity sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ== + dependencies: + xtend "^4.0.0" + prelude-ls@~1.1.2: version "1.1.2" resolved "https://registry.yarnpkg.com/prelude-ls/-/prelude-ls-1.1.2.tgz#21932a549f5e52ffd9a827f570e04be62a97da54" @@ -3882,6 +3986,11 @@ split-string@^3.0.1, split-string@^3.0.2: dependencies: extend-shallow "^3.0.0" +split2@^4.1.0: + version "4.2.0" + resolved "https://registry.yarnpkg.com/split2/-/split2-4.2.0.tgz#c9c5920904d148bab0b9f67145f245a86aadbfa4" + integrity sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg== + sprintf-js@~1.0.2: version "1.0.3" resolved "https://registry.yarnpkg.com/sprintf-js/-/sprintf-js-1.0.3.tgz#04e6926f662895354f3dd015203633b857297e2c" @@ -4638,6 +4747,11 @@ xmlcreate@^2.0.4: resolved "https://registry.yarnpkg.com/xmlcreate/-/xmlcreate-2.0.4.tgz#0c5ab0f99cdd02a81065fa9cd8f8ae87624889be" integrity "sha1-DFqw+ZzdAqgQZfqc2Piuh2JIib4= sha512-nquOebG4sngPmGPICTS5EnxqhKbCmz5Ox5hsszI2T6U5qdrJizBc+0ilYSEjTSzU0yZcmvppztXe/5Al5fUwdg==" +xtend@^4.0.0: + version "4.0.2" + resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" + integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== + y18n@^5.0.5: version "5.0.8" resolved "https://registry.yarnpkg.com/y18n/-/y18n-5.0.8.tgz#7f4934d0f7ca8c56f95314939ddcd2dd91ce1d55" From 430c88c42f4d089dd9d26755fbeb7cdfe130773b Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 12:22:19 -0500 Subject: [PATCH 17/20] =?UTF-8?q?refactor(postgres-adapter):=20relocate=20?= =?UTF-8?q?api/=20=E2=86=92=20cli/api/=20+=20update=20imports=20(Phase=202?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 of `adapter/postgres-first-class` per docs/postgres_first_class_design.md (Phases 1-2 inherited verbatim from docs/postgres_reintegration_assessment.md). Moves the restored Postgres adapter files into the modern Dataform/sqlanvil CLI layout: api/dbadapters/postgres.ts → cli/api/dbadapters/postgres.ts api/utils/postgres.ts → cli/api/utils/postgres.ts Updated imports (mechanical) to point at the current `cli/api/...` modules: sa/api/commands/credentials → sa/cli/api/commands/credentials sa/api/dbadapters/index → sa/cli/api/dbadapters/index sa/api/utils/postgres → sa/cli/api/utils/postgres sa/api/utils/results → sa/cli/api/utils/results sa/core/adapters → sa/cli/api/dbadapters/execution_sql Also dropped the (now-broken) imports for symbols that were deleted upstream when non-BQ warehouses were removed: sa/api/ssh_tunnel_proxy — gone, deleted via commit b4d2b2f2 sa/api/utils/error_parsing — gone (parseRedshiftEvalError) The file still references `SSHTunnelProxy` and `parseRedshiftEvalError` in the body; that's fine — file is **excluded from the BUILD glob** in both `cli/api/BUILD` and `cli/api/utils/BUILD` until Phase 3 does the proper interface alignment (executeRaw, deleteTable, schemas signature, sqlanvil.IJDBC → new PostgresConnection proto, SSH removal, error formatter swap). Verified: ./scripts/docker-bazel build //... → 139 targets, success Empty root-level api/ directory removed. Next: Phase 3a — implement IDbAdapter against the relocated file. --- cli/api/BUILD | 5 +++++ {api => cli/api}/dbadapters/postgres.ts | 10 ++++------ cli/api/utils/BUILD | 8 +++++++- {api => cli/api}/utils/postgres.ts | 2 +- 4 files changed, 17 insertions(+), 8 deletions(-) rename {api => cli/api}/dbadapters/postgres.ts (95%) rename {api => cli/api}/utils/postgres.ts (98%) diff --git a/cli/api/BUILD b/cli/api/BUILD index ce1504de..91d1c969 100644 --- a/cli/api/BUILD +++ b/cli/api/BUILD @@ -11,6 +11,11 @@ ts_library( exclude = [ "utils/**/*.*", "**/*_test.ts", + # Postgres adapter — relocated from api/ in Phase 2. + # Excluded until Phase 3 (interface alignment + proto IJDBC → + # PostgresConnection rewrite) makes it compile against the + # current IDbAdapter contract. + "dbadapters/postgres.ts", ], ), deps = [ diff --git a/api/dbadapters/postgres.ts b/cli/api/dbadapters/postgres.ts similarity index 95% rename from api/dbadapters/postgres.ts rename to cli/api/dbadapters/postgres.ts index 541f005e..975b248a 100644 --- a/api/dbadapters/postgres.ts +++ b/cli/api/dbadapters/postgres.ts @@ -1,12 +1,10 @@ import * as pg from "pg"; -import { Credentials } from "sa/api/commands/credentials"; -import { IDbAdapter, IDbClient } from "sa/api/dbadapters/index"; -import { SSHTunnelProxy } from "sa/api/ssh_tunnel_proxy"; -import { parseRedshiftEvalError } from "sa/api/utils/error_parsing"; -import { convertFieldType, PgPoolExecutor } from "sa/api/utils/postgres"; +import { Credentials } from "sa/cli/api/commands/credentials"; +import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters/index"; +import { collectEvaluationQueries, QueryOrAction } from "sa/cli/api/dbadapters/execution_sql"; +import { convertFieldType, PgPoolExecutor } from "sa/cli/api/utils/postgres"; import { ErrorWithCause } from "sa/common/errors/errors"; -import { collectEvaluationQueries, QueryOrAction } from "sa/core/adapters"; import { sqlanvil } from "sa/protos/ts"; interface IPostgresAdapterOptions { diff --git a/cli/api/utils/BUILD b/cli/api/utils/BUILD index 42393bcd..8f68d608 100644 --- a/cli/api/utils/BUILD +++ b/cli/api/utils/BUILD @@ -4,7 +4,13 @@ load("//tools:ts_library.bzl", "ts_library") ts_library( name = "utils", - srcs = glob(["**/*.ts"]), + srcs = glob( + ["**/*.ts"], + # Postgres helper — relocated from api/utils/ in Phase 2. + # Excluded until Phase 3 wires it through cli/api's BUILD with + # the right pg/pg-query-stream deps and an updated import surface. + exclude = ["postgres.ts"], + ), deps = [ "//common/protos", "//common/strings", diff --git a/api/utils/postgres.ts b/cli/api/utils/postgres.ts similarity index 98% rename from api/utils/postgres.ts rename to cli/api/utils/postgres.ts index 79755e24..8d05ffe1 100644 --- a/api/utils/postgres.ts +++ b/cli/api/utils/postgres.ts @@ -1,7 +1,7 @@ import * as pg from "pg"; import QueryStream from "pg-query-stream"; -import { LimitedResultSet } from "sa/api/utils/results"; +import { LimitedResultSet } from "sa/cli/api/utils/results"; import { sqlanvil } from "sa/protos/ts"; const maybeInitializePg = (() => { From f582416f7437d0aab3aeb9368495985da0fd6239 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 12:31:52 -0500 Subject: [PATCH 18/20] feat(protos): add Postgres + Supabase config messages (Phase 3c) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the proto messages that the Postgres-first-class adapter needs, per docs/postgres_first_class_design.md §3 + §8.2: Action-level option blocks: - PostgresOptions — tablespace, fillfactor, unlogged, native declarative partitioning, btree/gin/gist/hash/ brin indexes, materialized view refresh policy - SupabaseOptions — Realtime publication, RLS enable, owner role, pgvector convenience config; nests PostgresOptions Top-level connection config: - BigQueryConnection — project, location, default_dataset - PostgresConnection — host, port, database, user, password, ssl_mode, default_schema (libpq-style) - SupabaseConnection — project_ref, service_role_key, default_schema, optional direct connection_string override - WarehouseConfig — discriminated union over the three connection variants via `oneof connection` This PR is **additive only**. Nothing references these messages yet — wiring into ActionConfig sub-messages (TableConfig, IncrementalTableConfig, etc.) and WorkflowSettings happens in subsequent phases: - Phase 3a PostgresDbAdapter consumes PostgresConnection - Phase 4 WorkflowSettings gets a `warehouse: WarehouseConfig` field; CLI dispatches on warehouse.kind - Phase 5 SupabaseDbAdapter + rlsPolicy/realtimePublication/wrapper/ vectorIndex action types consume SupabaseOptions Verified: ./scripts/docker-bazel build //protos:ts → success ./scripts/docker-bazel build //... → success (139 targets) --- protos/configs.proto | 185 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 185 insertions(+) diff --git a/protos/configs.proto b/protos/configs.proto index 57c90b88..e50b38af 100644 --- a/protos/configs.proto +++ b/protos/configs.proto @@ -744,3 +744,188 @@ message NotebookRuntimeOptionsConfig { } } + +// ============================================================================= +// Postgres-first-class adapter — additions per +// docs/postgres_first_class_design.md. Phase 3c. +// +// These messages are declared here for wiring in subsequent phases: +// - PostgresOptions / SupabaseOptions → ActionConfig table-level blocks +// - PostgresConnection / SupabaseConnection / BigQueryConnection +// → WorkflowSettings.warehouse +// - WarehouseConfig → discriminated union over the +// connection variants +// +// Adding the messages alone does not change behavior — wiring happens in +// Phase 4 (CLI) and the parallel rewrite of ActionConfig's table/view/ +// incremental_table sub-messages. +// ============================================================================= + +// PostgresOptions — Postgres-native table-level options. Mirrors what +// BigQueryOptions-style fields do in TableConfig but in idiomatic Postgres. +// +// Used as a peer of the existing `bigquery: {...}` shape on action configs: +// publish("daily_orders", { postgres: { tablespace: "fast_ssd", ... } }) +message PostgresOptions { + // Physical storage placement (CREATE TABLE ... TABLESPACE ). + string tablespace = 1; + + // Storage parameter — fraction of each page to fill on insert (1-100). + uint32 fillfactor = 2; + + // CREATE UNLOGGED TABLE — faster writes, lost on crash. For staging/temp + // tables where durability isn't required. + bool unlogged = 3; + + // Native Postgres declarative partitioning. + message Partition { + enum Kind { + RANGE = 0; + LIST = 1; + HASH = 2; + } + Kind kind = 1; + repeated string columns = 2; + } + Partition partition = 4; + + // Indexes to create alongside the table. + message Index { + string name = 1; + repeated string columns = 2; + + enum Method { + BTREE = 0; + HASH = 1; + GIN = 2; + GIST = 3; + BRIN = 4; + } + Method method = 3; + + // Partial index predicate (WHERE ). + string where = 4; + + bool unique = 5; + + // INCLUDE non-key columns for covering indexes. + repeated string include = 6; + } + repeated Index indexes = 5; + + // Materialized view: WITH DATA vs WITH NO DATA on initial creation. + bool with_data = 6; + + // Materialized view refresh policy. "manual" | "on_dependency_change". + string refresh_policy = 7; +} + +// SupabaseOptions — Supabase-specific platform features layered on top of +// standard Postgres. Used as a peer of `postgres: {...}` for projects +// targeting `warehouse: { kind: supabase }`. +message SupabaseOptions { + // Standard Postgres options apply. Set these via `postgres:` directly or + // nest under `supabase.postgres:` — either is accepted. + PostgresOptions postgres = 1; + + // ALTER PUBLICATION supabase_realtime ADD TABLE . + // Implicitly sets REPLICA IDENTITY appropriately. + bool publish_to_realtime = 2; + + // ALTER TABLE ENABLE ROW LEVEL SECURITY. + // Note: only enables RLS — policies are declared via the `rlsPolicy` + // action type (see Phase 5). + bool enable_rls = 3; + + // OWNER TO . Typically "postgres" or "service_role". + string owner_role = 4; + + // pgvector convenience config. Equivalent to declaring a + // PostgresOptions.Index with method=HNSW or method=GIST + ivfflat ops, + // but more ergonomic for RAG pipelines. + message VectorConfig { + string column = 1; + uint32 dimensions = 2; + + enum IndexType { + IVFFLAT = 0; + HNSW = 1; + } + IndexType index_type = 3; + + // ivfflat: { lists }, hnsw: { m, ef_construction }. + map params = 4; + } + repeated VectorConfig vectors = 5; +} + +// BigQueryConnection — connection params for warehouse.kind = "bigquery". +// Mirrors the legacy flat fields on WorkflowSettings (default_project, +// default_location, default_dataset) but namespaced under warehouse. +message BigQueryConnection { + // The Google Cloud project (database). + string project = 1; + + // BigQuery location, e.g. "US", "EU", "europe-west4". + string location = 2; + + // Default dataset (schema). + string default_dataset = 3; +} + +// PostgresConnection — libpq-style connection params for +// warehouse.kind = "postgres". Standard Postgres host/port/database/user. +message PostgresConnection { + string host = 1; + uint32 port = 2; + string database = 3; + string user = 4; + string password = 5; + + // SSL mode: "disable" | "allow" | "prefer" | "require" | "verify-ca" + // | "verify-full". See https://www.postgresql.org/docs/current/libpq-ssl.html. + string ssl_mode = 6; + + string default_schema = 7; +} + +// SupabaseConnection — connection params for warehouse.kind = "supabase". +// Supabase projects expose a Postgres connection via project_ref + +// service_role_key, or a direct connection string for bypassing PostgREST. +message SupabaseConnection { + // From the Supabase dashboard (project URL host before .supabase.co). + string project_ref = 1; + + // Project service-role JWT. NEVER commit literally — use ${ENV_VAR} + // interpolation in workflow_settings.yaml. + string service_role_key = 2; + + string default_schema = 3; + + // Optional override — direct Postgres URL bypassing the PostgREST proxy. + // e.g. "postgresql://postgres:${PASSWORD}@db..supabase.co:5432/postgres". + // If set, takes precedence over project_ref + service_role_key for the + // direct DB connection. service_role_key is still used for RLS bypass. + string connection_string = 4; +} + +// WarehouseConfig — discriminated union over connection variants. The +// `kind:` YAML tag selects which `oneof` arm is unmarshalled. +// +// Example YAML: +// warehouse: +// kind: postgres +// host: db.example.com +// port: 5432 +// database: analytics +// user: sqlanvil_writer +// password: ${PG_PASSWORD} +// ssl_mode: require +// default_schema: public +message WarehouseConfig { + oneof connection { + BigQueryConnection bigquery = 1; + PostgresConnection postgres = 2; + SupabaseConnection supabase = 3; + } +} From 4463a858fe0b9e1bd7f5df7ec0b4970ba2dc4664 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 12:47:06 -0500 Subject: [PATCH 19/20] feat(postgres-adapter): implement IDbAdapter contract (Phase 3a) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites cli/api/dbadapters/postgres.ts so it compiles against the modern IDbAdapter interface, and wires the BUILD targets so the file is part of the //cli/api library again (drops the Phase 2 excludes). Interface alignment per docs/postgres_first_class_design.md §6 + docs/postgres_reintegration_assessment.md §Phase 3: - credentials sqlanvil.IJDBC → sqlanvil.IPostgresConnection (new proto from Phase 3c) - execute() returns IExecutionResult { rows, metadata } — was raw any[] - executeRaw() new, returns IExecutionResultRaw { rows, schema, metadata } — schema kept empty (we lean on the catalog query path for shape) - withClientLock() callback receives an IDbClient with both execute() and executeRaw() shimmed over the underlying pg pool client - schemas(database) now takes the database arg (ignored, since Postgres is database-scoped), and filters pg_catalog / pg_toast / pg_internal / information_schema - tables(database, schema) returns ITableMetadata[] (was ITarget[]) — hydrates full metadata per row. Optional schema filter is supported. - deleteTable(target) new — transactional `drop table|view if exists ... cascade` keyed on the actual catalog type - setMetadata() guards against null actualMetadata (table doesn't exist yet) instead of crashing Modernization (assessment doc §Phase 5): - Drop SSHTunnelProxy block — Supabase + most managed Postgres serve TLS on a public TCP port; SSH tunnel is no longer needed. The SSHTunnelProxy class was deleted upstream in commit b4d2b2f2. - Drop parseRedshiftEvalError — replaced with parsePostgresEvalError in cli/api/utils/error_parsing.ts. pg errors carry `.position` (byte offset) but not line/column, so we preserve the message only. - Drop preview() — wasn't in IDbAdapter. SSL handling tightened: ssl_mode = "disable" → no TLS ssl_mode = "verify-ca"|"verify-full" → strict cert verification ssl_mode = "" | "require" | "prefer" | "allow" → TLS with rejectUnauthorized=false (matches default Supabase / RDS UX) BUILD wiring: - cli/api/BUILD: drop `dbadapters/postgres.ts` exclude; add `@npm//pg` and `@npm//@types/pg` deps. - cli/api/utils/BUILD: drop `postgres.ts` exclude; add `@npm//pg`, `@npm//@types/pg`, `@npm//pg-query-stream` deps. Verified: ./scripts/docker-bazel build //cli/api/... → 8 targets, success ./scripts/docker-bazel build //... → 139 targets, success ./scripts/docker-bazel test //... → 38/41 pass (same 3 pre-existing failures: 2× missing BQ creds, 1× projects.spec `warehouse` validation issue — none regressed by this PR) Next phase (Phase 3b): Postgres SQL generator under core/compilation_sql/postgres/ — one generator per action type with golden-output tests. --- cli/api/BUILD | 7 +- cli/api/dbadapters/postgres.ts | 224 ++++++++++++++++++--------------- cli/api/utils/BUILD | 11 +- cli/api/utils/error_parsing.ts | 14 +++ 4 files changed, 144 insertions(+), 112 deletions(-) diff --git a/cli/api/BUILD b/cli/api/BUILD index 91d1c969..8871039a 100644 --- a/cli/api/BUILD +++ b/cli/api/BUILD @@ -11,11 +11,6 @@ ts_library( exclude = [ "utils/**/*.*", "**/*_test.ts", - # Postgres adapter — relocated from api/ in Phase 2. - # Excluded until Phase 3 (interface alignment + proto IJDBC → - # PostgresConnection rewrite) makes it compile against the - # current IDbAdapter contract. - "dbadapters/postgres.ts", ], ), deps = [ @@ -38,6 +33,7 @@ ts_library( "@npm//@types/js-yaml", "@npm//@types/long", "@npm//@types/node", + "@npm//@types/pg", "@npm//@types/semver", "@npm//@types/tmp", "@npm//deepmerge", @@ -46,6 +42,7 @@ ts_library( "@npm//google-sql-syntax-ts", "@npm//js-beautify", "@npm//js-yaml", + "@npm//pg", "@npm//promise-pool-executor", "@npm//protobufjs", "@npm//semver", diff --git a/cli/api/dbadapters/postgres.ts b/cli/api/dbadapters/postgres.ts index 975b248a..83fcc932 100644 --- a/cli/api/dbadapters/postgres.ts +++ b/cli/api/dbadapters/postgres.ts @@ -1,105 +1,110 @@ import * as pg from "pg"; -import { Credentials } from "sa/cli/api/commands/credentials"; -import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters/index"; import { collectEvaluationQueries, QueryOrAction } from "sa/cli/api/dbadapters/execution_sql"; +import { + IDbAdapter, + IDbClient, + IExecutionResult, + IExecutionResultRaw, + OnCancel +} from "sa/cli/api/dbadapters/index"; +import { parsePostgresEvalError } from "sa/cli/api/utils/error_parsing"; import { convertFieldType, PgPoolExecutor } from "sa/cli/api/utils/postgres"; import { ErrorWithCause } from "sa/common/errors/errors"; import { sqlanvil } from "sa/protos/ts"; -interface IPostgresAdapterOptions { - sshTunnel?: SSHTunnelProxy; -} +const INTERNAL_SCHEMAS = new Set(["information_schema", "pg_catalog", "pg_internal", "pg_toast"]); export class PostgresDbAdapter implements IDbAdapter { public static async create( - credentials: Credentials, + credentials: sqlanvil.IPostgresConnection, options?: { concurrencyLimit?: number; disableSslForTestsOnly?: boolean } - ) { - const jdbcCredentials = credentials as sqlanvil.IJDBC; - const baseClientConfig: Partial = { - user: jdbcCredentials.username, - password: jdbcCredentials.password, - database: jdbcCredentials.databaseName, - ssl: options?.disableSslForTestsOnly - ? false - : { - rejectUnauthorized: false, - ca: jdbcCredentials.ssl?.serverCertificate, - cert: jdbcCredentials.ssl?.clientCertificate, - key: jdbcCredentials.ssl?.clientPrivateKey + ): Promise { + const sslMode = (credentials.sslMode || "").toLowerCase(); + const sslEnabled = !options?.disableSslForTestsOnly && sslMode !== "disable"; + const clientConfig: pg.ClientConfig = { + host: credentials.host, + port: credentials.port, + database: credentials.database, + user: credentials.user, + password: credentials.password, + ssl: sslEnabled + ? { + // Supabase and most managed Postgres providers serve certs signed + // by their own CA. Skipping verification is the documented path + // for `sslmode=require`. Stricter `verify-ca` / `verify-full` + // requires a CA bundle that we don't ship today. + rejectUnauthorized: sslMode === "verify-ca" || sslMode === "verify-full" } + : false }; - if (jdbcCredentials.sshTunnel) { - const sshTunnel = await SSHTunnelProxy.create(jdbcCredentials.sshTunnel, { - host: jdbcCredentials.host, - port: jdbcCredentials.port - }); - const queryExecutor = new PgPoolExecutor( - { - ...baseClientConfig, - host: "127.0.0.1", - port: sshTunnel.localPort - }, - options - ); - return new PostgresDbAdapter(queryExecutor, { sshTunnel }); - } else { - const clientConfig: pg.ClientConfig = { - ...baseClientConfig, - host: jdbcCredentials.host, - port: jdbcCredentials.port - }; - const queryExecutor = new PgPoolExecutor(clientConfig, options); - return new PostgresDbAdapter(queryExecutor, {}); - } + const queryExecutor = new PgPoolExecutor(clientConfig, options); + return new PostgresDbAdapter(queryExecutor); } - private constructor( - private readonly queryExecutor: PgPoolExecutor, - private readonly options: IPostgresAdapterOptions - ) {} + private constructor(private readonly queryExecutor: PgPoolExecutor) {} public async execute( statement: string, options: { params?: any[]; - onCancel?: (handleCancel: () => void) => void; + onCancel?: OnCancel; rowLimit?: number; byteLimit?: number; includeQueryInError?: boolean; } = { rowLimit: 1000, byteLimit: 1024 * 1024 } - ) { - return await this.withClientLock(executor => executor.execute(statement, options)); + ): Promise { + return await this.withClientLock(client => client.execute(statement, options)); + } + + public async executeRaw( + statement: string, + options: { + params?: any[]; + rowLimit?: number; + } = { rowLimit: 1000 } + ): Promise { + const result = await this.execute(statement, options); + return { ...result, schema: [] }; } - public async withClientLock(callback: (client: IDbClient) => Promise) { + public async withClientLock(callback: (client: IDbClient) => Promise): Promise { return await this.queryExecutor.withClientLock(client => callback({ execute: async ( - statement: string, - options: { + stmt: string, + opts: { params?: any[]; + onCancel?: OnCancel; rowLimit?: number; byteLimit?: number; includeQueryInError?: boolean; } = { rowLimit: 1000, byteLimit: 1024 * 1024 } - ) => { + ): Promise => { try { - const rows = await client.execute(statement, options); + const rows = await client.execute(stmt, opts); return { rows, metadata: {} }; } catch (e) { - if (options.includeQueryInError) { - throw new Error(`Error encountered while running "${statement}": ${e.message}`); + if (opts.includeQueryInError) { + throw new Error(`Error encountered while running "${stmt}": ${e.message}`); } throw new ErrorWithCause(`Error executing postgres query: ${e.message}`, e); } + }, + executeRaw: async ( + stmt: string, + opts: { params?: { [name: string]: any }; rowLimit?: number } = { rowLimit: 1000 } + ): Promise => { + // Convert named param object to positional array — pg uses $1, $2 etc. + const positional = opts.params ? Object.values(opts.params) : undefined; + const rows = await client.execute(stmt, { params: positional, rowLimit: opts.rowLimit }); + return { rows, schema: [], metadata: {} }; } }) ); } - public async evaluate(queryOrAction: QueryOrAction) { + public async evaluate(queryOrAction: QueryOrAction): Promise { const validationQueries = collectEvaluationQueries(queryOrAction, false, (query: string) => !!query ? `explain ${query}` : "" ).map((validationQuery, index) => ({ index, validationQuery })); @@ -115,7 +120,7 @@ export class PostgresDbAdapter implements IDbAdapter { } catch (e) { evaluationResponse = { status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE, - error: parseRedshiftEvalError(validationQuery.query, e) + error: parsePostgresEvalError(validationQuery.query, e) }; } queryEvaluations.push( @@ -129,33 +134,45 @@ export class PostgresDbAdapter implements IDbAdapter { return queryEvaluations; } - public async tables(): Promise { + public async tables( + _database: string, + schema?: string + ): Promise { + const params: any[] = []; + let schemaClause = ""; + if (schema) { + schemaClause = "and table_schema = $1"; + params.push(schema); + } const queryResult = await this.execute( `select table_name, table_schema - from information_schema.tables - where table_schema != 'information_schema' - and table_schema != 'pg_catalog' - and table_schema != 'pg_internal'`, - { rowLimit: 10000, includeQueryInError: true } + from information_schema.tables + where table_schema not in ('information_schema', 'pg_catalog', 'pg_internal', 'pg_toast') + ${schemaClause}`, + { params, rowLimit: 10000, includeQueryInError: true } ); - const { rows } = queryResult; - return rows.map(row => ({ - schema: row.table_schema, - name: row.table_name + const targets = queryResult.rows.map(row => ({ + schema: row.table_schema as string, + name: row.table_name as string })); + // Hydrate full metadata for each target — IDbAdapter.tables returns + // ITableMetadata[], not ITarget[]. + return await Promise.all(targets.map(target => this.table(target))); } public async search( searchText: string, options: { limit: number } = { limit: 1000 } ): Promise { - // TODO: It would be nice to extend this to search through table/column descriptions. However, this involves - // a somewhat crazy 5-way join. const results = await this.execute( `select tables.table_schema as table_schema, tables.table_name as table_name from information_schema.tables as tables - left join information_schema.columns columns on tables.table_schema = columns.table_schema and tables.table_name = columns.table_name - where tables.table_schema ilike $1 or tables.table_name ilike $1 or columns.column_name ilike $1 + left join information_schema.columns columns + on tables.table_schema = columns.table_schema + and tables.table_name = columns.table_name + where tables.table_schema ilike $1 + or tables.table_name ilike $1 + or columns.column_name ilike $1 group by 1, 2`, { params: [`%${searchText}%`], @@ -186,13 +203,13 @@ export class PostgresDbAdapter implements IDbAdapter { { params, includeQueryInError: true } ), this.execute( - ` - select objsubid as column_number, description from pg_description - where objoid = ( - select oid from pg_class where relname = $2 and relnamespace = ( - select oid from pg_namespace where nspname = $1 - ) - )`, + `select objsubid as column_number, description + from pg_description + where objoid = ( + select oid from pg_class where relname = $2 and relnamespace = ( + select oid from pg_namespace where nspname = $1 + ) + )`, { params, includeQueryInError: true } ) ]); @@ -220,38 +237,42 @@ export class PostgresDbAdapter implements IDbAdapter { }); } - public async preview(target: sqlanvil.ITarget, limitRows: number = 10): Promise { - const { rows } = await this.execute( - `SELECT * FROM "${target.schema}"."${target.name}" LIMIT ${limitRows}` + public async deleteTable(target: sqlanvil.ITarget): Promise { + const metadata = await this.table(target); + if (!metadata) { + return; + } + const kind = metadata.type === sqlanvil.TableMetadata.Type.VIEW ? "view" : "table"; + await this.execute( + `drop ${kind} if exists "${target.schema}"."${target.name}" cascade`, + { includeQueryInError: true } ); - return rows; } - public async schemas(): Promise { - const schemas = await this.execute(`select nspname from pg_namespace`, { + public async schemas(_database: string): Promise { + const result = await this.execute(`select nspname from pg_namespace`, { includeQueryInError: true }); - return schemas.rows.map(row => row.nspname); - } - - public async createSchema(_: string, schema: string): Promise { - await this.execute(`create schema if not exists "${schema}"`, { includeQueryInError: true }); + return result.rows + .map(row => row.nspname as string) + .filter(name => !INTERNAL_SCHEMAS.has(name) && !name.startsWith("pg_")); } - public async close() { - await this.queryExecutor.close(); - if (this.options.sshTunnel) { - await this.options.sshTunnel.close(); - } + public async createSchema(_database: string, schema: string): Promise { + await this.execute(`create schema if not exists "${schema}"`, { + includeQueryInError: true + }); } public async setMetadata(action: sqlanvil.IExecutionAction): Promise { const { target, actionDescriptor, tableType } = action; - const actualMetadata = await this.table(target); + if (!actualMetadata) { + return; + } - const queries: Array> = []; - if (actionDescriptor.description) { + const queries: Array> = []; + if (actionDescriptor?.description) { queries.push( this.execute( `comment on ${tableType === "view" ? "view" : "table"} "${target.schema}"."${ @@ -260,7 +281,7 @@ export class PostgresDbAdapter implements IDbAdapter { ) ); } - if (actionDescriptor.columns?.length > 0) { + if (actionDescriptor?.columns?.length > 0) { actionDescriptor.columns .filter( column => @@ -277,7 +298,10 @@ export class PostgresDbAdapter implements IDbAdapter { ); }); } - await Promise.all(queries); } + + public async close(): Promise { + await this.queryExecutor.close(); + } } diff --git a/cli/api/utils/BUILD b/cli/api/utils/BUILD index 8f68d608..5202bc77 100644 --- a/cli/api/utils/BUILD +++ b/cli/api/utils/BUILD @@ -4,19 +4,16 @@ load("//tools:ts_library.bzl", "ts_library") ts_library( name = "utils", - srcs = glob( - ["**/*.ts"], - # Postgres helper — relocated from api/utils/ in Phase 2. - # Excluded until Phase 3 wires it through cli/api's BUILD with - # the right pg/pg-query-stream deps and an updated import surface. - exclude = ["postgres.ts"], - ), + srcs = glob(["**/*.ts"]), deps = [ "//common/protos", "//common/strings", "//core", "//protos:ts", "@npm//@types/node", + "@npm//@types/pg", "@npm//object-sizeof", + "@npm//pg", + "@npm//pg-query-stream", ], ) diff --git a/cli/api/utils/error_parsing.ts b/cli/api/utils/error_parsing.ts index e8876bb7..81de60b4 100644 --- a/cli/api/utils/error_parsing.ts +++ b/cli/api/utils/error_parsing.ts @@ -4,6 +4,20 @@ interface IBigqueryEvaluationError { message?: string; } +interface IPostgresEvaluationError { + message?: string; +} + +// Postgres-specific error parser. `pg` raises errors with `.message` plus +// optional `.position` (byte offset into the query). We don't try to map +// byte offset back to (line, column) — not all callers can supply the +// original query — so we just preserve the message. +export function parsePostgresEvalError(_query: string, error: IPostgresEvaluationError) { + return sqlanvil.QueryEvaluationError.create({ + message: error?.message ? String(error.message) : String(error) + }); +} + export function parseBigqueryEvalError(error: IBigqueryEvaluationError) { // expected error format: // e.message = Syntax error: Unexpected identifier "asda" at [2:1] From 7b97bf21f6120afd89cdaa9e6a74bb55bb6de953 Mon Sep 17 00:00:00 2001 From: Ivan Histand Date: Wed, 27 May 2026 13:07:06 -0500 Subject: [PATCH 20/20] docs: docs site sourcing & build plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the legal + structural plan for sqlanvil's documentation site: - Tier 1 source: upstream/main:docs/ — Apache 2.0, already inherited - Tier 2 source: docs.cloud.google.com/dataform — CC BY 4.0 - Tier 3: do-not-pull list (cloud.google.com marketing, screenshots, etc.) License compliance worked out for both tiers — Apache 2.0 covered by extending the existing NOTICE; CC BY 4.0 needs per-page attribution + ATTRIBUTIONS.md. Phased doc roadmap aligned to code phases: - D1: pull Tier 1 source as-is, rebrand - D2: rewrite reference pages Postgres-first - D3: net-new Supabase action type pages - D4: quickstart + CLI ref adapted from Tier 2 - D5: architecture & concept pages (mostly original) Site stack recommendation: Astro Starlight on Vercel (docs.sqlanvil.com), sibling repo ../sqlanvil-docs/ so doc deploys don't trigger code CI. Trademark scrubbing (dataform → sqlanvil) treated as separate concern from copyright — same playbook as the code rename PR (#1). --- docs/docs_site_plan.md | 246 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 246 insertions(+) create mode 100644 docs/docs_site_plan.md diff --git a/docs/docs_site_plan.md b/docs/docs_site_plan.md new file mode 100644 index 00000000..d918c6df --- /dev/null +++ b/docs/docs_site_plan.md @@ -0,0 +1,246 @@ +# sqlanvil Docs Site — Sourcing & Build Plan + +**Status:** Draft +**Owner:** Ivan +**Last updated:** 2026-05-27 +**Related:** +- [`postgres_first_class_design.md`](postgres_first_class_design.md) — what the docs need to describe +- [`hybrid_warehouses_supabase_bigquery.md`](hybrid_warehouses_supabase_bigquery.md) — already-drafted reference doc +- Root [`NOTICE`](../NOTICE) — Apache 2.0 attribution already wired up for the code; will extend to docs + +## 0. TL;DR + +Pull the upstream OSS Dataform repo's `docs/` (Apache 2.0, already +inherited via fork) as the foundation. Fill the gaps from +`docs.cloud.google.com/dataform` (CC BY 4.0). Rewrite all BigQuery +examples as Postgres-native per the postgres-first-class design. Ship a +docs site at `docs.sqlanvil.com` (Vercel static) — markdown-driven. + +## 1. Source Hierarchy + +### Tier 1 — Apache 2.0 markdown source (preferred) + +Already in upstream `dataform-co/dataform` repo at `docs/`: + +``` +docs/configs-reference.md +docs/packages.md +docs/reference/assertion.md +docs/reference/configs.md +docs/reference/declaration.md +docs/reference/incrementaltable.md +docs/reference/notebook.md +docs/reference/operation.md +docs/reference/session.md +docs/reference/table.md +docs/reference/test.md +docs/reference/view.md +``` + +- License: **Apache 2.0** (matches code license + existing NOTICE) +- Already inherited via fork +- Markdown source — easier to `git merge upstream/main -- docs/` +- Code samples already in code blocks (not embedded HTML) + +### Tier 2 — Google Cloud web docs (CC BY 4.0) + +`docs.cloud.google.com/dataform/docs/*` — for content that doesn't ship +in the OSS repo: + +- Quickstart / getting-started walkthrough +- CLI reference (full flag tables) +- Troubleshooting guide +- Locations / quotas / billing pages (sqlanvil-irrelevant — skip) +- Release notes (sqlanvil writes its own) +- Tutorial walkthroughs + +Footer of every page says: +> "Except as otherwise noted, the content of this page is licensed +> under the [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/), +> and code samples are licensed under the Apache 2.0 License" + +### Tier 3 — Do not pull + +- `cloud.google.com/dataform/*` (marketing pages — different terms, not + CC BY 4.0) +- GCP console screenshots (Google trademark + UI copyright) +- Anything behind login (TOS may restrict redistribution) +- Google's internal repos (not public anyway) + +## 2. License Compliance + +### Apache 2.0 (Tier 1 source) + +Already handled by root `NOTICE`. When pulling docs: + +1. Preserve copyright notices from any code samples. +2. Add a `NOTICE` entry for the docs in the root NOTICE: + ``` + This product includes documentation adapted from Google's Dataform + open source project (https://github.com/dataform-co/dataform), + licensed under the Apache License 2.0. See: + https://github.com/dataform-co/dataform/blob/main/LICENSE + ``` + +### CC BY 4.0 (Tier 2 source) + +Per page or per-section attribution. Recommended placement: + +- At the bottom of each pulled page: + ``` + --- + Adapted from Google Cloud Dataform documentation + (), licensed under CC BY 4.0. Modifications by sqlanvil + contributors to target PostgreSQL / Supabase semantics. + ``` +- In `docs/ATTRIBUTIONS.md`: line-item table of every pulled page, + source URL, snapshot date. + +### Trademark scrubbing (separate from copyright) + +Even with permissive license: +- `Dataform` → `sqlanvil` (entire word mark) +- Remove Google Cloud logos +- Replace GCP console screenshots +- No "Powered by Google" / no implied endorsement (CC BY 4.0 §3(a)(1)(iii)) + +This is the same playbook as the code rename (PR #1). + +## 3. Doc Pages by Phase + +### Phase D1 — Pull Tier 1 source + +Tickets: +- D1.1 — Copy `upstream/main:docs/` into `sqlanvil/docs/reference/` + (some already there — diff first, keep best) +- D1.2 — Rename surface sweep on docs: `dataform` → `sqlanvil`, + `@dataform/` → `@sqlanvil/`, `.df-credentials.json` → `.sa-credentials.json` +- D1.3 — Add `NOTICE` paragraph for inherited docs +- D1.4 — Verify nothing references BQ-only types/concepts unguarded + (cluster_by, NOT ENFORCED PKs, OPTIONS, MERGE) + +**Output:** every reference page exists, sqlanvil-branded, BQ-only +language flagged for D2 rewrite. + +### Phase D2 — Postgres-first reference rewrites + +For each action type, rewrite to be Postgres-first: + +| Page | BigQuery section to gate | Postgres section to add | +|---|---|---| +| `reference/table.md` | partition_by/cluster_by/OPTIONS | `PartitionConfig` (RANGE/LIST/HASH), indexes, tablespace, fillfactor | +| `reference/view.md` | (mostly portable) | Just rebrand | +| `reference/incrementaltable.md` | MERGE-based upsert | `INSERT ... ON CONFLICT (...) DO UPDATE` | +| `reference/operation.md` | (mostly portable) | Rebrand + transaction semantics note | +| `reference/assertion.md` | (mostly portable) | Add note: assertions can compile to CHECK constraints when user opts in | +| `reference/declaration.md` | (mostly portable) | Rebrand | +| `reference/test.md` | (mostly portable) | Rebrand | +| `reference/notebook.md` | GCP-specific — **drop** | Skip — no sqlanvil equivalent | +| `reference/session.md` | (mostly portable) | Rebrand | +| `configs-reference.md` | regenerate from new proto | New: `PostgresOptions`, `SupabaseOptions`, `WarehouseConfig` | + +**Output:** every page has Postgres-first content. BQ content gated +under explicit "BigQuery only" callouts. + +### Phase D3 — Supabase-specific pages (original content) + +No upstream source. Write from scratch: + +- `reference/rls_policy.md` — RLS policies as action types +- `reference/realtime_publication.md` — Realtime publications +- `reference/wrapper.md` — Supabase Wrappers (FDW) +- `reference/vector_index.md` — pgvector convenience action + +Plus concept pages: +- `concepts/supabase_target.md` — why Supabase is first-class +- `concepts/postgres_vs_bigquery.md` — BQ-isms that don't apply in PG + (companion to `postgres_first_class_design.md` §4 table) + +### Phase D4 — Top-of-funnel docs (CC BY 4.0 adaptation) + +Pull structure from `docs.cloud.google.com/dataform`, rewrite content: + +- `quickstart.md` — adapted from Google's quickstart; Postgres example + project instead of BQ +- `cli/reference.md` — CLI flag reference (table format from Google's + CLI ref page; sqlanvil flag values) +- `troubleshooting.md` — common errors (sqlanvil-specific; adapt + problem structure from Google's troubleshooting page) +- `installation.md` — npm install + docker dev container (original) + +### Phase D5 — Architecture & concepts (original) + +Already drafted: +- `hybrid_warehouses_supabase_bigquery.md` → publish as + `concepts/hybrid_warehouses.md` + +To draft: +- `concepts/why_sqlanvil.md` — positioning vs upstream Dataform OSS, + vs dbt, vs raw migrations +- `concepts/action_graph.md` — how actions form a DAG +- `concepts/incremental_strategies.md` — `ON CONFLICT` vs full reload + vs CDC-style append +- `architecture.md` — adapter / SQL generator / CLI layout + +## 4. Docs Site Build + +### Stack + +- **Source:** Markdown in `sqlanvil/docs/` +- **Site generator:** Pick between: + - **Astro Starlight** — TypeScript-friendly, modern, MDX support, + great search. Recommended. + - **Docusaurus** — React-based, more featureful, heavier. + - **VitePress** — Vue-based, lighter than Docusaurus. + - **mdBook** — Rust-based, very fast, simpler. + - **Plain Vercel + remark** — minimal, full control. +- **Host:** Vercel project `docs-sqlanvil-com`, domain `docs.sqlanvil.com` +- **Repo:** Sibling repo `../sqlanvil-docs/` (separate from monolith + `sqlanvil/` so docs deploys don't trigger code CI) + +### CI + +- GitHub Action on PR: build site, deploy preview URL +- Main branch → production +- Markdown lint + link check on every PR + +### Search + +- Algolia DocSearch (free for open source) +- Alternative: client-side Pagefind (Astro Starlight default) + +## 5. Sequencing vs Code PRs + +Doc work can happen in parallel with code Phases 3b-5 but should not +block them. Ordering: + +| Code phase | Doc work that depends on it | +|---|---| +| Phase 3a (adapter) | D2 — references that mention adapter methods | +| Phase 3b (SQL gen) | D2 — every action-type page's "compiled output" examples | +| Phase 4 (CLI wiring) | D4 — CLI reference, quickstart | +| Phase 5 (Supabase) | D3 — Supabase action type pages | + +Phase D1 + D5 can happen anytime — no code dependency. + +## 6. Open Questions + +- **Doc versioning.** If sqlanvil ships v0.x and breaking changes are + likely, do we version the docs? Defer until v1.0; for now, + `latest` only. +- **Hosting cost.** Vercel free tier easily handles a docs site. Algolia + DocSearch is free for OSS. Total cost: $0/mo at sqlanvil's scale. +- **Search index update cadence.** Tied to Vercel deploys — no separate + trigger. + +## 7. Action Items (immediate) + +- [ ] Decide site generator (recommend Astro Starlight) +- [ ] Create sibling repo `../sqlanvil-docs/` +- [ ] Wire Vercel project + domain `docs.sqlanvil.com` +- [ ] Phase D1.1: copy `upstream/main:docs/` into the new repo +- [ ] Add `NOTICE` paragraph + `ATTRIBUTIONS.md` +- [ ] First-pass rename sweep on inherited content + +Phase D1 is mostly mechanical — should fit in a single afternoon. +Subsequent phases scale with code maturity.