diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index a16cc846..ef278756 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1 +1 @@ -* @dataform-co/dataform-reviewers +* @ihistand diff --git a/BUILD b/BUILD index b427536a..df8e303c 100644 --- a/BUILD +++ b/BUILD @@ -52,7 +52,7 @@ nodejs_binary( load("@bazel_gazelle//:def.bzl", "gazelle") -# gazelle:prefix github.com/dataform-co/dataform +# gazelle:prefix github.com/ihistand/sqlanvil # gazelle:proto package # gazelle:proto_group go_package gazelle(name = "gazelle") @@ -64,5 +64,5 @@ load("//tools:ts_library.bzl", "ts_library") ts_library( name = "modules-fix", srcs = [], - module_name = "df", + module_name = "sa", ) diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..ed513357 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,104 @@ +# CLAUDE.md + +Guidance for Claude Code when working in the `sqlanvil/` project. + +## What This Is + +**sqlanvil** is Ivan's fork of [`dataform-co/dataform`](https://github.com/dataform-co/dataform), renamed and being repositioned as an open-source SQL workflow tool that runs against **both BigQuery and PostgreSQL/Supabase** (upstream Dataform OSS dropped Postgres support some time ago). + +- **Upstream**: `git@github.com:dataform-co/dataform.git` (Google's Dataform OSS — low activity since GA in BigQuery) +- **Origin**: `git@github.com:ihistand/sqlanvil.git` +- **Marketing site**: sibling repo `../sqlanvil-com/` — static HTML on Vercel (project: `sqlanvil-com`, team `Zlu36JPJdwPqwMeAWAqISllx`) + +## Stack + +- **Language**: TypeScript +- **Build**: Bazel (via Bazelisk) — old-style `WORKSPACE`, not `MODULE.bazel` +- **Protos**: protobuf (`protos/*.proto`) for core/configs/db_adapter/etc. +- **Target warehouses**: BigQuery (working), PostgreSQL (being reintegrated) +- **No npm `scripts`** in `package.json` — everything runs through Bazel. + +## Layout + +``` +core/ Compiler + action types (table/view/incremental/assertion/operation/notebook/declaration) +cli/ CLI entrypoint (cli/index.ts) and per-adapter glue (cli/api/dbadapters/) +protos/ Protobuf definitions for core/configs/execution/db_adapter +api/ Legacy directory — currently holds restored Postgres adapter files awaiting relocation +tools/ Bazel rules + the Postgres docker test fixture (tools/postgres/postgres_fixture.ts) +tests/ Integration specs (bigquery + postgres) against real warehouses +docs/ Reference + the two new Antigravity design docs (see below) +examples/ Sample Dataform projects +scripts/ `./scripts/run` is the CLI entrypoint wrapper +``` + +## Common Commands + +```bash +# One-time +npm i -g @bazel/bazelisk +sudo sysctl -w kern.maxfiles=65536 # Mac only — Bazel hits the default fd limit + +# Run the CLI (substitute for `dataform` from @dataform/cli) +./scripts/run help +./scripts/run compile path/to/project + +# Tests +bazel test //... # everything +bazel test //core/... # core only +bazel test //cli:index_test # CLI integration (needs GCP creds — see contributing.md) +``` + +## Design Directive — Rename Is Mandatory + +The fork must be fully renamed from `dataform` → `sqlanvil` before public-facing artifacts (npm packages, CLI binary, docs site, marketing) ship. Reason: avoid trademark conflict with Google's "Dataform" product. Scope of rename: + +- Proto package names (`dataform.*` → `sqlanvil.*`) +- npm package names (`@dataform/*` → `@sqlanvil/*` or unscoped `sqlanvil-*`) +- CLI binary (`dataform` → `sqlanvil`) +- Config files (`dataform.json` → `sqlanvil.json`, `workflow_settings.yaml` keys) +- Internal class names referencing `Dataform` +- Docs site references + +The rename is a hard prerequisite, not nice-to-have. Sequence it before — or in parallel with — the Postgres adapter work. + +## Design Directive — Postgres Is First-Class + +The Postgres adapter is **not** a BigQuery adapter with translated SQL. It generates idiomatic Postgres DDL/DML. Typical sqlanvil users may never have touched BigQuery; they should never see BigQuery quirks like `CREATE PRIMARY KEY mykey NOT ENFORCED`, `OPTIONS(...)` table options, `PARTITION BY`/`CLUSTER BY` clauses, or BigQuery's `MERGE` dialect. + +Two adapter variants ship: + +- **`postgres`** — standard Postgres. Idiomatic DDL, `INSERT ... ON CONFLICT` for upserts, native `CREATE INDEX`, tablespaces, fillfactor, partitioning via `PARTITION BY RANGE/LIST/HASH`. +- **`supabase`** — extends `postgres` with Supabase-specific surface area: RLS policies in actions, `auth.users` references, Realtime publications, `pgvector` indexes, `pg_cron` scheduling, Supabase Wrappers (FDW) declarations. + +Implications for the Antigravity reintegration doc: Phase 3 ("Interface Alignment") is **under-scoped**. It frames the work as making the restored adapter implement BigQuery's `IDbAdapter` interface. Real work also includes: + +- Postgres-specific action config blocks (e.g., `postgres: { tablespace, fillfactor, indexes, partition }` parallel to the existing `bigquery: { partitionBy, clusterBy, ... }`). +- A separate Postgres SQL generator path in `core/compilation_sql/` rather than reusing BigQuery's. +- Config schema additions in `protos/configs.proto` for both variants. + +## Active Work — Postgres Reintegration + +Current branch: `restore-postgres-adapter`. Recent commits (Ivan's, on top of upstream): + +1. `a220e2ed` — restored the Postgres adapter files from git history into `api/dbadapters/postgres.ts` and `api/utils/postgres.ts` (won't compile as-is) +2. `fcca60c1` — added `docs/postgres_reintegration_assessment.md` (Antigravity) +3. `1636e275` — added `docs/hybrid_warehouses_supabase_bigquery.md` (Antigravity) + +The two new design docs are Antigravity-authored and **load-bearing for the next sprint**: + +- **`docs/postgres_reintegration_assessment.md`** — 5-phase, ~1-2 day plan to make the restored adapter compile and wire into the CLI. Phases: deps (`pg`, `pg-query-stream`) → relocate `api/` → `cli/api/` → align `IDbAdapter` interface (implement `executeRaw`, `deleteTable`, full `ITableMetadata`) → branch CLI on `projectConfig.warehouse === "postgres"` → Bazel/docker fixture verification. +- **`docs/hybrid_warehouses_supabase_bigquery.md`** — marketing/architecture doc: three patterns for combining Supabase + BigQuery (federated queries, sequential pipeline, Supabase Wrappers / FDW). Reference material, no implementation required. + +When working on Postgres reintegration, **follow the assessment doc's phase order** — deps before relocation before interface work — because each phase's tests depend on the previous one passing through Bazel. + +## Fork Hygiene + +- Upstream changes still merge in cleanly today; the longer Ivan diverges (renames, Postgres adapter, future Supabase-specific features), the harder this gets. When pulling upstream, prefer rebasing feature branches onto `upstream/main` over merge commits to keep the history readable. +- Renaming Dataform → sqlanvil should be done in one sweep (package names, proto packages, CLI binary, docs) rather than incrementally — partial renames create grep ambiguity. + +## Things To Know + +- The `api/` directory at the repo root is **legacy**. The active CLI/adapter layout lives under `cli/api/`. Restored Postgres files are in the legacy location and need to move (see Phase 2 of the assessment doc). +- Integration tests need real warehouses: BigQuery creds in `test_credentials/bigquery.json`, Postgres via Docker container started by `tools/postgres/postgres_fixture.ts` inside the Bazel sandbox. +- The fork still references `@dataform/...` in many places — rename surface area is large. diff --git a/Dockerfile.dev b/Dockerfile.dev new file mode 100644 index 00000000..7a27624d --- /dev/null +++ b/Dockerfile.dev @@ -0,0 +1,55 @@ +# Dev container for building sqlanvil on Linux. +# +# Why: the pinned Bazel 5.4 + 2022-era rules_proto/protobuf chain does not +# build on macOS Tahoe (wrapped_clang missing LC_UUID, Apple SDK header +# conflicts with old protobuf). On Linux those issues don't apply. +# +# Build the image once: +# docker build -f Dockerfile.dev -t sqlanvil-dev . +# +# Run a one-shot Bazel command (caches persist in a named volume): +# docker run --rm -it \ +# -v "$PWD:/workspace" \ +# -v sqlanvil-bazel-cache:/root/.cache/bazel \ +# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \ +# sqlanvil-dev bazel build //protos:sqlanvil_proto +# +# Drop into a shell: +# docker run --rm -it \ +# -v "$PWD:/workspace" \ +# -v sqlanvil-bazel-cache:/root/.cache/bazel \ +# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \ +# sqlanvil-dev +# +# Or use the wrapper: ./scripts/docker-bazel build //protos:sqlanvil_proto + +FROM node:20-bookworm + +# Bazel needs: JDK, python (for some rules), git, build-essential, unzip, +# zip, and the usual C/C++ toolchain (gcc, g++, make). +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + git \ + gnupg \ + openjdk-17-jdk-headless \ + python3 \ + python-is-python3 \ + build-essential \ + unzip \ + zip \ + zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* + +# Bazelisk — reads .bazelversion and downloads the matching Bazel. +RUN npm install -g @bazel/bazelisk + +# Bazel writes a lot to ~/.cache/bazel; expose it as a volume target so +# the host docker volume sticks across runs. +ENV BAZEL_DISK_CACHE_DIR=/root/.cache/bazel-disk +RUN mkdir -p /root/.cache/bazel /root/.cache/bazel-disk + +WORKDIR /workspace + +# Default to an interactive shell. Override with the bazel command you want. +CMD ["bash"] diff --git a/NOTICE b/NOTICE new file mode 100644 index 00000000..f3efbc5e --- /dev/null +++ b/NOTICE @@ -0,0 +1,14 @@ +sqlanvil +Copyright 2026 Ivan Histand + +This product includes software originally developed as Dataform +(https://github.com/dataform-co/dataform) by Dataform Co and contributed +to by Google LLC, licensed under the Apache License, Version 2.0. + +The original Dataform copyright notice from upstream is preserved in +the LICENSE file. This derivative work (renamed sqlanvil) adds +PostgreSQL and Supabase warehouse adapter support, and is distributed +under the same Apache License, Version 2.0. + +Apache License, Version 2.0 +http://www.apache.org/licenses/LICENSE-2.0 diff --git a/WORKSPACE b/WORKSPACE index ee73c05f..e26d9f71 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -1,4 +1,4 @@ -workspace(name = "df") +workspace(name = "sa") load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -43,13 +43,13 @@ load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "yarn_install node_repositories( node_repositories = { - "24.13.0-darwin_amd64": ("node-v24.13.0-darwin-x64.tar.xz", "node-v24.13.0-darwin-x64", "4ca0a48233f091a2a69ec28dd58e59f394a1b2d4f052b6c6b10f760377fe266f"), - "24.13.0-darwin_arm64": ("node-v24.13.0-darwin-arm64.tar.xz", "node-v24.13.0-darwin-arm64", "c59a517e9147f25c6167426875a571432f1478c1d7ee7ecc10baa46b0d0e8545"), - "24.13.0-linux_amd64": ("node-v24.13.0-linux-x64.tar.xz", "node-v24.13.0-linux-x64", "e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e"), - "24.13.0-linux_arm64": ("node-v24.13.0-linux-arm64.tar.xz", "node-v24.13.0-linux-arm64", "e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e"), - "24.13.0-windows_amd64": ("node-v24.13.0-win-x64.zip", "node-v24.13.0-win-x64", "ca2742695be8de44027d71b3f53a4bdb36009b95575fe1ae6f7f0b5ce091cb88"), + "20.20.2-darwin_amd64": ("node-v20.20.2-darwin-x64.tar.xz", "node-v20.20.2-darwin-x64", "4d4c020eb534497e616de38f3733289ff33c615ddab38c048edec6547b7f76ea"), + "20.20.2-darwin_arm64": ("node-v20.20.2-darwin-arm64.tar.xz", "node-v20.20.2-darwin-arm64", "6375a1d4421bc04ab284ba89459df788a78c49c89e83c463d0eede47e2efc07b"), + "20.20.2-linux_amd64": ("node-v20.20.2-linux-x64.tar.xz", "node-v20.20.2-linux-x64", "df770b2a6f130ed8627c9782c988fda9669fa23898329a61a871e32f965e007d"), + "20.20.2-linux_arm64": ("node-v20.20.2-linux-arm64.tar.xz", "node-v20.20.2-linux-arm64", "73093db209e4e9e09dd7d15a47aeaab1b74833830df03efa5f942a1122c5fa71"), + "20.20.2-windows_amd64": ("node-v20.20.2-win-x64.zip", "node-v20.20.2-win-x64", "dc3700fdd57a63eedb8fd7e3c7baaa32e6a740a1b904167ff4204bc68ed8bf77"), }, - node_version = "24.13.0", + node_version = "20.20.2", package_json = ["//:package.json"], yarn_version = "1.13.0", ) diff --git a/api/dbadapters/postgres.ts b/api/dbadapters/postgres.ts deleted file mode 100644 index 228e5268..00000000 --- a/api/dbadapters/postgres.ts +++ /dev/null @@ -1,285 +0,0 @@ -import * as pg from "pg"; - -import { Credentials } from "df/api/commands/credentials"; -import { IDbAdapter, IDbClient } from "df/api/dbadapters/index"; -import { SSHTunnelProxy } from "df/api/ssh_tunnel_proxy"; -import { parseRedshiftEvalError } from "df/api/utils/error_parsing"; -import { convertFieldType, PgPoolExecutor } from "df/api/utils/postgres"; -import { ErrorWithCause } from "df/common/errors/errors"; -import { collectEvaluationQueries, QueryOrAction } from "df/core/adapters"; -import { dataform } from "df/protos/ts"; - -interface IPostgresAdapterOptions { - sshTunnel?: SSHTunnelProxy; -} - -export class PostgresDbAdapter implements IDbAdapter { - public static async create( - credentials: Credentials, - options?: { concurrencyLimit?: number; disableSslForTestsOnly?: boolean } - ) { - const jdbcCredentials = credentials as dataform.IJDBC; - const baseClientConfig: Partial = { - user: jdbcCredentials.username, - password: jdbcCredentials.password, - database: jdbcCredentials.databaseName, - ssl: options?.disableSslForTestsOnly - ? false - : { - rejectUnauthorized: false, - ca: jdbcCredentials.ssl?.serverCertificate, - cert: jdbcCredentials.ssl?.clientCertificate, - key: jdbcCredentials.ssl?.clientPrivateKey - } - }; - if (jdbcCredentials.sshTunnel) { - const sshTunnel = await SSHTunnelProxy.create(jdbcCredentials.sshTunnel, { - host: jdbcCredentials.host, - port: jdbcCredentials.port - }); - const queryExecutor = new PgPoolExecutor( - { - ...baseClientConfig, - host: "127.0.0.1", - port: sshTunnel.localPort - }, - options - ); - return new PostgresDbAdapter(queryExecutor, { sshTunnel }); - } else { - const clientConfig: pg.ClientConfig = { - ...baseClientConfig, - host: jdbcCredentials.host, - port: jdbcCredentials.port - }; - const queryExecutor = new PgPoolExecutor(clientConfig, options); - return new PostgresDbAdapter(queryExecutor, {}); - } - } - - private constructor( - private readonly queryExecutor: PgPoolExecutor, - private readonly options: IPostgresAdapterOptions - ) {} - - public async execute( - statement: string, - options: { - params?: any[]; - onCancel?: (handleCancel: () => void) => void; - rowLimit?: number; - byteLimit?: number; - includeQueryInError?: boolean; - } = { rowLimit: 1000, byteLimit: 1024 * 1024 } - ) { - return await this.withClientLock(executor => executor.execute(statement, options)); - } - - public async withClientLock(callback: (client: IDbClient) => Promise) { - return await this.queryExecutor.withClientLock(client => - callback({ - execute: async ( - statement: string, - options: { - params?: any[]; - rowLimit?: number; - byteLimit?: number; - includeQueryInError?: boolean; - } = { rowLimit: 1000, byteLimit: 1024 * 1024 } - ) => { - try { - const rows = await client.execute(statement, options); - return { rows, metadata: {} }; - } catch (e) { - if (options.includeQueryInError) { - throw new Error(`Error encountered while running "${statement}": ${e.message}`); - } - throw new ErrorWithCause(`Error executing postgres query: ${e.message}`, e); - } - } - }) - ); - } - - public async evaluate(queryOrAction: QueryOrAction) { - const validationQueries = collectEvaluationQueries(queryOrAction, false, (query: string) => - !!query ? `explain ${query}` : "" - ).map((validationQuery, index) => ({ index, validationQuery })); - const validationQueriesWithoutWrappers = collectEvaluationQueries(queryOrAction, false); - - const queryEvaluations = new Array(); - for (const { index, validationQuery } of validationQueries) { - let evaluationResponse: dataform.IQueryEvaluation = { - status: dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS - }; - try { - await this.execute(validationQuery.query); - } catch (e) { - evaluationResponse = { - status: dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE, - error: parseRedshiftEvalError(validationQuery.query, e) - }; - } - queryEvaluations.push( - dataform.QueryEvaluation.create({ - ...evaluationResponse, - incremental: validationQuery.incremental, - query: validationQueriesWithoutWrappers[index].query - }) - ); - } - return queryEvaluations; - } - - public async tables(): Promise { - const queryResult = await this.execute( - `select table_name, table_schema - from information_schema.tables - where table_schema != 'information_schema' - and table_schema != 'pg_catalog' - and table_schema != 'pg_internal'`, - { rowLimit: 10000, includeQueryInError: true } - ); - const { rows } = queryResult; - return rows.map(row => ({ - schema: row.table_schema, - name: row.table_name - })); - } - - public async search( - searchText: string, - options: { limit: number } = { limit: 1000 } - ): Promise { - // TODO: It would be nice to extend this to search through table/column descriptions. However, this involves - // a somewhat crazy 5-way join. - const results = await this.execute( - `select tables.table_schema as table_schema, tables.table_name as table_name - from information_schema.tables as tables - left join information_schema.columns columns on tables.table_schema = columns.table_schema and tables.table_name = columns.table_name - where tables.table_schema ilike $1 or tables.table_name ilike $1 or columns.column_name ilike $1 - group by 1, 2`, - { - params: [`%${searchText}%`], - rowLimit: options.limit - } - ); - return await Promise.all( - results.rows.map(row => - this.table({ - schema: row.table_schema, - name: row.table_name - }) - ) - ); - } - - public async table(target: dataform.ITarget): Promise { - const params = [target.schema, target.name]; - const [tableResults, columnResults, descriptionResults] = await Promise.all([ - this.execute( - `select table_type from information_schema.tables where table_schema = $1 and table_name = $2`, - { params, includeQueryInError: true } - ), - this.execute( - `select column_name, data_type, is_nullable, ordinal_position - from information_schema.columns - where table_schema = $1 and table_name = $2`, - { params, includeQueryInError: true } - ), - this.execute( - ` - select objsubid as column_number, description from pg_description - where objoid = ( - select oid from pg_class where relname = $2 and relnamespace = ( - select oid from pg_namespace where nspname = $1 - ) - )`, - { params, includeQueryInError: true } - ) - ]); - if (tableResults.rows.length === 0) { - return null; - } - return dataform.TableMetadata.create({ - target, - type: - tableResults.rows[0].table_type === "VIEW" - ? dataform.TableMetadata.Type.VIEW - : dataform.TableMetadata.Type.TABLE, - fields: columnResults.rows.map(row => - dataform.Field.create({ - name: row.column_name, - primitive: convertFieldType(row.data_type), - description: descriptionResults.rows.find( - descriptionRow => descriptionRow.column_number === row.ordinal_position - )?.description - }) - ), - description: descriptionResults.rows.find( - descriptionRow => descriptionRow.column_number === 0 - )?.description - }); - } - - public async preview(target: dataform.ITarget, limitRows: number = 10): Promise { - const { rows } = await this.execute( - `SELECT * FROM "${target.schema}"."${target.name}" LIMIT ${limitRows}` - ); - return rows; - } - - public async schemas(): Promise { - const schemas = await this.execute(`select nspname from pg_namespace`, { - includeQueryInError: true - }); - return schemas.rows.map(row => row.nspname); - } - - public async createSchema(_: string, schema: string): Promise { - await this.execute(`create schema if not exists "${schema}"`, { includeQueryInError: true }); - } - - public async close() { - await this.queryExecutor.close(); - if (this.options.sshTunnel) { - await this.options.sshTunnel.close(); - } - } - - public async setMetadata(action: dataform.IExecutionAction): Promise { - const { target, actionDescriptor, tableType } = action; - - const actualMetadata = await this.table(target); - - const queries: Array> = []; - if (actionDescriptor.description) { - queries.push( - this.execute( - `comment on ${tableType === "view" ? "view" : "table"} "${target.schema}"."${ - target.name - }" is '${actionDescriptor.description.replace(/'/g, "''")}'` - ) - ); - } - if (actionDescriptor.columns?.length > 0) { - actionDescriptor.columns - .filter( - column => - column.path.length === 1 && - actualMetadata.fields.some(field => field.name === column.path[0]) - ) - .forEach(column => { - queries.push( - this.execute( - `comment on column "${target.schema}"."${target.name}"."${ - column.path[0] - }" is '${column.description.replace(/'/g, "''")}'` - ) - ); - }); - } - - await Promise.all(queries); - } -} diff --git a/cli/BUILD b/cli/BUILD index 2eebdabb..3d61ae96 100644 --- a/cli/BUILD +++ b/cli/BUILD @@ -38,7 +38,7 @@ ts_library( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/cli:package_tar", + "//packages/@sqlanvil/cli:package_tar", ], ) @@ -69,8 +69,7 @@ ts_test_suite( ], data = [ ":node_modules", - "//packages/@dataform/core:package_tar", - "//test_credentials:bigquery.json", + "//packages/@sqlanvil/core:package_tar", "@nodejs//:node", "@nodejs//:npm", ], diff --git a/cli/api/BUILD b/cli/api/BUILD index 111ff2a8..8871039a 100644 --- a/cli/api/BUILD +++ b/cli/api/BUILD @@ -33,6 +33,7 @@ ts_library( "@npm//@types/js-yaml", "@npm//@types/long", "@npm//@types/node", + "@npm//@types/pg", "@npm//@types/semver", "@npm//@types/tmp", "@npm//deepmerge", @@ -41,6 +42,7 @@ ts_library( "@npm//google-sql-syntax-ts", "@npm//js-beautify", "@npm//js-yaml", + "@npm//pg", "@npm//promise-pool-executor", "@npm//protobufjs", "@npm//semver", @@ -51,7 +53,7 @@ ts_library( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) @@ -65,8 +67,7 @@ ts_test_suite( ], data = [ ":node_modules", - "//packages/@dataform/core:package_tar", - "//test_credentials:bigquery.json", + "//packages/@sqlanvil/core:package_tar", "@nodejs//:node", "@nodejs//:npm", ] + glob(["goldens/**"]), diff --git a/cli/api/commands/build.ts b/cli/api/commands/build.ts index 99538381..f1712b28 100644 --- a/cli/api/commands/build.ts +++ b/cli/api/commands/build.ts @@ -1,14 +1,14 @@ -import { prune } from "df/cli/api/commands/prune"; -import { state } from "df/cli/api/commands/state"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { targetStringifier } from "df/core/targets"; -import * as utils from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { prune } from "sa/cli/api/commands/prune"; +import { state } from "sa/cli/api/commands/state"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { targetStringifier } from "sa/core/targets"; +import * as utils from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; export async function build( - compiledGraph: dataform.ICompiledGraph, - runConfig: dataform.IRunConfig, + compiledGraph: sqlanvil.ICompiledGraph, + runConfig: sqlanvil.IRunConfig, dbadapter: dbadapters.IDbAdapter ) { const prunedGraph = prune(compiledGraph, runConfig); @@ -31,29 +31,29 @@ export class Builder { private readonly executionSql: ExecutionSql; constructor( - private readonly prunedGraph: dataform.ICompiledGraph, - private readonly runConfig: dataform.IRunConfig, - private readonly warehouseState: dataform.IWarehouseState + private readonly prunedGraph: sqlanvil.ICompiledGraph, + private readonly runConfig: sqlanvil.IRunConfig, + private readonly warehouseState: sqlanvil.IWarehouseState ) { this.executionSql = new ExecutionSql( prunedGraph.projectConfig, - prunedGraph.dataformCoreVersion || "1.0.0" + prunedGraph.sqlanvilCoreVersion || "1.0.0" ); prunedGraph.tables.forEach(utils.setOrValidateTableEnumType); } - public build(): dataform.ExecutionGraph { + public build(): sqlanvil.ExecutionGraph { if (utils.graphHasErrors(this.prunedGraph)) { throw new Error(`Project has unresolved compilation or validation errors.`); } - const tableMetadataByTarget = new Map(); + const tableMetadataByTarget = new Map(); this.warehouseState.tables.forEach(tableState => { tableMetadataByTarget.set(targetStringifier.stringify(tableState.target), tableState); }); - const actions: dataform.IExecutionAction[] = [].concat( + const actions: sqlanvil.IExecutionAction[] = [].concat( this.prunedGraph.tables.map(t => this.buildTable( t, @@ -64,7 +64,7 @@ export class Builder { this.prunedGraph.operations.map(o => this.buildOperation(o)), this.prunedGraph.assertions.map(a => this.buildAssertion(a)) ); - return dataform.ExecutionGraph.create({ + return sqlanvil.ExecutionGraph.create({ projectConfig: this.prunedGraph.projectConfig, runConfig: this.runConfig, warehouseState: this.warehouseState, @@ -74,9 +74,9 @@ export class Builder { } private buildTable( - table: dataform.ITable, - tableMetadata: dataform.ITableMetadata, - runConfig: dataform.IRunConfig + table: sqlanvil.ITable, + tableMetadata: sqlanvil.ITableMetadata, + runConfig: sqlanvil.IRunConfig ) { return { ...this.toPartialExecutionAction(table), @@ -85,36 +85,36 @@ export class Builder { tasks: table.disabled ? [] : this.executionSql.publishTasks(table, runConfig, tableMetadata).build(), - hermeticity: table.hermeticity || dataform.ActionHermeticity.HERMETIC + hermeticity: table.hermeticity || sqlanvil.ActionHermeticity.HERMETIC }; } - private buildOperation(operation: dataform.IOperation) { + private buildOperation(operation: sqlanvil.IOperation) { return { ...this.toPartialExecutionAction(operation), type: "operation", tasks: operation.disabled ? [] : operation.queries.map(statement => ({ type: "statement", statement })), - hermeticity: operation.hermeticity || dataform.ActionHermeticity.NON_HERMETIC + hermeticity: operation.hermeticity || sqlanvil.ActionHermeticity.NON_HERMETIC }; } - private buildAssertion(assertion: dataform.IAssertion) { + private buildAssertion(assertion: sqlanvil.IAssertion) { return { ...this.toPartialExecutionAction(assertion), type: "assertion", tasks: assertion.disabled ? [] : this.executionSql.assertTasks(assertion, this.prunedGraph.projectConfig).build(), - hermeticity: assertion.hermeticity || dataform.ActionHermeticity.HERMETIC + hermeticity: assertion.hermeticity || sqlanvil.ActionHermeticity.HERMETIC }; } private toPartialExecutionAction( - action: dataform.ITable | dataform.IOperation | dataform.IAssertion + action: sqlanvil.ITable | sqlanvil.IOperation | sqlanvil.IAssertion ) { - return dataform.ExecutionAction.create({ + return sqlanvil.ExecutionAction.create({ target: action.target, fileName: action.fileName, dependencyTargets: action.dependencyTargets, diff --git a/cli/api/commands/compile.ts b/cli/api/commands/compile.ts index ac863a9a..68a89dba 100644 --- a/cli/api/commands/compile.ts +++ b/cli/api/commands/compile.ts @@ -4,11 +4,11 @@ import * as path from "path"; import * as tmp from "tmp"; import { promisify } from "util"; -import { MISSING_CORE_VERSION_ERROR } from "df/cli/api/commands/install"; -import { readConfigFromWorkflowSettings } from "df/cli/api/utils"; -import { coerceAsError } from "df/common/errors/errors"; -import { decode64 } from "df/common/protos"; -import { dataform } from "df/protos/ts"; +import { MISSING_CORE_VERSION_ERROR } from "sa/cli/api/commands/install"; +import { readConfigFromWorkflowSettings } from "sa/cli/api/utils"; +import { coerceAsError } from "sa/common/errors/errors"; +import { decode64 } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; export class CompilationTimeoutError extends Error {} @@ -17,9 +17,9 @@ function print(text: string) { } export async function compile( - compileConfig: dataform.ICompileConfig = {} -): Promise { - let compiledGraph = dataform.CompiledGraph.create(); + compileConfig: sqlanvil.ICompileConfig = {} +): Promise { + let compiledGraph = sqlanvil.CompiledGraph.create(); const resolvedProjectPath = path.resolve(compileConfig.projectDir); const packageJsonPath = path.join(resolvedProjectPath, "package.json"); @@ -29,18 +29,18 @@ export async function compile( const temporaryProjectPath = tmp.dirSync().name; const workflowSettings = readConfigFromWorkflowSettings(resolvedProjectPath); - const workflowSettingsDataformCoreVersion = workflowSettings?.dataformCoreVersion; + const workflowSettingssqlanvilCoreVersion = workflowSettings?.sqlanvilCoreVersion; const workflowSettingsExtension = workflowSettings?.extension ?? undefined; compileConfig.extension = workflowSettingsExtension; - if (!workflowSettingsDataformCoreVersion && !fs.existsSync(packageJsonPath)) { + if (!workflowSettingssqlanvilCoreVersion && !fs.existsSync(packageJsonPath)) { throw new Error(MISSING_CORE_VERSION_ERROR); } // For stateless package installation, a temporary directory is used in order to avoid interfering // with user's project directories. - if (workflowSettingsDataformCoreVersion) { + if (workflowSettingssqlanvilCoreVersion) { [projectNodeModulesPath, packageJsonPath, packageLockJsonPath].forEach(npmPath => { if (fs.existsSync(npmPath)) { throw new Error(`'${npmPath}' unexpected; remove it and try again`); @@ -48,7 +48,7 @@ export async function compile( }); if (compileConfig.verbose) { - print(`Using isolated environment for @dataform/core@${workflowSettingsDataformCoreVersion}\n`); + print(`Using isolated environment for @sqlanvil/core@${workflowSettingssqlanvilCoreVersion}\n`); print(`Copying project to temporary directory: ${temporaryProjectPath}\n`); } const copyStartTime = performance.now(); @@ -64,7 +64,7 @@ export async function compile( path.join(temporaryProjectPath, "package.json"), `{ "dependencies": { - "@dataform/core": "${workflowSettingsDataformCoreVersion}" + "@sqlanvil/core": "${workflowSettingssqlanvilCoreVersion}" } }` ); @@ -88,10 +88,10 @@ export async function compile( const result = await CompileChildProcess.forkProcess().compile(compileConfig); - const decodedResult = decode64(dataform.CoreExecutionResponse, result); - compiledGraph = dataform.CompiledGraph.create(decodedResult.compile.compiledGraph); + const decodedResult = decode64(sqlanvil.CoreExecutionResponse, result); + compiledGraph = sqlanvil.CompiledGraph.create(decodedResult.compile.compiledGraph); - if (workflowSettingsDataformCoreVersion) { + if (workflowSettingssqlanvilCoreVersion) { fs.rmSync(temporaryProjectPath, { recursive: true }); } @@ -100,7 +100,7 @@ export async function compile( export class CompileChildProcess { public static forkProcess() { - // Runs the worker_bundle script we generate for the package (see packages/@dataform/cli/BUILD) + // Runs the worker_bundle script we generate for the package (see packages/@sqlanvil/cli/BUILD) // if it exists, otherwise run the bazel compile loader target. const findForkScript = () => { try { @@ -121,7 +121,7 @@ export class CompileChildProcess { this.childProcess = childProcess; } - public async compile(compileConfig: dataform.ICompileConfig) { + public async compile(compileConfig: sqlanvil.ICompileConfig) { const compileInChildProcess = new Promise(async (resolve, reject) => { this.childProcess.on("error", (e: Error) => reject(coerceAsError(e))); diff --git a/cli/api/commands/credentials.ts b/cli/api/commands/credentials.ts index 9a69c11d..370735a5 100644 --- a/cli/api/commands/credentials.ts +++ b/cli/api/commands/credentials.ts @@ -1,12 +1,12 @@ import * as fs from "fs"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { verifyObjectMatchesProto } from "df/common/protos"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { verifyObjectMatchesProto } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; export const CREDENTIALS_FILENAME = ".df-credentials.json"; -export function read(credentialsPath: string): dataform.IBigQuery { +export function read(credentialsPath: string): sqlanvil.IBigQuery { if (!fs.existsSync(credentialsPath)) { throw new Error(`Missing credentials JSON file; not found at path '${credentialsPath}'.`); } @@ -16,7 +16,7 @@ export function read(credentialsPath: string): dataform.IBigQuery { } catch (e) { throw new Error(`Error reading credentials file: ${e.message}`); } - const credentials = verifyObjectMatchesProto(dataform.BigQuery, credentialsAsJson); + const credentials = verifyObjectMatchesProto(sqlanvil.BigQuery, credentialsAsJson); if (!Object.keys(credentials).find(key => key === "projectId")?.length) { throw new Error(`Error reading credentials file: the projectId field is required`); } diff --git a/cli/api/commands/init.ts b/cli/api/commands/init.ts index 686e27e2..c2398a35 100644 --- a/cli/api/commands/init.ts +++ b/cli/api/commands/init.ts @@ -2,9 +2,9 @@ import * as fs from "fs"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { CREDENTIALS_FILENAME } from "df/cli/api/commands/credentials"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; +import { CREDENTIALS_FILENAME } from "sa/cli/api/commands/credentials"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; const gitIgnoreContents = ` ${CREDENTIALS_FILENAME} @@ -18,21 +18,15 @@ export interface IInitResult { export async function init( projectDir: string, - projectConfig: dataform.IProjectConfig + projectConfig: sqlanvil.IProjectConfig ): Promise { const workflowSettingsYamlPath = path.join(projectDir, "workflow_settings.yaml"); const packageJsonPath = path.join(projectDir, "package.json"); const gitignorePath = path.join(projectDir, ".gitignore"); - // dataform.json is Deprecated. - const dataformJsonPath = path.join(projectDir, "dataform.json"); - if ( - fs.existsSync(workflowSettingsYamlPath) || - fs.existsSync(packageJsonPath) || - fs.existsSync(dataformJsonPath) - ) { + if (fs.existsSync(workflowSettingsYamlPath) || fs.existsSync(packageJsonPath)) { throw new Error( - "Cannot init dataform project, this already appears to be an NPM or Dataform directory." + "Cannot init sqlanvil project, this already appears to be an NPM or sqlanvil directory." ); } @@ -45,12 +39,12 @@ export async function init( } // The order that fields are set here is preserved in the written yaml. - const workflowSettings: dataform.IWorkflowSettings = { - dataformCoreVersion: version, + const workflowSettings: sqlanvil.IWorkflowSettings = { + sqlanvilCoreVersion: version, defaultProject: projectConfig.defaultDatabase, defaultLocation: projectConfig.defaultLocation, - defaultDataset: projectConfig.defaultSchema || "dataform", - defaultAssertionDataset: projectConfig.assertionSchema || "dataform_assertions", + defaultDataset: projectConfig.defaultSchema || "sqlanvil", + defaultAssertionDataset: projectConfig.assertionSchema || "sqlanvil_assertions", defaultIcebergConfig: projectConfig.defaultIcebergConfig, }; if (projectConfig.databaseSuffix) { diff --git a/cli/api/commands/install.ts b/cli/api/commands/install.ts index f71f09a9..3b21a320 100644 --- a/cli/api/commands/install.ts +++ b/cli/api/commands/install.ts @@ -3,10 +3,10 @@ import * as fs from "fs"; import * as path from "path"; import { promisify } from "util"; -import { readDataformCoreVersionFromWorkflowSettings } from "df/cli/api/utils"; +import { readsqlanvilCoreVersionFromWorkflowSettings } from "sa/cli/api/utils"; export const MISSING_CORE_VERSION_ERROR = - "dataformCoreVersion must be specified either in workflow_settings.yaml or via a package.json"; + "sqlanvilCoreVersion must be specified either in workflow_settings.yaml or via a package.json"; export async function install(projectPath: string) { const resolvedProjectPath = path.resolve(projectPath); @@ -14,8 +14,8 @@ export async function install(projectPath: string) { // Core's readWorkflowSettings method cannot be used for this because Core assumes that // `require` can read YAML files directly. - const dataformCoreVersion = readDataformCoreVersionFromWorkflowSettings(resolvedProjectPath); - if (dataformCoreVersion) { + const sqlanvilCoreVersion = readsqlanvilCoreVersionFromWorkflowSettings(resolvedProjectPath); + if (sqlanvilCoreVersion) { throw new Error( "No installation is needed when using workflow_settings.yaml, as packages are installed at " + "runtime." diff --git a/cli/api/commands/jit/compiler.ts b/cli/api/commands/jit/compiler.ts index 651da96f..7ebc42eb 100644 --- a/cli/api/commands/jit/compiler.ts +++ b/cli/api/commands/jit/compiler.ts @@ -1,12 +1,12 @@ import { ChildProcess } from "child_process"; import * as path from "path"; -import { BaseWorker } from "df/cli/api/commands/base_worker"; -import { handleDbRequest } from "df/cli/api/commands/jit/rpc"; -import { IDbAdapter, IDbClient } from "df/cli/api/dbadapters"; -import { IBigQueryExecutionOptions } from "df/cli/api/dbadapters/bigquery"; -import { DEFAULT_COMPILATION_TIMEOUT_MILLIS } from "df/cli/api/utils/constants"; -import { dataform } from "df/protos/ts"; +import { BaseWorker } from "sa/cli/api/commands/base_worker"; +import { handleDbRequest } from "sa/cli/api/commands/jit/rpc"; +import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters"; +import { IBigQueryExecutionOptions } from "sa/cli/api/dbadapters/bigquery"; +import { DEFAULT_COMPILATION_TIMEOUT_MILLIS } from "sa/cli/api/utils/constants"; +import { sqlanvil } from "sa/protos/ts"; export interface IJitWorkerMessage { type: "rpc_request" | "jit_response" | "jit_error"; @@ -18,17 +18,17 @@ export interface IJitWorkerMessage { } export class JitCompileChildProcess extends BaseWorker< - dataform.IJitCompilationResponse, + sqlanvil.IJitCompilationResponse, IJitWorkerMessage > { public static async compile( - request: dataform.IJitCompilationRequest, + request: sqlanvil.IJitCompilationRequest, projectDir: string, dbadapter: IDbAdapter, dbclient: IDbClient, timeoutMillis: number = DEFAULT_COMPILATION_TIMEOUT_MILLIS, options?: IBigQueryExecutionOptions - ): Promise { + ): Promise { return await new JitCompileChildProcess().run( request, projectDir, @@ -44,13 +44,13 @@ export class JitCompileChildProcess extends BaseWorker< } private async run( - request: dataform.IJitCompilationRequest, + request: sqlanvil.IJitCompilationRequest, projectDir: string, dbadapter: IDbAdapter, dbclient: IDbClient, timeoutMillis: number, options?: IBigQueryExecutionOptions - ): Promise { + ): Promise { return await this.runWorker( timeoutMillis, child => { @@ -64,7 +64,7 @@ export class JitCompileChildProcess extends BaseWorker< if (message.type === "rpc_request") { await this.handleRpcRequest(message, child, dbadapter, dbclient, options); } else if (message.type === "jit_response") { - resolve(dataform.JitCompilationResponse.fromObject(message.response)); + resolve(sqlanvil.JitCompilationResponse.fromObject(message.response)); } else if (message.type === "jit_error") { reject(new Error(message.error)); } diff --git a/cli/api/commands/jit/rpc.ts b/cli/api/commands/jit/rpc.ts index c9bfea07..b5ae5218 100644 --- a/cli/api/commands/jit/rpc.ts +++ b/cli/api/commands/jit/rpc.ts @@ -1,9 +1,9 @@ import Long from "long"; -import { IDbAdapter, IDbClient } from "df/cli/api/dbadapters"; -import { IBigQueryExecutionOptions } from "df/cli/api/dbadapters/bigquery"; -import { Structs } from "df/common/protos/structs"; -import { dataform, google } from "df/protos/ts"; +import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters"; +import { IBigQueryExecutionOptions } from "sa/cli/api/dbadapters/bigquery"; +import { Structs } from "sa/common/protos/structs"; +import { sqlanvil, google } from "sa/protos/ts"; export async function handleDbRequest( dbadapter: IDbAdapter, @@ -31,8 +31,8 @@ async function handleExecute( request: Uint8Array, options?: IBigQueryExecutionOptions ): Promise { - const executeRequest = dataform.ExecuteRequest.decode(request); - const executeRequestObj = dataform.ExecuteRequest.toObject(executeRequest, { + const executeRequest = sqlanvil.ExecuteRequest.decode(request); + const executeRequestObj = sqlanvil.ExecuteRequest.toObject(executeRequest, { defaults: false }); const requestOptions = executeRequestObj.bigQueryOptions; @@ -51,31 +51,31 @@ async function handleExecute( } }); - return dataform.ExecuteResponse.encode({ + return sqlanvil.ExecuteResponse.encode({ rows: (results.rows || []).map(row => Structs.fromObject(row)), schemaFields: results.schema || [] } as any).finish(); } async function handleListTables(dbadapter: IDbAdapter, request: Uint8Array): Promise { - const listTablesRequest = dataform.ListTablesRequest.decode(request); + const listTablesRequest = sqlanvil.ListTablesRequest.decode(request); if (!listTablesRequest.database) { throw new Error("ListTablesRequest.database must be supplied"); } const tablesMetadata = await dbadapter.tables(listTablesRequest.database, listTablesRequest.schema); - const listTablesResponse = dataform.ListTablesResponse.create({ + const listTablesResponse = sqlanvil.ListTablesResponse.create({ tables: tablesMetadata }); - return dataform.ListTablesResponse.encode(listTablesResponse).finish(); + return sqlanvil.ListTablesResponse.encode(listTablesResponse).finish(); } async function handleGetTable(dbadapter: IDbAdapter, request: Uint8Array): Promise { - const getTableRequest = dataform.GetTableRequest.decode(request); + const getTableRequest = sqlanvil.GetTableRequest.decode(request); const tableMetadata = await dbadapter.table(getTableRequest.target); if (!tableMetadata) { throw new Error(`Table not found: ${JSON.stringify(getTableRequest.target)}`); } - return dataform.TableMetadata.encode(tableMetadata).finish(); + return sqlanvil.TableMetadata.encode(tableMetadata).finish(); } async function handleDeleteTable( @@ -83,7 +83,7 @@ async function handleDeleteTable( request: Uint8Array, dryRun?: boolean ): Promise { - const deleteTableRequest = dataform.DeleteTableRequest.decode(request); + const deleteTableRequest = sqlanvil.DeleteTableRequest.decode(request); if (dryRun) { return new Uint8Array(); } diff --git a/cli/api/commands/jit/rpc_test.ts b/cli/api/commands/jit/rpc_test.ts index f9fbd9bc..3ec21a81 100644 --- a/cli/api/commands/jit/rpc_test.ts +++ b/cli/api/commands/jit/rpc_test.ts @@ -2,10 +2,10 @@ import { expect } from "chai"; import Long from "long"; import { anything, capture, instance, mock, verify, when } from "ts-mockito"; -import { handleDbRequest } from "df/cli/api/commands/jit/rpc"; -import { IDbAdapter, IDbClient } from "df/cli/api/dbadapters"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { handleDbRequest } from "sa/cli/api/commands/jit/rpc"; +import { IDbAdapter, IDbClient } from "sa/cli/api/dbadapters"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("jit_rpc", () => { test("Execute RPC maps to client.execute with all options", async () => { @@ -13,7 +13,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT * FROM table"; - const executeRequest = dataform.ExecuteRequest.create({ + const executeRequest = sqlanvil.ExecuteRequest.create({ statement, rowLimit: Long.fromNumber(100), byteLimit: Long.fromNumber(1024), @@ -25,7 +25,7 @@ suite("jit_rpc", () => { dryRun: true } }); - const encodedRequest = dataform.ExecuteRequest.encode(executeRequest).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(executeRequest).finish(); // Real raw BigQuery f/v format const rawRows = [ @@ -40,10 +40,10 @@ suite("jit_rpc", () => { ]; const schema = [ - { name: "num", primitive: dataform.Field.Primitive.INTEGER }, - { name: "str", primitive: dataform.Field.Primitive.STRING }, - { name: "bool", primitive: dataform.Field.Primitive.BOOLEAN }, - { name: "n", primitive: dataform.Field.Primitive.STRING } + { name: "num", primitive: sqlanvil.Field.Primitive.INTEGER }, + { name: "str", primitive: sqlanvil.Field.Primitive.STRING }, + { name: "bool", primitive: sqlanvil.Field.Primitive.BOOLEAN }, + { name: "n", primitive: sqlanvil.Field.Primitive.STRING } ]; when(mockClient.executeRaw(statement, anything())).thenResolve({ rows: rawRows, @@ -52,7 +52,7 @@ suite("jit_rpc", () => { }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const row = decoded.rows[0]; @@ -81,14 +81,14 @@ suite("jit_rpc", () => { const mockClient = mock(); const target = { database: "db", schema: "sch", name: "tab" }; - const request = dataform.DeleteTableRequest.create({ target }); - const encodedRequest = dataform.DeleteTableRequest.encode(request).finish(); + const request = sqlanvil.DeleteTableRequest.create({ target }); + const encodedRequest = sqlanvil.DeleteTableRequest.encode(request).finish(); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "DeleteTable", encodedRequest); verify(mockAdapter.deleteTable(anything())).once(); const capturedTarget = capture(mockAdapter.deleteTable).last()[0]; - expect(dataform.Target.create(capturedTarget)).deep.equals(dataform.Target.create(target)); + expect(sqlanvil.Target.create(capturedTarget)).deep.equals(sqlanvil.Target.create(target)); expect(response.length).equals(0); }); @@ -97,7 +97,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT null as n"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); // Test with a null value when(mockClient.executeRaw(statement, anything())).thenResolve({ @@ -106,12 +106,12 @@ suite("jit_rpc", () => { f: [{ v: null }] } ], - schema: [{ name: "n", primitive: dataform.Field.Primitive.STRING }], + schema: [{ name: "n", primitive: sqlanvil.Field.Primitive.STRING }], metadata: {} }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const fListNull = decoded.rows[0].fields.f.listValue.values; @@ -129,7 +129,7 @@ suite("jit_rpc", () => { }); const responseEmpty = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decodedEmpty = dataform.ExecuteResponse.decode(responseEmpty); + const decodedEmpty = sqlanvil.ExecuteResponse.decode(responseEmpty); expect(decodedEmpty.rows.length).equals(0); verify(mockClient.executeRaw(statement, anything())).twice(); @@ -142,15 +142,15 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); - const request = dataform.ListTablesRequest.create({ database: "db", schema: "sch" }); - const encodedRequest = dataform.ListTablesRequest.encode(request).finish(); + const request = sqlanvil.ListTablesRequest.create({ database: "db", schema: "sch" }); + const encodedRequest = sqlanvil.ListTablesRequest.encode(request).finish(); const target1 = { database: "db", schema: "sch", name: "table1" }; - const metadata1 = { target: target1, type: dataform.TableMetadata.Type.TABLE } as any; + const metadata1 = { target: target1, type: sqlanvil.TableMetadata.Type.TABLE } as any; when(mockAdapter.tables("db", "sch")).thenResolve([metadata1]); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "ListTables", encodedRequest); - const decoded = dataform.ListTablesResponse.decode(response); + const decoded = sqlanvil.ListTablesResponse.decode(response); expect(decoded.tables.length).equals(1); expect(decoded.tables[0].target.name).equals("table1"); @@ -164,8 +164,8 @@ suite("jit_rpc", () => { const mockClient = mock(); // Request without database - const request = dataform.ListTablesRequest.create({ schema: "sch" }); - const encodedRequest = dataform.ListTablesRequest.encode(request).finish(); + const request = sqlanvil.ListTablesRequest.create({ schema: "sch" }); + const encodedRequest = sqlanvil.ListTablesRequest.encode(request).finish(); try { await handleDbRequest(instance(mockAdapter), instance(mockClient), "ListTables", encodedRequest); @@ -182,18 +182,18 @@ suite("jit_rpc", () => { const mockClient = mock(); const target = { database: "db", schema: "sch", name: "tab" }; - const request = dataform.GetTableRequest.create({ target }); - const encodedRequest = dataform.GetTableRequest.encode(request).finish(); + const request = sqlanvil.GetTableRequest.create({ target }); + const encodedRequest = sqlanvil.GetTableRequest.encode(request).finish(); when(mockAdapter.table(anything())).thenResolve({ target } as any); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "GetTable", encodedRequest); - const decoded = dataform.TableMetadata.decode(response); + const decoded = sqlanvil.TableMetadata.decode(response); expect(decoded.target.name).equals("tab"); verify(mockAdapter.table(anything())).once(); const capturedTarget = capture(mockAdapter.table).last()[0]; - expect(dataform.Target.create(capturedTarget)).deep.equals(dataform.Target.create(target)); + expect(sqlanvil.Target.create(capturedTarget)).deep.equals(sqlanvil.Target.create(target)); }); test("GetTable RPC throws error when table not found", async () => { @@ -201,8 +201,8 @@ suite("jit_rpc", () => { const mockClient = mock(); const target = { database: "db", schema: "sch", name: "missing" }; - const request = dataform.GetTableRequest.create({ target }); - const encodedRequest = dataform.GetTableRequest.encode(request).finish(); + const request = sqlanvil.GetTableRequest.create({ target }); + const encodedRequest = sqlanvil.GetTableRequest.encode(request).finish(); // Adapter returns null for missing table when(mockAdapter.table(anything())).thenResolve(null); @@ -222,10 +222,10 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); - const request = dataform.DeleteTableRequest.create({ + const request = sqlanvil.DeleteTableRequest.create({ target: { database: "db", schema: "sch", name: "tab" } }); - const encodedRequest = dataform.DeleteTableRequest.encode(request).finish(); + const encodedRequest = sqlanvil.DeleteTableRequest.encode(request).finish(); // Call with dryRun = true await handleDbRequest(instance(mockAdapter), instance(mockClient), "DeleteTable", encodedRequest, { dryRun: true }); @@ -239,7 +239,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); when(mockClient.executeRaw(anything(), anything())).thenResolve({ rows: [], metadata: {} }); @@ -268,7 +268,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const executeRequest = dataform.ExecuteRequest.create({ + const executeRequest = sqlanvil.ExecuteRequest.create({ statement, bigQueryOptions: { location: "EU", @@ -277,7 +277,7 @@ suite("jit_rpc", () => { dryRun: true } }); - const encodedRequest = dataform.ExecuteRequest.encode(executeRequest).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(executeRequest).finish(); const globalOptions = { labels: { global_label: "global_val" }, @@ -313,7 +313,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: { request_label: "request_val" } } }).finish(); @@ -338,7 +338,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: { request_label: "request_val" } } }).finish(); @@ -360,7 +360,7 @@ suite("jit_rpc", () => { const mockAdapter = mock(); const mockClient = mock(); const statement = "SELECT 1"; - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: { request_label: "request_val" } } }).finish(); @@ -383,7 +383,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT 1"; // Request has no labels - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { location: "US" } }).finish(); @@ -405,7 +405,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT 1"; // Request has empty labels - const encodedRequest = dataform.ExecuteRequest.encode({ + const encodedRequest = sqlanvil.ExecuteRequest.encode({ statement, bigQueryOptions: { labels: {} } }).finish(); @@ -427,7 +427,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT * FROM table"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); // Real raw BigQuery f/v format const rawRows = [ @@ -440,12 +440,12 @@ suite("jit_rpc", () => { when(mockClient.executeRaw(statement, anything())).thenResolve({ rows: rawRows, - schema: [{ name: "id", primitive: dataform.Field.Primitive.STRING }], + schema: [{ name: "id", primitive: sqlanvil.Field.Primitive.STRING }], metadata: { bigquery: { jobId: "job1" } } }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const row = decoded.rows[0]; @@ -461,7 +461,7 @@ suite("jit_rpc", () => { const mockClient = mock(); const statement = "SELECT complex_struct FROM table"; - const encodedRequest = dataform.ExecuteRequest.encode(dataform.ExecuteRequest.create({ statement })).finish(); + const encodedRequest = sqlanvil.ExecuteRequest.encode(sqlanvil.ExecuteRequest.create({ statement })).finish(); // Real raw BigQuery complex nested f/v format const rawRows = [ @@ -481,12 +481,12 @@ suite("jit_rpc", () => { when(mockClient.executeRaw(statement, anything())).thenResolve({ rows: rawRows, - schema: [{ name: "complex_struct", primitive: dataform.Field.Primitive.STRING }], + schema: [{ name: "complex_struct", primitive: sqlanvil.Field.Primitive.STRING }], metadata: { bigquery: { jobId: "job1" } } }); const response = await handleDbRequest(instance(mockAdapter), instance(mockClient), "Execute", encodedRequest); - const decoded = dataform.ExecuteResponse.decode(response); + const decoded = sqlanvil.ExecuteResponse.decode(response); expect(decoded.rows.length).equals(1); const row = decoded.rows[0]; diff --git a/cli/api/commands/prune.ts b/cli/api/commands/prune.ts index 71358ea0..cfb6bad6 100644 --- a/cli/api/commands/prune.ts +++ b/cli/api/commands/prune.ts @@ -1,13 +1,13 @@ -import { targetAsReadableString } from "df/core/targets"; -import * as utils from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { targetAsReadableString } from "sa/core/targets"; +import * as utils from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; -type CompileAction = dataform.ITable | dataform.IOperation | dataform.IAssertion; +type CompileAction = sqlanvil.ITable | sqlanvil.IOperation | sqlanvil.IAssertion; export function prune( - compiledGraph: dataform.ICompiledGraph, - runConfig: dataform.IRunConfig -): dataform.ICompiledGraph { + compiledGraph: sqlanvil.ICompiledGraph, + runConfig: sqlanvil.IRunConfig +): sqlanvil.ICompiledGraph { compiledGraph.tables.forEach(utils.setOrValidateTableEnumType); const includedActionNames = computeIncludedActionNames(compiledGraph, runConfig); return { @@ -25,8 +25,8 @@ export function prune( } function computeIncludedActionNames( - compiledGraph: dataform.ICompiledGraph, - runConfig: dataform.IRunConfig + compiledGraph: sqlanvil.ICompiledGraph, + runConfig: sqlanvil.IRunConfig ): Set { // Union all tables, operations, assertions. const allActions: CompileAction[] = [].concat( diff --git a/cli/api/commands/query.ts b/cli/api/commands/query.ts index 7921e205..66628fc5 100644 --- a/cli/api/commands/query.ts +++ b/cli/api/commands/query.ts @@ -1,12 +1,12 @@ -import * as dbadapters from "df/cli/api/dbadapters"; -import { CancellablePromise } from "df/cli/api/utils/cancellable_promise"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { CancellablePromise } from "sa/cli/api/utils/cancellable_promise"; +import { sqlanvil } from "sa/protos/ts"; export function run( dbadapter: dbadapters.IDbAdapter, query: string, options?: { - compileConfig?: dataform.ICompileConfig; + compileConfig?: sqlanvil.ICompileConfig; rowLimit?: number; byteLimit?: number; } @@ -29,6 +29,6 @@ export function run( export async function evaluate( dbadapter: dbadapters.IDbAdapter, query: string -): Promise { +): Promise { return (await dbadapter.evaluate(query))[0]; } diff --git a/cli/api/commands/run.ts b/cli/api/commands/run.ts index f618aa09..9c8bc395 100644 --- a/cli/api/commands/run.ts +++ b/cli/api/commands/run.ts @@ -1,26 +1,26 @@ import EventEmitter from "events"; import Long from "long"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { IBigQueryExecutionOptions } from "df/cli/api/dbadapters/bigquery"; -import { Flags } from "df/common/flags"; -import { retry } from "df/common/promises"; -import { deepClone, equals } from "df/common/protos"; -import { targetStringifier } from "df/core/targets"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { IBigQueryExecutionOptions } from "sa/cli/api/dbadapters/bigquery"; +import { Flags } from "sa/common/flags"; +import { retry } from "sa/common/promises"; +import { deepClone, equals } from "sa/common/protos"; +import { targetStringifier } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; const CANCEL_EVENT = "jobCancel"; const flags = { runnerNotificationPeriodMillis: Flags.number("runner-notification-period-millis", 5000) }; -const isSuccessfulAction = (actionResult: dataform.IActionResult) => - actionResult.status === dataform.ActionResult.ExecutionStatus.SUCCESSFUL || - actionResult.status === dataform.ActionResult.ExecutionStatus.DISABLED; +const isSuccessfulAction = (actionResult: sqlanvil.IActionResult) => + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL || + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.DISABLED; export interface IExecutedAction { - executionAction: dataform.IExecutionAction; - actionResult: dataform.IActionResult; + executionAction: sqlanvil.IExecutionAction; + actionResult: sqlanvil.IActionResult; } export interface IExecutionOptions { @@ -34,9 +34,9 @@ export interface IExecutionOptions { export function run( dbadapter: dbadapters.IDbAdapter, - graph: dataform.IExecutionGraph, + graph: sqlanvil.IExecutionGraph, executionOptions?: IExecutionOptions, - partiallyExecutedRunResult: dataform.IRunResult = {}, + partiallyExecutedRunResult: sqlanvil.IRunResult = {}, runnerNotificationPeriodMillis: number = flags.runnerNotificationPeriodMillis.get() ): Runner { return new Runner( @@ -49,27 +49,27 @@ export function run( } export class Runner { - private readonly warehouseStateByTarget: Map; + private readonly warehouseStateByTarget: Map; private readonly allActionTargets: Set; - private readonly runResult: dataform.IRunResult; - private readonly changeListeners: Array<(graph: dataform.IRunResult) => void> = []; + private readonly runResult: sqlanvil.IRunResult; + private readonly changeListeners: Array<(graph: sqlanvil.IRunResult) => void> = []; private readonly eEmitter: EventEmitter; private executedActionTargets: Set; private successfullyExecutedActionTargets: Set; - private pendingActions: dataform.IExecutionAction[]; + private pendingActions: sqlanvil.IExecutionAction[]; private lastNotificationTimestampMillis = 0; private stopped = false; private cancelled = false; private timeout: NodeJS.Timer; private timedOut = false; - private executionTask: Promise; + private executionTask: Promise; constructor( private readonly dbadapter: dbadapters.IDbAdapter, - private readonly graph: dataform.IExecutionGraph, + private readonly graph: sqlanvil.IExecutionGraph, private readonly executionOptions: IExecutionOptions = {}, - partiallyExecutedRunResult: dataform.IRunResult = {}, + partiallyExecutedRunResult: sqlanvil.IRunResult = {}, private readonly runnerNotificationPeriodMillis: number = flags.runnerNotificationPeriodMillis.get() ) { this.allActionTargets = new Set( @@ -79,7 +79,7 @@ export class Runner { actions: [], ...partiallyExecutedRunResult }; - this.warehouseStateByTarget = new Map(); + this.warehouseStateByTarget = new Map(); graph.warehouseState.tables?.forEach(tableMetadata => this.warehouseStateByTarget.set( targetStringifier.stringify(tableMetadata.target), @@ -88,7 +88,7 @@ export class Runner { ); this.executedActionTargets = new Set( this.runResult.actions - .filter(action => action.status !== dataform.ActionResult.ExecutionStatus.RUNNING) + .filter(action => action.status !== sqlanvil.ActionResult.ExecutionStatus.RUNNING) .map(action => targetStringifier.stringify(action.target)) ); this.successfullyExecutedActionTargets = new Set( @@ -104,7 +104,7 @@ export class Runner { this.eEmitter.setMaxListeners(0); } - public onChange(listener: (graph: dataform.IRunResult) => void): Runner { + public onChange(listener: (graph: sqlanvil.IRunResult) => void): Runner { this.changeListeners.push(listener); return this; } @@ -136,7 +136,7 @@ export class Runner { this.eEmitter.emit(CANCEL_EVENT, undefined, undefined); } - public async result(): Promise { + public async result(): Promise { try { return await this.executionTask; } finally { @@ -150,7 +150,7 @@ export class Runner { if (Date.now() - this.runnerNotificationPeriodMillis < this.lastNotificationTimestampMillis) { return; } - const runResultClone = deepClone(dataform.RunResult, this.runResult); + const runResultClone = deepClone(sqlanvil.RunResult, this.runResult); this.lastNotificationTimestampMillis = Date.now(); this.changeListeners.forEach(listener => listener(runResultClone)); } @@ -158,7 +158,7 @@ export class Runner { private async executeGraph() { const timer = Timer.start(this.runResult.timing); - this.runResult.status = dataform.RunResult.ExecutionStatus.RUNNING; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.RUNNING; this.runResult.timing = timer.current(); this.notifyListeners(); @@ -176,17 +176,17 @@ export class Runner { this.runResult.timing = timer.end(); - this.runResult.status = dataform.RunResult.ExecutionStatus.SUCCESSFUL; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL; if (this.timedOut) { - this.runResult.status = dataform.RunResult.ExecutionStatus.TIMED_OUT; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.TIMED_OUT; } else if (this.cancelled) { - this.runResult.status = dataform.RunResult.ExecutionStatus.CANCELLED; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.CANCELLED; } else if ( this.runResult.actions.some( - action => action.status === dataform.ActionResult.ExecutionStatus.FAILED + action => action.status === sqlanvil.ActionResult.ExecutionStatus.FAILED ) ) { - this.runResult.status = dataform.RunResult.ExecutionStatus.FAILED; + this.runResult.status = sqlanvil.RunResult.ExecutionStatus.FAILED; } return this.runResult; @@ -198,7 +198,7 @@ export class Runner { this.graph.actions .filter(action => !!action.target && !!action.target.schema) .forEach(({ target }) => { - // This field may not be present for older versions of dataform. + // This field may not be present for older versions of sqlanvil. const trueDatabase = target.database || this.graph.projectConfig.defaultDatabase; if (!databaseSchemas.has(trueDatabase)) { databaseSchemas.set(trueDatabase, new Set()); @@ -231,9 +231,9 @@ export class Runner { allPendingActions.forEach(pendingAction => this.runResult.actions.push({ target: pendingAction.target, - status: dataform.ActionResult.ExecutionStatus.SKIPPED, + status: sqlanvil.ActionResult.ExecutionStatus.SKIPPED, tasks: pendingAction.tasks.map(() => ({ - status: dataform.TaskResult.ExecutionStatus.SKIPPED + status: sqlanvil.TaskResult.ExecutionStatus.SKIPPED })) }) ); @@ -277,9 +277,9 @@ export class Runner { skippableActions.forEach(skippableAction => { this.runResult.actions.push({ target: skippableAction.target, - status: dataform.ActionResult.ExecutionStatus.SKIPPED, + status: sqlanvil.ActionResult.ExecutionStatus.SKIPPED, tasks: skippableAction.tasks.map(() => ({ - status: dataform.TaskResult.ExecutionStatus.SKIPPED + status: sqlanvil.TaskResult.ExecutionStatus.SKIPPED })) }); }); @@ -303,28 +303,28 @@ export class Runner { ]); } - private async executeAction(action: dataform.IExecutionAction): Promise { - let actionResult: dataform.IActionResult = { + private async executeAction(action: sqlanvil.IExecutionAction): Promise { + let actionResult: sqlanvil.IActionResult = { target: action.target, tasks: [] }; if (action.tasks.length === 0) { - actionResult.status = dataform.ActionResult.ExecutionStatus.DISABLED; + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.DISABLED; this.runResult.actions.push(actionResult); this.notifyListeners(); return actionResult; } const resumedActionResult = this.runResult.actions.find(existingActionResult => - equals(dataform.Target, existingActionResult.target, action.target) + equals(sqlanvil.Target, existingActionResult.target, action.target) ); if (resumedActionResult) { actionResult = resumedActionResult; } else { this.runResult.actions.push(actionResult); } - actionResult.status = dataform.ActionResult.ExecutionStatus.RUNNING; + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.RUNNING; const timer = Timer.start(resumedActionResult?.timing); actionResult.timing = timer.current(); this.notifyListeners(); @@ -336,7 +336,7 @@ export class Runner { return actionResult; } if ( - actionResult.status === dataform.ActionResult.ExecutionStatus.RUNNING && + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.RUNNING && !this.cancelled ) { const taskStatus = await this.executeTask(client, task, actionResult, { @@ -354,14 +354,14 @@ export class Runner { this.graph.projectConfig?.defaultReservation } }); - if (taskStatus === dataform.TaskResult.ExecutionStatus.FAILED) { - actionResult.status = dataform.ActionResult.ExecutionStatus.FAILED; - } else if (taskStatus === dataform.TaskResult.ExecutionStatus.CANCELLED) { - actionResult.status = dataform.ActionResult.ExecutionStatus.CANCELLED; + if (taskStatus === sqlanvil.TaskResult.ExecutionStatus.FAILED) { + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.FAILED; + } else if (taskStatus === sqlanvil.TaskResult.ExecutionStatus.CANCELLED) { + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.CANCELLED; } } else { actionResult.tasks.push({ - status: dataform.TaskResult.ExecutionStatus.SKIPPED + status: sqlanvil.TaskResult.ExecutionStatus.SKIPPED }); } } @@ -375,7 +375,7 @@ export class Runner { action.actionDescriptor && // Only set metadata if we expect the action to complete in SUCCESSFUL state // (i.e. it must still be RUNNING, and not FAILED). - actionResult.status === dataform.ActionResult.ExecutionStatus.RUNNING && + actionResult.status === sqlanvil.ActionResult.ExecutionStatus.RUNNING && !(this.graph.runConfig && this.graph.runConfig.disableSetMetadata) && // Only set metadata if not using BigQuery dry run !this.executionOptions.bigquery?.dryRun && @@ -392,16 +392,16 @@ export class Runner { actionResult.tasks.length - 1 ].errorMessage = `Error setting metadata: ${e.message}`; actionResult.tasks[actionResult.tasks.length - 1].status = - dataform.TaskResult.ExecutionStatus.FAILED; + sqlanvil.TaskResult.ExecutionStatus.FAILED; } - actionResult.status = dataform.ActionResult.ExecutionStatus.FAILED; + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.FAILED; } } this.warehouseStateByTarget.delete(targetStringifier.stringify(action.target)); - if (actionResult.status === dataform.ActionResult.ExecutionStatus.RUNNING) { - actionResult.status = dataform.ActionResult.ExecutionStatus.SUCCESSFUL; + if (actionResult.status === sqlanvil.ActionResult.ExecutionStatus.RUNNING) { + actionResult.status = sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL; } actionResult.timing = timer.end(); @@ -411,20 +411,20 @@ export class Runner { private async executeTask( client: dbadapters.IDbClient, - task: dataform.IExecutionTask, - parentAction: dataform.IActionResult, - options: { bigquery?: dataform.IBigQueryOptions & IBigQueryExecutionOptions } - ): Promise { + task: sqlanvil.IExecutionTask, + parentAction: sqlanvil.IActionResult, + options: { bigquery?: sqlanvil.IBigQueryOptions & IBigQueryExecutionOptions } + ): Promise { const timer = Timer.start(); - const taskResult: dataform.ITaskResult = { - status: dataform.TaskResult.ExecutionStatus.RUNNING, + const taskResult: sqlanvil.ITaskResult = { + status: sqlanvil.TaskResult.ExecutionStatus.RUNNING, timing: timer.current(), metadata: {} }; parentAction.tasks.push(taskResult); this.notifyListeners(); if (options.bigquery?.dryRun && task.type === "assertion") { - taskResult.status = dataform.TaskResult.ExecutionStatus.SUCCESSFUL; + taskResult.status = sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL; } else { try { @@ -447,11 +447,11 @@ export class Runner { throw new Error(`Assertion failed: query returned ${rowCount} row(s).`); } } - taskResult.status = dataform.TaskResult.ExecutionStatus.SUCCESSFUL; + taskResult.status = sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL; } catch (e) { taskResult.status = this.cancelled - ? dataform.TaskResult.ExecutionStatus.CANCELLED - : dataform.TaskResult.ExecutionStatus.FAILED; + ? sqlanvil.TaskResult.ExecutionStatus.CANCELLED + : sqlanvil.TaskResult.ExecutionStatus.FAILED; taskResult.errorMessage = `${this.graph.projectConfig.warehouse} error: ${e.message}`; if (e.metadata?.bigquery?.jobId) { taskResult.metadata = { @@ -469,18 +469,18 @@ export class Runner { } class Timer { - public static start(existingTiming?: dataform.ITiming) { + public static start(existingTiming?: sqlanvil.ITiming) { return new Timer(existingTiming?.startTimeMillis.toNumber() || new Date().valueOf()); } private constructor(readonly startTimeMillis: number) { } - public current(): dataform.ITiming { + public current(): sqlanvil.ITiming { return { startTimeMillis: Long.fromNumber(this.startTimeMillis) }; } - public end(): dataform.ITiming { + public end(): sqlanvil.ITiming { return { startTimeMillis: Long.fromNumber(this.startTimeMillis), endTimeMillis: Long.fromNumber(new Date().valueOf()) diff --git a/cli/api/commands/state.ts b/cli/api/commands/state.ts index 94f5fe8a..dfce4dda 100644 --- a/cli/api/commands/state.ts +++ b/cli/api/commands/state.ts @@ -1,10 +1,10 @@ -import { IDbAdapter } from "df/cli/api/dbadapters"; -import { dataform } from "df/protos/ts"; +import { IDbAdapter } from "sa/cli/api/dbadapters"; +import { sqlanvil } from "sa/protos/ts"; export async function state( dbadapter: IDbAdapter, - targets: dataform.ITarget[] -): Promise { + targets: sqlanvil.ITarget[] +): Promise { const allTables = await Promise.all(targets.map(async target => dbadapter.table(target))); // Filter out datasets that don't exist. diff --git a/cli/api/commands/test.ts b/cli/api/commands/test.ts index 03330c26..6d6d2f35 100644 --- a/cli/api/commands/test.ts +++ b/cli/api/commands/test.ts @@ -1,17 +1,17 @@ -import * as dbadapters from "df/cli/api/dbadapters"; -import { dataform } from "df/protos/ts"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { sqlanvil } from "sa/protos/ts"; export async function test( dbadapter: dbadapters.IDbAdapter, - tests: dataform.ITest[] -): Promise { + tests: sqlanvil.ITest[] +): Promise { return await Promise.all(tests.map(testCase => runTest(dbadapter, testCase))); } async function runTest( dbadapter: dbadapters.IDbAdapter, - testCase: dataform.ITest -): Promise { + testCase: sqlanvil.ITest +): Promise { // TODO: Test results are currently limited to 1MB. // We should paginate test results to remove this limit. let actualResults; diff --git a/cli/api/dbadapters/bigquery.ts b/cli/api/dbadapters/bigquery.ts index 178c0c7e..aa12d139 100644 --- a/cli/api/dbadapters/bigquery.ts +++ b/cli/api/dbadapters/bigquery.ts @@ -2,7 +2,7 @@ import { BigQuery, GetTablesResponse, TableField, TableMetadata } from "@google- import Long from "long"; import { PromisePoolExecutor } from "promise-pool-executor"; -import { collectEvaluationQueries, QueryOrAction } from "df/cli/api/dbadapters/execution_sql"; +import { collectEvaluationQueries, QueryOrAction } from "sa/cli/api/dbadapters/execution_sql"; import { IBigQueryError, IDbAdapter, @@ -10,12 +10,12 @@ import { IExecutionResult, IExecutionResultRaw, OnCancel -} from "df/cli/api/dbadapters/index"; -import { parseBigqueryEvalError } from "df/cli/api/utils/error_parsing"; -import { LimitedResultSet } from "df/cli/api/utils/results"; -import { coerceAsError } from "df/common/errors/errors"; -import { retry } from "df/common/promises"; -import { dataform } from "df/protos/ts"; +} from "sa/cli/api/dbadapters/index"; +import { parseBigqueryEvalError } from "sa/cli/api/utils/error_parsing"; +import { LimitedResultSet } from "sa/cli/api/utils/results"; +import { coerceAsError } from "sa/common/errors/errors"; +import { retry } from "sa/common/promises"; +import { sqlanvil } from "sa/protos/ts"; const EXTRA_GOOGLE_SCOPES = ["https://www.googleapis.com/auth/drive"]; @@ -38,14 +38,14 @@ export interface IBigQueryExecutionOptions { } export class BigQueryDbAdapter implements IDbAdapter { - private bigQueryCredentials: dataform.IBigQuery; + private bigQueryCredentials: sqlanvil.IBigQuery; private pool: PromisePoolExecutor; private readonly clients = new Map(); private readonly bigqueryClient?: BigQuery; constructor( - credentials: dataform.IBigQuery, + credentials: sqlanvil.IBigQuery, options?: { concurrencyLimit?: number; bigqueryClient?: BigQuery } ) { this.bigQueryCredentials = credentials; @@ -149,14 +149,14 @@ export class BigQueryDbAdapter implements IDbAdapter { }) }) .promise(); - return dataform.QueryEvaluation.create({ - status: dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS, + return sqlanvil.QueryEvaluation.create({ + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS, incremental, query }); } catch (e) { return { - status: dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE, + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE, error: parseBigqueryEvalError(e), incremental, query @@ -166,9 +166,9 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - public async tables(database: string, schema?: string): Promise { + public async tables(database: string, schema?: string): Promise { const datasetIds = schema ? [schema] : await this.schemas(database); - const tablesMetadata: dataform.ITableMetadata[] = []; + const tablesMetadata: sqlanvil.ITableMetadata[] = []; await Promise.all( datasetIds.map(async datasetId => { @@ -196,7 +196,7 @@ export class BigQueryDbAdapter implements IDbAdapter { public async search( searchText: string, options: { limit: number } = { limit: 1000 } - ): Promise { + ): Promise { const results = await this.execute( `select table_catalog, table_schema, table_name from region-${this.bigQueryCredentials.location}.INFORMATION_SCHEMA.COLUMN_FIELD_PATHS @@ -221,7 +221,7 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - public async table(target: dataform.ITarget): Promise { + public async table(target: sqlanvil.ITarget): Promise { const metadata = await this.getMetadata(target); if (!metadata) { @@ -246,13 +246,13 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - return dataform.TableMetadata.create({ + return sqlanvil.TableMetadata.create({ type: metadata.type === "TABLE" - ? dataform.TableMetadata.Type.TABLE + ? sqlanvil.TableMetadata.Type.TABLE : metadata.type === "VIEW" - ? dataform.TableMetadata.Type.VIEW - : dataform.TableMetadata.Type.UNKNOWN, + ? sqlanvil.TableMetadata.Type.VIEW + : sqlanvil.TableMetadata.Type.UNKNOWN, target, fields: metadata.schema.fields?.map(field => convertField(field)), lastUpdatedMillis: Long.fromString(metadata.lastModifiedTime), @@ -264,7 +264,7 @@ export class BigQueryDbAdapter implements IDbAdapter { }); } - public async deleteTable(target: dataform.ITarget): Promise { + public async deleteTable(target: sqlanvil.ITarget): Promise { await this.getClient(target.database) .dataset(target.schema) .table(target.name) @@ -283,7 +283,7 @@ export class BigQueryDbAdapter implements IDbAdapter { ); } - public async setMetadata(action: dataform.IExecutionAction): Promise { + public async setMetadata(action: sqlanvil.IExecutionAction): Promise { const { target, actionDescriptor } = action; const metadata = await this.getMetadata(target); @@ -302,7 +302,7 @@ export class BigQueryDbAdapter implements IDbAdapter { }); } - private async getMetadata(target: dataform.ITarget): Promise { + private async getMetadata(target: sqlanvil.ITarget): Promise { try { const table = await this.getClient(target.database) .dataset(target.schema) @@ -379,7 +379,7 @@ export class BigQueryDbAdapter implements IDbAdapter { return { query, useLegacySql: false, - jobPrefix: "dataform-" + (bigqueryOptions?.jobPrefix ? `${bigqueryOptions.jobPrefix}-` : ""), + jobPrefix: "sqlanvil-" + (bigqueryOptions?.jobPrefix ? `${bigqueryOptions.jobPrefix}-` : ""), location: bigqueryOptions?.location, maxResults: rowLimit, labels: bigqueryOptions?.labels, @@ -518,20 +518,20 @@ function cleanRows(rows: any[]) { return rows; } -function convertField(field: TableField): dataform.IField { - const result: dataform.IField = { +function convertField(field: TableField): sqlanvil.IField { + const result: sqlanvil.IField = { name: field.name, - flags: field.mode === "REPEATED" ? [dataform.Field.Flag.REPEATED] : [], + flags: field.mode === "REPEATED" ? [sqlanvil.Field.Flag.REPEATED] : [], description: field.description }; if (field.type === "RECORD" || field.type === "STRUCT") { - result.struct = dataform.Fields.create({ + result.struct = sqlanvil.Fields.create({ fields: field.fields.map(innerField => convertField(innerField)) }); } else { result.primitive = convertFieldType(field.type); } - return dataform.Field.create(result); + return sqlanvil.Field.create(result); } // See: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema @@ -539,42 +539,42 @@ function convertFieldType(type: string) { switch (String(type).toUpperCase()) { case "FLOAT": case "FLOAT64": - return dataform.Field.Primitive.FLOAT; + return sqlanvil.Field.Primitive.FLOAT; case "INTEGER": case "INT64": - return dataform.Field.Primitive.INTEGER; + return sqlanvil.Field.Primitive.INTEGER; case "NUMERIC": - return dataform.Field.Primitive.NUMERIC; + return sqlanvil.Field.Primitive.NUMERIC; case "BOOL": case "BOOLEAN": - return dataform.Field.Primitive.BOOLEAN; + return sqlanvil.Field.Primitive.BOOLEAN; case "STRING": - return dataform.Field.Primitive.STRING; + return sqlanvil.Field.Primitive.STRING; case "DATE": - return dataform.Field.Primitive.DATE; + return sqlanvil.Field.Primitive.DATE; case "DATETIME": - return dataform.Field.Primitive.DATETIME; + return sqlanvil.Field.Primitive.DATETIME; case "TIMESTAMP": - return dataform.Field.Primitive.TIMESTAMP; + return sqlanvil.Field.Primitive.TIMESTAMP; case "TIME": - return dataform.Field.Primitive.TIME; + return sqlanvil.Field.Primitive.TIME; case "BYTES": - return dataform.Field.Primitive.BYTES; + return sqlanvil.Field.Primitive.BYTES; case "GEOGRAPHY": - return dataform.Field.Primitive.GEOGRAPHY; + return sqlanvil.Field.Primitive.GEOGRAPHY; case "BIGNUMERIC": - return dataform.Field.Primitive.BIGNUMERIC; + return sqlanvil.Field.Primitive.BIGNUMERIC; case "JSON": - return dataform.Field.Primitive.JSON; + return sqlanvil.Field.Primitive.JSON; case "INTERVAL": - return dataform.Field.Primitive.INTERVAL; + return sqlanvil.Field.Primitive.INTERVAL; default: - return dataform.Field.Primitive.UNKNOWN; + return sqlanvil.Field.Primitive.UNKNOWN; } } function addDescriptionToMetadata( - columnDescriptions: dataform.IColumnDescriptor[], + columnDescriptions: sqlanvil.IColumnDescriptor[], metadataArray: TableField[] ): TableField[] { if (!columnDescriptions) { diff --git a/cli/api/dbadapters/bigquery_test.ts b/cli/api/dbadapters/bigquery_test.ts index ecff5ed3..986b9d1f 100644 --- a/cli/api/dbadapters/bigquery_test.ts +++ b/cli/api/dbadapters/bigquery_test.ts @@ -2,9 +2,9 @@ import { Dataset, Table } from "@google-cloud/bigquery"; import { expect } from "chai"; import { anything, instance, mock, verify, when } from "ts-mockito"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("BigQueryDbAdapter", () => { test("tables() with schema filters correctly", async () => { @@ -16,7 +16,7 @@ suite("BigQueryDbAdapter", () => { const schemaName = "schema1"; const projectId = "project1"; - const credentials = dataform.BigQuery.create({ projectId, location: "US" }); + const credentials = sqlanvil.BigQuery.create({ projectId, location: "US" }); const adapter = new BigQueryDbAdapter(credentials, { bigqueryClient: instance(mockBigQuery) }); when(mockBigQuery.dataset(schemaName)).thenReturn(instance(mockDataset)); @@ -53,7 +53,7 @@ suite("BigQueryDbAdapter", () => { const tableName = "table1"; const projectId = "project"; - const credentials = dataform.BigQuery.create({ projectId, location: "US" }); + const credentials = sqlanvil.BigQuery.create({ projectId, location: "US" }); const adapter = new BigQueryDbAdapter(credentials, { bigqueryClient: instance(mockBigQuery) }); when(mockBigQuery.dataset(schemaName)).thenReturn(instance(mockDataset)); diff --git a/cli/api/dbadapters/execution_sql.ts b/cli/api/dbadapters/execution_sql.ts index 33d1990e..9fce2653 100644 --- a/cli/api/dbadapters/execution_sql.ts +++ b/cli/api/dbadapters/execution_sql.ts @@ -1,12 +1,12 @@ import * as semver from "semver"; -import { concatenateQueries, Task, Tasks } from "df/cli/api/dbadapters/tasks"; -import { ErrorWithCause } from "df/common/errors/errors"; -import { CompilationSql } from "df/core/compilation_sql"; -import { tableTypeEnumToString } from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { concatenateQueries, Task, Tasks } from "sa/cli/api/dbadapters/tasks"; +import { ErrorWithCause } from "sa/common/errors/errors"; +import { CompilationSql } from "sa/core/compilation_sql"; +import { tableTypeEnumToString } from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; -export type QueryOrAction = string | dataform.Table | dataform.Operation | dataform.Assertion; +export type QueryOrAction = string | sqlanvil.Table | sqlanvil.Operation | sqlanvil.Assertion; export interface IValidationQuery { query?: string; @@ -17,37 +17,37 @@ export class ExecutionSql { private readonly CompilationSql: CompilationSql; constructor( - private readonly project: dataform.IProjectConfig, - private readonly dataformCoreVersion: string, + private readonly project: sqlanvil.IProjectConfig, + private readonly sqlanvilCoreVersion: string, private readonly uniqueIdGenerator: () => string = () => Math.random().toString(36).substring(2) ) { - this.CompilationSql = new CompilationSql(project, dataformCoreVersion); + this.CompilationSql = new CompilationSql(project, sqlanvilCoreVersion); } - public baseTableType(enumType: dataform.TableType) { + public baseTableType(enumType: sqlanvil.TableType) { switch (enumType) { - case dataform.TableType.TABLE: - case dataform.TableType.INCREMENTAL: - return dataform.TableMetadata.Type.TABLE; - case dataform.TableType.VIEW: - return dataform.TableMetadata.Type.VIEW; + case sqlanvil.TableType.TABLE: + case sqlanvil.TableType.INCREMENTAL: + return sqlanvil.TableMetadata.Type.TABLE; + case sqlanvil.TableType.VIEW: + return sqlanvil.TableMetadata.Type.VIEW; default: throw new Error(`Unexpected table type: ${tableTypeEnumToString(enumType)}`); } } - public tableTypeAsSql(type: dataform.TableMetadata.Type) { + public tableTypeAsSql(type: sqlanvil.TableMetadata.Type) { switch (type) { - case dataform.TableMetadata.Type.TABLE: + case sqlanvil.TableMetadata.Type.TABLE: return "table"; - case dataform.TableMetadata.Type.VIEW: + case sqlanvil.TableMetadata.Type.VIEW: return "view"; default: throw new Error(`Unexpected table type: ${type}`); } } - public insertInto(target: dataform.ITarget, columns: string[], query: string) { + public insertInto(target: sqlanvil.ITarget, columns: string[], query: string) { return ` insert into ${this.resolveTarget(target)} (${columns.join(",")}) @@ -55,12 +55,12 @@ select ${columns.join(",")} from (${query}) as insertions`; } - public oppositeTableType(type: dataform.TableMetadata.Type) { + public oppositeTableType(type: sqlanvil.TableMetadata.Type) { switch (type) { - case dataform.TableMetadata.Type.TABLE: - return dataform.TableMetadata.Type.VIEW; - case dataform.TableMetadata.Type.VIEW: - return dataform.TableMetadata.Type.TABLE; + case sqlanvil.TableMetadata.Type.TABLE: + return sqlanvil.TableMetadata.Type.VIEW; + case sqlanvil.TableMetadata.Type.VIEW: + return sqlanvil.TableMetadata.Type.TABLE; default: throw new Error(`Unexpected table type: ${type}`); } @@ -75,26 +75,26 @@ from (${query}) as insertions`; } public shouldWriteIncrementally( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ) { return ( (!runConfig.fullRefresh || table.protected) && tableMetadata && - tableMetadata.type !== dataform.TableMetadata.Type.VIEW + tableMetadata.type !== sqlanvil.TableMetadata.Type.VIEW ); } public preOps( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ): Task[] { let preOps = table.preOps; if ( - semver.gt(this.dataformCoreVersion, "1.4.8") && - table.enumType === dataform.TableType.INCREMENTAL && + semver.gt(this.sqlanvilCoreVersion, "1.4.8") && + table.enumType === sqlanvil.TableType.INCREMENTAL && this.shouldWriteIncrementally(table, runConfig, tableMetadata) ) { preOps = table.incrementalPreOps; @@ -103,14 +103,14 @@ from (${query}) as insertions`; } public postOps( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ): Task[] { let postOps = table.postOps; if ( - semver.gt(this.dataformCoreVersion, "1.4.8") && - table.enumType === dataform.TableType.INCREMENTAL && + semver.gt(this.sqlanvilCoreVersion, "1.4.8") && + table.enumType === sqlanvil.TableType.INCREMENTAL && this.shouldWriteIncrementally(table, runConfig, tableMetadata) ) { postOps = table.incrementalPostOps; @@ -118,18 +118,18 @@ from (${query}) as insertions`; return (postOps || []).map(post => Task.statement(post)); } - public resolveTarget(target: dataform.ITarget) { + public resolveTarget(target: sqlanvil.ITarget) { return this.CompilationSql.resolveTarget(target); } - public getIncrementalQuery(table: dataform.ITable): string { + public getIncrementalQuery(table: sqlanvil.ITable): string { return this.where(table.incrementalQuery || table.query, table.where); } public publishTasks( - table: dataform.ITable, - runConfig: dataform.IRunConfig, - tableMetadata?: dataform.ITableMetadata + table: sqlanvil.ITable, + runConfig: sqlanvil.IRunConfig, + tableMetadata?: sqlanvil.ITableMetadata ): Tasks { const tasks = new Tasks(); @@ -142,18 +142,18 @@ from (${query}) as insertions`; ); } - if (table.enumType === dataform.TableType.INCREMENTAL) { + if (table.enumType === sqlanvil.TableType.INCREMENTAL) { if (!this.shouldWriteIncrementally(table, runConfig, tableMetadata)) { tasks.add(Task.statement(this.createOrReplace(table))); } else { - const onSchemaChange = table.onSchemaChange ?? dataform.OnSchemaChange.IGNORE; + const onSchemaChange = table.onSchemaChange ?? sqlanvil.OnSchemaChange.IGNORE; switch (onSchemaChange) { - case dataform.OnSchemaChange.FAIL: - case dataform.OnSchemaChange.EXTEND: - case dataform.OnSchemaChange.SYNCHRONIZE: + case sqlanvil.OnSchemaChange.FAIL: + case sqlanvil.OnSchemaChange.EXTEND: + case sqlanvil.OnSchemaChange.SYNCHRONIZE: this.buildIncrementalSchemaChangeTasks(tasks, table); // Fall through to run the static DML after the procedure alters the schema - case dataform.OnSchemaChange.IGNORE: + case sqlanvil.OnSchemaChange.IGNORE: default: tasks.add( Task.statement( @@ -185,8 +185,8 @@ from (${query}) as insertions`; } public assertTasks( - assertion: dataform.IAssertion, - projectConfig: dataform.IProjectConfig, + assertion: sqlanvil.IAssertion, + projectConfig: sqlanvil.IProjectConfig, ): Tasks { const tasks = new Tasks(); const target = assertion.target; @@ -198,11 +198,11 @@ from (${query}) as insertions`; return tasks; } - public dropIfExists(target: dataform.ITarget, type: dataform.TableMetadata.Type) { + public dropIfExists(target: sqlanvil.ITarget, type: sqlanvil.TableMetadata.Type) { return `drop ${this.tableTypeAsSql(type)} if exists ${this.resolveTarget(target)}`; } - private buildIncrementalSchemaChangeTasks(tasks: Tasks, table: dataform.ITable) { + private buildIncrementalSchemaChangeTasks(tasks: Tasks, table: sqlanvil.ITable) { const uniqueId = this.uniqueIdGenerator(); const emptyTempTableTarget = { @@ -231,7 +231,7 @@ END;`; tasks.add(Task.statement(callProcedureSql)); } - private createProcedureName(target: dataform.ITarget, uniqueId: string): string { + private createProcedureName(target: sqlanvil.ITarget, uniqueId: string): string { return this.resolveTarget({ ...target, name: `df_osc_${uniqueId}` @@ -262,17 +262,17 @@ CREATE OR REPLACE TABLE ${emptyTempTableName} AS ( } private compareSchemasSql( - target: dataform.ITarget, - emptyTempTableTarget: dataform.ITarget + target: sqlanvil.ITarget, + emptyTempTableTarget: sqlanvil.ITarget ): string { return ` -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM \`${target.database}.${target.schema}.INFORMATION_SCHEMA.COLUMNS\` WHERE table_name = '${target.name}' @@ -287,25 +287,25 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) );`; } private applySchemaChangeStrategySql( - table: dataform.ITable, + table: sqlanvil.ITable, qualifiedTargetTableName: string ): string { - const onSchemaChange = table.onSchemaChange || dataform.OnSchemaChange.IGNORE; + const onSchemaChange = table.onSchemaChange || sqlanvil.OnSchemaChange.IGNORE; let sql = ` --- Apply schema change strategy (${dataform.OnSchemaChange[onSchemaChange]}).`; +-- Apply schema change strategy (${sqlanvil.OnSchemaChange[onSchemaChange]}).`; switch (onSchemaChange) { - case dataform.OnSchemaChange.FAIL: + case sqlanvil.OnSchemaChange.FAIL: sql += ` IF ARRAY_LENGTH(columns_added) > 0 OR ARRAY_LENGTH(columns_removed) > 0 THEN RAISE USING MESSAGE = FORMAT( @@ -316,7 +316,7 @@ IF ARRAY_LENGTH(columns_added) > 0 OR ARRAY_LENGTH(columns_removed) > 0 THEN END IF; `; break; - case dataform.OnSchemaChange.EXTEND: + case sqlanvil.OnSchemaChange.EXTEND: sql += ` IF ARRAY_LENGTH(columns_removed) > 0 THEN RAISE USING MESSAGE = FORMAT( @@ -328,7 +328,7 @@ END IF; ${this.alterTableAddColumnsSql(qualifiedTargetTableName)} `; break; - case dataform.OnSchemaChange.SYNCHRONIZE: + case sqlanvil.OnSchemaChange.SYNCHRONIZE: const uniqueKeys = table.uniqueKey || []; sql += ` DECLARE invalid_removed_columns ARRAY; @@ -380,9 +380,9 @@ DROP TABLE IF EXISTS ${emptyTempTableName}; } private incrementalSchemaChangeBody( - table: dataform.ITable, + table: sqlanvil.ITable, qualifiedTargetTableName: string, - emptyTempTableTarget: dataform.ITarget + emptyTempTableTarget: sqlanvil.ITarget ): string { const emptyTempTableName = this.resolveTarget(emptyTempTableTarget); const query = this.getIncrementalQuery(table); @@ -399,7 +399,7 @@ DROP TABLE IF EXISTS ${emptyTempTableName}; return statements.join("\n\n"); } - private createOrReplace(table: dataform.ITable) { + private createOrReplace(table: sqlanvil.ITable) { const options = []; if (table.bigquery && table.bigquery.partitionBy && table.bigquery.partitionExpirationDays) { options.push(`partition_expiration_days=${table.bigquery.partitionExpirationDays}`); @@ -426,13 +426,13 @@ DROP TABLE IF EXISTS ${emptyTempTableName}; }${options.length > 0 ? `OPTIONS(${options.join(",")})` : ""}as ${table.query}`; } - private createOrReplaceView(target: dataform.ITarget, query: string) { + private createOrReplaceView(target: sqlanvil.ITarget, query: string) { return ` create or replace view ${this.resolveTarget(target)} as ${query}`; } private mergeInto( - target: dataform.ITarget, + target: sqlanvil.ITarget, columns: string[], query: string, uniqueKey: string[], @@ -470,8 +470,8 @@ export function collectEvaluationQueries( validationQueries.push({ query: queryModifier(queryOrAction) }); } else { try { - if (queryOrAction instanceof dataform.Table) { - if (queryOrAction.enumType === dataform.TableType.INCREMENTAL) { + if (queryOrAction instanceof sqlanvil.Table) { + if (queryOrAction.enumType === sqlanvil.TableType.INCREMENTAL) { const incrementalTableQueries = queryOrAction.incrementalPreOps.concat( queryOrAction.incrementalQuery, queryOrAction.incrementalPostOps @@ -498,7 +498,7 @@ export function collectEvaluationQueries( } else { tableQueries.forEach(q => validationQueries.push({ query: queryModifier(q) })); } - } else if (queryOrAction instanceof dataform.Operation) { + } else if (queryOrAction instanceof sqlanvil.Operation) { if (concatenate) { validationQueries.push({ query: concatenateQueries(queryOrAction.queries, queryModifier) @@ -506,7 +506,7 @@ export function collectEvaluationQueries( } else { queryOrAction.queries.forEach(q => validationQueries.push({ query: queryModifier(q) })); } - } else if (queryOrAction instanceof dataform.Assertion) { + } else if (queryOrAction instanceof sqlanvil.Assertion) { validationQueries.push({ query: queryModifier(queryOrAction.query) }); } else { throw new Error("Unrecognized evaluate type."); diff --git a/cli/api/dbadapters/index.ts b/cli/api/dbadapters/index.ts index 380f1dbf..2da2905a 100644 --- a/cli/api/dbadapters/index.ts +++ b/cli/api/dbadapters/index.ts @@ -1,19 +1,19 @@ -import { QueryOrAction } from "df/cli/api/dbadapters/execution_sql"; -import { dataform } from "df/protos/ts"; +import { QueryOrAction } from "sa/cli/api/dbadapters/execution_sql"; +import { sqlanvil } from "sa/protos/ts"; export type OnCancel = (handleCancel: () => void) => void; export interface IExecutionResult { rows: any[]; - metadata: dataform.IExecutionMetadata; + metadata: sqlanvil.IExecutionMetadata; } export interface IExecutionResultRaw extends IExecutionResult { - schema?: dataform.IField[]; + schema?: sqlanvil.IField[]; } export interface IBigQueryError extends Error { - metadata?: dataform.IExecutionMetadata + metadata?: sqlanvil.IExecutionMetadata } export interface IDbClient { @@ -53,15 +53,15 @@ export interface IDbClient { export interface IDbAdapter extends IDbClient { withClientLock(callback: (client: IDbClient) => Promise): Promise; - evaluate(queryOrAction: QueryOrAction): Promise; + evaluate(queryOrAction: QueryOrAction): Promise; schemas(database: string): Promise; createSchema(database: string, schema: string): Promise; - tables(database: string, schema?: string): Promise; - search(searchText: string, options?: { limit: number }): Promise; - table(target: dataform.ITarget): Promise; - deleteTable(target: dataform.ITarget): Promise; + tables(database: string, schema?: string): Promise; + search(searchText: string, options?: { limit: number }): Promise; + table(target: sqlanvil.ITarget): Promise; + deleteTable(target: sqlanvil.ITarget): Promise; - setMetadata(action: dataform.IExecutionAction): Promise; + setMetadata(action: sqlanvil.IExecutionAction): Promise; } diff --git a/cli/api/dbadapters/postgres.ts b/cli/api/dbadapters/postgres.ts new file mode 100644 index 00000000..83fcc932 --- /dev/null +++ b/cli/api/dbadapters/postgres.ts @@ -0,0 +1,307 @@ +import * as pg from "pg"; + +import { collectEvaluationQueries, QueryOrAction } from "sa/cli/api/dbadapters/execution_sql"; +import { + IDbAdapter, + IDbClient, + IExecutionResult, + IExecutionResultRaw, + OnCancel +} from "sa/cli/api/dbadapters/index"; +import { parsePostgresEvalError } from "sa/cli/api/utils/error_parsing"; +import { convertFieldType, PgPoolExecutor } from "sa/cli/api/utils/postgres"; +import { ErrorWithCause } from "sa/common/errors/errors"; +import { sqlanvil } from "sa/protos/ts"; + +const INTERNAL_SCHEMAS = new Set(["information_schema", "pg_catalog", "pg_internal", "pg_toast"]); + +export class PostgresDbAdapter implements IDbAdapter { + public static async create( + credentials: sqlanvil.IPostgresConnection, + options?: { concurrencyLimit?: number; disableSslForTestsOnly?: boolean } + ): Promise { + const sslMode = (credentials.sslMode || "").toLowerCase(); + const sslEnabled = !options?.disableSslForTestsOnly && sslMode !== "disable"; + const clientConfig: pg.ClientConfig = { + host: credentials.host, + port: credentials.port, + database: credentials.database, + user: credentials.user, + password: credentials.password, + ssl: sslEnabled + ? { + // Supabase and most managed Postgres providers serve certs signed + // by their own CA. Skipping verification is the documented path + // for `sslmode=require`. Stricter `verify-ca` / `verify-full` + // requires a CA bundle that we don't ship today. + rejectUnauthorized: sslMode === "verify-ca" || sslMode === "verify-full" + } + : false + }; + const queryExecutor = new PgPoolExecutor(clientConfig, options); + return new PostgresDbAdapter(queryExecutor); + } + + private constructor(private readonly queryExecutor: PgPoolExecutor) {} + + public async execute( + statement: string, + options: { + params?: any[]; + onCancel?: OnCancel; + rowLimit?: number; + byteLimit?: number; + includeQueryInError?: boolean; + } = { rowLimit: 1000, byteLimit: 1024 * 1024 } + ): Promise { + return await this.withClientLock(client => client.execute(statement, options)); + } + + public async executeRaw( + statement: string, + options: { + params?: any[]; + rowLimit?: number; + } = { rowLimit: 1000 } + ): Promise { + const result = await this.execute(statement, options); + return { ...result, schema: [] }; + } + + public async withClientLock(callback: (client: IDbClient) => Promise): Promise { + return await this.queryExecutor.withClientLock(client => + callback({ + execute: async ( + stmt: string, + opts: { + params?: any[]; + onCancel?: OnCancel; + rowLimit?: number; + byteLimit?: number; + includeQueryInError?: boolean; + } = { rowLimit: 1000, byteLimit: 1024 * 1024 } + ): Promise => { + try { + const rows = await client.execute(stmt, opts); + return { rows, metadata: {} }; + } catch (e) { + if (opts.includeQueryInError) { + throw new Error(`Error encountered while running "${stmt}": ${e.message}`); + } + throw new ErrorWithCause(`Error executing postgres query: ${e.message}`, e); + } + }, + executeRaw: async ( + stmt: string, + opts: { params?: { [name: string]: any }; rowLimit?: number } = { rowLimit: 1000 } + ): Promise => { + // Convert named param object to positional array — pg uses $1, $2 etc. + const positional = opts.params ? Object.values(opts.params) : undefined; + const rows = await client.execute(stmt, { params: positional, rowLimit: opts.rowLimit }); + return { rows, schema: [], metadata: {} }; + } + }) + ); + } + + public async evaluate(queryOrAction: QueryOrAction): Promise { + const validationQueries = collectEvaluationQueries(queryOrAction, false, (query: string) => + !!query ? `explain ${query}` : "" + ).map((validationQuery, index) => ({ index, validationQuery })); + const validationQueriesWithoutWrappers = collectEvaluationQueries(queryOrAction, false); + + const queryEvaluations = new Array(); + for (const { index, validationQuery } of validationQueries) { + let evaluationResponse: sqlanvil.IQueryEvaluation = { + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS + }; + try { + await this.execute(validationQuery.query); + } catch (e) { + evaluationResponse = { + status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE, + error: parsePostgresEvalError(validationQuery.query, e) + }; + } + queryEvaluations.push( + sqlanvil.QueryEvaluation.create({ + ...evaluationResponse, + incremental: validationQuery.incremental, + query: validationQueriesWithoutWrappers[index].query + }) + ); + } + return queryEvaluations; + } + + public async tables( + _database: string, + schema?: string + ): Promise { + const params: any[] = []; + let schemaClause = ""; + if (schema) { + schemaClause = "and table_schema = $1"; + params.push(schema); + } + const queryResult = await this.execute( + `select table_name, table_schema + from information_schema.tables + where table_schema not in ('information_schema', 'pg_catalog', 'pg_internal', 'pg_toast') + ${schemaClause}`, + { params, rowLimit: 10000, includeQueryInError: true } + ); + const targets = queryResult.rows.map(row => ({ + schema: row.table_schema as string, + name: row.table_name as string + })); + // Hydrate full metadata for each target — IDbAdapter.tables returns + // ITableMetadata[], not ITarget[]. + return await Promise.all(targets.map(target => this.table(target))); + } + + public async search( + searchText: string, + options: { limit: number } = { limit: 1000 } + ): Promise { + const results = await this.execute( + `select tables.table_schema as table_schema, tables.table_name as table_name + from information_schema.tables as tables + left join information_schema.columns columns + on tables.table_schema = columns.table_schema + and tables.table_name = columns.table_name + where tables.table_schema ilike $1 + or tables.table_name ilike $1 + or columns.column_name ilike $1 + group by 1, 2`, + { + params: [`%${searchText}%`], + rowLimit: options.limit + } + ); + return await Promise.all( + results.rows.map(row => + this.table({ + schema: row.table_schema, + name: row.table_name + }) + ) + ); + } + + public async table(target: sqlanvil.ITarget): Promise { + const params = [target.schema, target.name]; + const [tableResults, columnResults, descriptionResults] = await Promise.all([ + this.execute( + `select table_type from information_schema.tables where table_schema = $1 and table_name = $2`, + { params, includeQueryInError: true } + ), + this.execute( + `select column_name, data_type, is_nullable, ordinal_position + from information_schema.columns + where table_schema = $1 and table_name = $2`, + { params, includeQueryInError: true } + ), + this.execute( + `select objsubid as column_number, description + from pg_description + where objoid = ( + select oid from pg_class where relname = $2 and relnamespace = ( + select oid from pg_namespace where nspname = $1 + ) + )`, + { params, includeQueryInError: true } + ) + ]); + if (tableResults.rows.length === 0) { + return null; + } + return sqlanvil.TableMetadata.create({ + target, + type: + tableResults.rows[0].table_type === "VIEW" + ? sqlanvil.TableMetadata.Type.VIEW + : sqlanvil.TableMetadata.Type.TABLE, + fields: columnResults.rows.map(row => + sqlanvil.Field.create({ + name: row.column_name, + primitive: convertFieldType(row.data_type), + description: descriptionResults.rows.find( + descriptionRow => descriptionRow.column_number === row.ordinal_position + )?.description + }) + ), + description: descriptionResults.rows.find( + descriptionRow => descriptionRow.column_number === 0 + )?.description + }); + } + + public async deleteTable(target: sqlanvil.ITarget): Promise { + const metadata = await this.table(target); + if (!metadata) { + return; + } + const kind = metadata.type === sqlanvil.TableMetadata.Type.VIEW ? "view" : "table"; + await this.execute( + `drop ${kind} if exists "${target.schema}"."${target.name}" cascade`, + { includeQueryInError: true } + ); + } + + public async schemas(_database: string): Promise { + const result = await this.execute(`select nspname from pg_namespace`, { + includeQueryInError: true + }); + return result.rows + .map(row => row.nspname as string) + .filter(name => !INTERNAL_SCHEMAS.has(name) && !name.startsWith("pg_")); + } + + public async createSchema(_database: string, schema: string): Promise { + await this.execute(`create schema if not exists "${schema}"`, { + includeQueryInError: true + }); + } + + public async setMetadata(action: sqlanvil.IExecutionAction): Promise { + const { target, actionDescriptor, tableType } = action; + const actualMetadata = await this.table(target); + if (!actualMetadata) { + return; + } + + const queries: Array> = []; + if (actionDescriptor?.description) { + queries.push( + this.execute( + `comment on ${tableType === "view" ? "view" : "table"} "${target.schema}"."${ + target.name + }" is '${actionDescriptor.description.replace(/'/g, "''")}'` + ) + ); + } + if (actionDescriptor?.columns?.length > 0) { + actionDescriptor.columns + .filter( + column => + column.path.length === 1 && + actualMetadata.fields.some(field => field.name === column.path[0]) + ) + .forEach(column => { + queries.push( + this.execute( + `comment on column "${target.schema}"."${target.name}"."${ + column.path[0] + }" is '${column.description.replace(/'/g, "''")}'` + ) + ); + }); + } + await Promise.all(queries); + } + + public async close(): Promise { + await this.queryExecutor.close(); + } +} diff --git a/cli/api/dbadapters/tasks.ts b/cli/api/dbadapters/tasks.ts index 8991e5be..4edf3db8 100644 --- a/cli/api/dbadapters/tasks.ts +++ b/cli/api/dbadapters/tasks.ts @@ -1,4 +1,4 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; export function concatenateQueries(statements: string[], modifier?: (mod: string) => string) { return statements @@ -45,7 +45,7 @@ export class Task { public static assertion(statement: string) { return new Task().type("assertion").statement(statement); } - private proto: dataform.IExecutionTask = dataform.ExecutionTask.create(); + private proto: sqlanvil.IExecutionTask = sqlanvil.ExecutionTask.create(); public type(v: string) { this.proto.type = v; @@ -62,6 +62,6 @@ export class Task { } public build() { - return dataform.ExecutionTask.create(this.proto); + return sqlanvil.ExecutionTask.create(this.proto); } } diff --git a/cli/api/execution_sql_test.ts b/cli/api/execution_sql_test.ts index 4a8cf556..d84a44fa 100644 --- a/cli/api/execution_sql_test.ts +++ b/cli/api/execution_sql_test.ts @@ -1,9 +1,9 @@ import { expect } from "chai"; import * as fs from "fs-extra"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("ExecutionSql with 'onSchemaChange'", () => { const executionSql = new ExecutionSql( @@ -15,9 +15,9 @@ suite("ExecutionSql with 'onSchemaChange'", () => { () => "test_uuid" ); - const baseTable: dataform.ITable = { + const baseTable: sqlanvil.ITable = { type: "incremental", - enumType: dataform.TableType.INCREMENTAL, + enumType: sqlanvil.TableType.INCREMENTAL, target: { database: "project-id", schema: "dataset-id", @@ -27,16 +27,16 @@ suite("ExecutionSql with 'onSchemaChange'", () => { incrementalQuery: "select 1 as id, 'a' as field1, 'new' as field2" }; - const tableMetadata: dataform.ITableMetadata = { - type: dataform.TableMetadata.Type.TABLE, + const tableMetadata: sqlanvil.ITableMetadata = { + type: sqlanvil.TableMetadata.Type.TABLE, fields: [ { name: "id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }, { name: "field1", - primitive: dataform.Field.Primitive.STRING + primitive: sqlanvil.Field.Primitive.STRING } ] }; @@ -44,7 +44,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates procedure for FAIL strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.FAIL + onSchemaChange: sqlanvil.OnSchemaChange.FAIL }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); const procedureSql = tasks.build().map(t => t.statement).join("\n;\n"); @@ -55,7 +55,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates procedure for EXTEND strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.EXTEND + onSchemaChange: sqlanvil.OnSchemaChange.EXTEND }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); const procedureSql = tasks.build().map(t => t.statement).join("\n;\n"); @@ -66,7 +66,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates procedure for SYNCHRONIZE strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.SYNCHRONIZE, + onSchemaChange: sqlanvil.OnSchemaChange.SYNCHRONIZE, uniqueKey: ["id"] }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); @@ -78,7 +78,7 @@ suite("ExecutionSql with 'onSchemaChange'", () => { test("generates simple merge for IGNORE strategy", () => { const table = { ...baseTable, - onSchemaChange: dataform.OnSchemaChange.IGNORE, + onSchemaChange: sqlanvil.OnSchemaChange.IGNORE, uniqueKey: ["id"] }; const tasks = executionSql.publishTasks(table, { fullRefresh: false }, tableMetadata); diff --git a/cli/api/goldens/on_schema_change_extend.sql b/cli/api/goldens/on_schema_change_extend.sql index 21f9558a..aee4e062 100644 --- a/cli/api/goldens/on_schema_change_extend.sql +++ b/cli/api/goldens/on_schema_change_extend.sql @@ -9,12 +9,12 @@ CREATE OR REPLACE TABLE `project-id.dataset-id.incremental_on_schema_change_df_t -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM `project-id.dataset-id.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'incremental_on_schema_change' @@ -29,11 +29,11 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) ); diff --git a/cli/api/goldens/on_schema_change_fail.sql b/cli/api/goldens/on_schema_change_fail.sql index ea8d9054..ab37c108 100644 --- a/cli/api/goldens/on_schema_change_fail.sql +++ b/cli/api/goldens/on_schema_change_fail.sql @@ -9,12 +9,12 @@ CREATE OR REPLACE TABLE `project-id.dataset-id.incremental_on_schema_change_df_t -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM `project-id.dataset-id.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'incremental_on_schema_change' @@ -29,11 +29,11 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) ); diff --git a/cli/api/goldens/on_schema_change_synchronize.sql b/cli/api/goldens/on_schema_change_synchronize.sql index bcfc63ea..863bf7dc 100644 --- a/cli/api/goldens/on_schema_change_synchronize.sql +++ b/cli/api/goldens/on_schema_change_synchronize.sql @@ -9,12 +9,12 @@ CREATE OR REPLACE TABLE `project-id.dataset-id.incremental_on_schema_change_df_t -- Compare schemas -DECLARE dataform_columns ARRAY; +DECLARE sqlanvil_columns ARRAY; DECLARE temp_table_columns ARRAY>; DECLARE columns_added ARRAY>; DECLARE columns_removed ARRAY; -SET dataform_columns = ( +SET sqlanvil_columns = ( SELECT IFNULL(ARRAY_AGG(DISTINCT column_name), []) FROM `project-id.dataset-id.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'incremental_on_schema_change' @@ -29,11 +29,11 @@ SET temp_table_columns = ( SET columns_added = ( SELECT IFNULL(ARRAY_AGG(column_info), []) FROM UNNEST(temp_table_columns) AS column_info - WHERE column_info.column_name NOT IN UNNEST(dataform_columns) + WHERE column_info.column_name NOT IN UNNEST(sqlanvil_columns) ); SET columns_removed = ( SELECT IFNULL(ARRAY_AGG(column_name), []) - FROM UNNEST(dataform_columns) AS column_name + FROM UNNEST(sqlanvil_columns) AS column_name WHERE column_name NOT IN (SELECT col.column_name FROM UNNEST(temp_table_columns) AS col) ); diff --git a/cli/api/index.ts b/cli/api/index.ts index c4e85f79..fe74588f 100644 --- a/cli/api/index.ts +++ b/cli/api/index.ts @@ -1,11 +1,11 @@ -import { build, Builder } from "df/cli/api/commands/build"; -import { compile } from "df/cli/api/commands/compile"; -import * as credentials from "df/cli/api/commands/credentials"; -import { init } from "df/cli/api/commands/init"; -import { install } from "df/cli/api/commands/install"; -import { prune } from "df/cli/api/commands/prune"; -import * as query from "df/cli/api/commands/query"; -import { run, Runner } from "df/cli/api/commands/run"; -import { test } from "df/cli/api/commands/test"; +import { build, Builder } from "sa/cli/api/commands/build"; +import { compile } from "sa/cli/api/commands/compile"; +import * as credentials from "sa/cli/api/commands/credentials"; +import { init } from "sa/cli/api/commands/init"; +import { install } from "sa/cli/api/commands/install"; +import { prune } from "sa/cli/api/commands/prune"; +import * as query from "sa/cli/api/commands/query"; +import { run, Runner } from "sa/cli/api/commands/run"; +import { test } from "sa/cli/api/commands/test"; export { init, install, credentials, compile, test, build, run, query, Runner, Builder, prune }; diff --git a/cli/api/utils.ts b/cli/api/utils.ts index b70a641e..92272c57 100644 --- a/cli/api/utils.ts +++ b/cli/api/utils.ts @@ -2,21 +2,21 @@ import * as fs from "fs-extra"; import { load as loadYaml, YAMLException } from "js-yaml"; import * as path from "path"; -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; export function prettyJsonStringify(obj: object) { return JSON.stringify(obj, null, 4) + "\n"; } -export function readDataformCoreVersionFromWorkflowSettings( +export function readsqlanvilCoreVersionFromWorkflowSettings( resolvedProjectPath: string ): string | undefined { - return readConfigFromWorkflowSettings(resolvedProjectPath)?.dataformCoreVersion; + return readConfigFromWorkflowSettings(resolvedProjectPath)?.sqlanvilCoreVersion; } export function readConfigFromWorkflowSettings( resolvedProjectPath: string -): dataform.WorkflowSettings | undefined { +): sqlanvil.WorkflowSettings | undefined { const workflowSettingsPath = path.join(resolvedProjectPath, "workflow_settings.yaml"); if (!fs.existsSync(workflowSettingsPath)) { return; @@ -32,5 +32,5 @@ export function readConfigFromWorkflowSettings( } throw e; } - return dataform.WorkflowSettings.create(workflowSettingsAsJson); + return sqlanvil.WorkflowSettings.create(workflowSettingsAsJson); } diff --git a/cli/api/utils/BUILD b/cli/api/utils/BUILD index 42393bcd..5202bc77 100644 --- a/cli/api/utils/BUILD +++ b/cli/api/utils/BUILD @@ -11,6 +11,9 @@ ts_library( "//core", "//protos:ts", "@npm//@types/node", + "@npm//@types/pg", "@npm//object-sizeof", + "@npm//pg", + "@npm//pg-query-stream", ], ) diff --git a/cli/api/utils/error_parsing.ts b/cli/api/utils/error_parsing.ts index 551f2fad..81de60b4 100644 --- a/cli/api/utils/error_parsing.ts +++ b/cli/api/utils/error_parsing.ts @@ -1,13 +1,27 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; interface IBigqueryEvaluationError { message?: string; } +interface IPostgresEvaluationError { + message?: string; +} + +// Postgres-specific error parser. `pg` raises errors with `.message` plus +// optional `.position` (byte offset into the query). We don't try to map +// byte offset back to (line, column) — not all callers can supply the +// original query — so we just preserve the message. +export function parsePostgresEvalError(_query: string, error: IPostgresEvaluationError) { + return sqlanvil.QueryEvaluationError.create({ + message: error?.message ? String(error.message) : String(error) + }); +} + export function parseBigqueryEvalError(error: IBigqueryEvaluationError) { // expected error format: // e.message = Syntax error: Unexpected identifier "asda" at [2:1] - const evalError = dataform.QueryEvaluationError.create({ + const evalError = sqlanvil.QueryEvaluationError.create({ message: String(error) }); try { diff --git a/cli/api/utils/graphs.ts b/cli/api/utils/graphs.ts index 30271be3..aa66656a 100644 --- a/cli/api/utils/graphs.ts +++ b/cli/api/utils/graphs.ts @@ -1,37 +1,37 @@ -import { targetStringifier } from "df/core/targets"; -import { dataform } from "df/protos/ts"; +import { targetStringifier } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; type CoreProtoActionTypes = - | dataform.ITable - | dataform.IOperation - | dataform.IAssertion - | dataform.IDeclaration - | dataform.IDataPreparation; + | sqlanvil.ITable + | sqlanvil.IOperation + | sqlanvil.IAssertion + | sqlanvil.IDeclaration + | sqlanvil.IDataPreparation; -function combineAllActions(graph: dataform.ICompiledGraph) { +function combineAllActions(graph: sqlanvil.ICompiledGraph) { return ([] as CoreProtoActionTypes[]).concat( - graph.tables || ([] as dataform.ITable[]), - graph.operations || ([] as dataform.IOperation[]), - graph.assertions || ([] as dataform.IAssertion[]), - graph.declarations || ([] as dataform.IDeclaration[]), - graph.dataPreparations || ([] as dataform.IDataPreparation[]) + graph.tables || ([] as sqlanvil.ITable[]), + graph.operations || ([] as sqlanvil.IOperation[]), + graph.assertions || ([] as sqlanvil.IAssertion[]), + graph.declarations || ([] as sqlanvil.IDeclaration[]), + graph.dataPreparations || ([] as sqlanvil.IDataPreparation[]) ); } -export function actionsByTarget(compiledGraph: dataform.ICompiledGraph) { +export function actionsByTarget(compiledGraph: sqlanvil.ICompiledGraph) { const actionsMap = new Map(); combineAllActions(compiledGraph) - // Required for backwards compatibility with old versions of @dataform/core. + // Required for backwards compatibility with old versions of @sqlanvil/core. .filter(action => !!action.target) .forEach(action => { actionsMap.set(targetStringifier.stringify(action.target), action); }); } -export function actionsByCanonicalTarget(compiledGraph: dataform.ICompiledGraph) { +export function actionsByCanonicalTarget(compiledGraph: sqlanvil.ICompiledGraph) { const actionsMap = new Map(); combineAllActions(compiledGraph) - // Required for backwards compatibility with old versions of @dataform/core. + // Required for backwards compatibility with old versions of @sqlanvil/core. .filter(action => !!action.canonicalTarget) .forEach(action => { actionsMap.set(targetStringifier.stringify(action.canonicalTarget), action); diff --git a/api/utils/postgres.ts b/cli/api/utils/postgres.ts similarity index 92% rename from api/utils/postgres.ts rename to cli/api/utils/postgres.ts index 04265d55..8d05ffe1 100644 --- a/api/utils/postgres.ts +++ b/cli/api/utils/postgres.ts @@ -1,8 +1,8 @@ import * as pg from "pg"; import QueryStream from "pg-query-stream"; -import { LimitedResultSet } from "df/api/utils/results"; -import { dataform } from "df/protos/ts"; +import { LimitedResultSet } from "sa/cli/api/utils/results"; +import { sqlanvil } from "sa/protos/ts"; const maybeInitializePg = (() => { let initialized = false; @@ -144,7 +144,7 @@ export function convertFieldType(type: string) { case "FLOAT8": case "DOUBLE PRECISION": case "REAL": - return dataform.Field.Primitive.FLOAT; + return sqlanvil.Field.Primitive.FLOAT; case "INTEGER": case "INT": case "INT2": @@ -152,13 +152,13 @@ export function convertFieldType(type: string) { case "INT8": case "BIGINT": case "SMALLINT": - return dataform.Field.Primitive.INTEGER; + return sqlanvil.Field.Primitive.INTEGER; case "DECIMAL": case "NUMERIC": - return dataform.Field.Primitive.NUMERIC; + return sqlanvil.Field.Primitive.NUMERIC; case "BOOLEAN": case "BOOL": - return dataform.Field.Primitive.BOOLEAN; + return sqlanvil.Field.Primitive.BOOLEAN; case "STRING": case "VARCHAR": case "CHAR": @@ -168,15 +168,15 @@ export function convertFieldType(type: string) { case "TEXT": case "NCHAR": case "BPCHAR": - return dataform.Field.Primitive.STRING; + return sqlanvil.Field.Primitive.STRING; case "DATE": - return dataform.Field.Primitive.DATE; + return sqlanvil.Field.Primitive.DATE; case "TIMESTAMP": case "TIMESTAMPZ": case "TIMESTAMP WITHOUT TIME ZONE": case "TIMESTAMP WITH TIME ZONE": - return dataform.Field.Primitive.TIMESTAMP; + return sqlanvil.Field.Primitive.TIMESTAMP; default: - return dataform.Field.Primitive.UNKNOWN; + return sqlanvil.Field.Primitive.UNKNOWN; } } diff --git a/cli/api/utils_test.ts b/cli/api/utils_test.ts index 09926ede..d26a8aee 100644 --- a/cli/api/utils_test.ts +++ b/cli/api/utils_test.ts @@ -3,10 +3,10 @@ import * as fs from "fs-extra"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { readConfigFromWorkflowSettings } from "df/cli/api/utils"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { readConfigFromWorkflowSettings } from "sa/cli/api/utils"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -24,7 +24,7 @@ suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml({ defaultProject: "dataform" }) + dumpYaml({ defaultProject: "sqlanvil" }) ); expect(readExtensionConfigFromWorkflowSettings(projectDir)).to.equal(undefined); }); @@ -34,8 +34,8 @@ suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), dumpYaml({ - dataformCoreVersion: "3.0.0", - defaultProject: "dataform", + sqlanvilCoreVersion: "3.0.0", + defaultProject: "sqlanvil", extension: { name: "test-extension", compilationMode: "PROLOGUE", @@ -48,7 +48,7 @@ suite("readExtensionConfigFromWorkflowSettings", ({ afterEach }) => { if (typeof mode === "string") { expect(mode).to.equal("PROLOGUE"); } else { - expect(mode).to.equal(dataform.ExtensionCompilationMode.PROLOGUE); + expect(mode).to.equal(sqlanvil.ExtensionCompilationMode.PROLOGUE); } }); diff --git a/cli/console.ts b/cli/console.ts index 9b084eb6..3e410a32 100644 --- a/cli/console.ts +++ b/cli/console.ts @@ -1,10 +1,10 @@ import * as readlineSync from "readline-sync"; -import { IInitResult } from "df/cli/api/commands/init"; -import { prettyJsonStringify } from "df/cli/api/utils"; -import { formatBytesInHumanReadableFormat, formatExecutionSuffix } from "df/cli/util"; -import { setOrValidateTableEnumType, tableTypeEnumToString } from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { IInitResult } from "sa/cli/api/commands/init"; +import { prettyJsonStringify } from "sa/cli/api/utils"; +import { formatBytesInHumanReadableFormat, formatExecutionSuffix } from "sa/cli/util"; +import { setOrValidateTableEnumType, tableTypeEnumToString } from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; // Support disabling colors in CLI output by using informal standard from https://no-color.org/ // NO_COLOR=1, NO_COLOR=true, NO_COLOR=yes @@ -187,7 +187,7 @@ export enum compiledGraphOutputType { Summary = "summary" } -export function printCompiledGraph(graph: dataform.ICompiledGraph, outputType: compiledGraphOutputType, quietCompilation: boolean) { +export function printCompiledGraph(graph: sqlanvil.ICompiledGraph, outputType: compiledGraphOutputType, quietCompilation: boolean) { const interactive = isInteractive(); @@ -237,7 +237,7 @@ export function printCompiledGraph(graph: dataform.ICompiledGraph, outputType: c } } -function formatStackTraceForQuietCompilation(compileError: dataform.ICompilationError): string { +function formatStackTraceForQuietCompilation(compileError: sqlanvil.ICompilationError): string { // Show only first 3 or available lines for cleaner error output // which contains the information on the file where the error occurred and the sufficient metadata for the user to fix the error. For e.g. // (line: 1) Unexpected identifier : @@ -253,7 +253,7 @@ function formatStackTraceForQuietCompilation(compileError: dataform.ICompilation return ""; } -export function printCompiledGraphErrors(graphErrors: dataform.IGraphErrors, quietCompilation: boolean) { +export function printCompiledGraphErrors(graphErrors: sqlanvil.IGraphErrors, quietCompilation: boolean) { if (graphErrors.compilationErrors && graphErrors.compilationErrors.length > 0) { printError("Compilation errors:", 1); graphErrors.compilationErrors.forEach(compileError => { @@ -267,7 +267,7 @@ export function printCompiledGraphErrors(graphErrors: dataform.IGraphErrors, qui } } -export function printTestResult(testResult: dataform.ITestResult) { +export function printTestResult(testResult: sqlanvil.ITestResult) { writeStdOut( `${testResult.name}: ${testResult.successful ? successOutput("passed") : errorOutput("failed")}` ); @@ -276,14 +276,14 @@ export function printTestResult(testResult: dataform.ITestResult) { } } -export function printExecutionGraph(executionGraph: dataform.ExecutionGraph, asJson: boolean) { +export function printExecutionGraph(executionGraph: sqlanvil.ExecutionGraph, asJson: boolean) { if (asJson) { writeStdOut(prettyJsonStringify(executionGraph.toJSON())); } else { const actionsByType = { - table: [] as dataform.IExecutionAction[], - assertion: [] as dataform.IExecutionAction[], - operation: [] as dataform.IExecutionAction[] + table: [] as sqlanvil.IExecutionAction[], + assertion: [] as sqlanvil.IExecutionAction[], + operation: [] as sqlanvil.IExecutionAction[] }; executionGraph.actions.forEach(action => { if ( @@ -321,8 +321,8 @@ export function printExecutionGraph(executionGraph: dataform.ExecutionGraph, asJ } export function printExecutedAction( - executedAction: dataform.IActionResult, - executionAction: dataform.IExecutionAction, + executedAction: sqlanvil.IActionResult, + executionAction: sqlanvil.IExecutionAction, dryRun?: boolean ) { const jobIds = executedAction.tasks @@ -338,7 +338,7 @@ export function printExecutedAction( const executionSuffix = formatExecutionSuffix(jobIds, bytesBilled); switch (executedAction.status) { - case dataform.ActionResult.ExecutionStatus.SUCCESSFUL: { + case sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL: { switch (executionAction.type) { case "table": { writeStdOut( @@ -374,7 +374,7 @@ export function printExecutedAction( } } } - case dataform.ActionResult.ExecutionStatus.FAILED: { + case sqlanvil.ActionResult.ExecutionStatus.FAILED: { switch (executionAction.type) { case "table": { writeStdErr( @@ -408,7 +408,7 @@ export function printExecutedAction( printExecutedActionErrors(executedAction, executionAction); return; } - case dataform.ActionResult.ExecutionStatus.SKIPPED: { + case sqlanvil.ActionResult.ExecutionStatus.SKIPPED: { switch (executionAction.type) { case "table": { writeStdOut( @@ -441,7 +441,7 @@ export function printExecutedAction( } return; } - case dataform.ActionResult.ExecutionStatus.DISABLED: { + case sqlanvil.ActionResult.ExecutionStatus.DISABLED: { switch (executionAction.type) { case "table": { writeStdOut( @@ -507,27 +507,27 @@ export function printFormatFilesResult( } } -function datasetString(target: dataform.ITarget, datasetType: string, disabled: boolean) { +function datasetString(target: sqlanvil.ITarget, datasetType: string, disabled: boolean) { return `${targetString(target)} [${datasetType}]${disabled ? " [disabled]" : ""}`; } -function assertionString(target: dataform.ITarget, disabled: boolean) { +function assertionString(target: sqlanvil.ITarget, disabled: boolean) { return `${targetString(target)}${disabled ? " [disabled]" : ""}`; } -function operationString(target: dataform.ITarget, disabled: boolean) { +function operationString(target: sqlanvil.ITarget, disabled: boolean) { return `${targetString(target)}${disabled ? " [disabled]" : ""}`; } -function plainTargetString(target: dataform.ITarget) { +function plainTargetString(target: sqlanvil.ITarget) { return `${target.schema}.${target.name}`; } -function targetString(target: dataform.ITarget) { +function targetString(target: sqlanvil.ITarget) { return calloutOutput(`${target.schema}.${target.name}`); } -export function dotRepresentation(graph: dataform.ICompiledGraph, interactive: boolean): string { +export function dotRepresentation(graph: sqlanvil.ICompiledGraph, interactive: boolean): string { const nodes: string[] = []; const edges: string[] = []; @@ -561,11 +561,11 @@ export function dotRepresentation(graph: dataform.ICompiledGraph, interactive: b } function printExecutedActionErrors( - executedAction: dataform.IActionResult, - executionAction: dataform.IExecutionAction + executedAction: sqlanvil.IActionResult, + executionAction: sqlanvil.IExecutionAction ) { const failingTasks = executedAction.tasks.filter( - task => task.status === dataform.TaskResult.ExecutionStatus.FAILED + task => task.status === sqlanvil.TaskResult.ExecutionStatus.FAILED ); failingTasks.forEach((task, i) => { executionAction.tasks[i].statement.split("\n").forEach(line => { diff --git a/cli/credentials.ts b/cli/credentials.ts index 12e55c5e..78dd79ef 100644 --- a/cli/credentials.ts +++ b/cli/credentials.ts @@ -1,10 +1,10 @@ import * as fs from "fs"; -import { question, selectionQuestion } from "df/cli/console"; -import { actuallyResolve } from "df/cli/util"; -import { dataform } from "df/protos/ts"; +import { question, selectionQuestion } from "sa/cli/console"; +import { actuallyResolve } from "sa/cli/util"; +import { sqlanvil } from "sa/protos/ts"; -export function getBigQueryCredentials(): dataform.IBigQuery { +export function getBigQueryCredentials(): sqlanvil.IBigQuery { const locationIndex = selectionQuestion("Enter the location of your datasets:", [ "US (default)", "EU", @@ -27,8 +27,8 @@ export function getBigQueryCredentials(): dataform.IBigQuery { } const cloudCredentialsPath = actuallyResolve( question( - "Please follow the instructions at https://docs.dataform.co/dataform-cli#create-a-credentials-file/\n" + - "to create and download a private key from the Google Cloud Console in JSON format.\n" + + "Follow instructions for creating a service account key for BigQuery access\n" + + "(see https://cloud.google.com/iam/docs/keys-create-delete) and download as JSON.\n" + "(You can delete this file after credential initialization is complete.)\n\n" + "Enter the path to your Google Cloud private key file:" ) diff --git a/cli/index.ts b/cli/index.ts index 452d71f5..a80bd0e7 100644 --- a/cli/index.ts +++ b/cli/index.ts @@ -5,10 +5,10 @@ import parseDuration from "parse-duration"; import * as path from "path"; import yargs from "yargs"; -import { build, compile, credentials, init, install, run, test } from "df/cli/api"; -import { CREDENTIALS_FILENAME } from "df/cli/api/commands/credentials"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { prettyJsonStringify } from "df/cli/api/utils"; +import { build, compile, credentials, init, install, run, test } from "sa/cli/api"; +import { CREDENTIALS_FILENAME } from "sa/cli/api/commands/credentials"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { prettyJsonStringify } from "sa/cli/api/utils"; import { compiledGraphOutputType, print, @@ -22,18 +22,18 @@ import { printInitResult, printSuccess, printTestResult -} from "df/cli/console"; -import { getBigQueryCredentials } from "df/cli/credentials"; +} from "sa/cli/console"; +import { getBigQueryCredentials } from "sa/cli/credentials"; import { actuallyResolve, assertPathExists, compiledGraphHasErrors, promptForIcebergConfig, -} from "df/cli/util"; -import { createYargsCli, INamedOption } from "df/cli/yargswrapper"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { formatFile } from "df/sqlx/format"; +} from "sa/cli/util"; +import { createYargsCli, INamedOption } from "sa/cli/yargswrapper"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { formatFile } from "sa/sqlx/format"; const RECOMPILE_DELAY = 500; @@ -46,7 +46,7 @@ process.on("unhandledRejection", async (reason: any) => { const projectDirOption: INamedOption = { name: "project-dir", option: { - describe: "The Dataform project directory.", + describe: "The sqlanvil project directory.", default: ".", coerce: actuallyResolve } @@ -56,16 +56,15 @@ const projectDirMustExistOption = { ...projectDirOption, check: (argv: yargs.Arguments) => { assertPathExists(argv[projectDirOption.name]); - const dataformJsonPath = path.resolve(argv[projectDirOption.name], "dataform.json"); const workflowSettingsYamlPath = path.resolve( argv[projectDirOption.name], "workflow_settings.yaml" ); - if (!fs.existsSync(dataformJsonPath) && !fs.existsSync(workflowSettingsYamlPath)) { + if (!fs.existsSync(workflowSettingsYamlPath)) { throw new Error( `${ argv[projectDirOption.name] - } does not appear to be a dataform directory (missing workflow_settings.yaml file).` + } does not appear to be a sqlanvil directory (missing workflow_settings.yaml file).` ); } } @@ -181,7 +180,7 @@ const timeoutOption: INamedOption = { const jobPrefixOption: INamedOption = { name: "job-prefix", option: { - describe: "Adds an additional prefix in the form of `dataform-${jobPrefix}-`.", + describe: "Adds an additional prefix in the form of `sqlanvil-${jobPrefix}-`.", type: "string", default: null } @@ -210,7 +209,7 @@ const bigqueryJobLabelsOption: INamedOption = { const quietCompileOption: INamedOption = { name: "quiet", option: { - describe: "Less verbose compilation output. Example usage: 'dataform compile --quiet'", + describe: "Less verbose compilation output. Example usage: 'sqlanvil compile --quiet'", type: "boolean", default: false } @@ -258,7 +257,7 @@ export function runCli() { format: `init [${projectDirOption.name}] [${ProjectConfigOptions.defaultDatabase.name}]` + ` [${ProjectConfigOptions.defaultLocation.name}]`, - description: "Create a new dataform project.", + description: "Create a new sqlanvil project.", positionalOptions: [ projectDirOption, { @@ -270,7 +269,7 @@ export function runCli() { if (!argv[ProjectConfigOptions.defaultDatabase.name]) { throw new Error( `The ${ProjectConfigOptions.defaultDatabase.name} positional argument is ` + - `required. Use "dataform help init" for more info.` + `required. Use "sqlanvil help init" for more info.` ); } } @@ -286,7 +285,7 @@ export function runCli() { if (!argv[ProjectConfigOptions.defaultLocation.name]) { throw new Error( `The ${ProjectConfigOptions.defaultLocation.name} positional argument is ` + - `required. Use "dataform help init" for more info.` + `required. Use "sqlanvil help init" for more info.` ); } } @@ -295,7 +294,7 @@ export function runCli() { options: [icebergOption], processFn: async argv => { const projectDir = argv[projectDirOption.name]; - const projectConfig: dataform.IProjectConfig = { + const projectConfig: sqlanvil.IProjectConfig = { defaultDatabase: argv[ProjectConfigOptions.defaultDatabase.name], defaultLocation: argv[ProjectConfigOptions.defaultLocation.name], }; @@ -329,7 +328,7 @@ export function runCli() { { format: `init-creds [${projectDirMustExistOption.name}]`, description: - `Create a ${credentials.CREDENTIALS_FILENAME} file for Dataform to use when ` + + `Create a ${credentials.CREDENTIALS_FILENAME} file for sqlanvil to use when ` + `accessing BigQuery.`, positionalOptions: [projectDirMustExistOption], options: [ @@ -378,7 +377,7 @@ export function runCli() { { format: `compile [${projectDirMustExistOption.name}]`, description: - "Compile the dataform project. Produces JSON output describing the non-executable graph.", + "Compile the sqlanvil project. Produces JSON output describing the non-executable graph.", positionalOptions: [projectDirMustExistOption], options: [ { @@ -396,7 +395,7 @@ export function runCli() { { name: verboseOptionName, option: { - describe: "Enable verbose compilation output. Example usage: 'dataform compile --verbose'", + describe: "Enable verbose compilation output. Example usage: 'sqlanvil compile --verbose'", type: "boolean", default: false }, @@ -501,7 +500,7 @@ export function runCli() { }, { format: `test [${projectDirMustExistOption.name}]`, - description: "Run the dataform project's unit tests.", + description: "Run the sqlanvil project's unit tests.", positionalOptions: [projectDirMustExistOption], options: [credentialsOption, timeoutOption, jsonOutputOption, ...ProjectConfigOptions.allYargsOptions], processFn: async argv => { @@ -545,7 +544,7 @@ export function runCli() { }, { format: `run [${projectDirMustExistOption.name}]`, - description: "Run the dataform project.", + description: "Run the sqlanvil project.", positionalOptions: [projectDirMustExistOption], options: [ { @@ -653,7 +652,7 @@ export function runCli() { bigqueryOptions = { ...bigqueryOptions, labels: argv[bigqueryJobLabelsOption.name] }; } - const actionsByName = new Map(); + const actionsByName = new Map(); executionGraph.actions.forEach(action => { actionsByName.set(targetAsReadableString(action.target), action); }); @@ -676,11 +675,11 @@ export function runCli() { const alreadyPrintedActions = new Set(); - const printExecutedGraph = (executedGraph: dataform.IRunResult) => { + const printExecutedGraph = (executedGraph: sqlanvil.IRunResult) => { executedGraph.actions .filter( actionResult => - actionResult.status !== dataform.ActionResult.ExecutionStatus.RUNNING + actionResult.status !== sqlanvil.ActionResult.ExecutionStatus.RUNNING ) .filter( executedAction => @@ -699,12 +698,12 @@ export function runCli() { runner.onChange(printExecutedGraph); const runResult = await runner.result(); printExecutedGraph(runResult); - return runResult.status === dataform.RunResult.ExecutionStatus.SUCCESSFUL ? 0 : 1; + return runResult.status === sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL ? 0 : 1; } }, { format: `format [${projectDirMustExistOption.name}]`, - description: "Format the dataform project's files.", + description: "Format the sqlanvil project's files.", positionalOptions: [projectDirMustExistOption], options: [ actionsOption, @@ -791,16 +790,16 @@ export function runCli() { } ] }) - .scriptName("dataform") + .scriptName("sqlanvil") .strict() .wrap(null) .recommendCommands() .fail(async (msg: string, err: any) => { if (!!err && err.name === "VMError" && err.message.includes("Cannot find module")) { - printError("Could not find NPM dependencies. Have you run 'dataform install'?"); + printError("Could not find NPM dependencies. Have you run 'sqlanvil install'?"); } else { const message = err?.message ? err.message.split("\n")[0] : msg; - printError(`Dataform encountered an error: ${message}`); + printError(`sqlanvil encountered an error: ${message}`); if (err?.stack) { printError(err.stack); } @@ -862,7 +861,7 @@ class ProjectConfigOptions { option: { describe: "Override for variables to inject via '--vars=someKey=someValue,a=b', referenced by " + - "`dataform.projectConfig.vars.someValue`. If unset, the value from workflow_settings.yaml is used.", + "`sqlanvil.projectConfig.vars.someValue`. If unset, the value from workflow_settings.yaml is used.", type: "string", default: null, coerce: (rawVarsString: string | null) => { @@ -939,8 +938,8 @@ class ProjectConfigOptions { public static constructProjectConfigOverride( argv: yargs.Arguments - ): dataform.IProjectConfig { - const projectConfigOptions: dataform.IProjectConfig = {}; + ): sqlanvil.IProjectConfig { + const projectConfigOptions: sqlanvil.IProjectConfig = {}; if (argv[ProjectConfigOptions.defaultDatabase.name]) { projectConfigOptions.defaultDatabase = argv[ProjectConfigOptions.defaultDatabase.name]; diff --git a/cli/index_compile_test.ts b/cli/index_compile_test.ts index c4966c5a..3b87d201 100644 --- a/cli/index_compile_test.ts +++ b/cli/index_compile_test.ts @@ -4,30 +4,30 @@ import * as fs from "fs-extra"; import { dump as dumpYaml, load as loadYaml } from "js-yaml"; import * as path from "path"; -import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "df/cli/index_test_base"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "sa/cli/index_test_base"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("compile command", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); test( - "compile throws an error when dataformCoreVersion not in workflow_settings.yaml and no " + + "compile throws an error when sqlanvilCoreVersion not in workflow_settings.yaml and no " + "package.json exists", async () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create({ defaultProject: "dataform" })) + dumpYaml(sqlanvil.WorkflowSettings.create({ defaultProject: "sqlanvil" })) ); expect( (await getProcessResult(execFile(nodePath, [cliEntryPointPath, "compile", projectDir]))) .stderr ).contains( - "dataformCoreVersion must be specified either in workflow_settings.yaml or via a " + + "sqlanvilCoreVersion must be specified either in workflow_settings.yaml or via a " + "package.json" ); } @@ -39,18 +39,16 @@ suite("compile command", ({ afterEach }) => { path.join(projectDir, "package.json"), `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); fs.writeFileSync( - path.join(projectDir, "dataform.json"), - `{ - "defaultDatabase": "tada-analytics", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "defaultLocation": "${DEFAULT_LOCATION}" -} + path.join(projectDir, "workflow_settings.yaml"), + `defaultProject: tada-analytics +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +defaultLocation: "${DEFAULT_LOCATION}" ` ); @@ -58,22 +56,22 @@ suite("compile command", ({ afterEach }) => { (await getProcessResult(execFile(nodePath, [cliEntryPointPath, "compile", projectDir]))) .stderr ).contains( - "Could not find a recent installed version of @dataform/core in the project. Check that " + - "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + - "`@dataform/core` is specified in `package.json`. If using `package.json`, then run " + - "`dataform install`." + "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + + "`sqlanvil install`." ); }); ["package.json", "package-lock.json", "node_modules"].forEach(npmFile => { - test(`compile throws an error when dataformCoreVersion in workflow_settings.yaml and ${npmFile} is present`, async () => { + test(`compile throws an error when sqlanvilCoreVersion in workflow_settings.yaml and ${npmFile} is present`, async () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), dumpYaml( - dataform.WorkflowSettings.create({ - defaultProject: "dataform", - dataformCoreVersion: "3.0.0" + sqlanvil.WorkflowSettings.create({ + defaultProject: "sqlanvil", + sqlanvilCoreVersion: "3.0.0" }) ) ); @@ -105,17 +103,17 @@ suite("disable-assertions flag (compilation)", ({ afterEach, beforeEach }) => { ); const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -158,7 +156,7 @@ SELECT 1 as id async function setUpWorkflowSettings(disableAssertions: boolean): Promise { const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); workflowSettings.disableAssertions = disableAssertions; @@ -175,14 +173,14 @@ SELECT 1 as id { canonicalTarget: { database: DEFAULT_DATABASE, - name: "dataform_example_table_assertions_uniqueKey_0", - schema: "dataform_assertions" + name: "sqlanvil_example_table_assertions_uniqueKey_0", + schema: "sqlanvil_assertions" }, dependencyTargets: [ { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" } ], disabled: true, @@ -190,22 +188,22 @@ SELECT 1 as id parentAction: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, query: // tslint:disable-next-line:tsr-detect-sql-literal-injection - `\nSELECT\n *\nFROM (\n SELECT\n id,\n COUNT(1) AS index_row_count\n FROM \`${DEFAULT_DATABASE}.dataform.example_table\`\n GROUP BY id\n ) AS data\nWHERE index_row_count > 1\n`, + `\nSELECT\n *\nFROM (\n SELECT\n id,\n COUNT(1) AS index_row_count\n FROM \`${DEFAULT_DATABASE}.sqlanvil.example_table\`\n GROUP BY id\n ) AS data\nWHERE index_row_count > 1\n`, target: { database: DEFAULT_DATABASE, - name: "dataform_example_table_assertions_uniqueKey_0", - schema: "dataform_assertions" + name: "sqlanvil_example_table_assertions_uniqueKey_0", + schema: "sqlanvil_assertions" } }, { canonicalTarget: { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" }, disabled: true, fileName: "definitions/test_assertion.sqlx", @@ -213,18 +211,18 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" } } ], - dataformCoreVersion: version, + sqlanvilCoreVersion: version, graphErrors: {}, jitData: {}, projectConfig: { - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, - defaultSchema: "dataform", + defaultSchema: "sqlanvil", disableAssertions: true, warehouse: "bigquery" }, @@ -233,7 +231,7 @@ SELECT 1 as id canonicalTarget: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, disabled: false, enumType: "TABLE", @@ -243,7 +241,7 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, type: "table" } @@ -251,18 +249,18 @@ SELECT 1 as id targets: [ { database: DEFAULT_DATABASE, - name: "dataform_example_table_assertions_uniqueKey_0", - schema: "dataform_assertions" + name: "sqlanvil_example_table_assertions_uniqueKey_0", + schema: "sqlanvil_assertions" }, { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" } ] }; @@ -308,8 +306,8 @@ suite("extension config", ({ afterEach }) => { dumpYaml({ defaultProject: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, - defaultDataset: "dataform", - defaultAssertionDataset: "dataform_assertions", + defaultDataset: "sqlanvil", + defaultAssertionDataset: "sqlanvil_assertions", extension: { name: "test-extension", compilationMode: "PROLOGUE", @@ -323,7 +321,7 @@ suite("extension config", ({ afterEach }) => { path.join(projectDir, "package.json"), `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); diff --git a/cli/index_help_test.ts b/cli/index_help_test.ts index 1951c168..44fe9baa 100644 --- a/cli/index_help_test.ts +++ b/cli/index_help_test.ts @@ -1,29 +1,29 @@ import { expect } from "chai"; import { execFile } from "child_process"; -import { cliEntryPointPath } from "df/cli/index_test_base"; -import { getProcessResult, nodePath, suite, test } from "df/testing"; +import { cliEntryPointPath } from "sa/cli/index_test_base"; +import { getProcessResult, nodePath, suite, test } from "sa/testing"; suite("help command", () => { test("shows global help with the help command", async () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("dataform [command]"); - expect(output).to.include("dataform init [project-dir] [default-database] [default-location]"); - expect(output).to.include("dataform install [project-dir]"); - expect(output).to.include("dataform init-creds [project-dir]"); - expect(output).to.include("dataform compile [project-dir]"); - expect(output).to.include("dataform test [project-dir]"); - expect(output).to.include("dataform run [project-dir]"); - expect(output).to.include("dataform format [project-dir]"); + expect(output).to.include("sqlanvil [command]"); + expect(output).to.include("sqlanvil init [project-dir] [default-database] [default-location]"); + expect(output).to.include("sqlanvil install [project-dir]"); + expect(output).to.include("sqlanvil init-creds [project-dir]"); + expect(output).to.include("sqlanvil compile [project-dir]"); + expect(output).to.include("sqlanvil test [project-dir]"); + expect(output).to.include("sqlanvil run [project-dir]"); + expect(output).to.include("sqlanvil format [project-dir]"); }); test("shows help for 'init' command", async () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "init"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Create a new dataform project."); + expect(output).to.include("Create a new sqlanvil project."); expect(output).to.include("--iceberg"); expect(output).to.include("Initialize the project with workflow-level Iceberg tables configuration."); }); @@ -40,7 +40,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "init-creds"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Create a .df-credentials.json file for Dataform to use when accessing BigQuery."); + expect(output).to.include("Create a .df-credentials.json file for sqlanvil to use when accessing BigQuery."); expect(output).to.include("[project-dir]"); expect(output).to.include("--test-connection"); expect(output).to.include("If true, a test query will be run using your final credentials."); @@ -50,7 +50,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "compile"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Compile the dataform project."); + expect(output).to.include("Compile the sqlanvil project."); expect(output).to.include("--watch"); expect(output).to.include("--json"); expect(output).to.include("--quiet"); @@ -60,7 +60,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "test"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Run the dataform project's unit tests."); + expect(output).to.include("Run the sqlanvil project's unit tests."); expect(output).to.include("[project-dir]"); expect(output).to.include("--credentials"); expect(output).to.include("--timeout"); @@ -72,7 +72,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "run"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Run the dataform project."); + expect(output).to.include("Run the sqlanvil project."); expect(output).to.include("--dry-run"); expect(output).to.include("--run-tests"); expect(output).to.include("--action-retry-limit"); @@ -88,7 +88,7 @@ suite("help command", () => { const result = await getProcessResult(execFile(nodePath, [cliEntryPointPath, "help", "format"])); expect(result.exitCode).equals(0); const output = result.stdout; - expect(output).to.include("Format the dataform project's files."); + expect(output).to.include("Format the sqlanvil project's files."); expect(output).to.include("--check"); expect(output).to.include("Check if files are formatted correctly without modifying them."); expect(output).to.include("--actions"); diff --git a/cli/index_init_test.ts b/cli/index_init_test.ts index 5ca0006a..05b83e21 100644 --- a/cli/index_init_test.ts +++ b/cli/index_init_test.ts @@ -4,7 +4,7 @@ import * as fs from "fs-extra"; import { load as loadYaml } from "js-yaml"; import * as path from "path"; -import { cliEntryPointPath } from "df/cli/index_test_base"; +import { cliEntryPointPath } from "sa/cli/index_test_base"; import { ICEBERG_BUCKET_NAME_HINT, ICEBERG_BUCKET_NAME_PROMPT_QUESTION, @@ -17,11 +17,11 @@ import { ICEBERG_TABLE_FOLDER_ROOT_PROMPT_QUESTION, ICEBERG_TABLE_FOLDER_ROOT_SUBPATH_HINT, ICEBERG_TABLE_FOLDER_SUBPATH_PROMPT_QUESTION -} from "df/cli/util"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { getProcessResult, nodePath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/cli/util"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { getProcessResult, nodePath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("init command", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -34,17 +34,17 @@ suite("init command", ({ afterEach }) => { cliEntryPointPath, "init", projectDir, - "--default-database=dataform-database", + "--default-database=sqlanvil-database", "--default-location=us-central1" ]) ); expect(fs.readFileSync(path.join(projectDir, "workflow_settings.yaml"), "utf8")).to - .equal(`dataformCoreVersion: ${version} -defaultProject: dataform-database + .equal(`sqlanvilCoreVersion: ${version} +defaultProject: sqlanvil-database defaultLocation: us-central1 -defaultDataset: dataform -defaultAssertionDataset: dataform_assertions +defaultDataset: sqlanvil +defaultAssertionDataset: sqlanvil_assertions `); }); @@ -63,7 +63,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-test", + "sqlanvil-iceberg-test", "us-central1", "--iceberg" ], { @@ -79,7 +79,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -105,7 +105,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -126,7 +126,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -151,7 +151,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -172,7 +172,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -197,7 +197,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -218,7 +218,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -243,7 +243,7 @@ defaultAssertionDataset: dataform_assertions cliEntryPointPath, "init", projectDir, - "dataform-iceberg-partial", + "sqlanvil-iceberg-partial", "us-east1", "--iceberg" ], { @@ -264,7 +264,7 @@ defaultAssertionDataset: dataform_assertions const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); assert.isTrue(fs.existsSync(workflowSettingsPath)); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); @@ -280,14 +280,14 @@ defaultAssertionDataset: dataform_assertions suite("init-creds command", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); - test("init-creds fails for directory without dataform config", async () => { + test("init-creds fails for directory without sqlanvil config", async () => { const emptyDir = tmpDirFixture.createNewTmpDir(); const result = await getProcessResult( execFile(nodePath, [cliEntryPointPath, "init-creds", emptyDir]) ); expect(result.exitCode).to.not.equal(0); expect(result.stderr).to.include( - `${emptyDir} does not appear to be a dataform directory (missing workflow_settings.yaml file).` + `${emptyDir} does not appear to be a sqlanvil directory (missing workflow_settings.yaml file).` ); }); }); diff --git a/cli/index_project_test.ts b/cli/index_project_test.ts index 35059d0f..e13d8766 100644 --- a/cli/index_project_test.ts +++ b/cli/index_project_test.ts @@ -4,17 +4,17 @@ import * as fs from "fs-extra"; import { dump as dumpYaml, load as loadYaml } from "js-yaml"; import * as path from "path"; -import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "df/cli/index_test_base"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { cliEntryPointPath, DEFAULT_DATABASE, DEFAULT_LOCATION } from "sa/cli/index_test_base"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("project ops", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); suite("install command", () => { - test("install throws an error when dataformCoreVersion in workflow_settings.yaml", async () => { + test("install throws an error when sqlanvilCoreVersion in workflow_settings.yaml", async () => { const projectDir = tmpDirFixture.createNewTmpDir(); await getProcessResult( @@ -22,7 +22,7 @@ suite("project ops", ({ afterEach }) => { cliEntryPointPath, "init", projectDir, - "--default-database=dataform-database", + "--default-database=sqlanvil-database", "--default-location=us-central1" ]) ); @@ -50,16 +50,16 @@ suite("project ops", ({ afterEach }) => { ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); diff --git a/cli/index_run_e2e_test.ts b/cli/index_run_e2e_test.ts index da19b4af..b4e9d737 100644 --- a/cli/index_run_e2e_test.ts +++ b/cli/index_run_e2e_test.ts @@ -10,11 +10,11 @@ import { DEFAULT_DATABASE, DEFAULT_LOCATION, DEFAULT_RESERVATION -} from "df/cli/index_test_base"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; -import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/cli/index_test_base"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; +import { corePackageTarPath, getProcessResult, nodePath, npmPath, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; suite("run e2e", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -31,16 +31,16 @@ suite("run e2e", ({ afterEach }) => { ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -62,7 +62,7 @@ suite("run e2e", ({ afterEach }) => { filePath, ` config { type: "table", tags: ["someTag"] } -select 1 as \${dataform.projectConfig.vars.testVar2} +select 1 as \${sqlanvil.projectConfig.vars.testVar2} ` ); @@ -87,11 +87,11 @@ select 1 as \${dataform.projectConfig.vars.testVar2} enumType: "TABLE", target: { database: DEFAULT_DATABASE, - schema: "dataform_test_schema_suffix", + schema: "sqlanvil_test_schema_suffix", name: "example" }, canonicalTarget: { - schema: "dataform", + schema: "sqlanvil", name: "example", database: DEFAULT_DATABASE }, @@ -104,8 +104,8 @@ select 1 as \${dataform.projectConfig.vars.testVar2} ], projectConfig: { warehouse: "bigquery", - defaultSchema: "dataform", - assertionSchema: "dataform_assertions", + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, vars: { @@ -116,11 +116,11 @@ select 1 as \${dataform.projectConfig.vars.testVar2} }, graphErrors: {}, jitData: {}, - dataformCoreVersion: version, + sqlanvilCoreVersion: version, targets: [ { database: DEFAULT_DATABASE, - schema: "dataform", + schema: "sqlanvil", name: "example" } ] @@ -158,13 +158,13 @@ select 1 as \${dataform.projectConfig.vars.testVar2} target: { database: DEFAULT_DATABASE, name: "example", - schema: "dataform" + schema: "sqlanvil" }, tasks: [ { statement: // tslint:disable-next-line:tsr-detect-sql-literal-injection - `create or replace table \`${DEFAULT_DATABASE}.dataform.example\` as \n\nselect 1 as testValue2`, + `create or replace table \`${DEFAULT_DATABASE}.sqlanvil.example\` as \n\nselect 1 as testValue2`, type: "statement" } ], @@ -172,10 +172,10 @@ select 1 as \${dataform.projectConfig.vars.testVar2} } ], projectConfig: { - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: "europe", - defaultSchema: "dataform", + defaultSchema: "sqlanvil", warehouse: "bigquery", vars: { testVar1: "testValue1", @@ -203,17 +203,17 @@ select 1 as \${dataform.projectConfig.vars.testVar2} ); const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -256,10 +256,10 @@ SELECT 1 as id async function setUpWorkflowSettings(disableAssertions: boolean): Promise { const workflowSettingsPath = path.join(projectDir, "workflow_settings.yaml"); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; workflowSettings.disableAssertions = disableAssertions; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); } @@ -275,13 +275,13 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "example_table", - schema: "dataform" + schema: "sqlanvil" }, tasks: [ { statement: // tslint:disable-next-line:tsr-detect-sql-literal-injection - `create or replace table \`${DEFAULT_DATABASE}.dataform.example_table\` as \n\nSELECT 1 as id`, + `create or replace table \`${DEFAULT_DATABASE}.sqlanvil.example_table\` as \n\nSELECT 1 as id`, type: "statement" } ], @@ -293,16 +293,16 @@ SELECT 1 as id target: { database: DEFAULT_DATABASE, name: "test_assertion", - schema: "dataform_assertions" + schema: "sqlanvil_assertions" }, type: "assertion", } ], projectConfig: { - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, - defaultSchema: "dataform", + defaultSchema: "sqlanvil", disableAssertions: true, warehouse: "bigquery" }, @@ -376,7 +376,7 @@ SELECT 1 as id "--json", "--disable-assertions", "--actions=test_assertion,example_table", - "--job-labels=env=testing,team=dataform" + "--job-labels=env=testing,team=sqlanvil" ]) ); @@ -402,18 +402,18 @@ SELECT 1 as id execFile(nodePath, [cliEntryPointPath, "init", projectDir, DEFAULT_DATABASE, DEFAULT_LOCATION]) ); - // Remove dataformCoreVersion so we can use the local package. - const workflowSettings = dataform.WorkflowSettings.create( + // Remove sqlanvilCoreVersion so we can use the local package. + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -454,8 +454,8 @@ SELECT 1 as id const compiledGraph = JSON.parse(compileResult.stdout); expect(compiledGraph.projectConfig).deep.equals({ warehouse: "bigquery", - defaultSchema: "dataform", - assertionSchema: "dataform_assertions", + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, defaultReservation: DEFAULT_RESERVATION @@ -481,8 +481,8 @@ SELECT 1 as id const executionGraph = JSON.parse(runResult.stdout); expect(executionGraph.projectConfig).deep.equals({ warehouse: "bigquery", - defaultSchema: "dataform", - assertionSchema: "dataform_assertions", + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION, defaultReservation: DEFAULT_RESERVATION @@ -502,16 +502,16 @@ SELECT 1 as id ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -579,16 +579,16 @@ select 1 ); // Install packages manually to get around bazel read-only sandbox issues. - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); fs.writeFileSync( packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -649,7 +649,7 @@ select 2 suite("onSchemaChange", ({ beforeEach }) => { const projectDir = tmpDirFixture.createNewTmpDir(); - const uniqueDataset = `dataform_e2e_osc_${Math.random().toString(36).substring(7)}`; + const uniqueDataset = `sqlanvil_e2e_osc_${Math.random().toString(36).substring(7)}`; beforeEach("setup test project", async () => { const npmCacheDir = tmpDirFixture.createNewTmpDir(); @@ -660,10 +660,10 @@ select 2 execFile(nodePath, [cliEntryPointPath, "init", projectDir, DEFAULT_DATABASE, DEFAULT_LOCATION]) ); - const workflowSettings = dataform.WorkflowSettings.create( + const workflowSettings = sqlanvil.WorkflowSettings.create( loadYaml(fs.readFileSync(workflowSettingsPath, "utf8")) ); - delete workflowSettings.dataformCoreVersion; + delete workflowSettings.sqlanvilCoreVersion; workflowSettings.defaultDataset = uniqueDataset; fs.writeFileSync(workflowSettingsPath, dumpYaml(workflowSettings)); @@ -671,7 +671,7 @@ select 2 packageJsonPath, `{ "dependencies":{ - "@dataform/core": "${version}" + "@sqlanvil/core": "${version}" } }` ); @@ -693,7 +693,7 @@ select 2 config { type: "operations" } -CREATE OR REPLACE TABLE \`\${dataform.projectConfig.defaultDatabase}.\${dataform.projectConfig.defaultSchema}.example_incremental\` AS SELECT 1 AS id, 'old' AS field1 +CREATE OR REPLACE TABLE \`\${sqlanvil.projectConfig.defaultDatabase}.\${sqlanvil.projectConfig.defaultSchema}.example_incremental\` AS SELECT 1 AS id, 'old' AS field1 ` ); @@ -716,7 +716,7 @@ SELECT 1 as id, 'new' as field1, 'new2' as field2 config { type: "operations" } -DROP SCHEMA IF EXISTS \`\${dataform.projectConfig.defaultDatabase}.\${dataform.projectConfig.defaultSchema}\` CASCADE +DROP SCHEMA IF EXISTS \`\${sqlanvil.projectConfig.defaultDatabase}.\${sqlanvil.projectConfig.defaultSchema}\` CASCADE ` ); }); @@ -724,7 +724,7 @@ DROP SCHEMA IF EXISTS \`\${dataform.projectConfig.defaultDatabase}.\${dataform.p test("generates dynamic SQL for EXTEND when table exists in BigQuery", async () => { try { // Run setup operation to create the table in BigQuery. - // Dataform will automatically create the uniqueDataset schema. + // sqlanvil will automatically create the uniqueDataset schema. await getProcessResult( execFile(nodePath, [ cliEntryPointPath, @@ -737,7 +737,7 @@ DROP SCHEMA IF EXISTS \`\${dataform.projectConfig.defaultDatabase}.\${dataform.p ); // Run the incremental table in dry-run mode. - // Dataform will detect the table exists and generate the dynamic procedural SQL. + // sqlanvil will detect the table exists and generate the dynamic procedural SQL. const runResult = await getProcessResult( execFile(nodePath, [ cliEntryPointPath, @@ -759,7 +759,7 @@ DROP SCHEMA IF EXISTS \`\${dataform.projectConfig.defaultDatabase}.\${dataform.p projectConfig: { warehouse: "bigquery", defaultSchema: uniqueDataset, - assertionSchema: "dataform_assertions", + assertionSchema: "sqlanvil_assertions", defaultDatabase: DEFAULT_DATABASE, defaultLocation: DEFAULT_LOCATION }, diff --git a/cli/index_test_base.ts b/cli/index_test_base.ts index b47f53fe..af2fa431 100644 --- a/cli/index_test_base.ts +++ b/cli/index_test_base.ts @@ -1,9 +1,9 @@ // tslint:disable tsr-detect-non-literal-fs-filename import * as path from "path"; -export const DEFAULT_DATABASE = "dataform-open-source"; +export const DEFAULT_DATABASE = "your-bigquery-project"; export const DEFAULT_LOCATION = "US"; -export const DEFAULT_RESERVATION = "projects/dataform-open-source/locations/us/reservations/dataform-test"; -export const CREDENTIALS_PATH = path.resolve(process.env.RUNFILES, "df/test_credentials/bigquery.json"); +export const DEFAULT_RESERVATION = "projects/your-bigquery-project/locations/us/reservations/sqlanvil-test"; +export const CREDENTIALS_PATH = path.resolve(process.env.RUNFILES, "sa/test_credentials/bigquery.json"); -export const cliEntryPointPath = "cli/node_modules/@dataform/cli/bundle.js"; +export const cliEntryPointPath = "cli/node_modules/@sqlanvil/cli/bundle.js"; diff --git a/cli/util.ts b/cli/util.ts index dcd77b37..fa5574fc 100644 --- a/cli/util.ts +++ b/cli/util.ts @@ -7,9 +7,9 @@ import { print, printError, printSuccess, -} from "df/cli/console"; -import {validateConnectionFormat} from "df/core/utils" -import { dataform } from "df/protos/ts"; +} from "sa/cli/console"; +import {validateConnectionFormat} from "sa/core/utils" +import { sqlanvil } from "sa/protos/ts"; export function actuallyResolve(...filePaths: string[]) { return path.resolve(...filePaths.map(filePath => untildify(filePath))); @@ -21,7 +21,7 @@ export function assertPathExists(checkPath: string) { } } -export function compiledGraphHasErrors(graph: dataform.ICompiledGraph) { +export function compiledGraphHasErrors(graph: sqlanvil.ICompiledGraph) { return graph.graphErrors?.compilationErrors?.length > 0; } @@ -38,7 +38,7 @@ export function formatExecutionSuffix(jobIds: string[], bytesBilled: string[]): export function formatBytesInHumanReadableFormat(bytes: number): string { // we do not want to raise an error when bytes < 0 - // because it will fail Dataform run command when in fact the BQ job was executed. + // because it will fail sqlanvil run command when in fact the BQ job was executed. if (bytes <= 0) {return '0 B';} const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB', 'PiB']; @@ -60,11 +60,11 @@ export function formatBytesInHumanReadableFormat(bytes: number): string { * @returns Constructed DefaultIcebergConfig object, or undefined if no inputs * were provided. */ -export function promptForIcebergConfig(): dataform.IDefaultIcebergConfig | undefined { +export function promptForIcebergConfig(): sqlanvil.IDefaultIcebergConfig | undefined { print(ICEBERG_CONFIG_PROMPT_TEXT); print(ICEBERG_CONFIG_PROMPT_HINT); - const tempIcebergConfig: dataform.IDefaultIcebergConfig = {}; + const tempIcebergConfig: sqlanvil.IDefaultIcebergConfig = {}; let bucketName: string; while (true) { diff --git a/cli/util_test.ts b/cli/util_test.ts index b86f8937..45eadb90 100644 --- a/cli/util_test.ts +++ b/cli/util_test.ts @@ -6,16 +6,16 @@ import { validateIcebergConfigBucketName, validateIcebergConfigTableFolderRoot, validateIcebergConfigTableFolderSubpath, -} from "df/cli/util"; -import { suite, test } from "df/testing"; +} from "sa/cli/util"; +import { suite, test } from "sa/testing"; suite('format execution suffix', () => { test('format execution suffix', () => { expect(formatExecutionSuffix([], [])).deep.equals(''); - expect(formatExecutionSuffix(["dataform-915a03fe1"], [])).deep.equals(" \n \t jobId: dataform-915a03fe1"); + expect(formatExecutionSuffix(["sqlanvil-915a03fe1"], [])).deep.equals(" \n \t jobId: sqlanvil-915a03fe1"); expect(formatExecutionSuffix([], ["10 MiB"])).deep.equals(" \n \t Bytes billed: 10 MiB"); - expect(formatExecutionSuffix(["dataform-915a03fe1"], ["17 KiB"])).deep.equals(" \n \t jobId: dataform-915a03fe1,\n \t Bytes billed: 17 KiB"); - expect(formatExecutionSuffix(["dataform-915a03fe1", "dataform-915a03fe2"], ["17 KiB", "1 GiB"])).deep.equals(" \n \t jobId: dataform-915a03fe1, dataform-915a03fe2,\n \t Bytes billed: 17 KiB, 1 GiB"); + expect(formatExecutionSuffix(["sqlanvil-915a03fe1"], ["17 KiB"])).deep.equals(" \n \t jobId: sqlanvil-915a03fe1,\n \t Bytes billed: 17 KiB"); + expect(formatExecutionSuffix(["sqlanvil-915a03fe1", "sqlanvil-915a03fe2"], ["17 KiB", "1 GiB"])).deep.equals(" \n \t jobId: sqlanvil-915a03fe1, sqlanvil-915a03fe2,\n \t Bytes billed: 17 KiB, 1 GiB"); }); }); diff --git a/cli/vm/compile.ts b/cli/vm/compile.ts index 7faefb46..424a9415 100644 --- a/cli/vm/compile.ts +++ b/cli/vm/compile.ts @@ -4,21 +4,21 @@ import * as path from "path"; import * as semver from "semver"; import { CompilerFunction, NodeVM } from "vm2"; -import { encode64 } from "df/common/protos"; -import { dataform } from "df/protos/ts"; +import { encode64 } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; -export function compile(compileConfig: dataform.ICompileConfig) { +export function compile(compileConfig: sqlanvil.ICompileConfig) { compileConfig.projectDir = fs.realpathSync(path.resolve(compileConfig.projectDir)); if ( !fs.existsSync( - path.join(compileConfig.projectDir, "node_modules", "@dataform", "core", "bundle.js") + path.join(compileConfig.projectDir, "node_modules", "@sqlanvil", "core", "bundle.js") ) ) { throw new Error( - "Could not find a recent installed version of @dataform/core in the project. Check that " + - "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + - "`@dataform/core` is specified in `package.json`. If using `package.json`, then run " + - "`dataform install`." + "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + + "`sqlanvil install`." ); } const vmIndexFileName = path.resolve(path.join(compileConfig.projectDir, "index.js")); @@ -34,7 +34,7 @@ export function compile(compileConfig: dataform.ICompileConfig) { } }); const compiler: CompilerFunction = indexGeneratorVm.run( - 'return require("@dataform/core").compiler', + 'return require("@sqlanvil/core").compiler', vmIndexFileName ); @@ -51,44 +51,43 @@ export function compile(compileConfig: dataform.ICompileConfig) { }, sourceExtensions: ["js", "sql", "sqlx", "yaml", "yml"], // vm2 3.11.3 strips file paths from V8 CallSite objects inside the sandbox, - // which breaks getCallerFile() in @dataform/core. Wrap each compiled module so + // which breaks getCallerFile() in @sqlanvil/core. Wrap each compiled module so // the current file path is exposed via a global, used as a fallback when the // stack-trace path is unavailable. The try/finally restores the previous value // to keep nested requires (macros) consistent. compiler: (code, filePath) => { const compiledCode = compiler(code, filePath); return ` - var __old_file = global.__dataform_current_file; - global.__dataform_current_file = ${JSON.stringify(filePath)}; + var __old_file = global.__sqlanvil_current_file; + global.__sqlanvil_current_file = ${JSON.stringify(filePath)}; try { ${compiledCode} } finally { - global.__dataform_current_file = __old_file; + global.__sqlanvil_current_file = __old_file; } `; } }); - const dataformCoreVersion: string = userCodeVm.run( - 'return require("@dataform/core").version || "0.0.0"', + const sqlanvilCoreVersion: string = userCodeVm.run( + 'return require("@sqlanvil/core").version || "0.0.0"', vmIndexFileName ); - if (semver.lt(dataformCoreVersion, "3.0.0-alpha.0")) { - throw new Error("@dataform/core ^3.0.0 required."); + if (semver.lt(sqlanvilCoreVersion, "3.0.0-alpha.0")) { + throw new Error("@sqlanvil/core ^3.0.0 required."); } return userCodeVm.run( ` global.workflowSettingsYaml = (function() { try { return require("./workflow_settings.yaml"); } catch(e) { console.error("YAML require failed:", e); } })(); - global.dataformJson = (function() { try { return require("./dataform.json"); } catch(e) {} })(); - return require("@dataform/core").main("${createCoreExecutionRequest(compileConfig)}") + return require("@sqlanvil/core").main("${createCoreExecutionRequest(compileConfig)}") `, vmIndexFileName ); } export function listenForCompileRequest() { - process.on("message", (compileConfig: dataform.ICompileConfig) => { + process.on("message", (compileConfig: sqlanvil.ICompileConfig) => { try { const compiledResult = compile(compileConfig); process.send(compiledResult); @@ -107,14 +106,14 @@ if (require.main === module) { } /** - * @returns a base64 encoded @see {@link dataform.CoreExecutionRequest} proto. + * @returns a base64 encoded @see {@link sqlanvil.CoreExecutionRequest} proto. */ -function createCoreExecutionRequest(compileConfig: dataform.ICompileConfig): string { +function createCoreExecutionRequest(compileConfig: sqlanvil.ICompileConfig): string { const filePaths = Array.from( new Set(glob.sync("!(node_modules)/**/*.*", { cwd: compileConfig.projectDir })) ); - return encode64(dataform.CoreExecutionRequest, { + return encode64(sqlanvil.CoreExecutionRequest, { // Add the list of file paths to the compile config if not already set. compile: { compileConfig: { filePaths, ...compileConfig } } }); diff --git a/cli/vm/jit_loader.js b/cli/vm/jit_loader.js index cb325cee..30633376 100644 --- a/cli/vm/jit_loader.js +++ b/cli/vm/jit_loader.js @@ -1,7 +1,7 @@ 'use strict'; if (require.main === module) { - var entryPointPath = 'df/cli/vm/jit_worker.js'; + var entryPointPath = 'sa/cli/vm/jit_worker.js'; var mainScript = process.argv[1] = entryPointPath; try { module.constructor._load(mainScript, this, /*isMain=*/true); diff --git a/cli/vm/jit_worker.ts b/cli/vm/jit_worker.ts index 5b24439c..91ca9fac 100644 --- a/cli/vm/jit_worker.ts +++ b/cli/vm/jit_worker.ts @@ -2,7 +2,7 @@ import * as fs from "fs"; import * as path from "path"; import { NodeVM } from "vm2"; -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; const pendingRpcCallbacks = new Map void>(); let hasStartedProcessing = false; @@ -28,12 +28,12 @@ export async function handleJitRequest(message: { try { const { request, projectDir } = message; - if (!fs.existsSync(path.join(projectDir, "node_modules", "@dataform", "core", "bundle.js"))) { + if (!fs.existsSync(path.join(projectDir, "node_modules", "@sqlanvil", "core", "bundle.js"))) { throw new Error( - "Could not find a recent installed version of @dataform/core in the project. Check that " + - "either `dataformCoreVersion` is specified in `workflow_settings.yaml`, or " + - "`@dataform/core` is specified in `package.json`. If using `package.json`, then run " + - "`dataform install`." + "Could not find a recent installed version of @sqlanvil/core in the project. Check that " + + "either `sqlanvilCoreVersion` is specified in `workflow_settings.yaml`, or " + + "`@sqlanvil/core` is specified in `package.json`. If using `package.json`, then run " + + "`sqlanvil install`." ); } @@ -49,8 +49,8 @@ export async function handleJitRequest(message: { }); }; - const requestMessage = dataform.JitCompilationRequest.fromObject(request); - const requestBytes = dataform.JitCompilationRequest.encode(requestMessage).finish(); + const requestMessage = sqlanvil.JitCompilationRequest.fromObject(request); + const requestBytes = sqlanvil.JitCompilationRequest.encode(requestMessage).finish(); const vmFileName = path.resolve(projectDir, "index.js"); @@ -60,7 +60,7 @@ export async function handleJitRequest(message: { builtin: [], context: "sandbox", external: { - modules: ["@dataform/*"] + modules: ["@sqlanvil/*"] }, root: projectDir }, @@ -68,7 +68,7 @@ export async function handleJitRequest(message: { }); const jitCompileInVm = vm.run(` - const { jitCompiler } = require("@dataform/core"); + const { jitCompiler } = require("@sqlanvil/core"); global.require = require; @@ -89,7 +89,7 @@ export async function handleJitRequest(message: { `, vmFileName); const responseBytes = await jitCompileInVm(requestBytes, rpcCallback); - const response = dataform.JitCompilationResponse.decode(responseBytes); + const response = sqlanvil.JitCompilationResponse.decode(responseBytes); process.send({ type: "jit_response", response: response.toJSON() }); } catch (e) { diff --git a/cli/yargswrapper.ts b/cli/yargswrapper.ts index c659e309..0b979e17 100644 --- a/cli/yargswrapper.ts +++ b/cli/yargswrapper.ts @@ -59,8 +59,8 @@ function createOptionsChain(yargsChain: yargs.Argv, command: ICommand) { function fixArgvForHelp() { // Obviously this is a massive hack. // The outcome of this is that the following commands are interchangeable: - // $ dataform help run - // $ dataform --help run + // $ sqlanvil help run + // $ sqlanvil --help run // The problem is that yargs.help() only allows us to specify an alias for the "--help" built-in option (by default that alias is "help"). // But because "--help" is only an option, not a command, it appears to be impossible (?) to configure yargs to respond to "help" correctly // (or at least, to correctly print help strings for commands; it happily prints a top-level help string). diff --git a/cloudbuild-publish.yaml b/cloudbuild-publish.yaml deleted file mode 100644 index eaea1f75..00000000 --- a/cloudbuild-publish.yaml +++ /dev/null @@ -1,35 +0,0 @@ -steps: - - name: 'gcr.io/cloud-builders/gcloud' - script: | - #!/usr/bin/env bash - gcloud secrets versions access latest --secret=github-token-access --format='get(payload.data)' | tr '_-' '/+' | base64 -d > ~/token.txt - REPO_TOKEN="$(gcloud auth print-access-token)" NPM_TOKEN=$NPM_TOKEN ./scripts/create_npmrc - secretEnv: ['NPM_TOKEN'] - - name: gcr.io/cloud-builders/bazel:5.4.0 - script: | - #!/usr/bin/env bash - ./scripts/publish - - name: 'gcr.io/$PROJECT_ID/github' - entrypoint: 'bash' - args: - - '-c' - - | - set -e - version=$(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g") - echo "Creating release notes for $version" - gh auth login --with-token < ~/token.txt - gh release create $version --generate-notes -availableSecrets: - secretManager: - - versionName: projects/178487900909/secrets/npm-publish-token/versions/latest - env: 'NPM_TOKEN' -artifacts: - npmPackages: - - repository: 'https://us-central1-npm.pkg.dev/dataform-open-source/dataform-open-source' - packagePath: './bazel-bin/packages/@dataform/cli/package' - - repository: 'https://us-central1-npm.pkg.dev/dataform-open-source/dataform-open-source' - packagePath: './bazel-bin/packages/@dataform/core/package' -options: - machineType: E2_HIGHCPU_8 - requestedVerifyOption: VERIFIED -timeout: 3600s diff --git a/cloudbuild-test.yaml b/cloudbuild-test.yaml deleted file mode 100644 index 09207ec0..00000000 --- a/cloudbuild-test.yaml +++ /dev/null @@ -1,12 +0,0 @@ -steps: - - name: 'gcr.io/cloud-builders/gcloud' - script: | - #!/usr/bin/env bash - REPO_TOKEN="$(gcloud auth print-access-token)" NPM_TOKEN="test_only" ./scripts/create_npmrc - - name: gcr.io/cloud-builders/bazel:5.4.0 - script: | - #!/usr/bin/env bash - ./scripts/run_tests_on_cloudbuild -options: - machineType: E2_HIGHCPU_8 -timeout: 3600s diff --git a/cloudbuild-version.yaml b/cloudbuild-version.yaml deleted file mode 100644 index 2bbbf86a..00000000 --- a/cloudbuild-version.yaml +++ /dev/null @@ -1,23 +0,0 @@ -steps: -# We have github personal access token stored in `github-token-access` secret in our GCP project. -# This step downloads it and store it in token.txt file for later steps to use for authentication. -- name: gcr.io/cloud-builders/gcloud - entrypoint: 'bash' - args: [ '-c', "gcloud secrets versions access latest --secret=github-token-access --format='get(payload.data)' | tr '_-' '/+' | base64 -d > token.txt" ] -- name: gcr.io/cloud-builders/git - entrypoint: 'bash' - args: - - '-c' - - | - _GITHUB_USER=$_GITHUB_USER _GITHUB_EMAIL=$_GITHUB_EMAIL ./scripts/create_gh_pr -- name: 'gcr.io/$PROJECT_ID/github' - entrypoint: 'bash' - args: - - '-c' - - | - set -e - echo "Create PR..." - gh auth login --with-token < token.txt - gh pr create -t "Publishing Dataform security patches" -b "Updating NPM package version to $(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g")" -B $BRANCH_NAME -H $(cat git_branch_name.txt) -options: - automapSubstitutions: true diff --git a/common/errors/BUILD b/common/errors/BUILD index b806837f..e28687ba 100644 --- a/common/errors/BUILD +++ b/common/errors/BUILD @@ -1,4 +1,4 @@ -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//testing:index.bzl", "ts_test_suite") load("//tools:ts_library.bzl", "ts_library") package(default_visibility = ["//visibility:public"]) @@ -26,7 +26,7 @@ ts_test_suite( ], deps = [ ":errors", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/node", "@npm//chai", diff --git a/common/errors/errors.spec.ts b/common/errors/errors.spec.ts index a1a19c50..c4b0ea2c 100644 --- a/common/errors/errors.spec.ts +++ b/common/errors/errors.spec.ts @@ -1,8 +1,8 @@ import { expect } from "chai"; import { basename } from "path"; -import { coerceAsError, ErrorWithCause } from "df/common/errors/errors"; -import { suite, test } from "df/testing"; +import { coerceAsError, ErrorWithCause } from "sa/common/errors/errors"; +import { suite, test } from "sa/testing"; suite(basename(__filename), () => { suite("ErrorWithCause", () => { diff --git a/common/flags/testing/index.ts b/common/flags/testing/index.ts index fa8f7843..1a69d4b3 100644 --- a/common/flags/testing/index.ts +++ b/common/flags/testing/index.ts @@ -1,5 +1,5 @@ -import { Flags } from "df/common/flags"; -import { IHookHandler } from "df/testing"; +import { Flags } from "sa/common/flags"; +import { IHookHandler } from "sa/testing"; // FlagOverridesFixture can be used to temporarily override flag values for testing purposes. // At the end of the test run, any previous flag values will be restored. diff --git a/common/promises/BUILD b/common/promises/BUILD index d34deba1..73de9aba 100644 --- a/common/promises/BUILD +++ b/common/promises/BUILD @@ -1,5 +1,5 @@ -load("@df//tools:ts_library.bzl", "ts_library") -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//tools:ts_library.bzl", "ts_library") +load("@sa//testing:index.bzl", "ts_test_suite") package(default_visibility = ["//visibility:public"]) @@ -27,7 +27,7 @@ ts_test_suite( ], deps = [ ":promises", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/node", "@npm//chai", diff --git a/common/promises/index.spec.ts b/common/promises/index.spec.ts index 472fec4e..c2332982 100644 --- a/common/promises/index.spec.ts +++ b/common/promises/index.spec.ts @@ -1,8 +1,8 @@ import { fail } from "assert"; import { assert, expect } from "chai"; -import { retry, runWithTimeout, sleep } from "df/common/promises"; -import { suite, test } from "df/testing"; +import { retry, runWithTimeout, sleep } from "sa/common/promises"; +import { suite, test } from "sa/testing"; suite(__filename, () => { suite("runWithTimeout", () => { diff --git a/common/protos/BUILD b/common/protos/BUILD index ffb46ca4..f7f43657 100644 --- a/common/protos/BUILD +++ b/common/protos/BUILD @@ -1,7 +1,7 @@ package(default_visibility = ["//visibility:public"]) load("//tools:ts_library.bzl", "ts_library") -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//testing:index.bzl", "ts_test_suite") ts_library( name = "protos", @@ -30,7 +30,7 @@ ts_test_suite( deps = [ ":protos", "//protos:ts", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/node", "@npm//chai", diff --git a/common/protos/index.ts b/common/protos/index.ts index 2bf2777b..84ffa8b5 100644 --- a/common/protos/index.ts +++ b/common/protos/index.ts @@ -1,10 +1,10 @@ import { util } from "protobufjs"; -import { google } from "df/protos/ts"; +import { google } from "sa/protos/ts"; const CONFIGS_PROTO_DOCUMENTATION_URL = - "https://dataform-co.github.io/dataform/docs/configs-reference"; -const REPORT_ISSUE_URL = "https://github.com/dataform-co/dataform/issues"; + "https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md"; +const REPORT_ISSUE_URL = "https://github.com/ihistand/sqlanvil/issues"; export interface IProtoClass { new (): Proto; @@ -92,7 +92,7 @@ export function verifyObjectMatchesProto( throw ReferenceError( `Unexpected property "${presentKey}" for "${protoType .getTypeUrl("") - .replace("/", "")}", please report this to the Dataform team at ` + + .replace("/", "")}", please report this to the sqlanvil team at ` + `${REPORT_ISSUE_URL}.` ); } diff --git a/common/protos/index_test.ts b/common/protos/index_test.ts index 241c79a3..6db2fd1e 100644 --- a/common/protos/index_test.ts +++ b/common/protos/index_test.ts @@ -1,20 +1,20 @@ import { expect } from "chai"; import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "./index"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("verifyObjectMatchesProto", () => { test("throws error when top-level object is an array", () => { expect(() => { - verifyObjectMatchesProto(dataform.Target, [] as any); + verifyObjectMatchesProto(sqlanvil.Target, [] as any); }).to.throw(ReferenceError, "Expected a top-level object, but found an array"); }); test("throws error when null value provided for array field and SHOW_DOCS_LINK", () => { expect(() => { verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, { dependencyTargets: null } as any, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -24,19 +24,19 @@ suite("verifyObjectMatchesProto", () => { test("throws error on type mismatch with SUGGEST_REPORTING_TO_DATAFORM_TEAM", () => { expect(() => { verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, { actionDescriptor: 123 } as any, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); }).to.throw( ReferenceError, - /Unexpected property "actionDescriptor" for ".*Table".*please report this to the Dataform team/ + /Unexpected property "actionDescriptor" for ".*Table".*please report this to the sqlanvil team/ ); }); test("throws error on default type mismatch", () => { expect(() => { - verifyObjectMatchesProto(dataform.Table, { actionDescriptor: 123 } as any); + verifyObjectMatchesProto(sqlanvil.Table, { actionDescriptor: 123 } as any); }).to.throw( ReferenceError, /Unexpected property "actionDescriptor", or property value type of "number" is incorrect/ diff --git a/common/protos/structs.ts b/common/protos/structs.ts index c0b704f9..2e76c819 100644 --- a/common/protos/structs.ts +++ b/common/protos/structs.ts @@ -1,4 +1,4 @@ -import { google } from "df/protos/ts"; +import { google } from "sa/protos/ts"; export type AnyValue = null | number | string | boolean | undefined | AnyValue[] | { [key: string]: AnyValue }; diff --git a/common/strings/BUILD b/common/strings/BUILD index 6e171d1b..80a64b0e 100644 --- a/common/strings/BUILD +++ b/common/strings/BUILD @@ -1,4 +1,4 @@ -load("@df//testing:index.bzl", "ts_test_suite") +load("@sa//testing:index.bzl", "ts_test_suite") load("//tools:ts_library.bzl", "ts_library") package(default_visibility = ["//visibility:public"]) @@ -28,7 +28,7 @@ ts_test_suite( ], deps = [ ":strings", - "@df//testing", + "@sa//testing", "@npm//@types/chai", "@npm//@types/long", "@npm//@types/node", diff --git a/common/strings/stringifier.spec.ts b/common/strings/stringifier.spec.ts index 3d25abcd..9cc96ddc 100644 --- a/common/strings/stringifier.spec.ts +++ b/common/strings/stringifier.spec.ts @@ -1,8 +1,8 @@ import { expect } from "chai"; import { basename } from "path"; -import { JSONObjectStringifier } from "df/common/strings/stringifier"; -import { suite, test } from "df/testing"; +import { JSONObjectStringifier } from "sa/common/strings/stringifier"; +import { suite, test } from "sa/testing"; interface IKey { a: string; diff --git a/contributing.md b/contributing.md index 36831d06..04a9bc78 100644 --- a/contributing.md +++ b/contributing.md @@ -1,162 +1,110 @@ # Contributing -Dataform is a TypeScript project, using [Bazel](https://bazel.build) as a build tool. To scope out work, please [check existing issues (which includes feature requests)](https://github.com/dataform-co/dataform/issues) or [open a discussion thread](https://github.com/dataform-co/dataform/discussions)! +sqlanvil is a TypeScript project, using [Bazel](https://bazel.build) as a build tool. ## Getting Started -First :fork_and_knife: [fork this repository](https://github.com/dataform-co/dataform/fork), clone it to your desktop, and navigate within. +[Fork the repository](https://github.com/ihistand/sqlanvil/fork), clone it, and navigate inside. ### Requirements #### [Bazel](https://bazel.build) -Bazel is a build system which we used to build the project and run tests. +Bazel is the build system. Install via Bazelisk: -The easiest way to install the correct Bazel version is through Bazelisk via [NPM](https://nodejs.org/en/download/): - -``` +```bash +brew install bazelisk # macOS +# or: npm i -g @bazel/bazelisk ``` -### Run the CLI - -You can run the project as you would the `npm` installation of `@dataform/cli`, but replace `dataform` with `./scripts/run`. - -For example, to print out the default help information: +On macOS, increase the open-file limit (Bazel hits the default): ```bash -./scripts/run help +sudo sysctl -w kern.maxfiles=65536 ``` -Check the [docs](https://cloud.google.com/dataform/docs/reference/dataform-cli-reference) for more examples. - -_Note: If you are running Bazel on a **Mac**, this or any step that requires building may fail with a `Too many open files in system` error. This is [due to a limitation](https://github.com/angular/angular-bazel-example/issues/178) on the default maximum open file descriptors. You can increase the limit by running `sudo sysctl -w kern.maxfiles=` (we use `65536`)._ +##### macOS compatibility (important) -### Test +The currently pinned Bazel 5.4 + 2022-era protobuf chain inherited from +upstream **does not build natively on macOS Tahoe / Apple Silicon** — +`wrapped_clang` ships without `LC_UUID` (rejected by current dyld) and the +old protobuf headers conflict with Xcode 21's SDK. This will be fixed by +a future toolchain modernization PR (Bazel 7 + Bzlmod migration). -The following command runs tests for @dataform/core: +Until then, build via Docker on macOS: ```bash -bazel test //core/... +./scripts/docker-bazel build //protos:sqlanvil_proto +./scripts/docker-bazel test //core/... +./scripts/docker-bazel build //... +./scripts/docker-bazel # drops into an interactive shell ``` -### Integration Test - -To run the CLI integration test against your own GCP project: - -1. Comment out the following dependency in `cli/BUILD`: - - - `//test_credentials:bigquery.json` +The wrapper builds a `sqlanvil-dev` image (Debian + Node 20 + JDK 17 + +Bazelisk) on first invocation and reuses named volumes for the Bazel cache +so subsequent runs are fast. -2. Update the following constants in `cli/index_test.ts` to match your project: +Linux users can use Bazelisk directly without Docker. - - `DEFAULT_DATABASE` - - `DEFAULT_LOCATION` - - `CREDENTIALS_PATH` +### Run the CLI - Prepare a credentials JSON file referenced by `CREDENTIALS_PATH`. Set values as follows: +Substitute `./scripts/run` for the installed `sqlanvil` binary: - - `projectId`: the same string as `DEFAULT_DATABASE`. - - `credentials`: the entire content of your GCP service account key JSON file as a single string (you can generate it with `jq -Rsa < path/to/key.json`). - - `location`: the same string as `DEFAULT_LOCATION`. +```bash +./scripts/run help +./scripts/run compile path/to/project +``` - Example: +### Test - ```json - { - "projectId": "my-gcp-project", - "credentials": "{\"type\":\"service_account\",...}", - "location": "US" - } - ``` +```bash +bazel test //core/... # core compiler tests +bazel test //... # everything (slow on cold cache) +``` -3. Run the test: +### Integration Tests - ```bash - bazel test //cli:index_test - ``` +Integration tests require real warehouse credentials. The upstream +`test_credentials` GCP project is no longer accessible; you'll need to wire +your own. -### Lint +For BigQuery integration tests: -The following command to check for any linting errors +1. Create a GCP service account with BigQuery access. +2. Download the key JSON. +3. Update constants in `cli/index_test_base.ts` to match your project + (`DEFAULT_DATABASE`, `DEFAULT_LOCATION`, `CREDENTIALS_PATH`). +4. `bazel test //cli:index_test`. -```bash -./scripts/lint -``` +For Postgres integration tests, `tools/postgres/postgres_fixture.ts` boots a +Docker container inside the Bazel sandbox — requires Docker running locally. ### Building -Building the CLI will build most of the required components. - ```bash -bazel build cli +bazel build cli # build the CLI +bazel build //... # build everything ``` -The projects folder here is not built as it requires an environment file, which can be provided from the team. - -### Add New NPM Dependencies +### Adding NPM Dependencies -Global yarn installations will throw errors when installing packages, instead you should use: +Use Bazel-wrapped yarn: ```bash -$ bazel run @nodejs//:yarn add ... +bazel run @nodejs//:yarn add ``` -Additionally, installed NPM dependencies need to be added to the `deps` of `ts_library` rules by -prefixing them with `@npm//...`. - -## The Contribution Process - -1. Decide on what you'd like to contribute. The majority of open-source contributions come from: - - 1. Someone deciding they want a feature that is not currently present, and which isn't a priority for the team. - - 1. Embracing the community aspect of open source (or getting that commit count up) and solving an issue. - -1. Plan out the change, and whether it is feasible. - - 1. If you're unsure of the scope of the change, then ask in the issue or [create a discussion](https://github.com/dataform-co/dataform/discussions). - - 1. We'd much prefer multiple smaller code changes than a single large one. - - 1. Avoid changing core functionality over a long time frame. Our development process is very dynamic, so if your code depends on lots of other parts of the project, then it is likely to be out of date by the time you finish! - - 1. If you're solving an issue, be sure to comment to make it known that you are currently solving it. Unless we have worked with you before, it is unlikely that we will lock the issue to you. - -1. Begin materialising your masterpiece. - - 1. Create a feature branch based on `main` ([link](https://github.com/dataform-co/dataform/tree/main)) for development work. - -1. Once done, review your code, run the tests, **[check for common mistakes](#common-pull-request-mistakes)** and then open a pull request. - - 1. Tidy the code by removing erronous log statements. Comment difficult to interpret sections. Make sure functions are names appropriately. We will review the pull request mainly by the git difference. - - 1. Assign a reviewer. Pick anyone on the team who seems to contribute a lot and they will refer it onto whoever is most responsible for the given subsystem. - -1. Discuss and process any changes requested. - - 1. It's unlikely your pull request will be perfect immediately; there will likely be some changes requested, whether it's to do with style or a more fundamental issue. - - 1. The automated integration tests must pass. - - 1. Once a pull request is accepted and all automated integration tests are passing, we will merge it for you. - -### Reporting Issues (!) - -Another way we'd love for you to contribute is by flagging any issues you find. First check through the list of [existing issues](https://github.com/dataform-co/dataform/issues) for anything similar, in order to avoid duplicates. If not, then full steam ahead! - -### Promoting Dataform - -If you're using Dataform for interesting projects then please let people know! Reach out to [dataform-preview@google.com](dataform-preview@google.com) for marketing support. - -### Common Pull Request Mistakes - -1. Is it too long? Small pull requests are easier to review and merge. If you are planning on making a larger change, then talk to the team and write a document on the design. +After installation, add the package to relevant `ts_library` deps prefixed +with `@npm//`. -1. Have you appropriately increased test coverage? If the operation of the change is not already tested, then tests will need to be written. +## Pull Requests -1. Is it a hack? Does it solve the problem, but it is not reliable, reproducable or extendable? In other words, [does it smell](https://en.wikipedia.org/wiki/Code_smell)? +- Keep PRs small and focused — small diffs are easier to review. +- Add tests when changing behavior. +- Don't reformat unrelated code (makes diffs noisy). +- Branch off `main`. -1. Have you changed whitespace or touched unrelated code? Please avoid this as it makes pull requests far more difficult to review. +## Reporting Issues -1. Are the comments useful, and is the code readable? Are the function and variable names appropriate? +[Open an issue](https://github.com/ihistand/sqlanvil/issues) on GitHub. diff --git a/core/BUILD b/core/BUILD index db9219e8..fbde4ff2 100644 --- a/core/BUILD +++ b/core/BUILD @@ -106,7 +106,7 @@ ts_test_suite( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", "//packages/sample-extension:package_tar", ], ) diff --git a/core/actions/assertion.ts b/core/actions/assertion.ts index e638e8b3..01e188be 100644 --- a/core/actions/assertion.ts +++ b/core/actions/assertion.ts @@ -1,8 +1,8 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { IActionContext, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { IActionContext, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, configTargetToCompiledGraphTarget, @@ -12,8 +12,8 @@ import { resolveActionsConfigFilename, toResolvable, validateQueryString -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -21,7 +21,7 @@ import { dataform } from "df/protos/ts"; * This maintains backwards compatability with older versions. * Consider breaking backwards compatability of these in v4. */ -interface ILegacyAssertionConfig extends dataform.ActionConfig.AssertionConfig { +interface ILegacyAssertionConfig extends sqlanvil.ActionConfig.AssertionConfig { dependencies: Resolvable[]; database: string; schema: string; @@ -37,7 +37,7 @@ export type AContextable = T | ((ctx: AssertionContext) => T); * specified in the query. If the query returns any rows, the assertion fails. * * You can create assertions in the following ways. Available config options are defined in - * [AssertionConfig](configs#dataform-ActionConfig-AssertionConfig), and are shared across all the + * [AssertionConfig](configs#sqlanvil-ActionConfig-AssertionConfig), and are shared across all the * following ways of creating assertions. * * **Using a SQLX file:** @@ -52,7 +52,7 @@ export type AContextable = T | ((ctx: AssertionContext) => T); * * **Using built-in assertions in the config block of a table:** * - * See [TableConfig.assertions](configs#dataform-ActionConfig-TableConfig) + * See [TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig) * * **Using action configs files:** * @@ -78,14 +78,14 @@ export type AContextable = T | ((ctx: AssertionContext) => T); * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class Assertion extends ActionBuilder { +export class Assertion extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Assertion.create(); + private proto = sqlanvil.Assertion.create(); /** @hidden We delay contextification until the final compile step, so hold these here for now. */ private contextableQuery: AContextable; @@ -124,7 +124,7 @@ export class Assertion extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -168,7 +168,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.dependencies](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.dependencies](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets dependencies of the assertion. */ @@ -184,7 +184,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.hermetic](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.hermetic](configs#sqlanvil-ActionConfig-AssertionConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -192,13 +192,13 @@ export class Assertion extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [AssertionConfig.disabled](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.disabled](configs#sqlanvil-ActionConfig-AssertionConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -210,7 +210,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.tags](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.tags](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -226,7 +226,7 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.description](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.description](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets the description of this assertion. */ @@ -237,14 +237,14 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.project](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.project](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets the database (Google Cloud project ID) in which to create the corresponding view for this * assertion. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true, useDefaultAssertionDataset: true } @@ -254,14 +254,14 @@ export class Assertion extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [AssertionConfig.dataset](configs#dataform-ActionConfig-AssertionConfig). + * [AssertionConfig.dataset](configs#sqlanvil-ActionConfig-AssertionConfig). * * Sets the schema (BigQuery dataset) in which to create the corresponding view for this * assertion. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true, useDefaultAssertionDataset: true } @@ -276,16 +276,16 @@ export class Assertion extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ public getParentAction() { - return dataform.Target.create(this.proto.parentAction); + return sqlanvil.Target.create(this.proto.parentAction); } /** @hidden */ - public setParentAction(target: dataform.Target) { + public setParentAction(target: sqlanvil.Target) { this.proto.parentAction = target; } @@ -297,7 +297,7 @@ export class Assertion extends ActionBuilder { validateQueryString(this.session, this.proto.query, this.proto.fileName); return verifyObjectMatchesProto( - dataform.Assertion, + sqlanvil.Assertion, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -310,7 +310,7 @@ export class Assertion extends ActionBuilder { */ private verifyConfig( unverifiedConfig: ILegacyAssertionConfig - ): dataform.ActionConfig.AssertionConfig { + ): sqlanvil.ActionConfig.AssertionConfig { if (unverifiedConfig.dependencies) { unverifiedConfig.dependencyTargets = unverifiedConfig.dependencies.map( (dependency: string | object) => resolvableAsActionConfigTarget(dependency) @@ -335,7 +335,7 @@ export class Assertion extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.ActionConfig.AssertionConfig, + sqlanvil.ActionConfig.AssertionConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/assertion_test.ts b/core/actions/assertion_test.ts index 02ec6e63..d230e673 100644 --- a/core/actions/assertion_test.ts +++ b/core/actions/assertion_test.ts @@ -4,15 +4,15 @@ import * as fs from "fs-extra"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML, WorkflowSettingsTemplates -} from "df/testing/run_core"; +} from "sa/testing/run_core"; const EMPTY_NOTEBOOK_CONTENTS = '{ "cells": [] }'; @@ -301,7 +301,7 @@ actions: projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(testConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(testConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -928,7 +928,7 @@ actions: const coreRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ disableAssertions: true }) ); @@ -980,7 +980,7 @@ actions: const coreRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ disableAssertions: true }) ); diff --git a/core/actions/data_preparation.ts b/core/actions/data_preparation.ts index c0c924aa..737da53f 100644 --- a/core/actions/data_preparation.ts +++ b/core/actions/data_preparation.ts @@ -1,10 +1,10 @@ import { dump as dumpYaml } from "js-yaml"; -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { Contextable, ITableContext, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { Contextable, ITableContext, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -14,23 +14,23 @@ import { resolveActionsConfigFilename, toResolvable, validateQueryString -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden */ -export class DataPreparation extends ActionBuilder { +export class DataPreparation extends ActionBuilder { public session: Session; // We delay contextification until the final compile step, so hold these here for now. public contextableQuery: Contextable; - private proto = dataform.DataPreparation.create(); + private proto = sqlanvil.DataPreparation.create(); constructor( session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig, + config?: sqlanvil.ActionConfig.DataPreparationConfig, configPath?: string ) { super(session); @@ -114,7 +114,7 @@ export class DataPreparation extends ActionBuilder { */ public getTarget() { // Return only the first target for now. - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } public compile() { @@ -125,7 +125,7 @@ export class DataPreparation extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.DataPreparation, + sqlanvil.DataPreparation, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -133,7 +133,7 @@ export class DataPreparation extends ActionBuilder { public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -143,7 +143,7 @@ export class DataPreparation extends ActionBuilder { public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -156,9 +156,9 @@ export class DataPreparation extends ActionBuilder { [key: string]: any; }, session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig + config?: sqlanvil.ActionConfig.DataPreparationConfig ) { - const defaultTarget = dataform.Target.create({ name: config.name }); + const defaultTarget = sqlanvil.Target.create({ name: config.name }); this.proto.target = this.finalizeTarget( this.applySessionToTarget(defaultTarget, session.projectConfig, config.filename, { validateTarget: true @@ -175,12 +175,12 @@ export class DataPreparation extends ActionBuilder { } private configureYamlWithTargets( - targets: dataform.Target[], + targets: sqlanvil.Target[], dataPreparationAsJson: { [key: string]: any; }, session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig + config?: sqlanvil.ActionConfig.DataPreparationConfig ) { const resolvedTargets = targets.map(target => this.applySessionToTarget(target, session.projectConfig, config.filename, { @@ -245,13 +245,13 @@ export class DataPreparation extends ActionBuilder { } private applySessionToTableReference(tableReference: { [key: string]: string }): object { - const target: dataform.ITarget = { + const target: sqlanvil.ITarget = { database: tableReference.project, schema: tableReference.dataset, name: tableReference.table }; const resolvedTarget = this.applySessionToTarget( - dataform.Target.create(target), + sqlanvil.Target.create(target), this.session.projectConfig ); // Convert resolved target into a Data Preparation Table Reference @@ -275,19 +275,19 @@ export class DataPreparation extends ActionBuilder { return resolvedTableReference; } - private getTargets(definition: { [key: string]: any }): dataform.Target[] { - const targets: dataform.Target[] = []; + private getTargets(definition: { [key: string]: any }): sqlanvil.Target[] { + const targets: sqlanvil.Target[] = []; if (definition && definition.nodes) { (definition.nodes as Array<{ [key: string]: any }>).forEach(node => { const table = node.destination?.table; if (table) { - const compiledGraphTarget: dataform.ITarget = { + const compiledGraphTarget: sqlanvil.ITarget = { database: table.project, schema: table.dataset, name: table.table }; - targets.push(dataform.Target.create(compiledGraphTarget)); + targets.push(sqlanvil.Target.create(compiledGraphTarget)); } }); } @@ -296,7 +296,7 @@ export class DataPreparation extends ActionBuilder { } private configureYaml( session?: Session, - config?: dataform.ActionConfig.DataPreparationConfig, + config?: sqlanvil.ActionConfig.DataPreparationConfig, configPath?: string ) { config.filename = resolveActionsConfigFilename(config.filename, configPath); @@ -322,7 +322,7 @@ export class DataPreparation extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -332,14 +332,14 @@ export class DataPreparation extends ActionBuilder { } } - private configureSqlx(session?: Session, config?: dataform.ActionConfig.DataPreparationConfig) { - const targets: dataform.Target[] = []; + private configureSqlx(session?: Session, config?: sqlanvil.ActionConfig.DataPreparationConfig) { + const targets: sqlanvil.Target[] = []; // Add destination as target targets.push(actionConfigToCompiledGraphTarget(config)); // Add Error Table if specified as a secondary target if (config.errorTable != null) { - const errorTableConfig = dataform.ActionConfig.DataPreparationConfig.ErrorTableConfig.create( + const errorTableConfig = sqlanvil.ActionConfig.DataPreparationConfig.ErrorTableConfig.create( config.errorTable ); const errorTableTarget = actionConfigToCompiledGraphTarget(errorTableConfig); @@ -378,7 +378,7 @@ export class DataPreparation extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -399,26 +399,26 @@ export class DataPreparation extends ActionBuilder { loadMode?: string | number, incrementalColumn?: string, uniqueKey?: string[] - ): dataform.LoadConfiguration { + ): sqlanvil.LoadConfiguration { if (!loadMode) { - return dataform.LoadConfiguration.create({ replace: {} }); + return sqlanvil.LoadConfiguration.create({ replace: {} }); } switch (loadMode.toString().toUpperCase()) { case "REPLACE_TABLE": - return dataform.LoadConfiguration.create({ replace: {} }); + return sqlanvil.LoadConfiguration.create({ replace: {} }); case "APPEND": - return dataform.LoadConfiguration.create({ append: {} }); + return sqlanvil.LoadConfiguration.create({ append: {} }); case "MAXIMUM": - return dataform.LoadConfiguration.create({ + return sqlanvil.LoadConfiguration.create({ maximum: { columnName: this.validateLoadModeColumnName(incrementalColumn) } }); case "UNIQUE": - return dataform.LoadConfiguration.create({ + return sqlanvil.LoadConfiguration.create({ unique: { columnName: this.validateLoadModeColumnName(incrementalColumn) } }); case "MERGE": - return dataform.LoadConfiguration.create({ + return sqlanvil.LoadConfiguration.create({ merge: { uniqueKey: this.validateUniqueKey(uniqueKey) } }); default: @@ -446,7 +446,7 @@ export class DataPreparationContext implements ITableContext { constructor(private dataPreparation: DataPreparation, private isIncremental = false) {} - public config(config: dataform.ActionConfig.DataPreparationConfig) { + public config(config: sqlanvil.ActionConfig.DataPreparationConfig) { this.dataPreparation.config(config); return ""; } diff --git a/core/actions/data_preparation_test.ts b/core/actions/data_preparation_test.ts index fa92ee5b..3299142f 100644 --- a/core/actions/data_preparation_test.ts +++ b/core/actions/data_preparation_test.ts @@ -4,14 +4,14 @@ import * as fs from "fs-extra"; import { dump as dumpYaml, load as loadYaml } from "js-yaml"; import * as path from "path"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("data preparation", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -427,7 +427,7 @@ FROM x const coreExecutionRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ defaultDatabase: "projectOverride", defaultSchema: "datasetOverride" }) diff --git a/core/actions/declaration.ts b/core/actions/declaration.ts index adc482cd..bfc6fdb1 100644 --- a/core/actions/declaration.ts +++ b/core/actions/declaration.ts @@ -1,9 +1,9 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Session } from "df/core/session"; -import { actionConfigToCompiledGraphTarget } from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Session } from "sa/core/session"; +import { actionConfigToCompiledGraphTarget } from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -11,7 +11,7 @@ import { dataform } from "df/protos/ts"; * This maintains backwards compatability with older versions. * Consider breaking backwards compatability of these in v4. */ -interface ILegacyDeclarationConfig extends dataform.ActionConfig.DeclarationConfig { +interface ILegacyDeclarationConfig extends sqlanvil.ActionConfig.DeclarationConfig { database: string; schema: string; fileName: string; @@ -19,18 +19,18 @@ interface ILegacyDeclarationConfig extends dataform.ActionConfig.DeclarationConf } /** - * You can declare any BigQuery table as a data source in Dataform. Declaring BigQuery data - * sources that are external to Dataform lets you treat those data sources as Dataform objects. + * You can declare any BigQuery table as a data source in sqlanvil. Declaring BigQuery data + * sources that are external to sqlanvil lets you treat those data sources as sqlanvil objects. * * Declaring data sources is optional, but can be useful when you want to do the following: - * * Reference or resolve declared sources in the same way as any other table in Dataform. - * * View declared sources in the visualized Dataform graph. - * * Use Dataform to manage the table-level and column-level descriptions of externally created + * * Reference or resolve declared sources in the same way as any other table in sqlanvil. + * * View declared sources in the visualized sqlanvil graph. + * * Use sqlanvil to manage the table-level and column-level descriptions of externally created * tables. * * Trigger workflow invocations that include all the dependents of an external data source. * * You can create declarations in the following ways. Available config options are defined in - * [DeclarationConfig](configs#dataform-ActionConfig-DeclarationConfig), and are shared across all + * [DeclarationConfig](configs#sqlanvil-ActionConfig-DeclarationConfig), and are shared across all * the followiing ways of creating declarations. * * **Using a SQLX file:** @@ -59,14 +59,14 @@ interface ILegacyDeclarationConfig extends dataform.ActionConfig.DeclarationConf * declare("name") * ``` */ -export class Declaration extends ActionBuilder { +export class Declaration extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Declaration.create(); + private proto = sqlanvil.Declaration.create(); /** @hidden */ constructor(session?: Session, unverifiedConfig?: any, filename?: string) { @@ -97,7 +97,7 @@ export class Declaration extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -108,7 +108,7 @@ export class Declaration extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [DeclarationConfig.description](configs#dataform-ActionConfig-DeclarationConfig). + * [DeclarationConfig.description](configs#sqlanvil-ActionConfig-DeclarationConfig). * * Sets the description of this assertion. */ @@ -122,11 +122,11 @@ export class Declaration extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [DeclarationConfig.columns](configs#dataform-ActionConfig-DeclarationConfig). + * [DeclarationConfig.columns](configs#sqlanvil-ActionConfig-DeclarationConfig). * * Sets the column descriptors of columns in this table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -143,13 +143,13 @@ export class Declaration extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ public compile() { return verifyObjectMatchesProto( - dataform.Declaration, + sqlanvil.Declaration, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -162,7 +162,7 @@ export class Declaration extends ActionBuilder { */ private verifyConfig( unverifiedConfig: ILegacyDeclarationConfig - ): dataform.ActionConfig.DeclarationConfig { + ): sqlanvil.ActionConfig.DeclarationConfig { if (unverifiedConfig.database) { unverifiedConfig.project = unverifiedConfig.database; delete unverifiedConfig.database; @@ -182,7 +182,7 @@ export class Declaration extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.ActionConfig.DeclarationConfig, + sqlanvil.ActionConfig.DeclarationConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/declaration_test.ts b/core/actions/declaration_test.ts index 82a1c790..21a9f686 100644 --- a/core/actions/declaration_test.ts +++ b/core/actions/declaration_test.ts @@ -3,14 +3,14 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { exampleActionDescriptor } from "df/core/actions/index_test"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { exampleActionDescriptor } from "sa/core/actions/index_test"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("declaration", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -97,7 +97,7 @@ actions: ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Unexpected property "fileName", or property value type of "string" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfigs for allowed properties.` + `Unexpected property "fileName", or property value type of "string" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfigs for allowed properties.` ); }); diff --git a/core/actions/filename_override_test.ts b/core/actions/filename_override_test.ts index 80ca72c1..f3a86a85 100644 --- a/core/actions/filename_override_test.ts +++ b/core/actions/filename_override_test.ts @@ -3,13 +3,13 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("filename_override", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/actions/incremental_table.ts b/core/actions/incremental_table.ts index 9abd4477..a5a56975 100644 --- a/core/actions/incremental_table.ts +++ b/core/actions/incremental_table.ts @@ -1,4 +1,4 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; import { ActionBuilder, checkConfigAdditionalOptionsOverlap, @@ -6,14 +6,14 @@ import { ILegacyTableConfig, LegacyConfigConverter, TableType -} from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { JitTableResult, Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Contextable, ITableContext, JitContextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +} from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { JitTableResult, Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Contextable, ITableContext, JitContextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -35,16 +35,16 @@ import { validateNoMixedCompilationMode, validateQueryString, validateStorageUriFormat, -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** - * When you define an incremental table, Dataform builds the incremental table from scratch only for - * the first time. During subsequent executions, Dataform only inserts or merges new rows into the + * When you define an incremental table, sqlanvil builds the incremental table from scratch only for + * the first time. During subsequent executions, sqlanvil only inserts or merges new rows into the * incremental table according to the conditions that you configure. * * You can create incremental tables in the following ways. Available config options are defined in - * [IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig), and are shared across all the + * [IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig), and are shared across all the * following ways of creating tables. * * **Using a SQLX file:** @@ -74,7 +74,7 @@ import { dataform } from "df/protos/ts"; * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class IncrementalTable extends ActionBuilder { +export class IncrementalTable extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; @@ -94,9 +94,9 @@ export class IncrementalTable extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Table.create({ + private proto = sqlanvil.Table.create({ type: "incremental", - enumType: dataform.TableType.INCREMENTAL, + enumType: sqlanvil.TableType.INCREMENTAL, disabled: false, tags: [] }); @@ -146,7 +146,7 @@ export class IncrementalTable extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -171,7 +171,7 @@ export class IncrementalTable extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -182,7 +182,7 @@ export class IncrementalTable extends ActionBuilder { this.schema(config.dataset); } if (config.assertions) { - this.assertions(dataform.ActionConfig.TableAssertionsConfig.create(config.assertions)); + this.assertions(sqlanvil.ActionConfig.TableAssertionsConfig.create(config.assertions)); } if (config.uniqueKey) { this.uniqueKey(config.uniqueKey); @@ -208,7 +208,7 @@ export class IncrementalTable extends ActionBuilder { session.projectConfig.defaultIcebergConfig?.connection ), fileFormat: getFileFormatValueForIcebergTable(config.iceberg.fileFormat?.toString()), - tableFormat: dataform.TableFormat.ICEBERG, + tableFormat: sqlanvil.TableFormat.ICEBERG, storageUri: getStorageUriForIcebergTable( getEffectiveBucketName(session.projectConfig.defaultIcebergConfig?.bucketName, config.iceberg.bucketName), getEffectiveTableFolderRoot(session.projectConfig.defaultIcebergConfig?.tableFolderRoot, config.iceberg.tableFolderRoot), @@ -258,7 +258,7 @@ export class IncrementalTable extends ActionBuilder { const existingAction = this.session.actions.indexOf(this); if (existingAction === -1) { throw Error( - "Expected pre-existing action, but none found. Please report this to the Dataform team." + "Expected pre-existing action, but none found. Please report this to the sqlanvil team." ); } this.session.actions[existingAction] = newAction; @@ -282,7 +282,7 @@ export class IncrementalTable extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } @@ -327,7 +327,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.disabled](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.disabled](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -341,7 +341,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.protected](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.protected](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If called with `true`, prevents the dataset from being rebuilt from scratch. */ @@ -353,10 +353,10 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.uniqueKey](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.uniqueKey](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If set, unique key represents a set of names of columns that will act as a the unique key. To - * enforce this, when updating the incremental table, Dataform merges rows with `uniqueKey` + * enforce this, when updating the incremental table, sqlanvil merges rows with `uniqueKey` * instead of appending them. */ public uniqueKey(uniqueKey: string[]) { @@ -365,12 +365,12 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of options available directly on - * [IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig). For example: + * [IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig). For example: * `publish("name", { type: "table", partitionBy: "column" }`). * * Sets bigquery options for the action. */ - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { if (!!bigquery.labels && Object.keys(bigquery.labels).length > 0) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; @@ -380,14 +380,14 @@ export class IncrementalTable extends ActionBuilder { const bigqueryFiltered = LegacyConfigConverter.legacyConvertBigQueryOptions(bigquery); if (Object.values(bigqueryFiltered).length > 0) { - this.proto.bigquery = dataform.BigQueryOptions.create(bigqueryFiltered); + this.proto.bigquery = sqlanvil.BigQueryOptions.create(bigqueryFiltered); } return this; } /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.dependencies](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.dependencies](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets dependencies of the incremental table. */ @@ -401,7 +401,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.hermetic](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.hermetic](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -409,13 +409,13 @@ export class IncrementalTable extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.tags](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.tags](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -431,7 +431,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.description](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.description](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the description of this incremental table. */ @@ -445,11 +445,11 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.columns](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.columns](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the column descriptors of columns in this incremental table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -461,14 +461,14 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.project](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.project](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the * Sets the database (Google Cloud project ID) in which to create the output of this action. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -478,13 +478,13 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.dataset](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.dataset](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -494,7 +494,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.assertions](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.assertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * Sets in-line assertions for this incremental table. * @@ -503,7 +503,7 @@ export class IncrementalTable extends ActionBuilder { * needed --> */ public assertions( - tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig + tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig ): IncrementalTable { const inlineAssertions = this.generateInlineAssertions(tableAssertionsConfig, this.proto); this.uniqueKeyAssertions = inlineAssertions.uniqueKeyAssertions; @@ -513,7 +513,7 @@ export class IncrementalTable extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [IncrementalTableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-IncrementalTableConfig). + * [IncrementalTableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -530,7 +530,7 @@ export class IncrementalTable extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -551,7 +551,7 @@ export class IncrementalTable extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -620,8 +620,8 @@ export class IncrementalTable extends ActionBuilder { private verifyConfig( // `any` is used here to facilitate the type merging of the legacy table config, which is very // different to the new structure. - unverifiedConfig: dataform.ActionConfig.IncrementalTableConfig | ILegacyTableConfig | any - ): dataform.ActionConfig.IncrementalTableConfig { + unverifiedConfig: sqlanvil.ActionConfig.IncrementalTableConfig | ILegacyTableConfig | any + ): sqlanvil.ActionConfig.IncrementalTableConfig { // The "type" field only exists on legacy incremental table configs. Here we convert them to the // new format. if (unverifiedConfig.type) { @@ -687,7 +687,7 @@ export class IncrementalTable extends ActionBuilder { } const config = verifyObjectMatchesProto( - dataform.ActionConfig.IncrementalTableConfig, + sqlanvil.ActionConfig.IncrementalTableConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -703,21 +703,21 @@ export class IncrementalTable extends ActionBuilder { // - for sqlx it will have type "string" // - for action.yaml it will be converted to enum which is represented // in TypeScript as a "number". - private mapOnSchemaChange(onSchemaChange?: string | number): dataform.OnSchemaChange { + private mapOnSchemaChange(onSchemaChange?: string | number): sqlanvil.OnSchemaChange { if (!onSchemaChange) { - return dataform.OnSchemaChange.IGNORE; + return sqlanvil.OnSchemaChange.IGNORE; } if (typeof onSchemaChange === "number") { switch (onSchemaChange) { - case dataform.ActionConfig.OnSchemaChange.IGNORE: - return dataform.OnSchemaChange.IGNORE; - case dataform.ActionConfig.OnSchemaChange.FAIL: - return dataform.OnSchemaChange.FAIL; - case dataform.ActionConfig.OnSchemaChange.EXTEND: - return dataform.OnSchemaChange.EXTEND; - case dataform.ActionConfig.OnSchemaChange.SYNCHRONIZE: - return dataform.OnSchemaChange.SYNCHRONIZE; + case sqlanvil.ActionConfig.OnSchemaChange.IGNORE: + return sqlanvil.OnSchemaChange.IGNORE; + case sqlanvil.ActionConfig.OnSchemaChange.FAIL: + return sqlanvil.OnSchemaChange.FAIL; + case sqlanvil.ActionConfig.OnSchemaChange.EXTEND: + return sqlanvil.OnSchemaChange.EXTEND; + case sqlanvil.ActionConfig.OnSchemaChange.SYNCHRONIZE: + return sqlanvil.OnSchemaChange.SYNCHRONIZE; default: throw new Error(`OnSchemaChange value "${onSchemaChange}" is not supported`); } @@ -725,13 +725,13 @@ export class IncrementalTable extends ActionBuilder { switch (onSchemaChange.toString().toUpperCase()) { case "IGNORE": - return dataform.OnSchemaChange.IGNORE; + return sqlanvil.OnSchemaChange.IGNORE; case "FAIL": - return dataform.OnSchemaChange.FAIL; + return sqlanvil.OnSchemaChange.FAIL; case "EXTEND": - return dataform.OnSchemaChange.EXTEND; + return sqlanvil.OnSchemaChange.EXTEND; case "SYNCHRONIZE": - return dataform.OnSchemaChange.SYNCHRONIZE; + return sqlanvil.OnSchemaChange.SYNCHRONIZE; default: throw new Error(`OnSchemaChange value "${onSchemaChange}" is not supported`); } @@ -811,7 +811,7 @@ export class IncrementalTableContext implements ITableContext { return ""; } - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { this.incrementalTable.bigquery(bigquery); return ""; } diff --git a/core/actions/incremental_table_test.ts b/core/actions/incremental_table_test.ts index 9b444fbb..c6917a0c 100644 --- a/core/actions/incremental_table_test.ts +++ b/core/actions/incremental_table_test.ts @@ -7,15 +7,15 @@ import { exampleActionDescriptor, exampleBuiltInAssertions, exampleBuiltInAssertionsAsYaml -} from "df/core/actions/index_test"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/core/actions/index_test"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("incremental table", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -439,7 +439,7 @@ defaultIcebergConfig: wsContent: VALID_WORKFLOW_SETTINGS_YAML, }, { - testName: "defaults to \`_dataform\` for tableFolderRoot", + testName: "defaults to \`_sqlanvil\` for tableFolderRoot", configBlock: ` type: "incremental", name: "incremental_table3", @@ -458,7 +458,7 @@ defaultIcebergConfig: tableFormat: "ICEBERG", fileFormat: "PARQUET", connection: "gcp.us.conn-id", - storageUri: "gs://my-bucket/_dataform/my-subpath", + storageUri: "gs://my-bucket/_sqlanvil/my-subpath", }, }, expectError: false, @@ -882,7 +882,7 @@ defaultIcebergConfig: expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); const compiledTable = result.compile.compiledGraph.tables[0]; expect(compiledTable.type).equals("incremental"); - expect(compiledTable.enumType).equals(dataform.TableType.INCREMENTAL); + expect(compiledTable.enumType).equals(sqlanvil.TableType.INCREMENTAL); expect(compiledTable.target.name).equals(testCase.expected!.target.name); expect(compiledTable.target.schema).equals(testCase.expected!.target.schema); expect(compiledTable.target.database).equals(testCase.expected!.target.database); @@ -902,7 +902,7 @@ defaultIcebergConfig: // Create workflow_settings without defaultProject (only defaultDataset and defaultLocation) fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - `defaultDataset: dataform + `defaultDataset: sqlanvil defaultLocation: europe-west2 ` ); @@ -926,8 +926,8 @@ select \${incremental()} as is_incremental` // when defaultProject is not specified, ensuring that warehouse state targets match compiled targets. // This allows incremental appends to work correctly even when defaultProject is not in workflow_settings.yaml. expect(compiledTable.type).equals("incremental"); - expect(compiledTable.enumType).equals(dataform.TableType.INCREMENTAL); - expect(compiledTable.target.schema).equals("dataform"); + expect(compiledTable.enumType).equals(sqlanvil.TableType.INCREMENTAL); + expect(compiledTable.target.schema).equals("sqlanvil"); expect(compiledTable.target.name).equals("incremental_table_without_default_project"); expect(compiledTable.target.database).equals(""); }); diff --git a/core/actions/index.ts b/core/actions/index.ts index 63f214ca..cea3e80c 100644 --- a/core/actions/index.ts +++ b/core/actions/index.ts @@ -1,16 +1,16 @@ -import { Assertion } from "df/core/actions/assertion"; -import { DataPreparation } from "df/core/actions/data_preparation"; -import { Declaration } from "df/core/actions/declaration"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation } from "df/core/actions/operation"; -import { Table } from "df/core/actions/table"; -import { Test } from "df/core/actions/test"; -import { View } from "df/core/actions/view"; -import { IColumnsDescriptor } from "df/core/column_descriptors"; -import { Resolvable } from "df/core/contextables"; -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import { Assertion } from "sa/core/actions/assertion"; +import { DataPreparation } from "sa/core/actions/data_preparation"; +import { Declaration } from "sa/core/actions/declaration"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation } from "sa/core/actions/operation"; +import { Table } from "sa/core/actions/table"; +import { Test } from "sa/core/actions/test"; +import { View } from "sa/core/actions/view"; +import { IColumnsDescriptor } from "sa/core/column_descriptors"; +import { Resolvable } from "sa/core/contextables"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; export type Action = | Table @@ -24,13 +24,13 @@ export type Action = | Test; export type ActionProto = - | dataform.Table // core.proto's Table represents the Table, View or IncrementalTable action type. - | dataform.Operation - | dataform.Assertion - | dataform.Declaration - | dataform.Notebook - | dataform.DataPreparation - | dataform.Test; + | sqlanvil.Table // core.proto's Table represents the Table, View or IncrementalTable action type. + | sqlanvil.Operation + | sqlanvil.Assertion + | sqlanvil.Declaration + | sqlanvil.Notebook + | sqlanvil.DataPreparation + | sqlanvil.Test; // In v4, consider making methods on inheritors of this private, forcing users to use constructors // in order to populate actions. @@ -43,18 +43,18 @@ export abstract class ActionBuilder { } public applySessionToTarget( - targetFromConfig: dataform.Target, - projectConfig: dataform.ProjectConfig, + targetFromConfig: sqlanvil.Target, + projectConfig: sqlanvil.ProjectConfig, fileName?: string, options?: { validateTarget?: boolean; useDefaultAssertionDataset?: boolean; } - ): dataform.Target { + ): sqlanvil.Target { const defaultSchema = options?.useDefaultAssertionDataset ? projectConfig.assertionSchema || projectConfig.defaultSchema : projectConfig.defaultSchema; - const target = dataform.Target.create({ + const target = sqlanvil.Target.create({ name: targetFromConfig.name, schema: targetFromConfig.schema || defaultSchema || undefined, database: targetFromConfig.database || projectConfig.defaultDatabase || undefined @@ -65,8 +65,8 @@ export abstract class ActionBuilder { return target; } - public finalizeTarget(targetFromConfig: dataform.Target): dataform.Target { - return dataform.Target.create({ + public finalizeTarget(targetFromConfig: sqlanvil.Target): sqlanvil.Target { + return sqlanvil.Target.create({ name: this.session.finalizeName(targetFromConfig.name), schema: targetFromConfig.schema ? this.session.finalizeSchema(targetFromConfig.schema) @@ -81,14 +81,14 @@ export abstract class ActionBuilder { public abstract getFileName(): string; /** Retrieves the resolved target from the proto. */ - public abstract getTarget(): dataform.Target; + public abstract getTarget(): sqlanvil.Target; /** Creates the final protobuf representation. */ public abstract compile(): T; protected generateInlineAssertions( - tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig, - proto: dataform.Table + tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig, + proto: sqlanvil.Table ): { uniqueKeyAssertions: Assertion[]; rowConditionsAssertion?: Assertion } { const inlineAssertions: { uniqueKeyAssertions: Assertion[]; @@ -101,11 +101,11 @@ export abstract class ActionBuilder { } const assertionPrefix = !!this.session.projectConfig.builtinAssertionNamePrefix ? `${this.session.projectConfig.builtinAssertionNamePrefix}_` : ""; let uniqueKeys = tableAssertionsConfig.uniqueKeys.map(uniqueKey => - dataform.ActionConfig.TableAssertionsConfig.UniqueKey.create(uniqueKey) + sqlanvil.ActionConfig.TableAssertionsConfig.UniqueKey.create(uniqueKey) ); if (!!tableAssertionsConfig.uniqueKey?.length) { uniqueKeys = [ - dataform.ActionConfig.TableAssertionsConfig.UniqueKey.create({ + sqlanvil.ActionConfig.TableAssertionsConfig.UniqueKey.create({ uniqueKey: tableAssertionsConfig.uniqueKey }) ]; @@ -115,7 +115,7 @@ export abstract class ActionBuilder { const uniqueKeyAssertion = this.session .assert( `${assertionPrefix}${proto.target.schema}_${proto.target.name}_assertions_uniqueKey_${index}`, - dataform.ActionConfig.AssertionConfig.create({ filename: proto.fileName }) + sqlanvil.ActionConfig.AssertionConfig.create({ filename: proto.fileName }) ) .query(ctx => this.session.compilationSql().indexAssertion(ctx.ref(proto.target), uniqueKey) @@ -123,7 +123,7 @@ export abstract class ActionBuilder { if (proto.tags) { uniqueKeyAssertion.tags(proto.tags); } - uniqueKeyAssertion.setParentAction(dataform.Target.create(proto.target)); + uniqueKeyAssertion.setParentAction(sqlanvil.Target.create(proto.target)); if (proto.disabled) { uniqueKeyAssertion.disabled(); } @@ -142,13 +142,13 @@ export abstract class ActionBuilder { inlineAssertions.rowConditionsAssertion = this.session .assert(`${assertionPrefix}${proto.target.schema}_${proto.target.name}_assertions_rowConditions`, { filename: proto.fileName - } as dataform.ActionConfig.AssertionConfig) + } as sqlanvil.ActionConfig.AssertionConfig) .query(ctx => this.session .compilationSql() .rowConditionsAssertion(ctx.ref(proto.target), mergedRowConditions) ); - inlineAssertions.rowConditionsAssertion.setParentAction(dataform.Target.create(proto.target)); + inlineAssertions.rowConditionsAssertion.setParentAction(sqlanvil.Target.create(proto.target)); if (proto.disabled) { inlineAssertions.rowConditionsAssertion.disabled(); } @@ -159,7 +159,7 @@ export abstract class ActionBuilder { return inlineAssertions; } - private validateTarget(target: dataform.Target, fileName: string) { + private validateTarget(target: sqlanvil.Target, fileName: string) { if (target.name.includes(".")) { this.session.compileError( new Error("Action target names cannot include '.'"), @@ -185,10 +185,10 @@ export abstract class ActionBuilder { } export function checkConfigAdditionalOptionsOverlap( - config: dataform.ActionConfig.TableConfig | dataform.ActionConfig.IncrementalTableConfig, + config: sqlanvil.ActionConfig.TableConfig | sqlanvil.ActionConfig.IncrementalTableConfig, session: Session ) { - const target = dataform.Target.create({ + const target = sqlanvil.Target.create({ database: config.project, schema: config.dataset, name: config.name @@ -390,9 +390,9 @@ export class LegacyConfigConverter { // This is a workaround to make bigquery options output empty fields with the same behaviour as // they did previously. public static legacyConvertBigQueryOptions( - bigquery: dataform.IBigQueryOptions - ): dataform.IBigQueryOptions { - let bigqueryFiltered: dataform.IBigQueryOptions = {}; + bigquery: sqlanvil.IBigQueryOptions + ): sqlanvil.IBigQueryOptions { + let bigqueryFiltered: sqlanvil.IBigQueryOptions = {}; Object.entries(bigquery).forEach(([key, value]) => { if (Array.isArray(value) && value.length === 0) { return; @@ -422,7 +422,7 @@ export class LegacyConfigConverter { if (legacyConfig.assertions.uniqueKeys?.[0]?.length > 0) { legacyConfig.assertions.uniqueKeys = (legacyConfig.assertions .uniqueKeys as string[][]).map(uniqueKey => - dataform.ActionConfig.TableAssertionsConfig.UniqueKey.create({ uniqueKey }) + sqlanvil.ActionConfig.TableAssertionsConfig.UniqueKey.create({ uniqueKey }) ); } if (typeof legacyConfig.assertions.nonNull === "string") { diff --git a/core/actions/index_test.ts b/core/actions/index_test.ts index 07359f3e..1bc1caa0 100644 --- a/core/actions/index_test.ts +++ b/core/actions/index_test.ts @@ -3,14 +3,14 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; export const exampleActionDescriptor = { inputSqlxConfigBlock: ` @@ -43,7 +43,7 @@ export const exampleActionDescriptor = { } ], description: "description" - } as dataform.IColumnDescriptor + } as sqlanvil.IColumnDescriptor }; export const exampleBuiltInAssertions = { @@ -112,7 +112,7 @@ export const exampleBuiltInAssertions = { "\nSELECT\n 'rowConditions1' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions1)\nUNION ALL\nSELECT\n 'rowConditions2' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions2)\nUNION ALL\nSELECT\n 'nonNull IS NOT NULL' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (nonNull IS NOT NULL)\n", tags: ["tag1", "tag2"] } - ] as dataform.IAssertion[] + ] as sqlanvil.IAssertion[] }; export const exampleBuiltInAssertionsAsYaml = { @@ -191,13 +191,13 @@ export const exampleBuiltInAssertionsAsYaml = { "\nSELECT\n 'rowConditions1' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions1)\nUNION ALL\nSELECT\n 'rowConditions2' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (rowConditions2)\nUNION ALL\nSELECT\n 'nonNull IS NOT NULL' AS failing_row_condition,\n *\nFROM `project.dataset.name`\nWHERE NOT (nonNull IS NOT NULL)\n", tags: ["tag1", "tag2"] } - ] as dataform.IAssertion[] + ] as sqlanvil.IAssertion[] }; suite("actions", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); - const getActionsFromResult = (tableType: string, result: dataform.CoreExecutionResponse) => { + const getActionsFromResult = (tableType: string, result: sqlanvil.CoreExecutionResponse) => { switch (tableType) { case "table": case "view": @@ -234,7 +234,7 @@ SELECT 1` const result = runMainInVm( coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ defaultDatabase: "otherProject", defaultSchema: "otherDataset", assertionSchema: "otherDataset", diff --git a/core/actions/notebook.ts b/core/actions/notebook.ts index fa6af7a6..b169dd30 100644 --- a/core/actions/notebook.ts +++ b/core/actions/notebook.ts @@ -1,23 +1,23 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, configTargetToCompiledGraphTarget, nativeRequire, resolveActionsConfigFilename -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * Notebooks run Jupyter Notebook files, and can output content to the storage buckets defined in * `workflow_settings.yaml` files. * * You can create notebooks in the following ways. Available config options are defined in - * [NotebookConfig](configs#dataform-ActionConfig-NotebookConfig), and are shared across all the + * [NotebookConfig](configs#sqlanvil-ActionConfig-NotebookConfig), and are shared across all the * following ways of creating notebooks. * * **Using action configs files:** @@ -46,7 +46,7 @@ import { dataform } from "df/protos/ts"; * { "cells": [] } * ``` */ -export class Notebook extends ActionBuilder { +export class Notebook extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** @@ -58,7 +58,7 @@ export class Notebook extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Notebook.create(); + private proto = sqlanvil.Notebook.create(); /** @hidden */ constructor(session?: Session, unverifiedConfig?: any, configPath?: string) { @@ -82,7 +82,7 @@ export class Notebook extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -125,13 +125,13 @@ export class Notebook extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ public compile() { return verifyObjectMatchesProto( - dataform.Notebook, + sqlanvil.Notebook, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -141,9 +141,9 @@ export class Notebook extends ActionBuilder { * @hidden Verify config checks that the constructor provided config matches the expected proto * structure. */ - private verifyConfig(unverifiedConfig: any): dataform.ActionConfig.NotebookConfig { + private verifyConfig(unverifiedConfig: any): sqlanvil.ActionConfig.NotebookConfig { return verifyObjectMatchesProto( - dataform.ActionConfig.NotebookConfig, + sqlanvil.ActionConfig.NotebookConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/notebook_test.ts b/core/actions/notebook_test.ts index 354b2af2..ab36ad8b 100644 --- a/core/actions/notebook_test.ts +++ b/core/actions/notebook_test.ts @@ -3,13 +3,13 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; const EMPTY_NOTEBOOK_CONTENTS = '{ "cells": [] }'; @@ -136,7 +136,7 @@ actions: test(`notebook default runtime options are loaded`, () => { const projectDir = createSimpleNotebookProject(` -defaultProject: dataform +defaultProject: sqlanvil defaultLocation: US defaultNotebookRuntimeOptions: outputBucket: gs://some-bucket @@ -150,7 +150,7 @@ defaultNotebookRuntimeOptions: expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); expect(asPlainObject(result.compile.compiledGraph.projectConfig)).deep.equals({ - defaultDatabase: "dataform", + defaultDatabase: "sqlanvil", defaultLocation: "US", defaultNotebookRuntimeOptions: { outputBucket: "gs://some-bucket", @@ -166,7 +166,7 @@ defaultNotebookRuntimeOptions: test(`notebook default runtime options snapshot destination defaults to output bucket`, () => { const projectDir = createSimpleNotebookProject(` -defaultProject: dataform +defaultProject: sqlanvil defaultLocation: US defaultNotebookRuntimeOptions: outputBucket: gs://some-bucket @@ -179,7 +179,7 @@ defaultNotebookRuntimeOptions: expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); expect(asPlainObject(result.compile.compiledGraph.projectConfig)).deep.equals({ - defaultDatabase: "dataform", + defaultDatabase: "sqlanvil", defaultLocation: "US", defaultNotebookRuntimeOptions: { outputBucket: "gs://some-bucket", @@ -195,7 +195,7 @@ defaultNotebookRuntimeOptions: test(`notebook default runtime options throw for snapshot destination with no uri or output bucket`, () => { const projectDir = createSimpleNotebookProject(` -defaultProject: dataform +defaultProject: sqlanvil defaultLocation: US defaultNotebookRuntimeOptions: runtimeTemplateName: projects/test-project/locations/us-central1/notebookRuntimeTemplates/test-template diff --git a/core/actions/operation.ts b/core/actions/operation.ts index 1c9f0056..523bb358 100644 --- a/core/actions/operation.ts +++ b/core/actions/operation.ts @@ -1,9 +1,9 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder } from "df/core/actions"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Contextable, IActionContext, JitContextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder } from "sa/core/actions"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Contextable, IActionContext, JitContextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -13,8 +13,8 @@ import { resolvableAsTarget, resolveActionsConfigFilename, toResolvable -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -22,7 +22,7 @@ import { dataform } from "df/protos/ts"; * This maintains backwards compatability with older versions. * Consider breaking backwards compatability of these in v4. */ -interface ILegacyOperationConfig extends dataform.ActionConfig.OperationConfig { +interface ILegacyOperationConfig extends sqlanvil.ActionConfig.OperationConfig { dependencies: Resolvable[]; database: string; schema: string; @@ -30,14 +30,14 @@ interface ILegacyOperationConfig extends dataform.ActionConfig.OperationConfig { type: string; } -export type JitOperationResult = string | string[] | dataform.IJitOperationResult; +export type JitOperationResult = string | string[] | sqlanvil.IJitOperationResult; /** - * Operations define custom SQL operations that don't fit into the Dataform model of publishing a + * Operations define custom SQL operations that don't fit into the sqlanvil model of publishing a * table or writing an assertion. * * You can create operations in the following ways. Available config options are defined in - * [OperationConfig](configs#dataform-ActionConfig-OperationConfig), and are shared across all the + * [OperationConfig](configs#sqlanvil-ActionConfig-OperationConfig), and are shared across all the * following ways of creating operations. * * **Using a SQLX file:** @@ -74,7 +74,7 @@ export type JitOperationResult = string | string[] | dataform.IJitOperationResul * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class Operation extends ActionBuilder { +export class Operation extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; @@ -87,7 +87,7 @@ export class Operation extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Operation.create(); + private proto = sqlanvil.Operation.create(); /** @hidden We delay contextification until the final compile step, so hold these here for now. */ private contextableQueries: Contextable; @@ -124,7 +124,7 @@ export class Operation extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -146,7 +146,7 @@ export class Operation extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -183,14 +183,14 @@ export class Operation extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } /** * @deprecated Deprecated in favor of - * [OperationConfig.dependencies](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.dependencies](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets dependencies of the table. */ @@ -207,7 +207,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.hermetic](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.hermetic](configs#sqlanvil-ActionConfig-OperationConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -215,13 +215,13 @@ export class Operation extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [OperationConfig.disabled](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.disabled](configs#sqlanvil-ActionConfig-OperationConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -233,7 +233,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.tags](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.tags](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -249,7 +249,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.hasOutput](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.hasOutput](configs#sqlanvil-ActionConfig-OperationConfig). * * Declares that this action creates a dataset which should be referenceable as a dependency * target, for example by using the `ref` function. @@ -261,7 +261,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.description](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.description](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the description of this assertion. */ @@ -275,11 +275,11 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.columns](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.columns](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the column descriptors of columns in this table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -291,14 +291,14 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.project](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.project](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the database (Google Cloud project ID) in which to create the corresponding view for this * operation. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -308,13 +308,13 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.dataset](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.dataset](configs#sqlanvil-ActionConfig-OperationConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -324,7 +324,7 @@ export class Operation extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [OperationConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-OperationConfig). + * [OperationConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-OperationConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -341,7 +341,7 @@ export class Operation extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -370,7 +370,7 @@ export class Operation extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Operation, + sqlanvil.Operation, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -395,7 +395,7 @@ export class Operation extends ActionBuilder { */ private verifyConfig( unverifiedConfig: ILegacyOperationConfig - ): dataform.ActionConfig.OperationConfig { + ): sqlanvil.ActionConfig.OperationConfig { // The "type" field only exists on legacy view configs. Here we convert them to the new format. if (unverifiedConfig.type) { delete unverifiedConfig.type; @@ -424,7 +424,7 @@ export class Operation extends ActionBuilder { } } return verifyObjectMatchesProto( - dataform.ActionConfig.OperationConfig, + sqlanvil.ActionConfig.OperationConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); diff --git a/core/actions/operation_test.ts b/core/actions/operation_test.ts index 5c1e4213..c17f1b1e 100644 --- a/core/actions/operation_test.ts +++ b/core/actions/operation_test.ts @@ -3,14 +3,14 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { exampleActionDescriptor } from "df/core/actions/index_test"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { exampleActionDescriptor } from "sa/core/actions/index_test"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("operation", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/actions/table.ts b/core/actions/table.ts index ee156252..ad1f5182 100644 --- a/core/actions/table.ts +++ b/core/actions/table.ts @@ -1,4 +1,4 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; import { ActionBuilder, checkConfigAdditionalOptionsOverlap, @@ -6,19 +6,19 @@ import { ILegacyTableConfig, LegacyConfigConverter, TableType -} from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { View } from "df/core/actions/view"; -import { ColumnDescriptors } from "df/core/column_descriptors"; +} from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { View } from "sa/core/actions/view"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; import { Contextable, ITableContext, JitContextable, Resolvable, -} from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +} from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -39,19 +39,19 @@ import { validateNoMixedCompilationMode, validateQueryString, validateStorageUriFormat, -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** JiT compilation stage result. String is equivalint to {query: value}. */ -export type JitTableResult = string | dataform.IJitTableResult; +export type JitTableResult = string | sqlanvil.IJitTableResult; /** - * Tables are the fundamental building block for storing data when using Dataform. Dataform compiles - * your Dataform core code into SQL, executes the SQL code, and creates your defined tables in + * Tables are the fundamental building block for storing data when using sqlanvil. sqlanvil compiles + * your sqlanvil core code into SQL, executes the SQL code, and creates your defined tables in * BigQuery. * * You can create tables in the following ways. Available config options are defined in - * [TableConfig](configs#dataform-ActionConfig-TableConfig), and are shared across all the + * [TableConfig](configs#sqlanvil-ActionConfig-TableConfig), and are shared across all the * following ways of creating tables. * * **Using a SQLX file:** @@ -88,7 +88,7 @@ export type JitTableResult = string | dataform.IJitTableResult; * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class Table extends ActionBuilder { +export class Table extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** @@ -107,9 +107,9 @@ export class Table extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Table.create({ + private proto = sqlanvil.Table.create({ type: "table", - enumType: dataform.TableType.TABLE, + enumType: sqlanvil.TableType.TABLE, disabled: false, tags: [] }); @@ -179,12 +179,12 @@ export class Table extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } if (config.assertions) { - this.assertions(dataform.ActionConfig.TableAssertionsConfig.create(config.assertions)); + this.assertions(sqlanvil.ActionConfig.TableAssertionsConfig.create(config.assertions)); } if (config.preOperations) { this.preOps(config.preOperations); @@ -205,7 +205,7 @@ export class Table extends ActionBuilder { session.projectConfig.defaultIcebergConfig?.connection ), fileFormat: getFileFormatValueForIcebergTable(config.iceberg.fileFormat?.toString()), - tableFormat: dataform.TableFormat.ICEBERG, + tableFormat: sqlanvil.TableFormat.ICEBERG, storageUri: getStorageUriForIcebergTable( getEffectiveBucketName(session.projectConfig.defaultIcebergConfig?.bucketName, config.iceberg.bucketName), getEffectiveTableFolderRoot(session.projectConfig.defaultIcebergConfig?.tableFolderRoot, config.iceberg.tableFolderRoot), @@ -254,7 +254,7 @@ export class Table extends ActionBuilder { const existingAction = this.session.actions.indexOf(this); if (existingAction === -1) { throw Error( - "Expected pre-existing action, but none found. Please report this to the Dataform team." + "Expected pre-existing action, but none found. Please report this to the sqlanvil team." ); } this.session.actions[existingAction] = newAction; @@ -278,7 +278,7 @@ export class Table extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } @@ -323,7 +323,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.disabled](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.disabled](configs#sqlanvil-ActionConfig-TableConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -337,12 +337,12 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of options available directly on - * [TableConfig](configs#dataform-ActionConfig-TableConfig). For example: + * [TableConfig](configs#sqlanvil-ActionConfig-TableConfig). For example: * `publish("name", { type: "table", partitionBy: "column" }`). * * Sets bigquery options for the action. */ - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { if (!!bigquery.labels && Object.keys(bigquery.labels).length > 0) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; @@ -352,14 +352,14 @@ export class Table extends ActionBuilder { const bigqueryFiltered = LegacyConfigConverter.legacyConvertBigQueryOptions(bigquery); if (Object.values(bigqueryFiltered).length > 0) { - this.proto.bigquery = dataform.BigQueryOptions.create(bigqueryFiltered); + this.proto.bigquery = sqlanvil.BigQueryOptions.create(bigqueryFiltered); } return this; } /** * @deprecated Deprecated in favor of - * [TableConfig.dependencies](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.dependencies](configs#sqlanvil-ActionConfig-TableConfig). * * Sets dependencies of the table. */ @@ -376,7 +376,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.hermetic](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.hermetic](configs#sqlanvil-ActionConfig-TableConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -384,13 +384,13 @@ export class Table extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [TableConfig.tags](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.tags](configs#sqlanvil-ActionConfig-TableConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -406,7 +406,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.description](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.description](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the description of this assertion. */ @@ -420,11 +420,11 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.columns](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.columns](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the column descriptors of columns in this table. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -436,13 +436,13 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.project](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.project](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the database (Google Cloud project ID) in which to create the output of this action. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -452,13 +452,13 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.dataset](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.dataset](configs#sqlanvil-ActionConfig-TableConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -468,7 +468,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.assertions](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig). * * Sets in-line assertions for this table. * @@ -476,7 +476,7 @@ export class Table extends ActionBuilder { * Usage of it via the JS API is deprecated, but the way it applies in-line assertions is still * needed --> */ - public assertions(tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig): Table { + public assertions(tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig): Table { const inlineAssertions = this.generateInlineAssertions(tableAssertionsConfig, this.proto); this.uniqueKeyAssertions = inlineAssertions.uniqueKeyAssertions; this.rowConditionsAssertion = inlineAssertions.rowConditionsAssertion; @@ -485,7 +485,7 @@ export class Table extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [TableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-TableConfig). + * [TableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-TableConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -502,7 +502,7 @@ export class Table extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -522,7 +522,7 @@ export class Table extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -550,7 +550,7 @@ export class Table extends ActionBuilder { this.proto.query = context.apply(this.contextableQuery); - if (this.proto.enumType === dataform.TableType.INCREMENTAL) { + if (this.proto.enumType === sqlanvil.TableType.INCREMENTAL) { this.proto.incrementalQuery = incrementalContext.apply(this.contextableQuery); this.proto.incrementalPreOps = this.contextifyOps(this.contextablePreOps, incrementalContext); @@ -596,8 +596,8 @@ export class Table extends ActionBuilder { private verifyConfig( // `any` is used here to facilitate the type merging of the legacy table config, which is very // different to the new structure. - unverifiedConfig: dataform.ActionConfig.TableConfig | ILegacyTableConfig | any - ): dataform.ActionConfig.TableConfig { + unverifiedConfig: sqlanvil.ActionConfig.TableConfig | ILegacyTableConfig | any + ): sqlanvil.ActionConfig.TableConfig { // The "type" field only exists on legacy table configs. Here we convert them to the // new format. if (unverifiedConfig.type) { @@ -669,7 +669,7 @@ export class Table extends ActionBuilder { } const config = verifyObjectMatchesProto( - dataform.ActionConfig.TableConfig, + sqlanvil.ActionConfig.TableConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -678,7 +678,7 @@ export class Table extends ActionBuilder { this.session.compileError( `requirePartitionFilter/partitionExpirationDays are not valid for non partitioned BigQuery tables`, config.filename, - dataform.Target.create({ + sqlanvil.Target.create({ database: config.project, schema: config.dataset, name: config.name @@ -768,7 +768,7 @@ export class TableContext implements ITableContext { return ""; } - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { this.table.bigquery(bigquery); return ""; } diff --git a/core/actions/table_test.ts b/core/actions/table_test.ts index 38a010d1..094dd4bd 100644 --- a/core/actions/table_test.ts +++ b/core/actions/table_test.ts @@ -7,15 +7,15 @@ import { exampleActionDescriptor, exampleBuiltInAssertions, exampleBuiltInAssertionsAsYaml -} from "df/core/actions/index_test"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/core/actions/index_test"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("table", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); @@ -564,7 +564,7 @@ defaultIcebergConfig: wsContent: VALID_WORKFLOW_SETTINGS_YAML, }, { - testName: "defaults to \`_dataform\` for tableFolderRoot", + testName: "defaults to \`_sqlanvil\` for tableFolderRoot", configBlock: ` name: "table3", dataset: "dataset3", @@ -582,7 +582,7 @@ defaultIcebergConfig: tableFormat: "ICEBERG", fileFormat: "PARQUET", connection: "gcp.us.conn-id", - storageUri: "gs://my-bucket/_dataform/my-subpath", + storageUri: "gs://my-bucket/_sqlanvil/my-subpath", }, }, expectError: false, diff --git a/core/actions/test.ts b/core/actions/test.ts index 2034069f..854cb172 100644 --- a/core/actions/test.ts +++ b/core/actions/test.ts @@ -1,11 +1,11 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { ActionBuilder, INamedConfig, TableType } from "df/core/actions"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { Contextable, IActionContext, ITableContext, Resolvable } from "df/core/contextables"; -import { Session } from "df/core/session"; -import { targetStringifier } from "df/core/targets"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { ActionBuilder, INamedConfig, TableType } from "sa/core/actions"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { Contextable, IActionContext, ITableContext, Resolvable } from "sa/core/contextables"; +import { Session } from "sa/core/session"; +import { targetStringifier } from "sa/core/targets"; import { ambiguousActionNameMsg, checkExcessProperties, @@ -13,8 +13,8 @@ import { strictKeysOf, stringifyResolvable, toResolvable -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * Configuration options for unit tests. @@ -41,7 +41,7 @@ export interface ITestConfig extends INamedConfig { const ITestConfigProperties = strictKeysOf()(["type", "dataset", "name", "filename", "tags"]); /** - * Dataform test actions can be used to write unit tests for your generated SQL + * sqlanvil test actions can be used to write unit tests for your generated SQL * * You can create unit tests in the following ways. * @@ -74,19 +74,19 @@ const ITestConfigProperties = strictKeysOf()(["type", "dataset", "n * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `input` and `expect` come from. */ -export class Test extends ActionBuilder { +export class Test extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; /** @hidden We delay contextification until the final compile step, so hold these here for now. */ public contextableInputs = new Map>(); private contextableQuery: Contextable; - private testTarget: dataform.ITarget; + private testTarget: sqlanvil.ITarget; /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Test.create(); + private proto = sqlanvil.Test.create(); /** @hidden */ constructor(session?: Session, config?: ITestConfig) { @@ -111,7 +111,7 @@ export class Test extends ActionBuilder { } if (config.dataset) { // Determine target from the parent dataset name - this.testTarget = dataform.Target.create( + this.testTarget = sqlanvil.Target.create( this.applySessionToTarget( resolvableAsTarget( toResolvable(config.dataset) @@ -119,7 +119,7 @@ export class Test extends ActionBuilder { this.session.projectConfig ) ); - const canonicalTestTarget = dataform.Target.create( + const canonicalTestTarget = sqlanvil.Target.create( this.applySessionToTarget( resolvableAsTarget( toResolvable(config.dataset) @@ -174,12 +174,12 @@ export class Test extends ActionBuilder { } /** @hidden */ - public getTarget(): dataform.Target { - return dataform.Target.create(this.proto.target); + public getTarget(): sqlanvil.Target { + return sqlanvil.Target.create(this.proto.target); } - public getTestTarget(): dataform.Target { - return dataform.Target.create(this.testTarget); + public getTestTarget(): sqlanvil.Target { + return sqlanvil.Target.create(this.testTarget); } public setFilename(filename: string) { @@ -239,7 +239,7 @@ export class Test extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Test, + sqlanvil.Test, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -358,8 +358,8 @@ class RefReplacingContext implements ITableContext { } } -function overrideTargetWithNewName(target: dataform.ITarget, testName: string): dataform.Target { - return dataform.Target.create({ +function overrideTargetWithNewName(target: sqlanvil.ITarget, testName: string): sqlanvil.Target { + return sqlanvil.Target.create({ database: target.database, schema: target.schema, name: testName diff --git a/core/actions/test_test.ts b/core/actions/test_test.ts index 8eb776c7..e9328483 100644 --- a/core/actions/test_test.ts +++ b/core/actions/test_test.ts @@ -3,13 +3,13 @@ import { expect } from "chai"; import * as fs from "fs-extra"; import * as path from "path"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("test", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/actions/view.ts b/core/actions/view.ts index 97f1b5ce..fd71a4d9 100644 --- a/core/actions/view.ts +++ b/core/actions/view.ts @@ -1,17 +1,17 @@ -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; import { ActionBuilder, ILegacyBigQueryOptions, LegacyConfigConverter, TableType -} from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Table } from "df/core/actions/table"; -import { ColumnDescriptors } from "df/core/column_descriptors"; -import { Contextable, ITableContext, JitContextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; +} from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Table } from "sa/core/actions/table"; +import { ColumnDescriptors } from "sa/core/column_descriptors"; +import { Contextable, ITableContext, JitContextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; import { actionConfigToCompiledGraphTarget, checkAssertionsForDependency, @@ -25,8 +25,8 @@ import { toResolvable, validateNoMixedCompilationMode, validateQueryString, -} from "df/core/utils"; -import { dataform } from "df/protos/ts"; +} from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @hidden @@ -41,7 +41,7 @@ export interface ILegacyViewBigqueryConfig { additionalOptions: { [key: string]: string }; } -export type JitViewResult = string | dataform.IJitTableResult; +export type JitViewResult = string | sqlanvil.IJitTableResult; /** * Views are virtualised tables. They are useful for creating a new structured table without having @@ -49,7 +49,7 @@ export type JitViewResult = string | dataform.IJitTableResult; * processing and storage. * * You can create views in the following ways. Available config options are defined in - * [ViewConfig](configs#dataform-ActionConfig-ViewConfig), and are shared across all the + * [ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig), and are shared across all the * following ways of creating tables. * * **Using a SQLX file:** @@ -86,7 +86,7 @@ export type JitViewResult = string | dataform.IJitTableResult; * Note: When using the Javascript API, methods in this class can be accessed by the returned value. * This is where `query` comes from. */ -export class View extends ActionBuilder { +export class View extends ActionBuilder { /** @hidden Hold a reference to the Session instance. */ public session: Session; @@ -106,9 +106,9 @@ export class View extends ActionBuilder { /** * @hidden Stores the generated proto for the compiled graph. */ - private proto = dataform.Table.create({ + private proto = sqlanvil.Table.create({ type: "view", - enumType: dataform.TableType.VIEW, + enumType: sqlanvil.TableType.VIEW, disabled: false, tags: [] }); @@ -158,7 +158,7 @@ export class View extends ActionBuilder { if (config.dependencyTargets) { this.dependencies( config.dependencyTargets.map(dependencyTarget => - configTargetToCompiledGraphTarget(dataform.ActionConfig.Target.create(dependencyTarget)) + configTargetToCompiledGraphTarget(sqlanvil.ActionConfig.Target.create(dependencyTarget)) ) ); } @@ -183,7 +183,7 @@ export class View extends ActionBuilder { if (config.columns?.length) { this.columns( config.columns.map(columnDescriptor => - dataform.ActionConfig.ColumnDescriptor.create(columnDescriptor) + sqlanvil.ActionConfig.ColumnDescriptor.create(columnDescriptor) ) ); } @@ -194,7 +194,7 @@ export class View extends ActionBuilder { this.schema(config.dataset); } if (config.assertions) { - this.assertions(dataform.ActionConfig.TableAssertionsConfig.create(config.assertions)); + this.assertions(sqlanvil.ActionConfig.TableAssertionsConfig.create(config.assertions)); } if (config.materialized) { this.materialized(config.materialized); @@ -253,7 +253,7 @@ export class View extends ActionBuilder { const existingAction = this.session.actions.indexOf(this); if (existingAction === -1) { throw Error( - "Expected pre-existing action, but none found. Please report this to the Dataform team." + "Expected pre-existing action, but none found. Please report this to the sqlanvil team." ); } this.session.actions[existingAction] = newAction; @@ -313,7 +313,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.disabled](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.disabled](configs#sqlanvil-ActionConfig-ViewConfig). * * If called with `true`, this action is not executed. The action can still be depended upon. * Useful for temporarily turning off broken actions. @@ -327,7 +327,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.materialized](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.materialized](configs#sqlanvil-ActionConfig-ViewConfig). * * Applies the materialized view optimization, see * https://cloud.google.com/bigquery/docs/materialized-views-intro. @@ -338,12 +338,12 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of options available directly on - * [ViewConfig](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets bigquery options for the action. */ - public bigquery(bigquery: dataform.IBigQueryOptions) { - this.proto.bigquery = dataform.BigQueryOptions.create(bigquery); + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { + this.proto.bigquery = sqlanvil.BigQueryOptions.create(bigquery); if (!!bigquery.labels) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; @@ -355,7 +355,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.dependencies](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.dependencies](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets dependencies of the view. */ @@ -372,7 +372,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.hermetic](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.hermetic](configs#sqlanvil-ActionConfig-ViewConfig). * * If true, this indicates that the action only depends on data from explicitly-declared * dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -380,13 +380,13 @@ export class View extends ActionBuilder { */ public hermetic(hermetic: boolean) { this.proto.hermeticity = hermetic - ? dataform.ActionHermeticity.HERMETIC - : dataform.ActionHermeticity.NON_HERMETIC; + ? sqlanvil.ActionHermeticity.HERMETIC + : sqlanvil.ActionHermeticity.NON_HERMETIC; } /** * @deprecated Deprecated in favor of - * [ViewConfig.tags](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.tags](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets a list of user-defined tags applied to this action. */ @@ -402,7 +402,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.description](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.description](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the description of this view. */ @@ -416,11 +416,11 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.columns](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.columns](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the column descriptors of columns in this view. */ - public columns(columns: dataform.ActionConfig.ColumnDescriptor[]) { + public columns(columns: sqlanvil.ActionConfig.ColumnDescriptor[]) { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } @@ -432,14 +432,14 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.project](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.project](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the * Sets the database (Google Cloud project ID) in which to create the output of this action. */ public database(database: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, database }), + sqlanvil.Target.create({ ...this.proto.target, database }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -449,13 +449,13 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.dataset](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.dataset](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets the schema (BigQuery dataset) in which to create the output of this action. */ public schema(schema: string) { this.proto.target = this.applySessionToTarget( - dataform.Target.create({ ...this.proto.target, schema }), + sqlanvil.Target.create({ ...this.proto.target, schema }), this.session.projectConfig, this.proto.fileName, { validateTarget: true } @@ -465,7 +465,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.assertions](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.assertions](configs#sqlanvil-ActionConfig-ViewConfig). * * Sets in-line assertions for this view. * @@ -473,7 +473,7 @@ export class View extends ActionBuilder { * Usage of it via the JS API is deprecated, but the way it applies in-line assertions is still * needed --> */ - public assertions(tableAssertionsConfig: dataform.ActionConfig.TableAssertionsConfig): View { + public assertions(tableAssertionsConfig: sqlanvil.ActionConfig.TableAssertionsConfig): View { const inlineAssertions = this.generateInlineAssertions(tableAssertionsConfig, this.proto); this.uniqueKeyAssertions = inlineAssertions.uniqueKeyAssertions; this.rowConditionsAssertion = inlineAssertions.rowConditionsAssertion; @@ -482,7 +482,7 @@ export class View extends ActionBuilder { /** * @deprecated Deprecated in favor of - * [ViewConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-ViewConfig). + * [ViewConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-ViewConfig). * * When called with `true`, assertions dependent upon any dependency will be add as dedpendency * to this action. @@ -496,7 +496,7 @@ export class View extends ActionBuilder { if (!this.proto.actionDescriptor) { this.proto.actionDescriptor = {}; } - this.proto.actionDescriptor.compilationMode = dataform.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; + this.proto.actionDescriptor.compilationMode = sqlanvil.ActionCompilationMode.ACTION_COMPILATION_MODE_JIT; this.contextableJitCode = jitCode; return this; } @@ -508,7 +508,7 @@ export class View extends ActionBuilder { /** @hidden */ public getTarget() { - return dataform.Target.create(this.proto.target); + return sqlanvil.Target.create(this.proto.target); } /** @hidden */ @@ -520,7 +520,7 @@ export class View extends ActionBuilder { } return verifyObjectMatchesProto( - dataform.Table, + sqlanvil.Table, this.proto, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -549,7 +549,7 @@ export class View extends ActionBuilder { this.proto.query = context.apply(this.contextableQuery); - if (this.proto.enumType === dataform.TableType.INCREMENTAL) { + if (this.proto.enumType === sqlanvil.TableType.INCREMENTAL) { this.proto.incrementalQuery = incrementalContext.apply(this.contextableQuery); this.proto.incrementalPreOps = this.contextifyOps(this.contextablePreOps, incrementalContext); @@ -595,8 +595,8 @@ export class View extends ActionBuilder { private verifyConfig( // `any` is used here to facilitate the type merging of the legacy table config, which is very // different to the new structure. - unverifiedConfig: dataform.ActionConfig.ViewConfig | ILegacyBigQueryOptions | any - ): dataform.ActionConfig.ViewConfig { + unverifiedConfig: sqlanvil.ActionConfig.ViewConfig | ILegacyBigQueryOptions | any + ): sqlanvil.ActionConfig.ViewConfig { // The "type" field only exists on legacy view configs. Here we convert them to the new format. if (unverifiedConfig.type) { delete unverifiedConfig.type; @@ -655,7 +655,7 @@ export class View extends ActionBuilder { } const config = verifyObjectMatchesProto( - dataform.ActionConfig.ViewConfig, + sqlanvil.ActionConfig.ViewConfig, unverifiedConfig, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); @@ -664,7 +664,7 @@ export class View extends ActionBuilder { this.session.compileError( `partitionBy/clusterBy can be applied only to materialized views`, config.filename, - dataform.Target.create({ + sqlanvil.Target.create({ database: config.project, schema: config.dataset, name: config.name @@ -745,7 +745,7 @@ export class ViewContext implements ITableContext { return ""; } - public bigquery(bigquery: dataform.IBigQueryOptions) { + public bigquery(bigquery: sqlanvil.IBigQueryOptions) { this.view.bigquery(bigquery); return ""; } diff --git a/core/actions/view_test.ts b/core/actions/view_test.ts index b72c4444..131caa71 100644 --- a/core/actions/view_test.ts +++ b/core/actions/view_test.ts @@ -7,14 +7,14 @@ import { exampleActionDescriptor, exampleBuiltInAssertions, exampleBuiltInAssertionsAsYaml -} from "df/core/actions/index_test"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +} from "sa/core/actions/index_test"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, VALID_WORKFLOW_SETTINGS_YAML -} from "df/testing/run_core"; +} from "sa/testing/run_core"; suite("view", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); diff --git a/core/column_descriptors.ts b/core/column_descriptors.ts index 95636310..b9efd0cf 100644 --- a/core/column_descriptors.ts +++ b/core/column_descriptors.ts @@ -1,5 +1,5 @@ -import * as utils from "df/core/utils"; -import { dataform } from "df/protos/ts"; +import * as utils from "sa/core/utils"; +import { sqlanvil } from "sa/protos/ts"; /** * @deprecated @@ -70,10 +70,10 @@ export const IRecordDescriptorProperties = () => */ export class ColumnDescriptors { public static mapConfigProtoToCompilationProto( - columns: dataform.ActionConfig.ColumnDescriptor[] - ): dataform.IColumnDescriptor[] { + columns: sqlanvil.ActionConfig.ColumnDescriptor[] + ): sqlanvil.IColumnDescriptor[] { return columns.map(column => { - return dataform.ColumnDescriptor.create({ + return sqlanvil.ColumnDescriptor.create({ path: column.path, description: column.description, tags: column.tags, @@ -84,7 +84,7 @@ export class ColumnDescriptors { public static mapLegacyObjectToConfigProto( columns: IColumnsDescriptor - ): dataform.ActionConfig.ColumnDescriptor[] { + ): sqlanvil.ActionConfig.ColumnDescriptor[] { return Object.keys(columns) .map(column => ColumnDescriptors.mapColumnDescriptionToProto([column], columns[column])) .flat(); @@ -93,18 +93,18 @@ export class ColumnDescriptors { public static mapColumnDescriptionToProto( currentPath: string[], description: string | IRecordDescriptor - ): dataform.ActionConfig.ColumnDescriptor[] { + ): sqlanvil.ActionConfig.ColumnDescriptor[] { if (typeof description === "string") { return [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description, path: currentPath }) ]; } - const columnDescriptor: dataform.ActionConfig.ColumnDescriptor[] = !!description + const columnDescriptor: sqlanvil.ActionConfig.ColumnDescriptor[] = !!description ? [ - dataform.ActionConfig.ColumnDescriptor.create({ + sqlanvil.ActionConfig.ColumnDescriptor.create({ path: currentPath, description: description.description, tags: typeof description.tags === "string" ? [description.tags] : description.tags, @@ -136,7 +136,7 @@ export class LegacyColumnDescriptors { public static mapToColumnProtoArray( columns: IColumnsDescriptor, reportError: (e: Error) => void - ): dataform.IColumnDescriptor[] { + ): sqlanvil.IColumnDescriptor[] { return Object.keys(columns) .map(column => LegacyColumnDescriptors.mapColumnDescriptionToProto([column], columns[column], reportError) @@ -148,10 +148,10 @@ export class LegacyColumnDescriptors { currentPath: string[], description: string | IRecordDescriptor, reportError: (e: Error) => void - ): dataform.IColumnDescriptor[] { + ): sqlanvil.IColumnDescriptor[] { if (typeof description === "string") { return [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description, path: currentPath }) @@ -163,9 +163,9 @@ export class LegacyColumnDescriptors { IRecordDescriptorProperties(), `${currentPath.join(".")} column descriptor` ); - const columnDescriptor: dataform.IColumnDescriptor[] = !!description + const columnDescriptor: sqlanvil.IColumnDescriptor[] = !!description ? [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ path: currentPath, description: description.description, displayName: description.displayName, diff --git a/core/compilation_sql/index.ts b/core/compilation_sql/index.ts index 4639a545..9eec8329 100644 --- a/core/compilation_sql/index.ts +++ b/core/compilation_sql/index.ts @@ -1,12 +1,12 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; export class CompilationSql { constructor( - private readonly project: dataform.IProjectConfig, - private readonly dataformCoreVersion: string + private readonly project: sqlanvil.IProjectConfig, + private readonly sqlanvilCoreVersion: string ) {} - public resolveTarget(target: dataform.ITarget) { + public resolveTarget(target: sqlanvil.ITarget) { const database = target.database || this.project.defaultDatabase; if (!database) { return `\`${target.schema || this.project.defaultSchema}.${target.name}\``; diff --git a/core/compilers.ts b/core/compilers.ts index a760153f..d0620be5 100644 --- a/core/compilers.ts +++ b/core/compilers.ts @@ -1,7 +1,7 @@ import { load as loadYaml, YAMLException } from "js-yaml"; -import * as Path from "df/core/path"; -import { SyntaxTreeNode, SyntaxTreeNodeType } from "df/sqlx/lexer"; +import * as Path from "sa/core/path"; +import { SyntaxTreeNode, SyntaxTreeNodeType } from "sa/sqlx/lexer"; const CONTEXT_FUNCTIONS = [ "self", @@ -87,7 +87,7 @@ function compileSqlx(rootNode: SyntaxTreeNode, path: string): string { rootNode ); - return `dataform.sqlxAction({ + return `sqlanvil.sqlxAction({ sqlxConfig: { name: "${Path.escapedBasename(path)}", type: "operations", diff --git a/core/compilers_test.ts b/core/compilers_test.ts index f5410e55..16a9f3c0 100644 --- a/core/compilers_test.ts +++ b/core/compilers_test.ts @@ -1,7 +1,7 @@ import { expect } from "chai"; -import { compile } from "df/core/compilers"; -import { suite, test } from "df/testing"; +import { compile } from "sa/core/compilers"; +import { suite, test } from "sa/testing"; suite("core/compilers", () => { suite("compile", () => { @@ -9,7 +9,7 @@ suite("core/compilers", () => { const code = `config { type: "table" } select 1`; const path = "definitions/foo.sqlx"; const result = compile(code, path); - expect(result).to.include("dataform.sqlxAction"); + expect(result).to.include("sqlanvil.sqlxAction"); expect(result).to.include("name: \"foo\""); expect(result).to.include("{ type: \"table\" }"); // The sqlx compiler will format the query. diff --git a/core/contextables.ts b/core/contextables.ts index 3347e664..75c968fe 100644 --- a/core/contextables.ts +++ b/core/contextables.ts @@ -1,10 +1,10 @@ -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; /** * A resolvable is a reference to an action, and it can be either the string representation of the * action's target, or the target of the action. */ -export type Resolvable = string | dataform.ITarget; +export type Resolvable = string | sqlanvil.ITarget; /** * Contextable arguments can either pass a plain value for their generic type `T` or can pass a @@ -89,13 +89,13 @@ export interface ITableContext extends IActionContext { /** JiT context, accessible at JiT compilation stage. */ export type JitContext = T & { /** Direct access to adapter. */ - adapter: dataform.DbAdapter, + adapter: sqlanvil.DbAdapter, /** JiT data object. */ data?: { [k: string]: any }, /** Original JiT compilation request. */ - request: dataform.IJitCompilationRequest, + request: sqlanvil.IJitCompilationRequest, /** Current execution information for introspection. */ - executionData: dataform.IRunningExecutionData, + executionData: sqlanvil.IRunningExecutionData, }; /** diff --git a/core/extension.ts b/core/extension.ts index c3f63760..de034642 100644 --- a/core/extension.ts +++ b/core/extension.ts @@ -1,13 +1,13 @@ -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; /** * Extension interface. */ -export interface IDataformExtension { +export interface ISqlanvilExtension { /** * Run additional compilation steps. * Passed session should be used for both new nodes creation and persisting errors. */ - compile(request: dataform.ICompileExecutionRequest, session: Session): void; + compile(request: sqlanvil.ICompileExecutionRequest, session: Session): void; } diff --git a/core/index.ts b/core/index.ts index fcade976..99d74ecc 100644 --- a/core/index.ts +++ b/core/index.ts @@ -1,14 +1,14 @@ -import { compile as compiler } from "df/core/compilers"; -import { IDataformExtension } from "df/core/extension"; -import { IJitCompiler, jitCompiler } from "df/core/jit_compiler"; -import { main } from "df/core/main"; -import { Session } from "df/core/session"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; +import { compile as compiler } from "sa/core/compilers"; +import { ISqlanvilExtension } from "sa/core/extension"; +import { IJitCompiler, jitCompiler } from "sa/core/jit_compiler"; +import { main } from "sa/core/main"; +import { Session } from "sa/core/session"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; // Create static session object. // This hack just enforces the singleton session object to -// be the same, regardless of the @dataform/core package that is running. +// be the same, regardless of the @sqlanvil/core package that is running. function globalSession() { if (!(global as any)._DF_SESSION) { (global as any)._DF_SESSION = new Session(); @@ -17,17 +17,17 @@ function globalSession() { } const session = globalSession(); -const supportedFeatures = [dataform.SupportedFeatures.ARRAY_BUFFER_IPC]; +const supportedFeatures = [sqlanvil.SupportedFeatures.ARRAY_BUFFER_IPC]; // Older versions of the CLI are not compatible with Core version ^3.0.0, and throw when this method // is not available. Instead this more interpretable error message is thrown. // Note: for future backwards compatability breaking changes, the exported "version" variable should // be used instead. function indexFileGenerator() { - throw new Error("@dataform/cli ^3.0.0 required."); + throw new Error("@sqlanvil/cli ^3.0.0 required."); } -// These exports constitute the public API of @dataform/core. -// They must also be listed in packages/@dataform/core/index.ts. -// Changes to these will break @dataform/cli, so take care! -export { compiler, IDataformExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version }; +// These exports constitute the public API of @sqlanvil/core. +// They must also be listed in packages/@sqlanvil/core/index.ts. +// Changes to these will break @sqlanvil/cli, so take care! +export { compiler, ISqlanvilExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version }; diff --git a/core/jit_compiler.ts b/core/jit_compiler.ts index c7993528..a91871a3 100644 --- a/core/jit_compiler.ts +++ b/core/jit_compiler.ts @@ -1,10 +1,10 @@ import * as $protobuf from "protobufjs"; -import { JitOperationResult } from "df/core/actions/operation"; -import { JitTableResult } from "df/core/actions/table"; -import { IActionContext, ITableContext, JitContext } from "df/core/contextables"; -import { IncrementalTableJitContext, SqlActionJitContext, TableJitContext } from "df/core/jit_context"; -import { dataform } from "df/protos/ts"; +import { JitOperationResult } from "sa/core/actions/operation"; +import { JitTableResult } from "sa/core/actions/table"; +import { IActionContext, ITableContext, JitContext } from "sa/core/contextables"; +import { IncrementalTableJitContext, SqlActionJitContext, TableJitContext } from "sa/core/jit_context"; +import { sqlanvil } from "sa/protos/ts"; function makeMainBody(code: string): (jctx: JitContext) => Promise { return ( @@ -17,21 +17,21 @@ function makeMainBody(code: string): (jctx: JitContext) => }); } -function makeJitTableResult(result: JitTableResult): dataform.IJitTableResult { - let jitResult: dataform.IJitTableResult = {}; +function makeJitTableResult(result: JitTableResult): sqlanvil.IJitTableResult { + let jitResult: sqlanvil.IJitTableResult = {}; if (typeof result === "string") { jitResult.query = result; } else { jitResult = result; } - return dataform.JitTableResult.create(jitResult); + return sqlanvil.JitTableResult.create(jitResult); } function jitCompileOperation( - request: dataform.IJitCompilationRequest, - adapter: dataform.DbAdapter, -): Promise { + request: sqlanvil.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, +): Promise { const mainBody = makeMainBody(request.jitCode); const jctx: JitContext = new SqlActionJitContext( @@ -47,14 +47,14 @@ function jitCompileOperation( queries = mainResult.queries; } - return dataform.JitOperationResult.create({ queries }); + return sqlanvil.JitOperationResult.create({ queries }); }); } function jitCompileTable( - request: dataform.IJitCompilationRequest, - adapter: dataform.DbAdapter, -): Promise { + request: sqlanvil.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, +): Promise { const mainBody = makeMainBody(request.jitCode); const jctx: JitContext = new TableJitContext( @@ -64,9 +64,9 @@ function jitCompileTable( } function jitCompileIncrementalTable( - request: dataform.IJitCompilationRequest, - adapter: dataform.DbAdapter, -): Promise { + request: sqlanvil.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, +): Promise { const mainBody = makeMainBody(request.jitCode); const incrementalJctx = new IncrementalTableJitContext( @@ -80,7 +80,7 @@ function jitCompileIncrementalTable( mainBody(incrementalJctx), mainBody(regularJctx), ]).then(([incrementalResult, regularResult]) => { - return dataform.JitIncrementalTableResult.create({ + return sqlanvil.JitIncrementalTableResult.create({ incremental: makeJitTableResult(incrementalResult), regular: makeJitTableResult(regularResult), }); @@ -94,22 +94,22 @@ export interface IJitCompiler { /** RPC callback, implementing DbAdapter. */ export type RpcCallback = (method: string, request: Uint8Array, callback: (error: Error | null, response: Uint8Array) => void) => void; -export function jitCompile(request: dataform.IJitCompilationRequest, rpcCallback: RpcCallback): Promise { +export function jitCompile(request: sqlanvil.IJitCompilationRequest, rpcCallback: RpcCallback): Promise { const rpcImpl: $protobuf.RPCImpl = (method, internalRequest, callback) => { rpcCallback(method.name, internalRequest, callback); }; - const dbAdapter = dataform.DbAdapter.create(rpcImpl); + const dbAdapter = sqlanvil.DbAdapter.create(rpcImpl); switch (request.compilationTargetType) { - case dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION: + case sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION: return jitCompileOperation(request, dbAdapter).then( - operation => dataform.JitCompilationResponse.create({ operation })); - case dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE: + operation => sqlanvil.JitCompilationResponse.create({ operation })); + case sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE: return jitCompileTable(request, dbAdapter).then( - table => dataform.JitCompilationResponse.create({ table })); - case dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE: + table => sqlanvil.JitCompilationResponse.create({ table })); + case sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE: return jitCompileIncrementalTable(request, dbAdapter).then( - incrementalTable => dataform.JitCompilationResponse.create({ incrementalTable })); + incrementalTable => sqlanvil.JitCompilationResponse.create({ incrementalTable })); default: throw new Error(`Unrecognized compilation target type: ${request.compilationTargetType}`); } @@ -119,9 +119,9 @@ export function jitCompile(request: dataform.IJitCompilationRequest, rpcCallback export function jitCompiler(rpcCallback: RpcCallback): IJitCompiler { return { compile: (request: Uint8Array) => { - const requestMessage = dataform.JitCompilationRequest.decode(request); + const requestMessage = sqlanvil.JitCompilationRequest.decode(request); return jitCompile(requestMessage, rpcCallback).then( - response => dataform.JitCompilationResponse.encode(response).finish() + response => sqlanvil.JitCompilationResponse.encode(response).finish() ); } }; diff --git a/core/jit_compiler_test.ts b/core/jit_compiler_test.ts index 9d252e05..19947c1b 100644 --- a/core/jit_compiler_test.ts +++ b/core/jit_compiler_test.ts @@ -1,15 +1,15 @@ import { expect } from "chai"; import Long from "long"; -import { jitCompile } from "df/core/jit_compiler"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; +import { jitCompile } from "sa/core/jit_compiler"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; suite("jit_compiler", () => { const rpcCallback: (method: string, request: Uint8Array, callback: (error: Error | null, response: Uint8Array) => void) => void = (method, request, callback) => { callback(null, new Uint8Array()); }; - const target = dataform.Target.create({ + const target = sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" @@ -17,77 +17,77 @@ suite("jit_compiler", () => { suite("jitCompileOperation", () => { test("compiles operation returning string", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => "SELECT 1"`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation returning array", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ["SELECT 1", "SELECT 2"]`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1", "SELECT 2"]); }); test("compiles operation returning object", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ({ queries: ["SELECT 1"] })`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation using context", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ({ queries: [\`SELECT "\${ctx.name()}"\`] })`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(['SELECT "name"']); }); test("compiles operation with arrow function returning promise", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `(ctx) => Promise.resolve("SELECT 1")`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation with async function", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async function(ctx) { return "SELECT 1"; }`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); }); test("compiles operation with regular function returning promise", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `function(ctx) { return Promise.resolve("SELECT 1"); }`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_OPERATION, }); const result = await jitCompile(request, rpcCallback); expect(result.operation.queries).to.deep.equal(["SELECT 1"]); @@ -96,22 +96,22 @@ suite("jit_compiler", () => { suite("jitCompileTable", () => { test("compiles table returning string", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => "SELECT 1"`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, }); const result = await jitCompile(request, rpcCallback); expect(result.table.query).to.equal("SELECT 1"); }); test("compiles table returning object", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => ({ query: "SELECT 1", preOps: ["PRE"], postOps: ["POST"] })`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, }); const result = await jitCompile(request, rpcCallback); expect(result.table.query).to.equal("SELECT 1"); @@ -122,7 +122,7 @@ suite("jit_compiler", () => { suite("jitCompileIncrementalTable", () => { test("compiles incremental table", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (ctx) => { if (ctx.incremental()) { return { query: "SELECT INC" }; @@ -131,7 +131,7 @@ suite("jit_compiler", () => { }`, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_INCREMENTAL_TABLE, }); const result = await jitCompile(request, rpcCallback); expect(result.incrementalTable.incremental?.query).to.equal("SELECT INC"); @@ -141,16 +141,16 @@ suite("jit_compiler", () => { suite("jitCompileContext", () => { test("can reference self and other tables", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (jctx) => \`$\{jctx.self()\}\n$\{jctx.ref('other')\}\``, target, jitData: {}, - dependencies: [dataform.Target.create({ + dependencies: [sqlanvil.Target.create({ database: "db", schema: "schema", name: "other", })], - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, }); const result = await jitCompile(request, rpcCallback); @@ -158,11 +158,11 @@ suite("jit_compiler", () => { }); test("can reference execution info data", async () => { - const request = dataform.JitCompilationRequest.create({ + const request = sqlanvil.JitCompilationRequest.create({ jitCode: `async (jctx) => \`$\{jctx.executionData.executionStartTime.seconds\}\n$\{jctx.executionData.executionId\}\``, target, jitData: {}, - compilationTargetType: dataform.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, + compilationTargetType: sqlanvil.JitCompilationTargetType.JIT_COMPILATION_TARGET_TYPE_TABLE, executionData: { executionId: "test-id", executionStartTime: {seconds: Long.fromNumber(1774974514), nanos: 481}, diff --git a/core/jit_context.ts b/core/jit_context.ts index ed042326..e75e2eec 100644 --- a/core/jit_context.ts +++ b/core/jit_context.ts @@ -1,23 +1,23 @@ -import { Structs } from "df/common/protos/structs"; -import { IActionContext, ITableContext, JitContext, Resolvable } from "df/core/contextables"; -import { ambiguousActionNameMsg, resolvableAsTarget, ResolvableMap, stringifyResolvable, toResolvable } from "df/core/utils"; -import { dataform, google } from "df/protos/ts"; +import { Structs } from "sa/common/protos/structs"; +import { IActionContext, ITableContext, JitContext, Resolvable } from "sa/core/contextables"; +import { ambiguousActionNameMsg, resolvableAsTarget, ResolvableMap, stringifyResolvable, toResolvable } from "sa/core/utils"; +import { sqlanvil, google } from "sa/protos/ts"; -function canonicalTargetValue(target: dataform.ITarget): string { +function canonicalTargetValue(target: sqlanvil.ITarget): string { return `${target.database}.${target.schema}.${target.name}`; } /** Generate SQL action JiT context. */ export class SqlActionJitContext implements JitContext { public readonly data: { [k: string]: any } | undefined; - public readonly executionData: dataform.IRunningExecutionData; + public readonly executionData: sqlanvil.IRunningExecutionData; - private readonly target: dataform.ITarget; + private readonly target: sqlanvil.ITarget; private readonly resolvableMap: ResolvableMap; constructor( - public readonly adapter: dataform.DbAdapter, - public readonly request: dataform.IJitCompilationRequest, + public readonly adapter: sqlanvil.DbAdapter, + public readonly request: sqlanvil.IJitCompilationRequest, ) { this.target = request.target; const dependencies = request.dependencies; @@ -77,8 +77,8 @@ export class SqlActionJitContext implements JitContext { /** JiT context for table and view actions. */ export class TableJitContext extends SqlActionJitContext implements JitContext { constructor( - adapter: dataform.DbAdapter, - request: dataform.IJitCompilationRequest, + adapter: sqlanvil.DbAdapter, + request: sqlanvil.IJitCompilationRequest, ) { super(adapter, request); } @@ -94,8 +94,8 @@ export class TableJitContext extends SqlActionJitContext implements JitContext { suite("SqlActionJitContext", () => { - const adapter = {} as dataform.DbAdapter; + const adapter = {} as sqlanvil.DbAdapter; const jitData = google.protobuf.Struct.create({ fields: { key: google.protobuf.Value.create({ @@ -42,14 +42,14 @@ suite("jit_context", () => { }) } }); - const request = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const request = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" }), dependencies: [ - dataform.Target.create({ + sqlanvil.Target.create({ database: "db", schema: "schema", name: "dep" @@ -58,8 +58,8 @@ suite("jit_context", () => { filePaths: [], jitData, }); - const withoutDependenciesRequest = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const withoutDependenciesRequest = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" @@ -136,9 +136,9 @@ suite("jit_context", () => { }); suite("TableJitContext", () => { - const adapter = {} as dataform.DbAdapter; - const request = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const adapter = {} as sqlanvil.DbAdapter; + const request = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" @@ -159,9 +159,9 @@ suite("jit_context", () => { }); suite("IncrementalTableJitContext", () => { - const adapter = {} as dataform.DbAdapter; - const request = dataform.JitCompilationRequest.create({ - target: dataform.Target.create({ + const adapter = {} as sqlanvil.DbAdapter; + const request = sqlanvil.JitCompilationRequest.create({ + target: sqlanvil.Target.create({ database: "db", schema: "schema", name: "name" diff --git a/core/main.ts b/core/main.ts index b4ead165..46483b11 100644 --- a/core/main.ts +++ b/core/main.ts @@ -3,38 +3,38 @@ import { encode64, verifyObjectMatchesProto, VerifyProtoErrorBehaviour -} from "df/common/protos"; -import { Assertion } from "df/core/actions/assertion"; -import { DataPreparation } from "df/core/actions/data_preparation"; -import { Declaration } from "df/core/actions/declaration"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation } from "df/core/actions/operation"; -import { Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { IDataformExtension } from "df/core/extension"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; -import { nativeRequire } from "df/core/utils"; -import { readWorkflowSettings } from "df/core/workflow_settings"; -import { dataform } from "df/protos/ts"; +} from "sa/common/protos"; +import { Assertion } from "sa/core/actions/assertion"; +import { DataPreparation } from "sa/core/actions/data_preparation"; +import { Declaration } from "sa/core/actions/declaration"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation } from "sa/core/actions/operation"; +import { Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { ISqlanvilExtension } from "sa/core/extension"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; +import { nativeRequire } from "sa/core/utils"; +import { readWorkflowSettings } from "sa/core/workflow_settings"; +import { sqlanvil } from "sa/protos/ts"; /** * This is the main entry point into the user space code that should be invoked by the compilation wrapper sandbox. * - * @param coreExecutionRequest an encoded @see {@link dataform.CoreExecutionRequest} proto. - * @returns an encoded @see {@link dataform.CoreExecutionResponse} proto. + * @param coreExecutionRequest an encoded @see {@link sqlanvil.CoreExecutionRequest} proto. + * @returns an encoded @see {@link sqlanvil.CoreExecutionResponse} proto. */ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | string { const globalAny = global as any; - let request: dataform.CoreExecutionRequest; + let request: sqlanvil.CoreExecutionRequest; if (typeof coreExecutionRequest === "string") { - // Older versions of the Dataform CLI send a base64 encoded string. + // Older versions of the sqlanvil CLI send a base64 encoded string. // See https://github.com/dataform-co/dataform/pull/1570. - request = decode64(dataform.CoreExecutionRequest, coreExecutionRequest); + request = decode64(sqlanvil.CoreExecutionRequest, coreExecutionRequest); } else { - request = dataform.CoreExecutionRequest.decode(coreExecutionRequest); + request = sqlanvil.CoreExecutionRequest.decode(coreExecutionRequest); } const compileRequest = request.compile; @@ -45,34 +45,34 @@ export function main(coreExecutionRequest: Uint8Array | string): Uint8Array | st // Merge in project config overrides. const projectConfigOverride = compileRequest.compileConfig.projectConfigOverride ?? {}; - projectConfig = dataform.ProjectConfig.create({ + projectConfig = sqlanvil.ProjectConfig.create({ ...projectConfig, ...projectConfigOverride, vars: { ...projectConfig.vars, ...projectConfigOverride.vars } }); // Initialize the compilation session. - const session = nativeRequire("@dataform/core").session as Session; + const session = nativeRequire("@sqlanvil/core").session as Session; session.init(compileRequest.compileConfig.projectDir, projectConfig, projectConfig); // Allow "includes" files to use the current session object. - globalAny.dataform = session; + globalAny.sqlanvil = session; prologueCompile(compileRequest, session); mainCompile(compileRequest, session); - const coreExecutionResponse = dataform.CoreExecutionResponse.create({ + const coreExecutionResponse = sqlanvil.CoreExecutionResponse.create({ compile: { compiledGraph: session.compile() } }); if (typeof coreExecutionRequest === "string") { - // Older versions of the Dataform CLI expect a base64 encoded string to be returned. + // Older versions of the sqlanvil CLI expect a base64 encoded string to be returned. // See https://github.com/dataform-co/dataform/pull/1570. - return encode64(dataform.CoreExecutionResponse, coreExecutionResponse); + return encode64(sqlanvil.CoreExecutionResponse, coreExecutionResponse); } - return dataform.CoreExecutionResponse.encode(coreExecutionResponse).finish(); + return sqlanvil.CoreExecutionResponse.encode(coreExecutionResponse).finish(); } function loadActionConfigs(session: Session, filePaths: string[]) { @@ -87,13 +87,13 @@ function loadActionConfigs(session: Session, filePaths: string[]) { .forEach(actionConfigsPath => { const actionConfigs = loadActionConfigsFile(session, actionConfigsPath); actionConfigs.actions.forEach(nonProtoActionConfig => { - const actionConfig = dataform.ActionConfig.create(nonProtoActionConfig); + const actionConfig = sqlanvil.ActionConfig.create(nonProtoActionConfig); if (actionConfig.table) { session.actions.push( new Table( session, - dataform.ActionConfig.TableConfig.create(actionConfig.table), + sqlanvil.ActionConfig.TableConfig.create(actionConfig.table), actionConfigsPath ) ); @@ -101,7 +101,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new View( session, - dataform.ActionConfig.ViewConfig.create(actionConfig.view), + sqlanvil.ActionConfig.ViewConfig.create(actionConfig.view), actionConfigsPath ) ); @@ -109,7 +109,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new IncrementalTable( session, - dataform.ActionConfig.IncrementalTableConfig.create(actionConfig.incrementalTable), + sqlanvil.ActionConfig.IncrementalTableConfig.create(actionConfig.incrementalTable), actionConfigsPath ) ); @@ -117,7 +117,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new Assertion( session, - dataform.ActionConfig.AssertionConfig.create(actionConfig.assertion), + sqlanvil.ActionConfig.AssertionConfig.create(actionConfig.assertion), actionConfigsPath ) ); @@ -125,7 +125,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new Operation( session, - dataform.ActionConfig.OperationConfig.create(actionConfig.operation), + sqlanvil.ActionConfig.OperationConfig.create(actionConfig.operation), actionConfigsPath ) ); @@ -133,14 +133,14 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new Declaration( session, - dataform.ActionConfig.DeclarationConfig.create(actionConfig.declaration) + sqlanvil.ActionConfig.DeclarationConfig.create(actionConfig.declaration) ) ); } else if (actionConfig.notebook) { session.actions.push( new Notebook( session, - dataform.ActionConfig.NotebookConfig.create(actionConfig.notebook), + sqlanvil.ActionConfig.NotebookConfig.create(actionConfig.notebook), actionConfigsPath ) ); @@ -148,7 +148,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { session.actions.push( new DataPreparation( session, - dataform.ActionConfig.DataPreparationConfig.create(actionConfig.dataPreparation), + sqlanvil.ActionConfig.DataPreparationConfig.create(actionConfig.dataPreparation), actionConfigsPath ) ); @@ -162,7 +162,7 @@ function loadActionConfigs(session: Session, filePaths: string[]) { function loadActionConfigsFile( session: Session, actionConfigsPath: string -): dataform.ActionConfigs { +): sqlanvil.ActionConfigs { let actionConfigsAsJson = {}; try { // tslint:disable-next-line: tsr-detect-non-literal-require @@ -171,39 +171,39 @@ function loadActionConfigsFile( session.compileError(e, actionConfigsPath); } verifyObjectMatchesProto( - dataform.ActionConfigs, + sqlanvil.ActionConfigs, actionConfigsAsJson, VerifyProtoErrorBehaviour.SHOW_DOCS_LINK ); - return dataform.ActionConfigs.fromObject(actionConfigsAsJson); + return sqlanvil.ActionConfigs.fromObject(actionConfigsAsJson); } -function prologueCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { - if (compileRequest?.compileConfig?.extension?.compilationMode === dataform.ExtensionCompilationMode.PROLOGUE) { +function prologueCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { + if (compileRequest?.compileConfig?.extension?.compilationMode === sqlanvil.ExtensionCompilationMode.PROLOGUE) { extensionCompile(compileRequest, session); } } -function mainCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { - if (compileRequest?.compileConfig?.extension?.compilationMode === dataform.ExtensionCompilationMode.APPLICATION_CODE) { +function mainCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { + if (compileRequest?.compileConfig?.extension?.compilationMode === sqlanvil.ExtensionCompilationMode.APPLICATION_CODE) { extensionCompile(compileRequest, session); return; } - dataformCompile(compileRequest, session); + sqlanvilCompile(compileRequest, session); } -function extensionCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { +function extensionCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { try { const module = nativeRequire(compileRequest?.compileConfig?.extension.name); - const extension: () => IDataformExtension = module.extension; + const extension: () => ISqlanvilExtension = module.extension; extension().compile(compileRequest, session); } catch (e) { session.compileError(e, compileRequest?.compileConfig?.extension.name); } } -function dataformCompile(compileRequest: dataform.ICompileExecutionRequest, session: Session) { +function sqlanvilCompile(compileRequest: sqlanvil.ICompileExecutionRequest, session: Session) { const globalAny = global as any; // Require "includes/*.js" files, attaching them (by file basename) to the `global` object. @@ -224,7 +224,7 @@ function dataformCompile(compileRequest: dataform.ICompileExecutionRequest, sess }); Object.assign(globalAny, topLevelIncludes); - // Bind various @dataform/core APIs to the 'global' object. + // Bind various @sqlanvil/core APIs to the 'global' object. globalAny.publish = session.publish.bind(session); globalAny.operate = session.operate.bind(session); globalAny.assert = session.assert.bind(session); diff --git a/core/main_test.ts b/core/main_test.ts index 1b53e0e9..79c76b9e 100644 --- a/core/main_test.ts +++ b/core/main_test.ts @@ -4,29 +4,28 @@ import * as fs from "fs-extra"; import { dump as dumpYaml } from "js-yaml"; import * as path from "path"; -import { version } from "df/core/version"; -import { dataform, google } from "df/protos/ts"; -import { asPlainObject, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { version } from "sa/core/version"; +import { sqlanvil, google } from "sa/protos/ts"; +import { asPlainObject, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; import { coreExecutionRequestFromPath, runMainInVm, - VALID_DATAFORM_JSON, VALID_WORKFLOW_SETTINGS_YAML, WorkflowSettingsTemplates -} from "df/testing/run_core"; +} from "sa/testing/run_core"; const EMPTY_NOTEBOOK_CONTENTS = '{ "cells": [] }'; interface IVerifiableAction { type?: string | null, - target?: dataform.ITarget | null; - canonicalTarget?: dataform.ITarget | null; - dependencyTargets?: dataform.ITarget[] | null + target?: sqlanvil.ITarget | null; + canonicalTarget?: sqlanvil.ITarget | null; + dependencyTargets?: sqlanvil.ITarget[] | null } -function toVerifiableAction(graph: dataform.ICompiledGraph, actionType: string): IVerifiableAction { - let action: dataform.IAssertion | dataform.ITable +function toVerifiableAction(graph: sqlanvil.ICompiledGraph, actionType: string): IVerifiableAction { + let action: sqlanvil.IAssertion | sqlanvil.ITable switch (actionType) { case "assertion": action = graph.assertions[0]; @@ -48,7 +47,7 @@ function toVerifiableAction(graph: dataform.ICompiledGraph, actionType: string): // INFO: if you want to see an overview of the tests in this file, press cmd-k-3 while in // VSCode, to collapse everything below the third level of indentation. -suite("@dataform/core", ({ afterEach }) => { +suite("@sqlanvil/core", ({ afterEach }) => { const tmpDirFixture = new TmpDirFixture(afterEach); suite("session", () => { @@ -62,7 +61,7 @@ suite("@dataform/core", ({ afterEach }) => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(testConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(testConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync(path.join(projectDir, "definitions/e.sqlx"), `config {type: "view"}`); @@ -209,7 +208,7 @@ publish("b", {"schema": "foo"}).dependencies("a")` const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(testConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(testConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -235,7 +234,7 @@ publish("b", {"schema": "foo"}).dependencies("a")` const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -301,7 +300,7 @@ publish("name")` const result = runMainInVm( coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ defaultSchema: "otherDataset" }) ) @@ -396,7 +395,7 @@ publish("b", "SELECT 1;");` const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -446,7 +445,7 @@ SELECT 1`); const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); const fileContents = `select @@ -465,7 +464,7 @@ from \`location\``; const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); const sqlContents = `select @@ -490,7 +489,7 @@ from \`location\``; const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); const sqlContents = `select @@ -542,23 +541,6 @@ quotes ); }); - // dataform.json for workflow settings is deprecated, but still currently supported. - test(`a valid dataform.json is present`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync(path.join(projectDir, "dataform.json"), VALID_DATAFORM_JSON); - - const result = runMainInVm(coreExecutionRequestFromPath(projectDir)); - - expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); - expect(asPlainObject(result.compile.compiledGraph.projectConfig)).deep.equals( - asPlainObject({ - defaultDatabase: "defaultProject", - defaultLocation: "US", - defaultSchema: "defaultDataset" - }) - ); - }); - test(`fails when no workflow settings file is present`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); @@ -567,19 +549,6 @@ quotes ); }); - test(`fails when both workflow settings and dataform.json files are present`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync(path.join(projectDir, "dataform.json"), VALID_DATAFORM_JSON); - fs.writeFileSync( - path.join(projectDir, "workflow_settings.yaml"), - VALID_WORKFLOW_SETTINGS_YAML - ); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - "dataform.json has been deprecated and cannot be defined alongside workflow_settings.yaml" - ); - }); - test(`fails when workflow_settings.yaml cannot be represented in JSON format`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync(path.join(projectDir, "workflow_settings.yaml"), "&*19132sdS:asd:"); @@ -603,15 +572,6 @@ someKey: and an extra: colon ); }); - test(`fails when dataform.json is an invalid json file`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync(path.join(projectDir, "dataform.json"), '{keyWithNoQuotes: "validValue"}'); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - "Expected property name or '}' in JSON at position 1 (line 1 column 2)" - ); - }); - test(`fails when a valid workflow_settings.yaml contains unknown fields`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( @@ -620,7 +580,7 @@ someKey: and an extra: colon ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Workflow settings error: Unexpected property "notAProjectConfigField", or property value type of "string" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-WorkflowSettings for allowed properties.` + `Workflow settings error: Unexpected property "notAProjectConfigField", or property value type of "string" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-WorkflowSettings for allowed properties.` ); }); @@ -633,24 +593,12 @@ someKey: and an extra: colon ); }); - test(`fails when a valid dataform.json contains unknown fields`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync( - path.join(projectDir, "dataform.json"), - `{"notAProjectConfigField": "value"}` - ); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Dataform json error: Unexpected property "notAProjectConfigField", or property value type of "string" is incorrect.` - ); - }); - test(`does not fail when defaultLocation is not present in workflow_settings.yaml`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), ` -dataformCoreVersion: ${version} +sqlanvilCoreVersion: ${version} defaultProject: project` ); @@ -682,11 +630,11 @@ vars: ` config { type: "table", - database: dataform.projectConfig.vars.projectVar, + database: sqlanvil.projectConfig.vars.projectVar, } -select 1 AS \${dataform.projectConfig.vars.selectVar}` +select 1 AS \${sqlanvil.projectConfig.vars.selectVar}` ); - const coreExecutionRequest = dataform.CoreExecutionRequest.create({ + const coreExecutionRequest = sqlanvil.CoreExecutionRequest.create({ compile: { compileConfig: { projectDir, @@ -706,7 +654,7 @@ select 1 AS \${dataform.projectConfig.vars.selectVar}` expect(result.compile.compiledGraph.graphErrors.compilationErrors).deep.equals([]); expect(asPlainObject(result.compile.compiledGraph)).deep.equals( asPlainObject({ - dataformCoreVersion: version, + sqlanvilCoreVersion: version, graphErrors: {}, jitData: {}, projectConfig: { @@ -746,14 +694,14 @@ select 1 AS \${dataform.projectConfig.vars.selectVar}` ); }); - suite("dataform core version", () => { + suite("sqlanvil core version", () => { test(`main fails when the workflow settings version is not the installed current version`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), ` -dataformCoreVersion: 1.0.0 -defaultProject: dataform` +sqlanvilCoreVersion: 1.0.0 +defaultProject: sqlanvil` ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( @@ -766,7 +714,7 @@ defaultProject: dataform` fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), ` -dataformCoreVersion: ${version} +sqlanvilCoreVersion: ${version} defaultProject: project defaultLocation: US` ); @@ -800,18 +748,6 @@ vars: ); }); - test(`variables in dataform.json must be strings`, () => { - const projectDir = tmpDirFixture.createNewTmpDir(); - fs.writeFileSync( - path.join(projectDir, "dataform.json"), - `{"vars": { "intVar": 1, "strVar": "str" } }` - ); - - expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - "Custom variables defined in workflow settings can only be strings." - ); - }); - test(`variables can be referenced in SQLX`, () => { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( @@ -830,16 +766,16 @@ vars: ` config { type: "table", - database: dataform.projectConfig.vars.databaseVar, + database: sqlanvil.projectConfig.vars.databaseVar, schema: "tableSchema", - description: dataform.projectConfig.vars.descriptionVar, + description: sqlanvil.projectConfig.vars.descriptionVar, assertions: { - nonNull: [dataform.projectConfig.vars.columnVar], + nonNull: [sqlanvil.projectConfig.vars.columnVar], } } -select 1 AS \${dataform.projectConfig.vars.columnVar}` +select 1 AS \${sqlanvil.projectConfig.vars.columnVar}` ); - const coreExecutionRequest = dataform.CoreExecutionRequest.create({ + const coreExecutionRequest = sqlanvil.CoreExecutionRequest.create({ compile: { compileConfig: { projectDir, @@ -883,7 +819,7 @@ select 1 AS \${dataform.projectConfig.vars.columnVar}` } } ], - dataformCoreVersion: version, + sqlanvilCoreVersion: version, graphErrors: {}, jitData: {}, projectConfig: { @@ -972,7 +908,7 @@ actions: ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfigs for allowed properties.` + `Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfigs for allowed properties.` ); }); @@ -990,7 +926,7 @@ actions:` ); expect(() => runMainInVm(coreExecutionRequestFromPath(projectDir))).to.throw( - `Unexpected empty value for "actions". See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfigs for allowed properties.` + `Unexpected empty value for "actions". See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfigs for allowed properties.` ); }); @@ -1153,7 +1089,7 @@ actions: const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(projectConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(projectConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -1339,7 +1275,7 @@ publish("name", { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(projectConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(projectConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -1448,7 +1384,7 @@ operate("name", { const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(projectConfig)) + dumpYaml(sqlanvil.WorkflowSettings.create(projectConfig)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync( @@ -1558,7 +1494,7 @@ assert("name", { const coreRequest = coreExecutionRequestFromPath( projectDir, - dataform.ProjectConfig.create({ + sqlanvil.ProjectConfig.create({ disableAssertions: true }) ); @@ -1606,7 +1542,7 @@ assert("name", { fs.writeFileSync( path.join(projectDir, "definitions/jit.js"), ` -dataform.jitData("key", { +sqlanvil.jitData("key", { "number": 123, "string": "value", "boolean": true, @@ -1674,8 +1610,8 @@ dataform.jitData("key", { fs.writeFileSync( path.join(projectDir, "definitions/jit.js"), ` -dataform.jitData("key", 1); -dataform.jitData("key", 2); +sqlanvil.jitData("key", 1); +sqlanvil.jitData("key", 2); ` ); const result = runMainInVm(coreExecutionRequestFromPath(projectDir)); @@ -1695,7 +1631,7 @@ dataform.jitData("key", 2); fs.writeFileSync( path.join(projectDir, "definitions/jit.js"), ` -dataform.jitData("key", {test: () => {}}); +sqlanvil.jitData("key", {test: () => {}}); ` ); const result = runMainInVm(coreExecutionRequestFromPath(projectDir)); @@ -1792,7 +1728,7 @@ publish("name", { materialized: true, })`, expectedError: - 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-TableConfig for allowed properties.' + 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-TableConfig for allowed properties.' }, { testName: "partitionExpirationDays invalid for BigQuery tables", @@ -1869,7 +1805,7 @@ publish("name", { fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), dumpYaml( - dataform.WorkflowSettings.create( + sqlanvil.WorkflowSettings.create( WorkflowSettingsTemplates.bigqueryWithDefaultProjectAndDataset ) ) @@ -1902,7 +1838,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = tmpDirFixture.createNewTmpDir(); fs.writeFileSync( path.join(projectDir, "workflow_settings.yaml"), - dumpYaml(dataform.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) + dumpYaml(sqlanvil.WorkflowSettings.create(WorkflowSettingsTemplates.bigquery)) ); fs.mkdirSync(path.join(projectDir, "definitions")); fs.writeFileSync(path.join(projectDir, "definitions/e.sqlx"), `config {type: "view"}`); @@ -1926,7 +1862,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "some-extension", - compilationMode: dataform.ExtensionCompilationMode.COMPILATION_MODE_UNSPECIFIED, + compilationMode: sqlanvil.ExtensionCompilationMode.COMPILATION_MODE_UNSPECIFIED, }; const result = runMainInVm(request); @@ -1939,8 +1875,8 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = setUpProjectWithExtension(); const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + name: "@sqlanvil/sample-extension", + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -1953,8 +1889,8 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const projectDir = setUpProjectWithExtension(); const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.APPLICATION_CODE, + name: "@sqlanvil/sample-extension", + compilationMode: sqlanvil.ExtensionCompilationMode.APPLICATION_CODE, }; const result = runMainInVm(request); @@ -1971,8 +1907,8 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.APPLICATION_CODE + name: "@sqlanvil/sample-extension", + compilationMode: sqlanvil.ExtensionCompilationMode.APPLICATION_CODE }; const result = runMainInVm(request); @@ -1991,8 +1927,8 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + name: "@sqlanvil/sample-extension", + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -2006,7 +1942,7 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} const request = coreExecutionRequestFromPath(projectDir); request.compile.compileConfig.extension = { name: "does-not-exist", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -2018,10 +1954,10 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} test("catches exceptions thrown from extension", () => { const projectDir = setUpProjectWithExtension(); - const request = coreExecutionRequestFromPath(projectDir, dataform.ProjectConfig.create({vars: {"throw-error": "true"}})); + const request = coreExecutionRequestFromPath(projectDir, sqlanvil.ProjectConfig.create({vars: {"throw-error": "true"}})); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + name: "@sqlanvil/sample-extension", + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); @@ -2033,10 +1969,10 @@ publish("name", {type: "${fromType}", schema: "schemaOverride"}).type("${toType} test("persists extension compilation errors", () => { const projectDir = setUpProjectWithExtension(); - const request = coreExecutionRequestFromPath(projectDir, dataform.ProjectConfig.create({vars: {"store-compile-error": "true"}})); + const request = coreExecutionRequestFromPath(projectDir, sqlanvil.ProjectConfig.create({vars: {"store-compile-error": "true"}})); request.compile.compileConfig.extension = { - name: "@dataform/sample-extension", - compilationMode: dataform.ExtensionCompilationMode.PROLOGUE, + name: "@sqlanvil/sample-extension", + compilationMode: sqlanvil.ExtensionCompilationMode.PROLOGUE, }; const result = runMainInVm(request); diff --git a/core/session.ts b/core/session.ts index b29bc971..f00ea4d8 100644 --- a/core/session.ts +++ b/core/session.ts @@ -1,52 +1,52 @@ import { default as TarjanGraphConstructor, Graph as TarjanGraph } from "tarjan-graph"; -import { encode64, unknownToValue, verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { Action, ActionProto, ILegacyTableConfig, TableType } from "df/core/actions"; -import { AContextable, Assertion, AssertionContext } from "df/core/actions/assertion"; +import { encode64, unknownToValue, verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { Action, ActionProto, ILegacyTableConfig, TableType } from "sa/core/actions"; +import { AContextable, Assertion, AssertionContext } from "sa/core/actions/assertion"; import { DataPreparation, DataPreparationContext, -} from "df/core/actions/data_preparation"; -import { Declaration } from "df/core/actions/declaration"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation, OperationContext } from "df/core/actions/operation"; -import { Table, TableContext } from "df/core/actions/table"; -import { Test } from "df/core/actions/test"; -import { View } from "df/core/actions/view"; -import { CompilationSql } from "df/core/compilation_sql"; -import { Contextable, IActionContext, ITableContext, Resolvable } from "df/core/contextables"; -import { targetAsReadableString, targetStringifier } from "df/core/targets"; -import * as utils from "df/core/utils"; -import { ResolvableMap, toResolvable } from "df/core/utils"; -import { version as dataformCoreVersion } from "df/core/version"; -import { dataform, google } from "df/protos/ts"; +} from "sa/core/actions/data_preparation"; +import { Declaration } from "sa/core/actions/declaration"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation, OperationContext } from "sa/core/actions/operation"; +import { Table, TableContext } from "sa/core/actions/table"; +import { Test } from "sa/core/actions/test"; +import { View } from "sa/core/actions/view"; +import { CompilationSql } from "sa/core/compilation_sql"; +import { Contextable, IActionContext, ITableContext, Resolvable } from "sa/core/contextables"; +import { targetAsReadableString, targetStringifier } from "sa/core/targets"; +import * as utils from "sa/core/utils"; +import { ResolvableMap, toResolvable } from "sa/core/utils"; +import { version as sqlanvilCoreVersion } from "sa/core/version"; +import { sqlanvil, google } from "sa/protos/ts"; const DEFAULT_CONFIG = { - defaultSchema: "dataform", - assertionSchema: "dataform_assertions" + defaultSchema: "sqlanvil", + assertionSchema: "sqlanvil_assertions" }; /** * Contains methods that are published globally, so can be invoked anywhere in the `/definitions` - * folder of a Dataform project. + * folder of a sqlanvil project. */ export class Session { public rootDir: string; /** - * Stores the Dataform project configuration of the current Dataform project. Can be accessed via - * the `dataform` global variable. + * Stores the sqlanvil project configuration of the current sqlanvil project. Can be accessed via + * the `sqlanvil` global variable. * * Example: * * ```js - * dataform.projectConfig.vars.myVariableName === "myVariableValue" + * sqlanvil.projectConfig.vars.myVariableName === "myVariableValue" * ``` */ - public projectConfig: dataform.ProjectConfig; + public projectConfig: sqlanvil.ProjectConfig; // The canonical project config contains the project config before schema and database overrides. - public canonicalProjectConfig: dataform.ProjectConfig; + public canonicalProjectConfig: sqlanvil.ProjectConfig; public actions: Action[]; public indexedActions: ResolvableMap; @@ -58,28 +58,28 @@ export class Session { // upon a certain action in our actions list. We use this later to resolve dependencies. public actionAssertionMap = new ResolvableMap(); - public graphErrors: dataform.IGraphErrors; + public graphErrors: sqlanvil.IGraphErrors; // jit_context.data, avilable at jit stage. public jitContextData: google.protobuf.Struct | undefined; constructor( rootDir?: string, - projectConfig?: dataform.ProjectConfig, - originalProjectConfig?: dataform.ProjectConfig + projectConfig?: sqlanvil.ProjectConfig, + originalProjectConfig?: sqlanvil.ProjectConfig ) { this.init(rootDir, projectConfig, originalProjectConfig); } public init( rootDir: string, - projectConfig?: dataform.ProjectConfig, - originalProjectConfig?: dataform.ProjectConfig + projectConfig?: sqlanvil.ProjectConfig, + originalProjectConfig?: sqlanvil.ProjectConfig ) { this.rootDir = rootDir; - this.projectConfig = dataform.ProjectConfig.create(projectConfig || DEFAULT_CONFIG); + this.projectConfig = sqlanvil.ProjectConfig.create(projectConfig || DEFAULT_CONFIG); this.canonicalProjectConfig = getCanonicalProjectConfig( - dataform.ProjectConfig.create(originalProjectConfig || projectConfig || DEFAULT_CONFIG) + sqlanvil.ProjectConfig.create(originalProjectConfig || projectConfig || DEFAULT_CONFIG) ); this.actions = []; this.tests = []; @@ -88,7 +88,7 @@ export class Session { } public compilationSql(): CompilationSql { - return new CompilationSql(this.projectConfig, dataformCoreVersion); + return new CompilationSql(this.projectConfig, sqlanvilCoreVersion); } public sqlxAction(actionOptions: { @@ -264,7 +264,7 @@ export class Session { name: string, queryOrConfig?: | Contextable - | dataform.ActionConfig.OperationConfig + | sqlanvil.ActionConfig.OperationConfig ): Operation { const filename = utils.getCallerFile(this.rootDir); let operation: Operation; @@ -293,9 +293,9 @@ export class Session { name: string, queryOrConfig?: | Contextable - | dataform.ActionConfig.TableConfig - | dataform.ActionConfig.ViewConfig - | dataform.ActionConfig.IncrementalTableConfig + | sqlanvil.ActionConfig.TableConfig + | sqlanvil.ActionConfig.ViewConfig + | sqlanvil.ActionConfig.IncrementalTableConfig | ILegacyTableConfig // `any` is used here to facilitate the type merging of legacy table configs, which are very // different to the new structures. @@ -334,7 +334,7 @@ export class Session { } /** - * Adds a Dataform assertion the compiled graph. + * Adds a sqlanvil assertion the compiled graph. * * Available only in the `/definitions` directory. * @@ -342,9 +342,9 @@ export class Session { */ public assert( name: string, - queryOrConfig?: AContextable | dataform.ActionConfig.AssertionConfig + queryOrConfig?: AContextable | sqlanvil.ActionConfig.AssertionConfig // // `any` is used here to facilitate the type merging of legacy declaration configs options, - // // without breaking typescript consumers of Dataform. + // // without breaking typescript consumers of sqlanvil. // | any ): Assertion { const filename = utils.getCallerFile(this.rootDir); @@ -362,7 +362,7 @@ export class Session { } /** - * Declares the dataset as a Dataform data source. + * Declares the dataset as a sqlanvil data source. * * Available only in the `/definitions` directory. * @@ -370,9 +370,9 @@ export class Session { */ public declare( config: - | dataform.ActionConfig.DeclarationConfig + | sqlanvil.ActionConfig.DeclarationConfig // `any` is used here to facilitate the type merging of legacy declaration configs options, - // without breaking typescript consumers of Dataform. + // without breaking typescript consumers of sqlanvil. | any ): Declaration { const declaration = new Declaration(this, config, utils.getCallerFile(this.rootDir)); @@ -407,7 +407,7 @@ export class Session { * * @see [Notebook](Notebook) for examples on how to use. */ - public notebook(config: dataform.ActionConfig.NotebookConfig): Notebook { + public notebook(config: sqlanvil.ActionConfig.NotebookConfig): Notebook { const configFileName = utils.getCallerFile(this.rootDir); const notebook = new Notebook(this, config, configFileName); this.actions.push(notebook); @@ -423,10 +423,10 @@ export class Session { this.jitContextData.fields[key] = unknownToValue(data); } - public compileError(err: Error | string, path?: string, actionTarget?: dataform.ITarget) { + public compileError(err: Error | string, path?: string, actionTarget?: sqlanvil.ITarget) { const fileName = path || utils.getCallerFile(this.rootDir) || __filename; - const compileError = dataform.CompilationError.create({ + const compileError = sqlanvil.CompilationError.create({ fileName, actionName: !!actionTarget ? targetAsReadableString(actionTarget) : undefined, actionTarget @@ -440,7 +440,7 @@ export class Session { this.graphErrors.compilationErrors.push(compileError); } - public compile(): dataform.CompiledGraph { + public compile(): sqlanvil.CompiledGraph { this.actions.push(...this.tests); this.indexedActions = new ResolvableMap( this.actions.map(action => ({ actionTarget: action.getTarget(), value: action })) @@ -454,7 +454,7 @@ export class Session { throw new Error("Custom variables defined in workflow settings can only be strings."); } - const compiledGraph = dataform.CompiledGraph.create({ + const compiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: this.projectConfig, tables: this.compileGraphChunk( this.actions.filter( @@ -479,7 +479,7 @@ export class Session { this.actions.filter(action => action instanceof DataPreparation) ), graphErrors: this.graphErrors, - dataformCoreVersion, + sqlanvilCoreVersion, targets: this.actions.map(action => action.getTarget()), jitData: this.jitContextData, }); @@ -526,7 +526,7 @@ export class Session { ) ); verifyObjectMatchesProto( - dataform.CompiledGraph, + sqlanvil.CompiledGraph, compiledGraph, VerifyProtoErrorBehaviour.SUGGEST_REPORTING_TO_DATAFORM_TEAM ); @@ -534,7 +534,7 @@ export class Session { } public compileToBase64() { - return encode64(dataform.CompiledGraph, this.compile()); + return encode64(sqlanvil.CompiledGraph, this.compile()); } public finalizeDatabase(database: string): string { @@ -578,8 +578,8 @@ export class Session { private fullyQualifyDependencies(actions: ActionProto[]) { actions.forEach(action => { - const fullyQualifiedDependencies: { [name: string]: dataform.ITarget } = {}; - if (action instanceof dataform.Declaration || !action.dependencyTargets) { + const fullyQualifiedDependencies: { [name: string]: sqlanvil.ITarget } = {}; + if (action instanceof sqlanvil.Declaration || !action.dependencyTargets) { // Declarations cannot have dependencies. return; } @@ -624,14 +624,14 @@ export class Session { }); } - private alterActionName(actions: ActionProto[], declarationTargets: dataform.ITarget[]) { + private alterActionName(actions: ActionProto[], declarationTargets: sqlanvil.ITarget[]) { const { tablePrefix, schemaSuffix, databaseSuffix } = this.projectConfig; if (!tablePrefix && !schemaSuffix && !databaseSuffix) { return; } - const newTargetByOriginalTarget = new Map(); + const newTargetByOriginalTarget = new Map(); declarationTargets.forEach(declarationTarget => newTargetByOriginalTarget.set( targetStringifier.stringify(declarationTarget), @@ -652,7 +652,7 @@ export class Session { }); // Fix up dependencies in case those dependencies' names have changed. - const getUpdatedTarget = (originalTarget: dataform.ITarget) => { + const getUpdatedTarget = (originalTarget: sqlanvil.ITarget) => { // It's possible that we don't have a new Target for a dependency that failed to compile, // so fall back to the original Target. if (!newTargetByOriginalTarget.has(targetStringifier.stringify(originalTarget))) { @@ -661,18 +661,18 @@ export class Session { return newTargetByOriginalTarget.get(targetStringifier.stringify(originalTarget)); }; actions.forEach(action => { - if (!(action instanceof dataform.Declaration)) { + if (!(action instanceof sqlanvil.Declaration)) { // Declarations cannot have dependencies. action.dependencyTargets = (action.dependencyTargets || []).map(getUpdatedTarget); } - if (action instanceof dataform.Assertion && !!action.parentAction) { + if (action instanceof sqlanvil.Assertion && !!action.parentAction) { action.parentAction = getUpdatedTarget(action.parentAction); } }); } - private checkTestNameUniqueness(tests: dataform.ITest[]) { + private checkTestNameUniqueness(tests: sqlanvil.ITest[]) { const allNames: string[] = []; tests.forEach(testProto => { if (allNames.includes(testProto.name)) { @@ -694,7 +694,7 @@ export class Session { const tarjanGraph: TarjanGraph = new (TarjanGraphConstructor as any)(); actions.forEach(action => { // Declarations cannot have dependencies. - const cleanedDependencies = (action instanceof dataform.Declaration || + const cleanedDependencies = (action instanceof sqlanvil.Declaration || !action.dependencyTargets ? [] : action.dependencyTargets @@ -740,8 +740,8 @@ export class Session { }); } - private removeNonUniqueActionsFromCompiledGraph(compiledGraph: dataform.CompiledGraph) { - function getNonUniqueTargets(targets: dataform.ITarget[]): Set { + private removeNonUniqueActionsFromCompiledGraph(compiledGraph: sqlanvil.CompiledGraph) { + function getNonUniqueTargets(targets: sqlanvil.ITarget[]): Set { const allTargets = new Set(); const nonUniqueTargets = new Set(); @@ -816,8 +816,8 @@ function definesDataset(type: string) { return type === "view" || type === "table" || type === "incremental"; } -function getCanonicalProjectConfig(originalProjectConfig: dataform.ProjectConfig) { - return dataform.ProjectConfig.create({ +function getCanonicalProjectConfig(originalProjectConfig: sqlanvil.ProjectConfig) { + return sqlanvil.ProjectConfig.create({ warehouse: originalProjectConfig.warehouse, defaultSchema: originalProjectConfig.defaultSchema, defaultDatabase: originalProjectConfig.defaultDatabase, diff --git a/core/targets.ts b/core/targets.ts index 9d1eef04..384ce869 100644 --- a/core/targets.ts +++ b/core/targets.ts @@ -1,8 +1,8 @@ -import { JSONObjectStringifier } from "df/common/strings/stringifier"; -import { dataform } from "df/protos/ts"; +import { JSONObjectStringifier } from "sa/common/strings/stringifier"; +import { sqlanvil } from "sa/protos/ts"; /** Produces an unambigous mapping to and from a string representation. */ -export const targetStringifier = new JSONObjectStringifier(); +export const targetStringifier = new JSONObjectStringifier(); /** * Provides a readable string representation of the target which is used for e.g. specifying @@ -12,7 +12,7 @@ export const targetStringifier = new JSONObjectStringifier(); * This is an ambiguous transformation, multiple targets may map to the same string * and it should not be used for indexing. Use @see {@link targetStringifier} instead. */ -export function targetAsReadableString(target: dataform.ITarget): string { +export function targetAsReadableString(target: sqlanvil.ITarget): string { const nameParts = [target.name, target.schema]; if (!!target.database) { nameParts.push(target.database); diff --git a/core/utils.ts b/core/utils.ts index 9068ad66..526c2abc 100644 --- a/core/utils.ts +++ b/core/utils.ts @@ -1,15 +1,15 @@ -import { Action, ActionProto } from "df/core/actions"; -import { Assertion } from "df/core/actions/assertion"; -import { DataPreparation } from "df/core/actions/data_preparation"; -import { IncrementalTable } from "df/core/actions/incremental_table"; -import { Notebook } from "df/core/actions/notebook"; -import { Operation } from "df/core/actions/operation"; -import { Table } from "df/core/actions/table"; -import { View } from "df/core/actions/view"; -import { Contextable, Resolvable } from "df/core/contextables"; -import * as Path from "df/core/path"; -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import { Action, ActionProto } from "sa/core/actions"; +import { Assertion } from "sa/core/actions/assertion"; +import { DataPreparation } from "sa/core/actions/data_preparation"; +import { IncrementalTable } from "sa/core/actions/incremental_table"; +import { Notebook } from "sa/core/actions/notebook"; +import { Operation } from "sa/core/actions/operation"; +import { Table } from "sa/core/actions/table"; +import { View } from "sa/core/actions/view"; +import { Contextable, Resolvable } from "sa/core/contextables"; +import * as Path from "sa/core/path"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; declare var __webpack_require__: any; declare var __non_webpack_require__: any; @@ -75,8 +75,8 @@ export function getCallerFile(rootDir: string) { break; } if (!lastfile) { - if ((global as any).__dataform_current_file) { - lastfile = (global as any).__dataform_current_file; + if ((global as any).__sqlanvil_current_file) { + lastfile = (global as any).__sqlanvil_current_file; } else { // This is likely caused by Session.compileError() being called inside Session.compile(). // If so, explicitly pass the filename to Session.compileError(). @@ -101,7 +101,7 @@ function getCurrentStack(): NodeJS.CallSite[] { } } -export function graphHasErrors(graph: dataform.ICompiledGraph) { +export function graphHasErrors(graph: sqlanvil.ICompiledGraph) { return graph.graphErrors?.compilationErrors.length > 0; } @@ -133,28 +133,28 @@ function isResolvableArray(parts: any[]): parts is [string, string?, string?] { } export function resolvableAsTarget( - resolvable: Resolvable | dataform.ActionConfig.Target -): dataform.Target { + resolvable: Resolvable | sqlanvil.ActionConfig.Target +): sqlanvil.Target { if (typeof resolvable === "string") { - return dataform.Target.create({ + return sqlanvil.Target.create({ name: resolvable }); } - const actionConfigTarget = (resolvable as dataform.ActionConfig.ITarget); - if (actionConfigTarget instanceof dataform.ActionConfig.Target || actionConfigTarget.dataset !== undefined || actionConfigTarget.project !== undefined) { - return dataform.Target.create({ + const actionConfigTarget = (resolvable as sqlanvil.ActionConfig.ITarget); + if (actionConfigTarget instanceof sqlanvil.ActionConfig.Target || actionConfigTarget.dataset !== undefined || actionConfigTarget.project !== undefined) { + return sqlanvil.Target.create({ name: actionConfigTarget.name, schema: actionConfigTarget.dataset, database: actionConfigTarget.project, includeDependentAssertions: actionConfigTarget.includeDependentAssertions, }); } - return dataform.Target.create(resolvable); + return sqlanvil.Target.create(resolvable); } export function resolvableAsActionConfigTarget( resolvable: string | object -): dataform.ActionConfig.ITarget { +): sqlanvil.ActionConfig.ITarget { if (typeof resolvable === "string") { const parts = resolvable.split(".").reverse(); if (!isResolvableArray(parts)) { @@ -169,7 +169,7 @@ export function resolvableAsActionConfigTarget( }; } - return resolvable as dataform.ActionConfig.ITarget; + return resolvable as sqlanvil.ActionConfig.ITarget; } export function stringifyResolvable(res: Resolvable) { @@ -192,12 +192,12 @@ export function ambiguousActionNameMsg(act: Resolvable, allActs: Action[] | stri * @deprecated use ActionBuilder.applySessionToTarget() instead. */ export function target( - config: dataform.IProjectConfig, + config: sqlanvil.IProjectConfig, name: string, schema?: string, database?: string -): dataform.ITarget { - return dataform.Target.create({ +): sqlanvil.ITarget { + return sqlanvil.Target.create({ name, schema: schema || config.defaultSchema || undefined, database: database || config.defaultDatabase || undefined @@ -331,15 +331,15 @@ export function validateStorageUriFormat( */ export function getFileFormatValueForIcebergTable( configFileFormat?: string, -): dataform.FileFormat { +): sqlanvil.FileFormat { if (!configFileFormat) { // Default to PARQUET if fileFormat is undefined. - return dataform.FileFormat.PARQUET; + return sqlanvil.FileFormat.PARQUET; } switch (configFileFormat.toUpperCase()) { case "PARQUET": - return dataform.FileFormat.PARQUET; + return sqlanvil.FileFormat.PARQUET; default: throw new Error( @@ -411,7 +411,7 @@ export function getEffectiveBucketName( * Iceberg table. If the tableFolderRoot is provided in the config block, that * value will be used. Otherwise, defaultTableFolderRoot defined in * workflow_settings.yaml will be used. If none of those two values are - * defined, "_dataform" will be used. + * defined, "_sqlanvil" will be used. * @param defaultTableFolderRoot defined in workflow_settings.yaml * @param configTableFolderRoot defined in the config block * @returns tableFolderRoot used to construct storageUri for Iceberg tables @@ -425,7 +425,7 @@ export function getEffectiveTableFolderRoot( } else if (defaultTableFolderRoot) { return defaultTableFolderRoot; } else { - return "_dataform"; + return "_sqlanvil"; } } @@ -458,30 +458,30 @@ export function getEffectiveTableFolderSubpath( export function tableTypeStringToEnum(type: string, throwIfUnknown: boolean) { switch (type) { case "table": - return dataform.TableType.TABLE; + return sqlanvil.TableType.TABLE; case "incremental": - return dataform.TableType.INCREMENTAL; + return sqlanvil.TableType.INCREMENTAL; case "view": - return dataform.TableType.VIEW; + return sqlanvil.TableType.VIEW; default: { if (throwIfUnknown) { throw new Error(`Unexpected table type: ${type}`); } - return dataform.TableType.UNKNOWN_TYPE; + return sqlanvil.TableType.UNKNOWN_TYPE; } } } -export function tableTypeEnumToString(enumType: dataform.TableType) { - return dataform.TableType[enumType].toLowerCase(); +export function tableTypeEnumToString(enumType: sqlanvil.TableType) { + return sqlanvil.TableType[enumType].toLowerCase(); } -export function setOrValidateTableEnumType(table: dataform.ITable) { - let enumTypeFromStr: dataform.TableType | null = null; +export function setOrValidateTableEnumType(table: sqlanvil.ITable) { + let enumTypeFromStr: sqlanvil.TableType | null = null; if (table.type !== "" && table.type !== undefined) { enumTypeFromStr = tableTypeStringToEnum(table.type, true); } - if (table.enumType === dataform.TableType.UNKNOWN_TYPE || table.enumType === undefined) { + if (table.enumType === sqlanvil.TableType.UNKNOWN_TYPE || table.enumType === undefined) { table.enumType = enumTypeFromStr!; } else if (enumTypeFromStr !== null && table.enumType !== enumTypeFromStr) { throw new Error( @@ -501,8 +501,8 @@ export function extractActionDetailsFromFileName( } // Converts the config proto's target proto to the compiled graph proto's representation. -export function configTargetToCompiledGraphTarget(configTarget: dataform.ActionConfig.Target) { - const compiledGraphTarget: dataform.ITarget = { name: configTarget.name }; +export function configTargetToCompiledGraphTarget(configTarget: sqlanvil.ActionConfig.Target) { + const compiledGraphTarget: sqlanvil.ITarget = { name: configTarget.name }; if (configTarget.project) { compiledGraphTarget.database = configTarget.project; } @@ -512,25 +512,25 @@ export function configTargetToCompiledGraphTarget(configTarget: dataform.ActionC if (configTarget.hasOwnProperty("includeDependentAssertions")) { compiledGraphTarget.includeDependentAssertions = configTarget.includeDependentAssertions; } - return dataform.Target.create(compiledGraphTarget); + return sqlanvil.Target.create(compiledGraphTarget); } // Converts a config proto's action config proto to the compiled graph proto's representation. // Action config protos roughly contain target protos fields. export function actionConfigToCompiledGraphTarget( actionConfig: - | dataform.ActionConfig.TableConfig - | dataform.ActionConfig.ViewConfig - | dataform.ActionConfig.IncrementalTableConfig - | dataform.ActionConfig.OperationConfig - | dataform.ActionConfig.AssertionConfig - | dataform.ActionConfig.DeclarationConfig - | dataform.ActionConfig.NotebookConfig - | dataform.ActionConfig.DataPreparationConfig - | dataform.ActionConfig.DataPreparationConfig.ErrorTableConfig - | dataform.ActionConfig.Target -): dataform.Target { - const compiledGraphTarget = dataform.Target.create({ name: actionConfig.name }); + | sqlanvil.ActionConfig.TableConfig + | sqlanvil.ActionConfig.ViewConfig + | sqlanvil.ActionConfig.IncrementalTableConfig + | sqlanvil.ActionConfig.OperationConfig + | sqlanvil.ActionConfig.AssertionConfig + | sqlanvil.ActionConfig.DeclarationConfig + | sqlanvil.ActionConfig.NotebookConfig + | sqlanvil.ActionConfig.DataPreparationConfig + | sqlanvil.ActionConfig.DataPreparationConfig.ErrorTableConfig + | sqlanvil.ActionConfig.Target +): sqlanvil.Target { + const compiledGraphTarget = sqlanvil.Target.create({ name: actionConfig.name }); if ("project" in actionConfig && actionConfig.project !== undefined) { compiledGraphTarget.database = actionConfig.project; } @@ -552,7 +552,7 @@ export function resolveActionsConfigFilename(configFilename: string, configPath: export function checkAssertionsForDependency( action: actionsWithDependencies, resolvable: Resolvable -): dataform.Target { +): sqlanvil.Target { const dependencyTarget = resolvableAsTarget(resolvable); if ( !dependencyTarget.hasOwnProperty("includeDependentAssertions") && @@ -594,7 +594,7 @@ export class ResolvableMap { private byDatabaseAndName: Map> = new Map(); private byDatabaseSchemaAndName: Map>> = new Map(); - public constructor(values?: Array<{ actionTarget: dataform.ITarget, value: T }>) { + public constructor(values?: Array<{ actionTarget: sqlanvil.ITarget, value: T }>) { if (values) { for (const { actionTarget, value } of values) { this.set(actionTarget, value); @@ -602,7 +602,7 @@ export class ResolvableMap { } } - public set(actionTarget: dataform.ITarget, value: T) { + public set(actionTarget: sqlanvil.ITarget, value: T) { this.setByNameLevel(this.byName, actionTarget.name, value); if (!!actionTarget.schema) { @@ -626,7 +626,7 @@ export class ResolvableMap { } } - public find(actionTarget: dataform.ITarget): T[] { + public find(actionTarget: sqlanvil.ITarget): T[] { if (!!actionTarget.database) { if (!!actionTarget.schema) { return ( @@ -653,7 +653,7 @@ export class ResolvableMap { private setBySchemaLevel( targetMap: Map>, - actionTarget: dataform.ITarget, + actionTarget: sqlanvil.ITarget, value: T ) { if (!targetMap.has(actionTarget.schema)) { diff --git a/core/utils_test.ts b/core/utils_test.ts index 88c281f9..5473a546 100644 --- a/core/utils_test.ts +++ b/core/utils_test.ts @@ -11,8 +11,8 @@ import { validateStorageUriFormat, } from './utils'; -import {dataform} from "df/protos/ts"; -import {suite, test} from 'df/testing'; +import {sqlanvil} from "sa/protos/ts"; +import {suite, test} from 'sa/testing'; /** * Executes a function and returns the Error if one is thrown, otherwise returns undefined. @@ -55,7 +55,7 @@ function assertThrowsWithMessage(testFn: () => void, expectedMessage: string) { } } -suite('Dataform Utility Validations', () => { +suite('sqlanvil Utility Validations', () => { const CONNECTION_ERROR_MSG = 'The connection must be in the format `{project}.{location}.{connection_id}` or `projects/{project}/locations/{location}/connections/{connection_id}`, or be set to `DEFAULT`.'; const STORAGE_ERROR_MSG = 'The storage URI must be in the format `gs://{bucket_name}/{path_to_data}`.'; @@ -66,12 +66,12 @@ suite('Dataform Utility Validations', () => { test('does not throw for a valid dot-separated connection format', () => { assertNoThrow(() => validateConnectionFormat('my-project.us-central1.my-connection')); - assertNoThrow(() => validateConnectionFormat('gcp-proj-123.europe-west4.dataform-conn_id')); + assertNoThrow(() => validateConnectionFormat('gcp-proj-123.europe-west4.sqlanvil-conn_id')); }); test('does not throw for a valid resource-formatted connection format', () => { assertNoThrow(() => validateConnectionFormat('projects/my-project/locations/us-central1/connections/my-connection')); - assertNoThrow(() => validateConnectionFormat('projects/gcp-proj-123/locations/europe-west4/connections/dataform-conn_id')); + assertNoThrow(() => validateConnectionFormat('projects/gcp-proj-123/locations/europe-west4/connections/sqlanvil-conn_id')); }); test('throws for an empty connection string', () => { @@ -170,12 +170,12 @@ suite('Dataform Utility Validations', () => { expect(getEffectiveTableFolderRoot('ws-root', '')).to.equal('ws-root'); }); - test('returns "_dataform" when both config and default are undefined', () => { - expect(getEffectiveTableFolderRoot(undefined, undefined)).to.equal('_dataform'); + test('returns "_sqlanvil" when both config and default are undefined', () => { + expect(getEffectiveTableFolderRoot(undefined, undefined)).to.equal('_sqlanvil'); }); - test('returns "_dataform" when both config and default are empty strings', () => { - expect(getEffectiveTableFolderRoot('', '')).to.equal('_dataform'); + test('returns "_sqlanvil" when both config and default are empty strings', () => { + expect(getEffectiveTableFolderRoot('', '')).to.equal('_sqlanvil'); }); }); @@ -228,20 +228,20 @@ suite('Dataform Utility Validations', () => { suite('getFileFormatValueForIcebergTable', () => { test('returns PARQUET when configFileFormat is undefined', () => { - expect(getFileFormatValueForIcebergTable(undefined)).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable(undefined)).to.equal(sqlanvil.FileFormat.PARQUET); }); test('returns PARQUET when configFileFormat is "PARQUET"', () => { - expect(getFileFormatValueForIcebergTable('PARQUET')).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('PARQUET')).to.equal(sqlanvil.FileFormat.PARQUET); }); test('returns PARQUET when fileFormat is an empty string', () => { - expect(getFileFormatValueForIcebergTable('')).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('')).to.equal(sqlanvil.FileFormat.PARQUET); }); test('is case insensitive ', () => { - expect(getFileFormatValueForIcebergTable('parquet')).to.equal(dataform.FileFormat.PARQUET); - expect(getFileFormatValueForIcebergTable('pArQuEt')).to.equal(dataform.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('parquet')).to.equal(sqlanvil.FileFormat.PARQUET); + expect(getFileFormatValueForIcebergTable('pArQuEt')).to.equal(sqlanvil.FileFormat.PARQUET); }); test('throws an error for an unsupported file format string', () => { @@ -254,15 +254,15 @@ suite('Dataform Utility Validations', () => { suite('getStorageUriForIcebergTable', () => { const testBucket = 'my-iceberg-bucket'; - const testRoot = '_dataform'; + const testRoot = '_sqlanvil'; const testSubpath = 'data/v1'; test('constructs the URI with a provided tableFolderRoot', () => { - expect(getStorageUriForIcebergTable(testBucket, testRoot, testSubpath)).to.equal('gs://my-iceberg-bucket/_dataform/data/v1'); + expect(getStorageUriForIcebergTable(testBucket, testRoot, testSubpath)).to.equal('gs://my-iceberg-bucket/_sqlanvil/data/v1'); }); test('handles empty bucket name, resulting in an invalid but formed URI', () => { - expect(getStorageUriForIcebergTable('', testRoot, testSubpath)).to.equal('gs:///_dataform/data/v1'); + expect(getStorageUriForIcebergTable('', testRoot, testSubpath)).to.equal('gs:///_sqlanvil/data/v1'); }); test('handles empty tableFolderRoot, resulting in an invalid but formed URI', () => { @@ -270,7 +270,7 @@ suite('Dataform Utility Validations', () => { }); test('handles empty tableFolderSubpath', () => { - expect(getStorageUriForIcebergTable(testBucket, testRoot, '')).to.equal('gs://my-iceberg-bucket/_dataform/'); + expect(getStorageUriForIcebergTable(testBucket, testRoot, '')).to.equal('gs://my-iceberg-bucket/_sqlanvil/'); }); }); diff --git a/core/workflow_settings.ts b/core/workflow_settings.ts index 37fb54db..301e4487 100644 --- a/core/workflow_settings.ts +++ b/core/workflow_settings.ts @@ -1,25 +1,17 @@ import { YAMLException } from "js-yaml"; -import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "df/common/protos"; -import { INVALID_YAML_ERROR_STRING } from "df/core/compilers"; -import { version } from "df/core/version"; -import { dataform } from "df/protos/ts"; +import { verifyObjectMatchesProto, VerifyProtoErrorBehaviour } from "sa/common/protos"; +import { INVALID_YAML_ERROR_STRING } from "sa/core/compilers"; +import { version } from "sa/core/version"; +import { sqlanvil } from "sa/protos/ts"; declare var __webpack_require__: any; declare var __non_webpack_require__: any; const nativeRequire = typeof __webpack_require__ === "function" ? __non_webpack_require__ : require; -export function readWorkflowSettings(failIfMissing: boolean = true): dataform.ProjectConfig { +export function readWorkflowSettings(failIfMissing: boolean = true): sqlanvil.ProjectConfig { const globalAny = global as any; const workflowSettingsYaml = globalAny.workflowSettingsYaml || maybeRequire("workflow_settings.yaml"); - // `dataform.json` is deprecated; new versions of Dataform Core prefer `workflow_settings.yaml`. - const dataformJson = globalAny.dataformJson || maybeRequire("dataform.json"); - - if (workflowSettingsYaml && dataformJson) { - throw Error( - "dataform.json has been deprecated and cannot be defined alongside workflow_settings.yaml" - ); - } if (workflowSettingsYaml) { const workflowSettingsAsJson = workflowSettingsYaml.asJson; @@ -29,32 +21,18 @@ export function readWorkflowSettings(failIfMissing: boolean = true): dataform.Pr return workflowSettingsAsProjectConfig(verifyWorkflowSettingsAsJson(workflowSettingsAsJson)); } - if (dataformJson) { - // Dataform JSON used the compiled graph's config proto, rather than workflow settings. - try { - return dataform.ProjectConfig.create( - verifyObjectMatchesProto(dataform.ProjectConfig, dataformJson) - ); - } catch (e) { - if (e instanceof ReferenceError) { - throw ReferenceError(`Dataform json error: ${e.message}`); - } - throw e; - } - } - if (failIfMissing) { throw Error("Failed to resolve workflow_settings.yaml"); } - return dataform.ProjectConfig.create(); + return sqlanvil.ProjectConfig.create(); } -function verifyWorkflowSettingsAsJson(workflowSettingsAsJson: object): dataform.WorkflowSettings { - let workflowSettings = dataform.WorkflowSettings.create(); +function verifyWorkflowSettingsAsJson(workflowSettingsAsJson: object): sqlanvil.WorkflowSettings { + let workflowSettings = sqlanvil.WorkflowSettings.create(); try { - workflowSettings = dataform.WorkflowSettings.create( + workflowSettings = sqlanvil.WorkflowSettings.create( verifyObjectMatchesProto( - dataform.WorkflowSettings, + sqlanvil.WorkflowSettings, workflowSettingsAsJson as { [key: string]: any; }, @@ -68,10 +46,10 @@ function verifyWorkflowSettingsAsJson(workflowSettingsAsJson: object): dataform. throw e; } - // The caller of Dataform Core should ensure that the correct version is installed. - if (!!workflowSettings.dataformCoreVersion && workflowSettings.dataformCoreVersion !== version) { + // The caller of sqlanvil Core should ensure that the correct version is installed. + if (!!workflowSettings.sqlanvilCoreVersion && workflowSettings.sqlanvilCoreVersion !== version) { throw Error( - `Version mismatch: workflow settings specifies version ${workflowSettings.dataformCoreVersion}` + + `Version mismatch: workflow settings specifies version ${workflowSettings.sqlanvilCoreVersion}` + `, but ${version} was found` ); } @@ -96,9 +74,9 @@ function maybeRequire(file: string): any { } export function workflowSettingsAsProjectConfig( - workflowSettings: dataform.WorkflowSettings -): dataform.ProjectConfig { - const projectConfig = dataform.ProjectConfig.create(); + workflowSettings: sqlanvil.WorkflowSettings +): sqlanvil.ProjectConfig { + const projectConfig = sqlanvil.ProjectConfig.create(); if (workflowSettings.defaultProject) { projectConfig.defaultDatabase = workflowSettings.defaultProject; } diff --git a/docs/configs-reference.md b/docs/configs-reference.md index 12a897ce..d4318f24 100644 --- a/docs/configs-reference.md +++ b/docs/configs-reference.md @@ -1,3 +1,8 @@ # Configs Reference -Relocated to [docs/reference/configs](https://dataform-co.github.io/dataform/docs/reference/configs). +TODO: write the sqlanvil configs reference once the new proto schemas +(`WarehouseConfig`, `PostgresOptions`, `SupabaseOptions`) are merged. + +For the upstream Dataform configs reference (which still describes the +BigQuery-only behavior sqlanvil inherits today), see: +https://dataform-co.github.io/dataform/docs/reference/configs diff --git a/docs/docs_site_plan.md b/docs/docs_site_plan.md new file mode 100644 index 00000000..d918c6df --- /dev/null +++ b/docs/docs_site_plan.md @@ -0,0 +1,246 @@ +# sqlanvil Docs Site — Sourcing & Build Plan + +**Status:** Draft +**Owner:** Ivan +**Last updated:** 2026-05-27 +**Related:** +- [`postgres_first_class_design.md`](postgres_first_class_design.md) — what the docs need to describe +- [`hybrid_warehouses_supabase_bigquery.md`](hybrid_warehouses_supabase_bigquery.md) — already-drafted reference doc +- Root [`NOTICE`](../NOTICE) — Apache 2.0 attribution already wired up for the code; will extend to docs + +## 0. TL;DR + +Pull the upstream OSS Dataform repo's `docs/` (Apache 2.0, already +inherited via fork) as the foundation. Fill the gaps from +`docs.cloud.google.com/dataform` (CC BY 4.0). Rewrite all BigQuery +examples as Postgres-native per the postgres-first-class design. Ship a +docs site at `docs.sqlanvil.com` (Vercel static) — markdown-driven. + +## 1. Source Hierarchy + +### Tier 1 — Apache 2.0 markdown source (preferred) + +Already in upstream `dataform-co/dataform` repo at `docs/`: + +``` +docs/configs-reference.md +docs/packages.md +docs/reference/assertion.md +docs/reference/configs.md +docs/reference/declaration.md +docs/reference/incrementaltable.md +docs/reference/notebook.md +docs/reference/operation.md +docs/reference/session.md +docs/reference/table.md +docs/reference/test.md +docs/reference/view.md +``` + +- License: **Apache 2.0** (matches code license + existing NOTICE) +- Already inherited via fork +- Markdown source — easier to `git merge upstream/main -- docs/` +- Code samples already in code blocks (not embedded HTML) + +### Tier 2 — Google Cloud web docs (CC BY 4.0) + +`docs.cloud.google.com/dataform/docs/*` — for content that doesn't ship +in the OSS repo: + +- Quickstart / getting-started walkthrough +- CLI reference (full flag tables) +- Troubleshooting guide +- Locations / quotas / billing pages (sqlanvil-irrelevant — skip) +- Release notes (sqlanvil writes its own) +- Tutorial walkthroughs + +Footer of every page says: +> "Except as otherwise noted, the content of this page is licensed +> under the [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/), +> and code samples are licensed under the Apache 2.0 License" + +### Tier 3 — Do not pull + +- `cloud.google.com/dataform/*` (marketing pages — different terms, not + CC BY 4.0) +- GCP console screenshots (Google trademark + UI copyright) +- Anything behind login (TOS may restrict redistribution) +- Google's internal repos (not public anyway) + +## 2. License Compliance + +### Apache 2.0 (Tier 1 source) + +Already handled by root `NOTICE`. When pulling docs: + +1. Preserve copyright notices from any code samples. +2. Add a `NOTICE` entry for the docs in the root NOTICE: + ``` + This product includes documentation adapted from Google's Dataform + open source project (https://github.com/dataform-co/dataform), + licensed under the Apache License 2.0. See: + https://github.com/dataform-co/dataform/blob/main/LICENSE + ``` + +### CC BY 4.0 (Tier 2 source) + +Per page or per-section attribution. Recommended placement: + +- At the bottom of each pulled page: + ``` + --- + Adapted from Google Cloud Dataform documentation + (), licensed under CC BY 4.0. Modifications by sqlanvil + contributors to target PostgreSQL / Supabase semantics. + ``` +- In `docs/ATTRIBUTIONS.md`: line-item table of every pulled page, + source URL, snapshot date. + +### Trademark scrubbing (separate from copyright) + +Even with permissive license: +- `Dataform` → `sqlanvil` (entire word mark) +- Remove Google Cloud logos +- Replace GCP console screenshots +- No "Powered by Google" / no implied endorsement (CC BY 4.0 §3(a)(1)(iii)) + +This is the same playbook as the code rename (PR #1). + +## 3. Doc Pages by Phase + +### Phase D1 — Pull Tier 1 source + +Tickets: +- D1.1 — Copy `upstream/main:docs/` into `sqlanvil/docs/reference/` + (some already there — diff first, keep best) +- D1.2 — Rename surface sweep on docs: `dataform` → `sqlanvil`, + `@dataform/` → `@sqlanvil/`, `.df-credentials.json` → `.sa-credentials.json` +- D1.3 — Add `NOTICE` paragraph for inherited docs +- D1.4 — Verify nothing references BQ-only types/concepts unguarded + (cluster_by, NOT ENFORCED PKs, OPTIONS, MERGE) + +**Output:** every reference page exists, sqlanvil-branded, BQ-only +language flagged for D2 rewrite. + +### Phase D2 — Postgres-first reference rewrites + +For each action type, rewrite to be Postgres-first: + +| Page | BigQuery section to gate | Postgres section to add | +|---|---|---| +| `reference/table.md` | partition_by/cluster_by/OPTIONS | `PartitionConfig` (RANGE/LIST/HASH), indexes, tablespace, fillfactor | +| `reference/view.md` | (mostly portable) | Just rebrand | +| `reference/incrementaltable.md` | MERGE-based upsert | `INSERT ... ON CONFLICT (...) DO UPDATE` | +| `reference/operation.md` | (mostly portable) | Rebrand + transaction semantics note | +| `reference/assertion.md` | (mostly portable) | Add note: assertions can compile to CHECK constraints when user opts in | +| `reference/declaration.md` | (mostly portable) | Rebrand | +| `reference/test.md` | (mostly portable) | Rebrand | +| `reference/notebook.md` | GCP-specific — **drop** | Skip — no sqlanvil equivalent | +| `reference/session.md` | (mostly portable) | Rebrand | +| `configs-reference.md` | regenerate from new proto | New: `PostgresOptions`, `SupabaseOptions`, `WarehouseConfig` | + +**Output:** every page has Postgres-first content. BQ content gated +under explicit "BigQuery only" callouts. + +### Phase D3 — Supabase-specific pages (original content) + +No upstream source. Write from scratch: + +- `reference/rls_policy.md` — RLS policies as action types +- `reference/realtime_publication.md` — Realtime publications +- `reference/wrapper.md` — Supabase Wrappers (FDW) +- `reference/vector_index.md` — pgvector convenience action + +Plus concept pages: +- `concepts/supabase_target.md` — why Supabase is first-class +- `concepts/postgres_vs_bigquery.md` — BQ-isms that don't apply in PG + (companion to `postgres_first_class_design.md` §4 table) + +### Phase D4 — Top-of-funnel docs (CC BY 4.0 adaptation) + +Pull structure from `docs.cloud.google.com/dataform`, rewrite content: + +- `quickstart.md` — adapted from Google's quickstart; Postgres example + project instead of BQ +- `cli/reference.md` — CLI flag reference (table format from Google's + CLI ref page; sqlanvil flag values) +- `troubleshooting.md` — common errors (sqlanvil-specific; adapt + problem structure from Google's troubleshooting page) +- `installation.md` — npm install + docker dev container (original) + +### Phase D5 — Architecture & concepts (original) + +Already drafted: +- `hybrid_warehouses_supabase_bigquery.md` → publish as + `concepts/hybrid_warehouses.md` + +To draft: +- `concepts/why_sqlanvil.md` — positioning vs upstream Dataform OSS, + vs dbt, vs raw migrations +- `concepts/action_graph.md` — how actions form a DAG +- `concepts/incremental_strategies.md` — `ON CONFLICT` vs full reload + vs CDC-style append +- `architecture.md` — adapter / SQL generator / CLI layout + +## 4. Docs Site Build + +### Stack + +- **Source:** Markdown in `sqlanvil/docs/` +- **Site generator:** Pick between: + - **Astro Starlight** — TypeScript-friendly, modern, MDX support, + great search. Recommended. + - **Docusaurus** — React-based, more featureful, heavier. + - **VitePress** — Vue-based, lighter than Docusaurus. + - **mdBook** — Rust-based, very fast, simpler. + - **Plain Vercel + remark** — minimal, full control. +- **Host:** Vercel project `docs-sqlanvil-com`, domain `docs.sqlanvil.com` +- **Repo:** Sibling repo `../sqlanvil-docs/` (separate from monolith + `sqlanvil/` so docs deploys don't trigger code CI) + +### CI + +- GitHub Action on PR: build site, deploy preview URL +- Main branch → production +- Markdown lint + link check on every PR + +### Search + +- Algolia DocSearch (free for open source) +- Alternative: client-side Pagefind (Astro Starlight default) + +## 5. Sequencing vs Code PRs + +Doc work can happen in parallel with code Phases 3b-5 but should not +block them. Ordering: + +| Code phase | Doc work that depends on it | +|---|---| +| Phase 3a (adapter) | D2 — references that mention adapter methods | +| Phase 3b (SQL gen) | D2 — every action-type page's "compiled output" examples | +| Phase 4 (CLI wiring) | D4 — CLI reference, quickstart | +| Phase 5 (Supabase) | D3 — Supabase action type pages | + +Phase D1 + D5 can happen anytime — no code dependency. + +## 6. Open Questions + +- **Doc versioning.** If sqlanvil ships v0.x and breaking changes are + likely, do we version the docs? Defer until v1.0; for now, + `latest` only. +- **Hosting cost.** Vercel free tier easily handles a docs site. Algolia + DocSearch is free for OSS. Total cost: $0/mo at sqlanvil's scale. +- **Search index update cadence.** Tied to Vercel deploys — no separate + trigger. + +## 7. Action Items (immediate) + +- [ ] Decide site generator (recommend Astro Starlight) +- [ ] Create sibling repo `../sqlanvil-docs/` +- [ ] Wire Vercel project + domain `docs.sqlanvil.com` +- [ ] Phase D1.1: copy `upstream/main:docs/` into the new repo +- [ ] Add `NOTICE` paragraph + `ATTRIBUTIONS.md` +- [ ] First-pass rename sweep on inherited content + +Phase D1 is mostly mechanical — should fit in a single afternoon. +Subsequent phases scale with code maturity. diff --git a/docs/packages.md b/docs/packages.md index db30b8a6..f88ecce7 100644 --- a/docs/packages.md +++ b/docs/packages.md @@ -1,6 +1,6 @@ # Packages -[<- Back to Home](https://dataform-co.github.io/dataform). +[<- Back to Home](https://github.com/ihistand/sqlanvil). ## Sample Packages @@ -13,11 +13,11 @@ ## Creating a package -Creating your own package is relatively easy, as long as you're relatively familiar with the Dataform framework. It may also help to understand the fundamentals of JavaScript, but you can probably muddle through without this! +Creating your own package is relatively easy, as long as you're relatively familiar with the sqlanvil framework. It may also help to understand the fundamentals of JavaScript, but you can probably muddle through without this! ### Clone the base package repo -[This repo](https://github.com/dataform-co/dataform-package-base) contains the building blocks of a package: +[This repo](https://github.com/ihistand/sqlanvil-package-base) contains the building blocks of a package: - index.js - example.js diff --git a/docs/postgres_first_class_design.md b/docs/postgres_first_class_design.md new file mode 100644 index 00000000..1e9055d5 --- /dev/null +++ b/docs/postgres_first_class_design.md @@ -0,0 +1,342 @@ +# Postgres-First-Class Adapter Design + +**Status:** Draft +**Replaces (in scope):** `docs/postgres_reintegration_assessment.md` Phase 3 framing +**Complements:** `docs/hybrid_warehouses_supabase_bigquery.md` (architectural patterns) — this doc is the implementation spec + +## 0. TL;DR + +The restored Postgres adapter must not be a BigQuery adapter with translated SQL. sqlanvil ships two warehouse variants — `postgres` (standard) and `supabase` (Postgres + Supabase platform features) — both generating idiomatic SQL with native action config blocks. A user who has never touched BigQuery must never see BigQuery quirks (`NOT ENFORCED` PKs, `OPTIONS(...)` table options, `PARTITION BY DATE_TRUNC(...)` clauses, `MERGE` dialect). + +## 1. Why Not a BigQuery-Shaped Swap + +The Antigravity assessment frames reintegration as: implement `IDbAdapter` for Postgres, branch on `projectConfig.warehouse`, done. That is necessary but not sufficient. It leaves three structural problems: + +1. **SQL generation is BQ-shaped.** `core/compilation_sql/` was written around BigQuery's dialect (`MERGE`, `CREATE OR REPLACE TABLE ... OPTIONS(...)`, partitioning DSL). A `case warehouse` branch inside each generator produces brittle, half-translated SQL. +2. **Action config blocks are BQ-shaped.** `table.bigquery = { partitionBy, clusterBy, requirePartitionFilter, ... }` exposes BigQuery concepts. Postgres equivalents (`tablespace`, `fillfactor`, native `PARTITION BY RANGE/LIST/HASH`, btree/gin/gist/hnsw indexes) don't map. Forcing them through BQ-shaped fields is leaky. +3. **Supabase isn't just "Postgres on a host."** RLS, Realtime publications, `auth.users` integration, pgvector, pg_cron, Supabase Wrappers — none of these are addressable via a generic Postgres adapter, and all of them are why someone picks Supabase. + +## 2. Adapter Architecture + +``` +PostgresDbAdapter (postgres) + │ + ├── implements IDbAdapter + ├── uses node-postgres (`pg`, `pg-query-stream`) + ├── delegates SQL generation to PostgresSqlGenerator + └── connection: standard libpq DSN / JDBC-style credential + +SupabaseDbAdapter extends PostgresDbAdapter (supabase) + │ + ├── inherits all Postgres behavior + ├── adds: RLS introspection, Realtime publication management, + │ Supabase Wrapper foreign-server discovery + ├── delegates SQL generation to SupabaseSqlGenerator + │ (extends PostgresSqlGenerator) + └── connection: Supabase project credentials + (project_ref + service_role_key OR direct DB url + service_role for RLS bypass) +``` + +Both adapters live under `cli/api/dbadapters/`. SQL generators live under `core/compilation_sql/postgres/` and `core/compilation_sql/supabase/`. + +### 2.1 `IDbAdapter` Surface + +Methods the BigQuery adapter exposes today (`executeRaw`, `tables`, `deleteTable`, `prepareSchema`, `dryRun`, ...) must all be implementable against `pg` without semantic distortion. Where BigQuery returns BQ-specific metadata (`tableType: "VIEW" | "TABLE" | "MATERIALIZED_VIEW" | "EXTERNAL"`), Postgres returns its equivalent set (`tableType: "TABLE" | "VIEW" | "MATERIALIZED_VIEW" | "FOREIGN_TABLE" | "PARTITIONED_TABLE"`). The `ITableMetadata` interface should be a union over warehouse-specific extensions, not a lowest-common-denominator struct. + +## 3. Action Config Schema + +### 3.1 Proto changes (`protos/configs.proto`) + +Add two new message types alongside the existing `BigQueryOptions`: + +```proto +message PostgresOptions { + // Physical storage + string tablespace = 1; + uint32 fillfactor = 2; + bool unlogged = 3; + + // Partitioning (native Postgres declarative partitioning) + message Partition { + enum Kind { RANGE = 0; LIST = 1; HASH = 2; } + Kind kind = 1; + repeated string columns = 2; + } + Partition partition = 4; + + // Indexes + message Index { + string name = 1; + repeated string columns = 2; + enum Method { BTREE = 0; HASH = 1; GIN = 2; GIST = 3; BRIN = 4; } + Method method = 3; + string where = 4; // partial index predicate + bool unique = 5; + repeated string include = 6; // INCLUDE columns + } + repeated Index indexes = 5; + + // Materialized view options + bool with_data = 6; // WITH DATA / WITH NO DATA on initial creation + string refresh_policy = 7; // "manual" | "on_dependency_change" +} + +message SupabaseOptions { + // Standard Postgres options apply + PostgresOptions postgres = 1; + + // Supabase platform + bool publish_to_realtime = 2; // ALTER PUBLICATION supabase_realtime + bool enable_rls = 3; // ALTER TABLE ... ENABLE ROW LEVEL SECURITY + string owner_role = 4; // typically "postgres" or "service_role" + + // pgvector convenience (otherwise expressible via PostgresOptions.indexes) + message VectorConfig { + string column = 1; + uint32 dimensions = 2; + enum IndexType { IVFFLAT = 0; HNSW = 1; } + IndexType index_type = 3; + map params = 4; // ivfflat: lists; hnsw: m, ef_construction + } + repeated VectorConfig vectors = 5; +} +``` + +### 3.2 TypeScript action surface + +```typescript +publish("daily_orders", { + type: "incremental", + uniqueKey: ["order_id"], + postgres: { + partition: { kind: "range", columns: ["order_date"] }, + indexes: [ + { name: "ix_daily_orders_customer", columns: ["customer_id"], method: "btree" }, + { name: "ix_daily_orders_search", columns: ["description"], method: "gin" } + ] + } +}).query(ctx => `SELECT ... FROM ${ctx.ref("raw_orders")} WHERE order_date >= ${ctx.incremental() ? "(SELECT MAX(order_date) FROM ${ctx.self()})" : "'2020-01-01'"}`); +``` + +Compare to existing BigQuery shape: + +```typescript +publish("daily_orders", { + type: "incremental", + bigquery: { partitionBy: "DATE(order_date)", clusterBy: ["customer_id"], requirePartitionFilter: true } +}).query(...); +``` + +Each warehouse owns its own config namespace. Compilation errors if the wrong block is used against the wrong warehouse. + +## 4. SQL Generation Differences + +| Concern | BigQuery (existing) | Postgres (this spec) | +| :--- | :--- | :--- | +| Create table | `CREATE OR REPLACE TABLE \`x.y.z\` OPTIONS(...) AS SELECT ...` | `CREATE TABLE schema.tbl (...); INSERT INTO ...` (atomic via `BEGIN; DROP IF EXISTS; CREATE; INSERT; COMMIT;`) | +| Replace table | Single-statement `CREATE OR REPLACE TABLE` | Transactional drop + create + populate | +| Incremental upsert | `MERGE ... USING ... WHEN MATCHED THEN UPDATE ... WHEN NOT MATCHED THEN INSERT` | `INSERT ... ON CONFLICT (cols) DO UPDATE SET ...` | +| View | `CREATE OR REPLACE VIEW` | `CREATE OR REPLACE VIEW` (works in PG) | +| Materialized view | `CREATE MATERIALIZED VIEW` (auto-refresh) | `CREATE MATERIALIZED VIEW ... WITH [NO] DATA;` + explicit `REFRESH MATERIALIZED VIEW [CONCURRENTLY]` | +| Partitioning | `PARTITION BY DATE(col)`, `PARTITION BY RANGE_BUCKET(col, ...)` | `PARTITION BY RANGE/LIST/HASH (cols)` + `CREATE TABLE part PARTITION OF parent FOR VALUES ...` | +| Clustering | `CLUSTER BY col1, col2` (storage layout) | No direct equivalent. Closest: `CLUSTER table USING index` (one-shot reorder) + appropriate btree index. Not silently translated. | +| Primary key | `CREATE PRIMARY KEY ... NOT ENFORCED` (informational only in BQ) | `PRIMARY KEY (...)` — actually enforced | +| Assertions | `SELECT ... FROM x WHERE failing_condition` | Same shape; can additionally compile to `CHECK` constraints when user opts in | + +**Rule of thumb:** if BigQuery has a concept Postgres lacks (clustering, NOT ENFORCED PKs, `OPTIONS(description = ...)`), the Postgres generator either translates to the nearest meaningful equivalent **and warns in compilation output**, or refuses with a clear error pointing at `postgres:`-namespaced alternatives. + +## 5. New Supabase-Native Action Types + +Add to `core/actions/`: + +### 5.1 `rlsPolicy` + +```typescript +publish("orders_policy", { + type: "rlsPolicy", + table: "orders", + name: "users_see_own_orders", + command: "select", // "all" | "select" | "insert" | "update" | "delete" + roles: ["authenticated"], + using: "user_id = auth.uid()", + withCheck: "user_id = auth.uid()" +}); +``` + +Compiles to: + +```sql +CREATE POLICY users_see_own_orders ON orders + FOR SELECT TO authenticated + USING (user_id = auth.uid()) + WITH CHECK (user_id = auth.uid()); +``` + +Refs into tables work via the standard action graph — declaring an RLS policy creates a dependency edge to the underlying table. + +### 5.2 `realtimePublication` + +```typescript +publish("orders_realtime", { + type: "realtimePublication", + table: "orders", + events: ["insert", "update", "delete"] +}); +``` + +Compiles to `ALTER PUBLICATION supabase_realtime ADD TABLE orders;` (with replica identity adjustments). + +### 5.3 `wrapper` (foreign-server / FDW) + +Pairs with the hybrid-warehouse doc's Pattern C (BigQuery → Supabase): + +```typescript +publish("bq_churn_predictions", { + type: "wrapper", + wrapper: "bigquery", + server: "bq_analytics", + options: { project: "my-bq-project", dataset: "models", table: "churn_predictions" } +}); +``` + +Compiles to `CREATE FOREIGN TABLE ... SERVER bq_analytics OPTIONS (...)`. + +### 5.4 `vectorIndex` + +Convenience wrapper for pgvector — could also be expressed as a `PostgresOptions.Index` with `method: HNSW`, but a dedicated type makes RAG pipelines first-class. + +## 6. Implementation Phases (Replaces Antigravity Phases 3-5) + +Phases 1-2 (deps + relocation) from the Antigravity doc stand as written. The remaining work is re-scoped: + +### Phase 3a — Adapter skeleton (1 day) +- Implement `PostgresDbAdapter` against `pg` and `pg-query-stream`. +- Map `IDbAdapter` methods to Postgres semantics. Where BigQuery returns warehouse-specific metadata, return Postgres equivalents — do not force into BQ shape. +- Credential format: standard `{ host, port, database, user, password, ssl }` (or DSN string). Separate from BigQuery's `{ projectId, credentials, location }`. + +### Phase 3b — Postgres SQL generator (2-3 days) +- New directory: `core/compilation_sql/postgres/`. +- One generator per action type (table, view, incremental, materialized view, operation, assertion, declaration). +- Tests: `core/compilation_sql/postgres/*_test.ts`. Snapshot output against expected idiomatic Postgres SQL. + +### Phase 3c — Action config schema additions (1 day) +- Add `PostgresOptions` and `SupabaseOptions` to `protos/configs.proto`. +- Surface in TypeScript action types alongside existing `bigquery:` namespace. +- Compilation errors for cross-warehouse misuse (`postgres:` block against `warehouse: "bigquery"`, etc.). + +### Phase 4 — CLI wiring (0.5 day) +- Branch `cli/index.ts` adapter instantiation on `projectConfig.warehouse ∈ {bigquery, postgres, supabase}`. +- Credential file format auto-detection. +- `dataform init` (rename target: `sqlanvil init`) templates for both new warehouses. + +### Phase 5 — Supabase variant (2-3 days) +- `SupabaseDbAdapter extends PostgresDbAdapter`. +- Supabase-specific SQL generator extensions. +- New action types: `rlsPolicy`, `realtimePublication`, `wrapper`, `vectorIndex`. +- Connection config supports both direct Postgres URL and Supabase project_ref + service_role_key. + +### Phase 6 — Integration tests (1-2 days) +- Extend `tools/postgres/postgres_fixture.ts` with a parallel `tools/supabase/` fixture (Docker-compose'd Supabase stack — `supabase/postgres` image + Realtime + PostgREST optional). +- Test specs: `tests/integration/postgres.spec.ts` (already restored), new `tests/integration/supabase.spec.ts`. + +**Revised total estimate: 7-10 engineering days.** Antigravity's 1-2 day estimate covers Phase 1-2 + a minimal Phase 3a only. + +## 7. Risks Specific to This Approach + +| Risk | Mitigation | +| :--- | :--- | +| Postgres + Supabase generator divergence over time | Supabase generator inherits from Postgres via class extension; share a fixture suite where behaviors overlap. | +| New action types (`rlsPolicy`, etc.) bloat the core graph | Gate them behind the Supabase variant; standard Postgres users never see them. | +| Connection-string format proliferation | Document one canonical format per variant; provide a `sqlanvil credentials check` subcommand that validates and reports which warehouse it inferred. | +| BigQuery users expecting feature parity (clustering, GA partition pruning) on Postgres | Compilation warnings + docs page mapping BQ concepts → Postgres equivalents, with explicit "no equivalent" markers. | + +## 8. Resolved Decisions + +### 8.1 Rename is mandatory and precedes public release + +The full `dataform` → `sqlanvil` rename happens before any public-facing artifact ships. Trademark risk from Google is the driver. Scope: + +- Proto package names: `dataform.proto.*` → `sqlanvil.proto.*`. Touches every `.proto` file's `package` line and every TS import of generated types. +- npm packages: `@dataform/core`, `@dataform/cli`, etc. → `@sqlanvil/core`, `@sqlanvil/cli`. Republish under new scope; old `@dataform/*` namespace was never Ivan's anyway. +- CLI binary: `dataform` → `sqlanvil`. Update `cli/BUILD`, `scripts/run`, install docs. +- Config files: `dataform.json` → `sqlanvil.json`. `workflow_settings.yaml` keys retained (already neutral). +- Internal class names: `IDataformConfig` → `ISqlanvilConfig`, etc. +- Docs, error messages, telemetry user-agent strings. + +Recommended sequencing: **rename first, in a single sweep PR on the `restore-postgres-adapter` branch**, then layer the adapter work on top. Rationale: a partial rename is worse than either state — grep ambiguity, broken imports, mixed branding. Get it done in one painful day. + +`sqlanvil-com/index.html` should also gain a one-line legal notice acknowledging the fork's origin per Apache 2.0 license terms (Dataform OSS is Apache-2.0; attribution is required, derivative naming is not — but credit upstream cleanly). + +### 8.2 Connection config is nested under `warehouse:` block + +Decision: **nested**. Shape: + +```yaml +# workflow_settings.yaml — BigQuery variant +warehouse: + kind: bigquery + project: my-bq-project + location: US + defaultDataset: analytics + +# workflow_settings.yaml — Postgres variant +warehouse: + kind: postgres + host: db.example.com + port: 5432 + database: analytics + user: sqlanvil_writer + password: ${PG_PASSWORD} # env interpolation + ssl: require + defaultSchema: public + +# workflow_settings.yaml — Supabase variant +warehouse: + kind: supabase + projectRef: abcdefghijklmnop # from supabase dashboard + serviceRoleKey: ${SUPABASE_SERVICE_ROLE_KEY} + defaultSchema: public + # alternative: direct DB URL bypassing PostgREST + # connectionString: postgresql://postgres:${PASSWORD}@db.${PROJECT_REF}.supabase.co:5432/postgres +``` + +Why nested over flat: + +- **Extensibility.** Adding AlloyDB / CockroachDB / Redshift later = add a new `kind` value, not invent new top-level keys. +- **No naming collisions.** Flat `warehouse: postgres` collides semantically with `defaultDatabase: postgres` (kind vs database name). Nested removes the ambiguity. +- **Grouping.** All connection-affecting fields live in one block. Credentials, defaults, dialect flags all co-located. +- **Validation.** Discriminated union on `kind` — strict per-variant field validation, no global field that's only meaningful for some warehouses. + +Proto representation (`protos/configs.proto`): + +```proto +message WarehouseConfig { + oneof connection { + BigQueryConnection bigquery = 1; + PostgresConnection postgres = 2; + SupabaseConnection supabase = 3; + } +} + +message BigQueryConnection { string project = 1; string location = 2; string default_dataset = 3; } +message PostgresConnection { string host = 1; uint32 port = 2; string database = 3; string user = 4; string password = 5; string ssl_mode = 6; string default_schema = 7; } +message SupabaseConnection { string project_ref = 1; string service_role_key = 2; string default_schema = 3; string connection_string = 4; /* optional override */ } +``` + +YAML parser uses the `kind:` tag to discriminate before unmarshalling into the appropriate `oneof` arm. + +### 8.3 No migration path needed from existing Dataform projects + +Acuantia (`~/projects/acuantia-gcp-dataform/`) and other BQ-only Dataform projects stay on Google Cloud / upstream Dataform. They are **not** migration targets. sqlanvil's audience is new personal/OSS projects, especially Supabase-backed ones like **listanvil** (in this monorepo at `../listanvil/`). + +This means: no need for `dataform-compat` translation layer, no need to accept `dataform.json` in addition to `sqlanvil.json`, no need to support `@dataform/...` action config blocks under the rename. Clean break. + +## 9. Recommended Branch Strategy + +Three sequential PRs on top of `restore-postgres-adapter`: + +1. **`rename/dataform-to-sqlanvil`** — pure rename, no behavior change. Mechanical. Reviewable as a diff against upstream. +2. **`adapter/postgres-first-class`** — Phases 1-4 of section 6. New proto messages, Postgres SQL generator, `PostgresDbAdapter`, CLI wiring. No Supabase code yet. +3. **`adapter/supabase-variant`** — Phase 5-6. `SupabaseDbAdapter`, new action types (`rlsPolicy`, `realtimePublication`, `wrapper`, `vectorIndex`), Supabase integration fixture. + +Each PR self-contained, mergeable independently. PR 1 unblocks any future public artifact; PRs 2-3 unblock listanvil-style projects using sqlanvil. diff --git a/docs/postgres_reintegration_assessment.md b/docs/postgres_reintegration_assessment.md index 3d5a0b48..9e03e3f6 100644 --- a/docs/postgres_reintegration_assessment.md +++ b/docs/postgres_reintegration_assessment.md @@ -1,6 +1,13 @@ -# Technical Assessment: PostgreSQL Reintegration in SqlAnvil +# Technical Assessment: PostgreSQL Reintegration in sqlanvil -This document provides a detailed technical assessment of what is required to re-integrate PostgreSQL database adapter support into the modern **SqlAnvil** (Dataform) codebase using the old restored files. +> **SUPERSEDED.** Phase 3+ of this assessment is replaced by +> [`postgres_first_class_design.md`](postgres_first_class_design.md), which +> treats the Postgres adapter as first-class (idiomatic SQL generation, native +> action config blocks, dedicated SQL generator path) rather than a +> BigQuery-shaped adapter swap. Phases 1-2 (deps + relocation) below remain +> applicable; Phases 3-5 are re-scoped in the design doc. + +This document provides a detailed technical assessment of what is required to re-integrate PostgreSQL database adapter support into the modern **sqlanvil** codebase using the old restored files. --- diff --git a/docs/reference/assertion.md b/docs/reference/assertion.md index 74acd5fa..85d22567 100644 --- a/docs/reference/assertion.md +++ b/docs/reference/assertion.md @@ -1,4 +1,4 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/assertion"](../modules/_core_actions_assertion_.md) › [Assertion](_core_actions_assertion_.assertion.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/assertion"](../modules/_core_actions_assertion_.md) › [Assertion](_core_actions_assertion_.assertion.md) # Class: Assertion @@ -6,7 +6,7 @@ An assertion is a data quality test query that finds rows that violate one or mo specified in the query. If the query returns any rows, the assertion fails. You can create assertions in the following ways. Available config options are defined in -[AssertionConfig](configs#dataform-ActionConfig-AssertionConfig), and are shared across all the +[AssertionConfig](configs#sqlanvil-ActionConfig-AssertionConfig), and are shared across all the following ways of creating assertions. **Using a SQLX file:** @@ -21,7 +21,7 @@ SELECT * FROM table WHERE a IS NULL **Using built-in assertions in the config block of a table:** -See [TableConfig.assertions](configs#dataform-ActionConfig-TableConfig) +See [TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig) **Using action configs files:** @@ -73,7 +73,7 @@ This is where `query` comes from. ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.project](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.project](configs#sqlanvil-ActionConfig-AssertionConfig). Sets the database (Google Cloud project ID) in which to create the corresponding view for this assertion. @@ -93,7 +93,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.dependencies](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.dependencies](configs#sqlanvil-ActionConfig-AssertionConfig). Sets dependencies of the assertion. @@ -112,7 +112,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.description](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.description](configs#sqlanvil-ActionConfig-AssertionConfig). Sets the description of this assertion. @@ -131,7 +131,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.disabled](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.disabled](configs#sqlanvil-ActionConfig-AssertionConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -151,7 +151,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[AssertionConfig.hermetic](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.hermetic](configs#sqlanvil-ActionConfig-AssertionConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -188,7 +188,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.dataset](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.dataset](configs#sqlanvil-ActionConfig-AssertionConfig). Sets the schema (BigQuery dataset) in which to create the corresponding view for this assertion. @@ -208,7 +208,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[AssertionConfig.tags](configs#dataform-ActionConfig-AssertionConfig). +[AssertionConfig.tags](configs#sqlanvil-ActionConfig-AssertionConfig). Sets a list of user-defined tags applied to this action. diff --git a/docs/reference/configs.md b/docs/reference/configs.md index de618ae7..473b26ae 100644 --- a/docs/reference/configs.md +++ b/docs/reference/configs.md @@ -6,30 +6,30 @@ - [configs.proto](#configs-proto) - - [ActionConfig](#dataform-ActionConfig) - - [ActionConfig.AssertionConfig](#dataform-ActionConfig-AssertionConfig) - - [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) - - [ActionConfig.DataPreparationConfig](#dataform-ActionConfig-DataPreparationConfig) - - [ActionConfig.DeclarationConfig](#dataform-ActionConfig-DeclarationConfig) - - [ActionConfig.IncrementalTableConfig](#dataform-ActionConfig-IncrementalTableConfig) - - [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) - - [ActionConfig.IncrementalTableConfig.LabelsEntry](#dataform-ActionConfig-IncrementalTableConfig-LabelsEntry) - - [ActionConfig.NotebookConfig](#dataform-ActionConfig-NotebookConfig) - - [ActionConfig.OnSchemaChange](#dataform-ActionConfig-OnSchemaChange) - - [ActionConfig.OperationConfig](#dataform-ActionConfig-OperationConfig) - - [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) - - [ActionConfig.TableAssertionsConfig.UniqueKey](#dataform-ActionConfig-TableAssertionsConfig-UniqueKey) - - [ActionConfig.TableConfig](#dataform-ActionConfig-TableConfig) - - [ActionConfig.TableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-TableConfig-AdditionalOptionsEntry) - - [ActionConfig.TableConfig.LabelsEntry](#dataform-ActionConfig-TableConfig-LabelsEntry) - - [ActionConfig.Target](#dataform-ActionConfig-Target) - - [ActionConfig.ViewConfig](#dataform-ActionConfig-ViewConfig) - - [ActionConfig.ViewConfig.AdditionalOptionsEntry](#dataform-ActionConfig-ViewConfig-AdditionalOptionsEntry) - - [ActionConfig.ViewConfig.LabelsEntry](#dataform-ActionConfig-ViewConfig-LabelsEntry) - - [ActionConfigs](#dataform-ActionConfigs) - - [NotebookRuntimeOptionsConfig](#dataform-NotebookRuntimeOptionsConfig) - - [WorkflowSettings](#dataform-WorkflowSettings) - - [WorkflowSettings.VarsEntry](#dataform-WorkflowSettings-VarsEntry) + - [ActionConfig](#sqlanvil-ActionConfig) + - [ActionConfig.AssertionConfig](#sqlanvil-ActionConfig-AssertionConfig) + - [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) + - [ActionConfig.DataPreparationConfig](#sqlanvil-ActionConfig-DataPreparationConfig) + - [ActionConfig.DeclarationConfig](#sqlanvil-ActionConfig-DeclarationConfig) + - [ActionConfig.IncrementalTableConfig](#sqlanvil-ActionConfig-IncrementalTableConfig) + - [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) + - [ActionConfig.IncrementalTableConfig.LabelsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-LabelsEntry) + - [ActionConfig.NotebookConfig](#sqlanvil-ActionConfig-NotebookConfig) + - [ActionConfig.OnSchemaChange](#sqlanvil-ActionConfig-OnSchemaChange) + - [ActionConfig.OperationConfig](#sqlanvil-ActionConfig-OperationConfig) + - [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) + - [ActionConfig.TableAssertionsConfig.UniqueKey](#sqlanvil-ActionConfig-TableAssertionsConfig-UniqueKey) + - [ActionConfig.TableConfig](#sqlanvil-ActionConfig-TableConfig) + - [ActionConfig.TableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-TableConfig-AdditionalOptionsEntry) + - [ActionConfig.TableConfig.LabelsEntry](#sqlanvil-ActionConfig-TableConfig-LabelsEntry) + - [ActionConfig.Target](#sqlanvil-ActionConfig-Target) + - [ActionConfig.ViewConfig](#sqlanvil-ActionConfig-ViewConfig) + - [ActionConfig.ViewConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-ViewConfig-AdditionalOptionsEntry) + - [ActionConfig.ViewConfig.LabelsEntry](#sqlanvil-ActionConfig-ViewConfig-LabelsEntry) + - [ActionConfigs](#sqlanvil-ActionConfigs) + - [NotebookRuntimeOptionsConfig](#sqlanvil-NotebookRuntimeOptionsConfig) + - [WorkflowSettings](#sqlanvil-WorkflowSettings) + - [WorkflowSettings.VarsEntry](#sqlanvil-WorkflowSettings-VarsEntry) - [Scalar Value Types](#scalar-value-types) @@ -39,7 +39,7 @@ ## configs.proto - + ### ActionConfig @@ -47,16 +47,16 @@ Action config defines the contents of `actions.yaml` configuration files. | Field | Type | Label | Description | | ---------------- | ------------------------------------------------------------------------------------ | ----- | ----------- | -| table | [ActionConfig.TableConfig](#dataform-ActionConfig-TableConfig) | | | -| view | [ActionConfig.ViewConfig](#dataform-ActionConfig-ViewConfig) | | | -| incrementalTable | [ActionConfig.IncrementalTableConfig](#dataform-ActionConfig-IncrementalTableConfig) | | | -| assertion | [ActionConfig.AssertionConfig](#dataform-ActionConfig-AssertionConfig) | | | -| operation | [ActionConfig.OperationConfig](#dataform-ActionConfig-OperationConfig) | | | -| declaration | [ActionConfig.DeclarationConfig](#dataform-ActionConfig-DeclarationConfig) | | | -| notebook | [ActionConfig.NotebookConfig](#dataform-ActionConfig-NotebookConfig) | | | -| dataPreparation | [ActionConfig.DataPreparationConfig](#dataform-ActionConfig-DataPreparationConfig) | | | +| table | [ActionConfig.TableConfig](#sqlanvil-ActionConfig-TableConfig) | | | +| view | [ActionConfig.ViewConfig](#sqlanvil-ActionConfig-ViewConfig) | | | +| incrementalTable | [ActionConfig.IncrementalTableConfig](#sqlanvil-ActionConfig-IncrementalTableConfig) | | | +| assertion | [ActionConfig.AssertionConfig](#sqlanvil-ActionConfig-AssertionConfig) | | | +| operation | [ActionConfig.OperationConfig](#sqlanvil-ActionConfig-OperationConfig) | | | +| declaration | [ActionConfig.DeclarationConfig](#sqlanvil-ActionConfig-DeclarationConfig) | | | +| notebook | [ActionConfig.NotebookConfig](#sqlanvil-ActionConfig-NotebookConfig) | | | +| dataPreparation | [ActionConfig.DataPreparationConfig](#sqlanvil-ActionConfig-DataPreparationConfig) | | | - + ### ActionConfig.AssertionConfig @@ -65,7 +65,7 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the assertion. | | dataset | [string](#string) | | The dataset (schema) of the assertion. | | project | [string](#string) | | The Google Cloud project (database) of the assertion. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | @@ -74,7 +74,7 @@ Action config defines the contents of `actions.yaml` configuration files. | dependOnDependencyAssertions | [bool](#bool) | | If true, assertions dependent upon any of the dependencies are added as dependencies as well. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.ColumnDescriptor @@ -85,21 +85,21 @@ Action config defines the contents of `actions.yaml` configuration files. | bigqueryPolicyTags | [string](#string) | repeated | A list of BigQuery policy tags that will be applied to the column. | | tags | [string](#string) | repeated | A list of tags for this column which will be applied. | - + ### ActionConfig.DataPreparationConfig | Field | Type | Label | Description | | ------------------- | ---------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | name | [string](#string) | | The name of the data preparation. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | description | [string](#string) | | Description of the data preparation. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.DeclarationConfig @@ -109,9 +109,9 @@ Action config defines the contents of `actions.yaml` configuration files. | dataset | [string](#string) | | The dataset (schema) of the declaration. | | project | [string](#string) | | The Google Cloud project (database) of the declaration. | | description | [string](#string) | | Description of the declaration. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the declaration. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the declaration. | - + ### ActionConfig.IncrementalTableConfig @@ -120,30 +120,30 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the incremental table. | | dataset | [string](#string) | | The dataset (schema) of the incremental table. | | project | [string](#string) | | The Google Cloud project (database) of the incremental table. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | preOperations | [string](#string) | repeated | Queries to run before `query`. This can be useful for granting permissions. | | postOperations | [string](#string) | repeated | Queries to run after `query`. | | protected | [bool](#bool) | | If true, prevents the dataset from being rebuilt from scratch. | -| uniqueKey | [string](#string) | repeated | If set, unique key represents a set of names of columns that will act as a the unique key. To enforce this, when updating the incremental table, Dataform merges rows with `uniqueKey` instead of appending them. | +| uniqueKey | [string](#string) | repeated | If set, unique key represents a set of names of columns that will act as a the unique key. To enforce this, when updating the incremental table, sqlanvil merges rows with `uniqueKey` instead of appending them. | | description | [string](#string) | | Description of the incremental table. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | | partitionBy | [string](#string) | | The key by which to partition the table. Typically the name of a timestamp or the date column. See https://cloud.google.com/dataform/docs/partitions-clusters. | | partitionExpirationDays | [int32](#int32) | | The number of days for which BigQuery stores data in each partition. The setting applies to all partitions in a table, but is calculated independently for each partition based on the partition time. | | requirePartitionFilter | [bool](#bool) | | Declares whether the partitioned table requires a WHERE clause predicate filter that filters the partitioning column. | | updatePartitionFilter | [string](#string) | | SQL-based filter for when incremental updates are applied. | | clusterBy | [string](#string) | repeated | The keys by which to cluster partitions by. See https://cloud.google.com/dataform/docs/partitions-clusters. | -| labels | [ActionConfig.IncrementalTableConfig.LabelsEntry](#dataform-ActionConfig-IncrementalTableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | -| additionalOptions | [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | +| labels | [ActionConfig.IncrementalTableConfig.LabelsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | +| additionalOptions | [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | -| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | +| assertions | [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | -| onSchemaChange | [ActionConfg.OnSchemaChange](#dataform-ActionConfig-OnSchemaChange) | | Defines the action behavior if the selected columns in query doesn't match columns in the target table. | +| onSchemaChange | [ActionConfg.OnSchemaChange](#sqlanvil-ActionConfig-OnSchemaChange) | | Defines the action behavior if the selected columns in query doesn't match columns in the target table. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry @@ -152,7 +152,7 @@ Action config defines the contents of `actions.yaml` configuration files. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.IncrementalTableConfig.LabelsEntry @@ -161,7 +161,7 @@ Action config defines the contents of `actions.yaml` configuration files. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.NotebookConfig @@ -170,7 +170,7 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the notebook. | | location | [string](#string) | | The Google Cloud location of the notebook. | | project | [string](#string) | | The Google Cloud project (database) of the notebook. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | @@ -178,7 +178,7 @@ Action config defines the contents of `actions.yaml` configuration files. | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.OnSchemaChange @@ -189,7 +189,7 @@ Action config defines the contents of `actions.yaml` configuration files. | EXTEND | New columns will be added to the target table. Fails if columns are deleted or renamed. | | SYNCHRONIZE | Does not block any new column(s) from being added, deleted or renamed. Partitioned or clustered columns cannot be deleted or renamed. | - + ### ActionConfig.OperationConfig @@ -198,18 +198,18 @@ Action config defines the contents of `actions.yaml` configuration files. | name | [string](#string) | | The name of the operation. | | dataset | [string](#string) | | The dataset (schema) of the operation. | | project | [string](#string) | | The Google Cloud project (database) of the operation. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | hasOutput | [bool](#bool) | | Declares that this action creates a dataset which should be referenceable as a dependency target, for example by using the `ref` function. | | description | [string](#string) | | Description of the operation. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the operation. Can only be set if hasOutput is true. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the operation. Can only be set if hasOutput is true. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.TableAssertionsConfig @@ -219,11 +219,11 @@ action types. | Field | Type | Label | Description | | ------------- | ------------------------------------------------------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | uniqueKey | [string](#string) | repeated | Column(s) which constitute the dataset's unique key index. If set, the resulting assertion will fail if there is more than one row in the dataset with the same values for all of these column(s). | -| uniqueKeys | [ActionConfig.TableAssertionsConfig.UniqueKey](#dataform-ActionConfig-TableAssertionsConfig-UniqueKey) | repeated | | +| uniqueKeys | [ActionConfig.TableAssertionsConfig.UniqueKey](#sqlanvil-ActionConfig-TableAssertionsConfig-UniqueKey) | repeated | | | nonNull | [string](#string) | repeated | Column(s) which may never be `NULL`. If set, the resulting assertion will fail if any row contains `NULL` values for these column(s). | | rowConditions | [string](#string) | repeated | General condition(s) which should hold true for all rows in the dataset. If set, the resulting assertion will fail if any row violates any of these condition(s). | - + ### ActionConfig.TableAssertionsConfig.UniqueKey @@ -236,7 +236,7 @@ the column(s) in the unique key(s). | --------- | ----------------- | -------- | ----------- | | uniqueKey | [string](#string) | repeated | | - + ### ActionConfig.TableConfig @@ -245,26 +245,26 @@ the column(s) in the unique key(s). | name | [string](#string) | | The name of the table. | | dataset | [string](#string) | | The dataset (schema) of the table. | | project | [string](#string) | | The Google Cloud project (database) of the table. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | | preOperations | [string](#string) | repeated | Queries to run before `query`. This can be useful for granting permissions. | | postOperations | [string](#string) | repeated | Queries to run after `query`. | | description | [string](#string) | | Description of the table. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | | partitionBy | [string](#string) | | The key by which to partition the table. Typically the name of a timestamp or the date column. See https://cloud.google.com/dataform/docs/partitions-clusters. | | partitionExpirationDays | [int32](#int32) | | The number of days for which BigQuery stores data in each partition. The setting applies to all partitions in a table, but is calculated independently for each partition based on the partition time. | | requirePartitionFilter | [bool](#bool) | | Declares whether the partitioned table requires a WHERE clause predicate filter that filters the partitioning column. | | clusterBy | [string](#string) | repeated | The keys by which to cluster partitions by. See https://cloud.google.com/dataform/docs/partitions-clusters. | -| labels | [ActionConfig.TableConfig.LabelsEntry](#dataform-ActionConfig-TableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | -| additionalOptions | [ActionConfig.TableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-TableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | +| labels | [ActionConfig.TableConfig.LabelsEntry](#sqlanvil-ActionConfig-TableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | +| additionalOptions | [ActionConfig.TableConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-TableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | -| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | +| assertions | [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.TableConfig.AdditionalOptionsEntry @@ -273,7 +273,7 @@ the column(s) in the unique key(s). | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.TableConfig.LabelsEntry @@ -282,7 +282,7 @@ the column(s) in the unique key(s). | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.Target @@ -295,7 +295,7 @@ Target represents a unique action identifier. | name | [string](#string) | | The name of the action. | | includeDependentAssertions | [bool](#bool) | | flag for when we want to add assertions of this dependency in dependency_targets as well. | - + ### ActionConfig.ViewConfig @@ -304,7 +304,7 @@ Target represents a unique action identifier. | name | [string](#string) | | The name of the view. | | dataset | [string](#string) | | The dataset (schema) of the view. | | project | [string](#string) | | The Google Cloud project (database) of the view. | -| dependencyTargets | [ActionConfig.Target](#dataform-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | +| dependencyTargets | [ActionConfig.Target](#sqlanvil-ActionConfig-Target) | repeated | Targets of actions that this action is dependent on. | | filename | [string](#string) | | Path to the source file that the contents of the action is loaded from. | | tags | [string](#string) | repeated | A list of user-defined tags with which the action should be labeled. | | disabled | [bool](#bool) | | If set to true, this action will not be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. | @@ -314,15 +314,15 @@ Target represents a unique action identifier. | partitionBy | [string](#string) | | Optional. Applicable only to materialized view. The key by which to partition the materialized view. Typically the name of a timestamp or the date column. See https://cloud.google.com/bigquery/docs/materialized-views-create#partitioned_materialized_views. | | clusterBy | [string](#string) | repeated | Optional. Applicable only to materialized view. The keys by which to cluster partitions by. See https://cloud.google.com/bigquery/docs/materialized-views-create#cluster_materialized_views. | | description | [string](#string) | | Description of the view. | -| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | -| labels | [ActionConfig.ViewConfig.LabelsEntry](#dataform-ActionConfig-ViewConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | -| additionalOptions | [ActionConfig.ViewConfig.AdditionalOptionsEntry](#dataform-ActionConfig-ViewConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | +| columns | [ActionConfig.ColumnDescriptor](#sqlanvil-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. | +| labels | [ActionConfig.ViewConfig.LabelsEntry](#sqlanvil-ActionConfig-ViewConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. | +| additionalOptions | [ActionConfig.ViewConfig.AdditionalOptionsEntry](#sqlanvil-ActionConfig-ViewConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. | | dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action | | hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. | -| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | +| assertions | [ActionConfig.TableAssertionsConfig](#sqlanvil-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. | | reservation | [string](#string) | | Optional. The BigQuery reservation to use for execution. | - + ### ActionConfig.ViewConfig.AdditionalOptionsEntry @@ -331,7 +331,7 @@ Target represents a unique action identifier. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfig.ViewConfig.LabelsEntry @@ -340,7 +340,7 @@ Target represents a unique action identifier. | key | [string](#string) | | | | value | [string](#string) | | | - + ### ActionConfigs @@ -348,9 +348,9 @@ Action configs defines the contents of `actions.yaml` configuration files. | Field | Type | Label | Description | | ------- | -------------------------------------- | -------- | ----------- | -| actions | [ActionConfig](#dataform-ActionConfig) | repeated | | +| actions | [ActionConfig](#sqlanvil-ActionConfig) | repeated | | - + ### NotebookRuntimeOptionsConfig @@ -358,7 +358,7 @@ Action configs defines the contents of `actions.yaml` configuration files. | ------------ | ----------------- | ----- | ------------------------------------------------------------ | | outputBucket | [string](#string) | | Storage bucket to output notebooks to after their execution. | - + ### WorkflowSettings @@ -367,20 +367,20 @@ configuration file. | Field | Type | Label | Description | | ----------------------------- | ---------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| dataformCoreVersion | [string](#string) | | The desired dataform core version to compile against. | +| sqlanvilCoreVersion | [string](#string) | | The desired sqlanvil core version to compile against. | | defaultProject | [string](#string) | | Required. The default Google Cloud project (database). | | defaultDataset | [string](#string) | | Required. The default dataset (schema). | | defaultLocation | [string](#string) | | Required. The default BigQuery location to use. For more information on BigQuery locations, see https://cloud.google.com/bigquery/docs/locations. | | defaultAssertionDataset | [string](#string) | | Required. The default dataset (schema) for assertions. | -| vars | [WorkflowSettings.VarsEntry](#dataform-WorkflowSettings-VarsEntry) | repeated | Optional. User-defined variables that are made available to project code during compilation. An object containing a list of "key": value pairs. | +| vars | [WorkflowSettings.VarsEntry](#sqlanvil-WorkflowSettings-VarsEntry) | repeated | Optional. User-defined variables that are made available to project code during compilation. An object containing a list of "key": value pairs. | | projectSuffix | [string](#string) | | Optional. The suffix to append to all Google Cloud project references. | | datasetSuffix | [string](#string) | | Optional. The suffix to append to all dataset references. | | namePrefix | [string](#string) | | Optional. The prefix to append to all action names. | -| defaultNotebookRuntimeOptions | [NotebookRuntimeOptionsConfig](#dataform-NotebookRuntimeOptionsConfig) | | Optional. Default runtime options for Notebook actions. | +| defaultNotebookRuntimeOptions | [NotebookRuntimeOptionsConfig](#sqlanvil-NotebookRuntimeOptionsConfig) | | Optional. Default runtime options for Notebook actions. | | builtinAssertionNamePrefix | [string](#string) | | Optional. The prefix to append to built-in assertion names. | | defaultReservation | [string](#string) | | Optional. The default BigQuery reservation to use for execution. | - + ### WorkflowSettings.VarsEntry diff --git a/docs/reference/declaration.md b/docs/reference/declaration.md index 0e043313..928d1b9c 100644 --- a/docs/reference/declaration.md +++ b/docs/reference/declaration.md @@ -1,19 +1,19 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/declaration"](../modules/_core_actions_declaration_.md) › [Declaration](_core_actions_declaration_.declaration.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/declaration"](../modules/_core_actions_declaration_.md) › [Declaration](_core_actions_declaration_.declaration.md) # Class: Declaration -You can declare any BigQuery table as a data source in Dataform. Declaring BigQuery data -sources that are external to Dataform lets you treat those data sources as Dataform objects. +You can declare any BigQuery table as a data source in sqlanvil. Declaring BigQuery data +sources that are external to sqlanvil lets you treat those data sources as sqlanvil objects. Declaring data sources is optional, but can be useful when you want to do the following: -* Reference or resolve declared sources in the same way as any other table in Dataform. -* View declared sources in the visualized Dataform graph. -* Use Dataform to manage the table-level and column-level descriptions of externally created +* Reference or resolve declared sources in the same way as any other table in sqlanvil. +* View declared sources in the visualized sqlanvil graph. +* Use sqlanvil to manage the table-level and column-level descriptions of externally created tables. * Trigger workflow invocations that include all the dependents of an external data source. You can create declarations in the following ways. Available config options are defined in -[DeclarationConfig](configs#dataform-ActionConfig-DeclarationConfig), and are shared across all +[DeclarationConfig](configs#sqlanvil-ActionConfig-DeclarationConfig), and are shared across all the followiing ways of creating declarations. **Using a SQLX file:** @@ -62,7 +62,7 @@ declare("name") ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[DeclarationConfig.columns](configs#dataform-ActionConfig-DeclarationConfig). +[DeclarationConfig.columns](configs#sqlanvil-ActionConfig-DeclarationConfig). Sets the column descriptors of columns in this table. @@ -81,7 +81,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[DeclarationConfig.description](configs#dataform-ActionConfig-DeclarationConfig). +[DeclarationConfig.description](configs#sqlanvil-ActionConfig-DeclarationConfig). Sets the description of this assertion. diff --git a/docs/reference/incrementaltable.md b/docs/reference/incrementaltable.md index 8f90a3e1..bea8e2c6 100644 --- a/docs/reference/incrementaltable.md +++ b/docs/reference/incrementaltable.md @@ -1,13 +1,13 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/incremental_table"](../modules/_core_actions_incremental_table_.md) › [IncrementalTable](_core_actions_incremental_table_.incrementaltable.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/incremental_table"](../modules/_core_actions_incremental_table_.md) › [IncrementalTable](_core_actions_incremental_table_.incrementaltable.md) # Class: IncrementalTable -When you define an incremental table, Dataform builds the incremental table from scratch only for -the first time. During subsequent executions, Dataform only inserts or merges new rows into the +When you define an incremental table, sqlanvil builds the incremental table from scratch only for +the first time. During subsequent executions, sqlanvil only inserts or merges new rows into the incremental table according to the conditions that you configure. You can create incremental tables in the following ways. Available config options are defined in -[IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig), and are shared across all the +[IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig), and are shared across all the following ways of creating tables. **Using a SQLX file:** @@ -72,7 +72,7 @@ This is where `query` comes from. ▸ **assertions**(`assertions`: TableAssertionsConfig): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.assertions](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.assertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets in-line assertions for this incremental table. @@ -95,7 +95,7 @@ ___ ▸ **bigquery**(`bigquery`: IBigQueryOptions): *this* **`deprecated`** Deprecated in favor of options available directly on -[IncrementalTableConfig](configs#dataform-ActionConfig-IncrementalTableConfig). For example: +[IncrementalTableConfig](configs#sqlanvil-ActionConfig-IncrementalTableConfig). For example: `publish("name", { type: "table", partitionBy: "column" }`). Sets bigquery options for the action. @@ -115,7 +115,7 @@ ___ ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.columns](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.columns](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the column descriptors of columns in this incremental table. @@ -134,7 +134,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.project](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.project](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the Sets the database (Google Cloud project ID) in which to create the output of this action. @@ -154,7 +154,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.dependencies](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.dependencies](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets dependencies of the incremental table. @@ -173,7 +173,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.description](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.description](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the description of this incremental table. @@ -192,7 +192,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.disabled](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.disabled](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -212,7 +212,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.hermetic](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.hermetic](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -287,7 +287,7 @@ ___ ▸ **protected**(`isProtected`: boolean): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.protected](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.protected](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If called with `true`, prevents the dataset from being rebuilt from scratch. @@ -322,7 +322,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.dataset](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.dataset](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -341,7 +341,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-IncrementalTableConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -361,7 +361,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.tags](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.tags](configs#sqlanvil-ActionConfig-IncrementalTableConfig). Sets a list of user-defined tags applied to this action. @@ -398,10 +398,10 @@ ___ ▸ **uniqueKey**(`uniqueKey`: string[]): *void* **`deprecated`** Deprecated in favor of -[IncrementalTableConfig.uniqueKey](configs#dataform-ActionConfig-IncrementalTableConfig). +[IncrementalTableConfig.uniqueKey](configs#sqlanvil-ActionConfig-IncrementalTableConfig). If set, unique key represents a set of names of columns that will act as a the unique key. To -enforce this, when updating the incremental table, Dataform merges rows with `uniqueKey` +enforce this, when updating the incremental table, sqlanvil merges rows with `uniqueKey` instead of appending them. **Parameters:** diff --git a/docs/reference/notebook.md b/docs/reference/notebook.md index 863ecebc..a0c9fab5 100644 --- a/docs/reference/notebook.md +++ b/docs/reference/notebook.md @@ -1,4 +1,4 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/notebook"](../modules/_core_actions_notebook_.md) › [Notebook](_core_actions_notebook_.notebook.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/notebook"](../modules/_core_actions_notebook_.md) › [Notebook](_core_actions_notebook_.notebook.md) # Class: Notebook @@ -6,7 +6,7 @@ Notebooks run Jupyter Notebook files, and can output content to the storage buck `workflow_settings.yaml` files. You can create notebooks in the following ways. Available config options are defined in -[NotebookConfig](configs#dataform-ActionConfig-NotebookConfig), and are shared across all the +[NotebookConfig](configs#sqlanvil-ActionConfig-NotebookConfig), and are shared across all the following ways of creating notebooks. **Using action configs files:** diff --git a/docs/reference/operation.md b/docs/reference/operation.md index 62c02b13..4c01bc31 100644 --- a/docs/reference/operation.md +++ b/docs/reference/operation.md @@ -1,12 +1,12 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/operation"](../modules/_core_actions_operation_.md) › [Operation](_core_actions_operation_.operation.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/operation"](../modules/_core_actions_operation_.md) › [Operation](_core_actions_operation_.operation.md) # Class: Operation -Operations define custom SQL operations that don't fit into the Dataform model of publishing a +Operations define custom SQL operations that don't fit into the sqlanvil model of publishing a table or writing an assertion. You can create operations in the following ways. Available config options are defined in -[OperationConfig](configs#dataform-ActionConfig-OperationConfig), and are shared across all the +[OperationConfig](configs#sqlanvil-ActionConfig-OperationConfig), and are shared across all the following ways of creating operations. **Using a SQLX file:** @@ -72,7 +72,7 @@ This is where `query` comes from. ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.columns](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.columns](configs#sqlanvil-ActionConfig-OperationConfig). Sets the column descriptors of columns in this table. @@ -91,7 +91,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.project](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.project](configs#sqlanvil-ActionConfig-OperationConfig). Sets the database (Google Cloud project ID) in which to create the corresponding view for this operation. @@ -111,7 +111,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.dependencies](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.dependencies](configs#sqlanvil-ActionConfig-OperationConfig). Sets dependencies of the table. @@ -130,7 +130,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.description](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.description](configs#sqlanvil-ActionConfig-OperationConfig). Sets the description of this assertion. @@ -149,7 +149,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.disabled](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.disabled](configs#sqlanvil-ActionConfig-OperationConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -169,7 +169,7 @@ ___ ▸ **hasOutput**(`hasOutput`: boolean): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.hasOutput](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.hasOutput](configs#sqlanvil-ActionConfig-OperationConfig). Declares that this action creates a dataset which should be referenceable as a dependency target, for example by using the `ref` function. @@ -189,7 +189,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[OperationConfig.hermetic](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.hermetic](configs#sqlanvil-ActionConfig-OperationConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -228,7 +228,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.dataset](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.dataset](configs#sqlanvil-ActionConfig-OperationConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -247,7 +247,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-OperationConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -267,7 +267,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[OperationConfig.tags](configs#dataform-ActionConfig-OperationConfig). +[OperationConfig.tags](configs#sqlanvil-ActionConfig-OperationConfig). Sets a list of user-defined tags applied to this action. diff --git a/docs/reference/session.md b/docs/reference/session.md index b0da1f98..efc4186e 100644 --- a/docs/reference/session.md +++ b/docs/reference/session.md @@ -1,9 +1,9 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/session"](../modules/_core_session_.md) › [Session](_core_session_.session.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/session"](../modules/_core_session_.md) › [Session](_core_session_.session.md) # Class: Session Contains methods that are published globally, so can be invoked anywhere in the `/definitions` -folder of a Dataform project. +folder of a sqlanvil project. ## Hierarchy @@ -30,13 +30,13 @@ folder of a Dataform project. • **projectConfig**: *ProjectConfig* -Stores the Dataform project configuration of the current Dataform project. Can be accessed via -the `dataform` global variable. +Stores the sqlanvil project configuration of the current sqlanvil project. Can be accessed via +the `sqlanvil` global variable. Example: ```js -dataform.projectConfig.vars.myVariableName === "myVariableValue" +sqlanvil.projectConfig.vars.myVariableName === "myVariableValue" ``` ## Methods @@ -45,7 +45,7 @@ dataform.projectConfig.vars.myVariableName === "myVariableValue" ▸ **assert**(`name`: string, `queryOrConfig?`: AContextable‹string› | AssertionConfig): *[Assertion](_core_actions_assertion_.assertion.md)* -Adds a Dataform assertion the compiled graph. +Adds a sqlanvil assertion the compiled graph. Available only in the `/definitions` directory. @@ -66,7 +66,7 @@ ___ ▸ **declare**(`config`: DeclarationConfig | any): *[Declaration](_core_actions_declaration_.declaration.md)* -Declares the dataset as a Dataform data source. +Declares the dataset as a sqlanvil data source. Available only in the `/definitions` directory. diff --git a/docs/reference/table.md b/docs/reference/table.md index 86f1f153..fab7fd4a 100644 --- a/docs/reference/table.md +++ b/docs/reference/table.md @@ -1,13 +1,13 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/table"](../modules/_core_actions_table_.md) › [Table](_core_actions_table_.table.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/table"](../modules/_core_actions_table_.md) › [Table](_core_actions_table_.table.md) # Class: Table -Tables are the fundamental building block for storing data when using Dataform. Dataform compiles -your Dataform core code into SQL, executes the SQL code, and creates your defined tables in +Tables are the fundamental building block for storing data when using sqlanvil. sqlanvil compiles +your sqlanvil core code into SQL, executes the SQL code, and creates your defined tables in BigQuery. You can create tables in the following ways. Available config options are defined in -[TableConfig](configs#dataform-ActionConfig-TableConfig), and are shared across all the +[TableConfig](configs#sqlanvil-ActionConfig-TableConfig), and are shared across all the following ways of creating tables. **Using a SQLX file:** @@ -77,7 +77,7 @@ This is where `query` comes from. ▸ **assertions**(`assertions`: TableAssertionsConfig): *this* **`deprecated`** Deprecated in favor of -[TableConfig.assertions](configs#dataform-ActionConfig-TableConfig). +[TableConfig.assertions](configs#sqlanvil-ActionConfig-TableConfig). Sets in-line assertions for this table. @@ -100,7 +100,7 @@ ___ ▸ **bigquery**(`bigquery`: IBigQueryOptions): *this* **`deprecated`** Deprecated in favor of options available directly on -[TableConfig](configs#dataform-ActionConfig-TableConfig). For example: +[TableConfig](configs#sqlanvil-ActionConfig-TableConfig). For example: `publish("name", { type: "table", partitionBy: "column" }`). Sets bigquery options for the action. @@ -120,7 +120,7 @@ ___ ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[TableConfig.columns](configs#dataform-ActionConfig-TableConfig). +[TableConfig.columns](configs#sqlanvil-ActionConfig-TableConfig). Sets the column descriptors of columns in this table. @@ -139,7 +139,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[TableConfig.project](configs#dataform-ActionConfig-TableConfig). +[TableConfig.project](configs#sqlanvil-ActionConfig-TableConfig). Sets the database (Google Cloud project ID) in which to create the output of this action. @@ -158,7 +158,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[TableConfig.dependencies](configs#dataform-ActionConfig-TableConfig). +[TableConfig.dependencies](configs#sqlanvil-ActionConfig-TableConfig). Sets dependencies of the table. @@ -177,7 +177,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[TableConfig.description](configs#dataform-ActionConfig-TableConfig). +[TableConfig.description](configs#sqlanvil-ActionConfig-TableConfig). Sets the description of this assertion. @@ -196,7 +196,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[TableConfig.disabled](configs#dataform-ActionConfig-TableConfig). +[TableConfig.disabled](configs#sqlanvil-ActionConfig-TableConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -216,7 +216,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[TableConfig.hermetic](configs#dataform-ActionConfig-TableConfig). +[TableConfig.hermetic](configs#sqlanvil-ActionConfig-TableConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -307,7 +307,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[TableConfig.dataset](configs#dataform-ActionConfig-TableConfig). +[TableConfig.dataset](configs#sqlanvil-ActionConfig-TableConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -326,7 +326,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[TableConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-TableConfig). +[TableConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-TableConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -346,7 +346,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[TableConfig.tags](configs#dataform-ActionConfig-TableConfig). +[TableConfig.tags](configs#sqlanvil-ActionConfig-TableConfig). Sets a list of user-defined tags applied to this action. diff --git a/docs/reference/test.md b/docs/reference/test.md index e8b45529..7952dd13 100644 --- a/docs/reference/test.md +++ b/docs/reference/test.md @@ -1,8 +1,8 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/test"](../modules/_core_actions_test_.md) › [Test](_core_actions_test_.test.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/test"](../modules/_core_actions_test_.md) › [Test](_core_actions_test_.test.md) # Class: Test -Dataform test actions can be used to write unit tests for your generated SQL +sqlanvil test actions can be used to write unit tests for your generated SQL You can create unit tests in the following ways. diff --git a/docs/reference/view.md b/docs/reference/view.md index 6d50b78b..f43ea1f6 100644 --- a/docs/reference/view.md +++ b/docs/reference/view.md @@ -1,4 +1,4 @@ -[Dataform Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/view"](../modules/_core_actions_view_.md) › [View](_core_actions_view_.view.md) +[sqlanvil Javascript API Reference](../README.md) › [Globals](../globals.md) › ["core/actions/view"](../modules/_core_actions_view_.md) › [View](_core_actions_view_.view.md) # Class: View @@ -7,7 +7,7 @@ to copy the original data to it, which can result in significant cost savings fo processing and storage. You can create views in the following ways. Available config options are defined in -[ViewConfig](configs#dataform-ActionConfig-ViewConfig), and are shared across all the +[ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig), and are shared across all the following ways of creating tables. **Using a SQLX file:** @@ -78,7 +78,7 @@ This is where `query` comes from. ▸ **assertions**(`assertions`: TableAssertionsConfig): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.assertions](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.assertions](configs#sqlanvil-ActionConfig-ViewConfig). Sets in-line assertions for this view. @@ -101,7 +101,7 @@ ___ ▸ **bigquery**(`bigquery`: IBigQueryOptions): *this* **`deprecated`** Deprecated in favor of options available directly on -[ViewConfig](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig](configs#sqlanvil-ActionConfig-ViewConfig). Sets bigquery options for the action. @@ -120,7 +120,7 @@ ___ ▸ **columns**(`columns`: ColumnDescriptor[]): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.columns](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.columns](configs#sqlanvil-ActionConfig-ViewConfig). Sets the column descriptors of columns in this view. @@ -139,7 +139,7 @@ ___ ▸ **database**(`database`: string): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.project](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.project](configs#sqlanvil-ActionConfig-ViewConfig). Sets the Sets the database (Google Cloud project ID) in which to create the output of this action. @@ -159,7 +159,7 @@ ___ ▸ **dependencies**(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.dependencies](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.dependencies](configs#sqlanvil-ActionConfig-ViewConfig). Sets dependencies of the view. @@ -178,7 +178,7 @@ ___ ▸ **description**(`description`: string): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.description](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.description](configs#sqlanvil-ActionConfig-ViewConfig). Sets the description of this view. @@ -197,7 +197,7 @@ ___ ▸ **disabled**(`disabled`: boolean): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.disabled](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.disabled](configs#sqlanvil-ActionConfig-ViewConfig). If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. @@ -217,7 +217,7 @@ ___ ▸ **hermetic**(`hermetic`: boolean): *void* **`deprecated`** Deprecated in favor of -[ViewConfig.hermetic](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.hermetic](configs#sqlanvil-ActionConfig-ViewConfig). If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source @@ -238,7 +238,7 @@ ___ ▸ **materialized**(`materialized`: boolean): *void* **`deprecated`** Deprecated in favor of -[ViewConfig.materialized](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.materialized](configs#sqlanvil-ActionConfig-ViewConfig). Applies the materialized view optimization, see https://cloud.google.com/bigquery/docs/materialized-views-intro. @@ -328,7 +328,7 @@ ___ ▸ **schema**(`schema`: string): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.dataset](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.dataset](configs#sqlanvil-ActionConfig-ViewConfig). Sets the schema (BigQuery dataset) in which to create the output of this action. @@ -347,7 +347,7 @@ ___ ▸ **setDependOnDependencyAssertions**(`dependOnDependencyAssertions`: boolean): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.dependOnDependencyAssertions](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.dependOnDependencyAssertions](configs#sqlanvil-ActionConfig-ViewConfig). When called with `true`, assertions dependent upon any dependency will be add as dedpendency to this action. @@ -367,7 +367,7 @@ ___ ▸ **tags**(`value`: string | string[]): *this* **`deprecated`** Deprecated in favor of -[ViewConfig.tags](configs#dataform-ActionConfig-ViewConfig). +[ViewConfig.tags](configs#sqlanvil-ActionConfig-ViewConfig). Sets a list of user-defined tags applied to this action. diff --git a/docs/rename_checklist.md b/docs/rename_checklist.md new file mode 100644 index 00000000..db0aaaca --- /dev/null +++ b/docs/rename_checklist.md @@ -0,0 +1,198 @@ +# Rename Checklist: `dataform` → `sqlanvil` + +**Status:** Draft / Spec +**Branch target:** `rename/dataform-to-sqlanvil` (first of three PRs per `postgres_first_class_design.md` §9) +**Driver:** Trademark risk from Google's "Dataform" product. + +## Conventions + +- **Package scope:** `@dataform/...` → `@sqlanvil/...` (keep scoped, just swap the scope owner). +- **Bazel workspace name:** `df` → `sa` (matches the existing 2-char convention). Implies tsconfig path `df/*` → `sa/*` and every import `from "df/core/..."` → `from "sa/core/..."`. +- **Proto package:** `dataform` → `sqlanvil`. +- **Proto Java package:** `com.dataform.protos` → `com.sqlanvil.protos` (cosmetic; no Java consumers in this fork). +- **Proto Go package:** `github.com/dataform-co/dataform/protos/dataform` → `github.com/ihistand/sqlanvil/protos/sqlanvil`. +- **Config file:** `dataform.json` is removed entirely (already deprecated upstream; clean break — no `sqlanvil.json` deprecated-fallback hybrid). `workflow_settings.yaml` is the only project config going forward; the key names are already neutral. +- **CLI binary:** `dataform` → `sqlanvil` (binary name and `./scripts/run` references). +- **NOT renamed (legitimate upstream references):** the git remote `upstream` pointing at `github.com/dataform-co/dataform`, attribution lines in LICENSE-equivalent files, historical commit messages. + +## Categorized Checklist + +### A. Proto schema (8 files) + +``` +protos/configs.proto +protos/core.proto +protos/db_adapter.proto +protos/evaluation.proto +protos/execution.proto +protos/extension.proto +protos/jit.proto +protos/profiles.proto +``` + +In each: + +- [ ] `package dataform;` → `package sqlanvil;` +- [ ] `option java_package = "com.dataform.protos";` → `option java_package = "com.sqlanvil.protos";` (where present) +- [ ] `option go_package = "github.com/dataform-co/dataform/protos/dataform";` → `option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil";` (where present) +- [ ] Field-level: `string dataform_core_version` → `string sqlanvil_core_version` (in `configs.proto` line 18 and `core.proto` line 414). **This is a wire-format-breaking change** — fine, since we're publishing fresh. +- [ ] Comment-level: `// The desired dataform core version to compile against.` → `// The desired sqlanvil core version to compile against.` +- [ ] URL comments pointing at `cloud.google.com/dataform/docs/...` — **keep** if they reference legitimate BigQuery/Dataform partitioning docs that are still factually correct, OR replace with sqlanvil's own docs once written. For the rename PR, keep them — annotate them as TODO for the Postgres-first-class doc work. + +### B. Bazel — WORKSPACE / BUILD / .bzl (every BUILD file) + +- [ ] `WORKSPACE` line 1: `workspace(name = "df")` → `workspace(name = "sa")` +- [ ] `protos/BUILD:7,27`: target `dataform_proto` → `sqlanvil_proto` (and the `:dataform_proto` reference in same file) +- [ ] Directory rename: `packages/@dataform/` → `packages/@sqlanvil/` (rename `packages/@dataform/cli` → `packages/@sqlanvil/cli`, same for `core`) +- [ ] Every `//packages/@dataform/core:package_tar` label → `//packages/@sqlanvil/core:package_tar` (15+ occurrences across BUILD files in `core/`, `cli/`, `cli/api/`, `examples/`, and every `tests/**/BUILD`) +- [ ] Every `//packages/@dataform/cli:package_tar` label → `//packages/@sqlanvil/cli:package_tar` (3+ occurrences) +- [ ] `packages/@dataform/core/BUILD:47`: `package_name = "@dataform/core"` → `package_name = "@sqlanvil/core"` +- [ ] `packages/@dataform/cli/BUILD:56`: `package_name = "@dataform/cli"` → `package_name = "@sqlanvil/cli"` +- [ ] `packages/sample-extension/BUILD:46`: `package_name = "@dataform/sample-extension"` → `package_name = "@sqlanvil/sample-extension"` +- [ ] `//packages/@dataform:package.layer.json` → `//packages/@sqlanvil:package.layer.json` (and rename the directory layer file) +- [ ] `BUILD` (root) line 55: `# gazelle:prefix github.com/dataform-co/dataform` → `# gazelle:prefix github.com/ihistand/sqlanvil` +- [ ] `vscode/BUILD:46`: `dataform_logo.png` reference (and the actual file rename — see §G) +- [ ] `test_credentials/BUILD:12-14`: GCP KMS keyring references (`dataform-open-source`, `dataform-builder-key`, `dataform-builder-keyring`) — these point at `dataform-co`'s GCP project which Ivan can't access. **Delete or replace** with his own GCP KMS setup. For the rename PR: delete `test_credentials/BUILD` entirely; integration tests against BQ will need fresh credentials wiring anyway. + +### C. tsconfig + TypeScript imports + +- [ ] `tsconfig.json` `paths`: `"df/*"` → `"sa/*"` (and the three array entries underneath — same key) +- [ ] All TS imports across `core/`, `cli/`, `tests/`, `testing/`, `tools/`, `packages/` of the form `from "df/..."` → `from "sa/..."`. Mechanical sed: + +```bash +find . -name '*.ts' -not -path '*/node_modules/*' -not -path './bazel-*' \ + -exec sed -i '' 's|from "df/|from "sa/|g; s|require("df/|require("sa/|g; s|import("df/|import("sa/|g' {} + +``` + +Estimated occurrences: hundreds. Verify after with `grep -r '"df/' --include='*.ts'` returns nothing outside generated bazel output. + +### D. npm package metadata + +- [ ] `package.json` (root): add `"name": "sqlanvil"` if absent (currently the file has no `name` field — confirms it's a workspace root, fine to leave nameless, but add `"private": true` + repo URL). +- [ ] Subdirectory `package.json` files for the published packages — generated from `tools/gen-package-json/` templates? Confirm whether the templates live there and update them. Otherwise update each: + - `packages/@sqlanvil/cli/package.json` (post-directory-rename) — `"name": "@sqlanvil/cli"`, `"bin": { "sqlanvil": "..." }` + - `packages/@sqlanvil/core/package.json` — `"name": "@sqlanvil/core"` + - `packages/sample-extension/package.json` — `"name": "@sqlanvil/sample-extension"` +- [ ] `tests/integration/bigquery_project/package.json` — drop or update `@dataform/core` dependency to `@sqlanvil/core`. +- [ ] `tests/api/projects/common_v2/package.json` — same. +- [ ] `tests/api/projects/never_finishes_compiling/package.json` — same. +- [ ] `tests/api/projects/invalid_dataform_json/package.json` — see §F (test deletion). +- [ ] `vscode/package.json` — VSCode extension manifest. Rename: `name`, `displayName`, `publisher`, command IDs (`dataform.*` → `sqlanvil.*`), activation events. + +### E. CLI binary + scripts + +- [ ] `scripts/run`: + ```bash + bazel build //packages/@dataform/cli:bin → //packages/@sqlanvil/cli:bin + ./bazel-bin/packages/@dataform/cli/bin.sh "$@" → ./bazel-bin/packages/@sqlanvil/cli/bin.sh "$@" + ``` +- [ ] CLI binary `name` in `packages/@sqlanvil/cli/BUILD` — confirm the binary target name; rename anything `dataform_bin` / `dataform_cli` → `sqlanvil_bin` / `sqlanvil_cli`. +- [ ] `cli/yargswrapper.ts` / `cli/index.ts` `scriptName` calls (yargs program name shown in `--help`) — search for `.scriptName("dataform")` → `.scriptName("sqlanvil")`. +- [ ] Help text strings referencing "Dataform" → "sqlanvil". + +### F. Drop the `dataform.json` legacy path + +The upstream code already comments `dataform.json` as deprecated. Clean-break decision: remove it entirely instead of accepting a `sqlanvil.json` parallel. + +- [ ] `cli/index.ts:59` — delete the `dataformJsonPath` resolution. +- [ ] `cli/vm/compile.ts:83` — delete `global.dataformJson = ...` line. +- [ ] `cli/api/commands/init.ts:26-27` — delete the dataform.json branch from project scaffolding. +- [ ] `core/workflow_settings.ts:15-20` — remove dataform.json fallback; `workflow_settings.yaml` is the only path. +- [ ] `testing/run_core.ts:113` — delete the dataformJson global injection. +- [ ] `core/main_test.ts` — delete tests at lines 545, 570, 606, 636, 803 (dataform.json validation tests). Keep the workflow_settings.yaml tests. +- [ ] `cli/index_compile_test.ts:47` — remove the dataform.json fixture usage from this test. +- [ ] Test fixture directory `tests/api/projects/invalid_dataform_json/` — delete entirely (its purpose was testing the legacy path). +- [ ] `tests/api/BUILD:9-10` — remove the `invalid_dataform_json` references. + +### G. Static assets / branding + +- [ ] `vscode/dataform_logo.png` → `vscode/sqlanvil_logo.png`. Replace the image asset itself before publishing the VSCode extension (Ivan needs to design/source one). For the rename PR, a placeholder is fine. +- [ ] `static/` directory — audit for any other branded assets (icons, banners). +- [ ] `LICENSE` — verify Apache-2.0 attribution is intact. Add a `NOTICE` file (Apache-2.0 §4 requirement when distributing derivative works): "sqlanvil is a derivative of Dataform, originally developed by Dataform Co and contributed to by Google LLC, licensed under Apache License 2.0." Required, not optional. + +### H. Documentation + +- [ ] `readme.md` (root) — currently empty per the `Dataform Core` indexed content (the README still has upstream's content). Rewrite for sqlanvil's positioning. +- [ ] `contributing.md` — update build/test instructions, replace `dataform` CLI references. +- [ ] `docs/configs-reference.md` — references `dataform` semantics. Update for sqlanvil and per the Postgres-first-class spec. +- [ ] `docs/packages.md` — package author guide; update `@dataform/...` patterns to `@sqlanvil/...`. +- [ ] `docs/reference/` — all existing reference docs (likely auto-generated; update generator templates rather than the output). +- [ ] `docs/postgres_reintegration_assessment.md` — mark "SUPERSEDED by postgres_first_class_design.md" at the top. +- [ ] `docs/hybrid_warehouses_supabase_bigquery.md` line 25 — `dataform.json` reference, update. +- [ ] `CLAUDE.md` — already names sqlanvil throughout; no change. + +### I. Test credentials + CI + +- [ ] `cloudbuild-publish.yaml`, `cloudbuild-test.yaml`, `cloudbuild-version.yaml` — these are upstream's Cloud Build configs targeting dataform-co's GCP project. Ivan can't run them. **Delete or replace** with GitHub Actions targeting his own infra. For the rename PR: delete. +- [ ] `test_credentials/bigquery.json` (referenced from `cli/BUILD` per contributing.md) — replace with Ivan's own GCP service account if he wants integration tests against a real BQ project; otherwise delete and remove the dependency from `cli/BUILD`. + +### J. Code-level identifiers + +- [ ] Class / interface / type names containing `Dataform`: + - `IDataformConfig` (if exists) → `ISqlanvilConfig` + - Any `DataformError`, `DataformProject`, etc. → `Sqlanvil...` + - Grep: `grep -rn 'class.*Dataform\|interface.*Dataform\|type.*Dataform\|enum.*Dataform' --include='*.ts'` +- [ ] Variable names: `dataformJson`, `dataformCoreVersion`, etc. Search-and-replace where the meaning is unambiguous. +- [ ] String literals in error messages: `"Dataform compilation failed"` → `"sqlanvil compilation failed"`. Search: `grep -rn '"[^"]*Dataform[^"]*"' --include='*.ts'`. +- [ ] User-agent strings sent to BigQuery / external services — change so server-side logs distinguish sqlanvil from upstream Dataform. + +### K. Repo metadata + +- [ ] `.gitignore` — no dataform references expected, but verify. +- [ ] GitHub repo description (set via `gh repo edit ihistand/sqlanvil --description ...`). +- [ ] GitHub repo topics: drop `dataform`, add `sqlanvil`, `postgres`, `supabase`, `bigquery`, `data-pipeline`. + +## Execution Order Within the Rename PR + +1. **Mechanical first** (low-risk, high-volume): tsconfig + every `from "df/` → `from "sa/"` import, all `//packages/@dataform/` → `//packages/@sqlanvil/` Bazel labels, proto package names. Run `bazel build //...` and `bazel test //...` after each to catch breakage early. +2. **Directory moves**: `packages/@dataform/` → `packages/@sqlanvil/`, `vscode/dataform_logo.png` → `vscode/sqlanvil_logo.png`. Commit as separate logical step. +3. **Removals**: `dataform.json` code path (§F), Cloud Build configs, `test_credentials/BUILD` if not replaced. +4. **Identifiers and strings**: class/interface renames, error messages, help text. +5. **Docs + LICENSE NOTICE**. +6. **Final sweep**: `grep -ri 'dataform' . --include='*' --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=bazel-*` — every remaining hit must be either (a) a legitimate upstream/attribution reference, (b) inside a comment URL pointing at cloud.google.com docs that's still factually valid, or (c) in this rename checklist itself. + +## Verification Commands + +After the rename PR is drafted, run all of these. Each must pass. + +```bash +# 1. Build +bazel build //... + +# 2. Test +bazel test //... + +# 3. CLI smoke test +./scripts/run help +./scripts/run init /tmp/sqlanvil-test +ls /tmp/sqlanvil-test/ # should contain workflow_settings.yaml, NOT dataform.json + +# 4. No stray references +grep -rn 'dataform' . \ + --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=bazel-out \ + --exclude-dir=bazel-bin --exclude-dir=bazel-testlogs --exclude-dir=bazel-sqlanvil \ + | grep -v 'docs/rename_checklist.md' \ + | grep -v 'upstream' \ + | grep -v 'NOTICE' \ + | grep -v 'cloud.google.com/dataform' + +# Expected: empty (or only allow-listed legitimate references) + +# 5. No `df/` import paths remain +grep -rn '"df/' --include='*.ts' --exclude-dir=node_modules --exclude-dir='bazel-*' . +# Expected: empty +``` + +## Out of Scope (Defer to Later PRs) + +- Postgres adapter implementation — that's PR 2 (`adapter/postgres-first-class`). +- Supabase variant — PR 3 (`adapter/supabase-variant`). +- New proto messages (`PostgresOptions`, `SupabaseOptions`, `WarehouseConfig`) — also PR 2. +- VSCode extension actual feature work — only the rename happens here. +- Marketing site (`sqlanvil-com/`) content rewrite — separate concern. + +## Risk Notes + +- **Upstream merges become harder after this PR.** Every cherry-pick from `upstream/main` will conflict on package names, imports, BUILD labels. Mitigation: pull all desired upstream changes first, merge cleanly, then start the rename. +- **The wire-format change to `dataform_core_version` → `sqlanvil_core_version`** means any compiled graph proto file from upstream Dataform won't deserialize. Acceptable since sqlanvil isn't claiming proto compatibility with upstream. +- **Bazel cache invalidates entirely** after directory moves. First post-rename build will be slow (cold cache). diff --git a/docs/rename_handoff.md b/docs/rename_handoff.md new file mode 100644 index 00000000..6c4ec0f1 --- /dev/null +++ b/docs/rename_handoff.md @@ -0,0 +1,152 @@ +# Rename Handoff + +Snapshot of the `rename/dataform-to-sqlanvil` branch state for picking +back up after a break. + +## Status + +**Logically complete + proto layer verified building.** Ready for review +and merge into `restore-postgres-adapter`. Larger Bazel targets blocked +by pre-existing upstream toolchain rot (separate from the rename — see +"Known pre-existing issues" below). + +## Where to pick up + +```bash +cd ~/projects-ivan/sqlanvil +git checkout rename/dataform-to-sqlanvil +git log --oneline restore-postgres-adapter..HEAD # see the 10 rename commits +``` + +## What was verified + +**Proto-layer build succeeded inside Docker:** + +```bash +./scripts/docker-bazel build //protos:sqlanvil_proto +# → Build completed successfully, 4 total actions +``` + +That proves: +- New WORKSPACE name `sa` (was `df`) is accepted by Bazel +- Target `//protos:sqlanvil_proto` (was `//protos:dataform_proto`) resolves +- All 8 `.proto` files parse with `package sqlanvil;` +- `java_package = "com.sqlanvil.protos"` + `go_package = "github.com/ihistand/sqlanvil/..."` accepted +- Wire-format-breaking field rename `sqlanvil_core_version` accepted +- `@com_google_protobuf//` external resolves and the C++ chain compiles + +## What was NOT verified (and why) + +| Target | Blocker | Type | +| :--- | :--- | :--- | +| `//protos:ts` | Node v24.13.0 pin in WORKSPACE has stale SHA256 — nodejs.org served a different binary | Pre-existing upstream pin drift | +| `//core/...` | Same node pin issue (transitive) | Pre-existing | +| `bazel test //...` | Same | Pre-existing | +| `./scripts/run help` (CLI smoke) | Same | Pre-existing | +| Native macOS build | Bazel 5.4 `wrapped_clang` lacks `LC_UUID` (rejected by Tahoe dyld); Bazel 6.x WORKSPACE `@bazel_tools//platforms` refactor breaks too | Pre-existing macOS+toolchain rot | + +All five would fail identically on `upstream/main` — not caused by this +PR. + +## Commits on the branch + +``` +63685541 fix(protos): strip_import_prefix + add timestamp_proto dep +05e2... build: Dockerfile.dev + scripts/docker-bazel for macOS users +ce6af2eb refactor: final rename sweep — proto auto-docs + goldens + examples + dead infra +31adafa6 refactor: rename interface ID + write NOTICE + rewrite root docs +9809e459 refactor: rename CLI binary + help text + VSCode extension +1399c23d refactor: drop dataform.json legacy config path (clean break) +513e6963 refactor: rename TS imports df/ → sa/ + proto namespace + framework defaults +2148bb3e refactor: rename packages/@dataform → @sqlanvil + Bazel labels + dead infra +f084b0ae refactor: rename proto packages dataform → sqlanvil + WORKSPACE +b4b24cd6 docs: rename plan + postgres-first-class design +``` + +Total: ~150 files changed, ~2400 lines net (mostly mechanical renames). + +## Known pre-existing issues (NOT introduced by this PR, but blocking +verification beyond proto layer) + +### 1. Node v24.13.0 SHA256 mismatch +`WORKSPACE` pins `node-v24.13.0-linux-arm64.tar.xz` with checksum +`e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e`. +nodejs.org currently serves a binary with checksum +`aa881151bd0f9f154a0424dd60a72e9ce10672619121658c278a24327ef46831`. +Fix: bump pin to a current Node release (probably 20 LTS to match the +Docker image, or 22 LTS) and regenerate SHA. Likely also needs matching +amd64 SHA. + +### 2. Bazel toolchain modernization needed for native macOS builds +The pinned Bazel 5.4 was released Dec 2022. Its `wrapped_clang` shim +binary doesn't include `LC_UUID` load commands, which macOS Tahoe's +dyld rejects. Bumping to Bazel 6+ hits a different problem +(`@bazel_tools//platforms` removed; WORKSPACE references it). + +Real fix is multi-day work: bump to Bazel 7 LTS, migrate WORKSPACE to +`MODULE.bazel` (Bzlmod), update `rules_proto`/`rules_nodejs`/`protobuf` +versions to match. + +### 3. proto_library bugs in upstream Dataform too +The `import "extension.proto"` (no `protos/` prefix) and missing +`timestamp_proto` dep were both present in `upstream/main`. Likely +masked by upstream CI only running through `ts_proto_library` rather +than building the proto_library target directly. + +Fixed in this PR (commit `63685541`). + +## Next steps (in priority order) + +1. **Open a PR** for the rename branch against `restore-postgres-adapter`. + Use the commit summaries as the PR body. Mark "verified at proto layer + via Docker, blocked on toolchain rot for fuller verification." + +2. **Small follow-up PR: refresh stale toolchain pins.** + - Update WORKSPACE node version (v24.13.0 → v20.x or v22.x LTS) with + matching SHA256 hashes for arm64 + amd64 + - This should unblock `//protos:ts`, then `//core/...`, then full tree + - Scope: 1-2 hour PR. Mechanical. + +3. **Then start the Postgres adapter PR** as planned in + `docs/postgres_first_class_design.md` §9 (PR 2: + `adapter/postgres-first-class`). + +4. **Defer until needed: Bazel 7 + Bzlmod modernization.** Would unblock + native macOS builds, but Docker dev container is functional for now. + Multi-day PR. Sequence after Postgres adapter is at least partially + working, to avoid stacking too much risky change. + +## Quick-reference commands + +```bash +# Switch to the branch +git checkout rename/dataform-to-sqlanvil + +# View what changed +git diff restore-postgres-adapter..HEAD --stat | tail -5 + +# Build inside Docker (first run ~5 min, subsequent <1 min) +./scripts/docker-bazel build //protos:sqlanvil_proto + +# Interactive shell inside the dev container +./scripts/docker-bazel + +# When Node pin gets fixed in WORKSPACE, try: +./scripts/docker-bazel build //protos:ts +./scripts/docker-bazel build //core/... +./scripts/docker-bazel run //packages/@sqlanvil/cli:bin -- help +``` + +## Files worth re-reading first thing next session + +- `docs/rename_checklist.md` — the original surface map +- `docs/postgres_first_class_design.md` — what comes after this rename +- `CLAUDE.md` — design directives + active-work context +- `Dockerfile.dev` + `scripts/docker-bazel` — verification workflow + +## Memory state (in `~/.claude/projects/-Users-ivan-projects-ivan/memory/`) + +- `user_dataform_expertise.md` — Ivan is experienced Dataform dev, skip 101 +- `project_sqlanvil_postgres_design.md` — first-class Postgres + Supabase, + nested config, rename mandatory, listanvil is target user +- `MEMORY.md` indexes them diff --git a/examples/BUILD b/examples/BUILD index 95e33682..b0c63fab 100644 --- a/examples/BUILD +++ b/examples/BUILD @@ -22,8 +22,7 @@ ts_test_suite( data = [ ":all_projects", ":node_modules", - "//packages/@dataform/core:package_tar", - "//test_credentials:bigquery.json", + "//packages/@sqlanvil/core:package_tar", "@nodejs//:node", ], deps = [ @@ -44,7 +43,7 @@ node_modules( deps = [ # The CLI bundle remains at this directory level's node_modules, whereas the core package # is copied into each example project's node_modules during testing. - "//packages/@dataform/cli:package_tar", - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/cli:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/examples/examples_test.ts b/examples/examples_test.ts index 5465e8f8..0d11906b 100644 --- a/examples/examples_test.ts +++ b/examples/examples_test.ts @@ -2,12 +2,12 @@ import { expect } from "chai"; import { execFile } from "child_process"; import * as fs from "fs-extra"; -import { verifyObjectMatchesProto } from "df/common/protos"; -import { dataform } from "df/protos/ts"; -import { getProcessResult, nodePath, suite, test } from "df/testing"; +import { verifyObjectMatchesProto } from "sa/common/protos"; +import { sqlanvil } from "sa/protos/ts"; +import { getProcessResult, nodePath, suite, test } from "sa/testing"; suite("examples", { parallel: true }, () => { - const cliEntryPointPath = "examples/node_modules/@dataform/cli/bundle.js"; + const cliEntryPointPath = "examples/node_modules/@sqlanvil/cli/bundle.js"; ["stackoverflow_reporter", "extreme_weather_programming"].forEach(exampleProject => { test(`${exampleProject} runs`, async () => { @@ -19,10 +19,10 @@ suite("examples", { parallel: true }, () => { const projectDir = `examples/${exampleProject}_copy`; fs.copySync(originalProjectDir, projectDir, { dereference: true }); fs.copySync( - "examples/node_modules/@dataform/core", - `${projectDir}/node_modules/@dataform/core` + "examples/node_modules/@sqlanvil/core", + `${projectDir}/node_modules/@sqlanvil/core` ); - // A blank `package.json` makes no `dataformCoreVersion` in `workflow_settings.yaml` be OK. + // A blank `package.json` makes no `sqlanvilCoreVersion` in `workflow_settings.yaml` be OK. // tslint:disable-next-line: tsr-detect-non-literal-fs-filename fs.writeFileSync(`${projectDir}/package.json`, ""); @@ -32,7 +32,7 @@ suite("examples", { parallel: true }, () => { expect(processResult.exitCode).equals(0); const compiledGraph = verifyObjectMatchesProto( - dataform.CompiledGraph, + sqlanvil.CompiledGraph, JSON.parse(processResult.stdout) ); expect(compiledGraph.graphErrors).deep.equals({}); diff --git a/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql b/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql index d8b3b795..41631296 100644 --- a/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql +++ b/examples/extreme_weather_programming/definitions/repositories_created_during_extreme_weather.sql @@ -1,7 +1,7 @@ SELECT * FROM - `dataform-open-source.was_there_extreme_weather` - LEFT OUTER JOIN `dataform-open-source.repositories_that_mention_extreme_weather` USING (date) + `your-bigquery-project.was_there_extreme_weather` + LEFT OUTER JOIN `your-bigquery-project.repositories_that_mention_extreme_weather` USING (date) ORDER BY date diff --git a/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb b/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb index a69e6ac4..0ae49ec8 100644 --- a/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb +++ b/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb @@ -22,8 +22,8 @@ "metadata": {}, "outputs": [], "source": [ - "%%bigquery results --project dataform-open-source\n", - "SELECT * FROM `dataform-open-source.dataform_examples.repositories_created_during_extreme_weather`" + "%%bigquery results --project your-bigquery-project\n", + "SELECT * FROM `your-bigquery-project.your_examples_dataset.repositories_created_during_extreme_weather`" ] }, { diff --git a/examples/extreme_weather_programming/workflow_settings.yaml b/examples/extreme_weather_programming/workflow_settings.yaml index fb2e7a09..6636df55 100644 --- a/examples/extreme_weather_programming/workflow_settings.yaml +++ b/examples/extreme_weather_programming/workflow_settings.yaml @@ -1,7 +1,7 @@ -defaultProject: dataform-open-source +defaultProject: your-bigquery-project defaultLocation: us -defaultDataset: dataform_extreme_weather_programming_example -defaultAssertionDataset: dataform_extreme_weather_programming_example_assertions +defaultDataset: sqlanvil_extreme_weather_programming_example +defaultAssertionDataset: sqlanvil_extreme_weather_programming_example_assertions defaultNotebookRuntimeOptions: outputBucket: gs://some-bucket runtimeTemplateName: projects/test-project/locations/us-central1/notebookRuntimeTemplates/test-template diff --git a/examples/stackoverflow_reporter/workflow_settings.yaml b/examples/stackoverflow_reporter/workflow_settings.yaml index df567eb7..8d911ed2 100644 --- a/examples/stackoverflow_reporter/workflow_settings.yaml +++ b/examples/stackoverflow_reporter/workflow_settings.yaml @@ -1,4 +1,4 @@ -defaultProject: dataform-open-source +defaultProject: your-bigquery-project defaultLocation: us -defaultDataset: dataform_stackoverflow_reporter_example -defaultAssertionDataset: dataform_stackoverflow_reporter_example_assertions +defaultDataset: sqlanvil_stackoverflow_reporter_example +defaultAssertionDataset: sqlanvil_stackoverflow_reporter_example_assertions diff --git a/package.json b/package.json index 44b58b9a..e3df4aaf 100644 --- a/package.json +++ b/package.json @@ -19,11 +19,13 @@ "@types/long": "^4.0.0", "@types/moo": "^0.5.0", "@types/node": "^16.16.0", + "@types/pg": "^8.11.0", "@types/readline-sync": "^1.4.3", "@types/request": "^2.48.3", "@types/rimraf": "^2.0.2", "@types/semver": "^7.3.13", "@types/tmp": "^0.2.0", + "@types/uuid": "^9.0.0", "@types/vscode": "^1.45.1", "@types/yargs": "^15.0.5", "chai": "^4.2.0", @@ -46,6 +48,8 @@ "moo": "^0.5.0", "object-sizeof": "^1.6.1", "parse-duration": "^1.0.0", + "pg": "^8.11.3", + "pg-query-stream": "^4.5.3", "prettier": "^1.14.2", "promise-pool-executor": "^1.1.1", "protobufjs": "^7.5.5", @@ -72,6 +76,7 @@ "uglify-js": "^3.7.7", "untildify": "^4.0.0", "url": "^0.11.0", + "uuid": "^9.0.0", "vm2": "3.11.3", "vsce": "^1.79.5", "vscode-jsonrpc": "^5.0.1", diff --git a/packages/@dataform/cli/index.ts b/packages/@dataform/cli/index.ts deleted file mode 100644 index d3263aa8..00000000 --- a/packages/@dataform/cli/index.ts +++ /dev/null @@ -1,2 +0,0 @@ -import { runCli } from "df/cli"; -runCli(); diff --git a/packages/@dataform/cli/package.layer.json b/packages/@dataform/cli/package.layer.json deleted file mode 100644 index 3d9c7062..00000000 --- a/packages/@dataform/cli/package.layer.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "bin": { - "dataform": "bundle.js" - } -} diff --git a/packages/@dataform/cli/worker.ts b/packages/@dataform/cli/worker.ts deleted file mode 100644 index bc5f2930..00000000 --- a/packages/@dataform/cli/worker.ts +++ /dev/null @@ -1,2 +0,0 @@ -import { listenForCompileRequest } from "df/cli/vm/compile"; -listenForCompileRequest(); diff --git a/packages/@dataform/core/index.ts b/packages/@dataform/core/index.ts deleted file mode 100644 index 2c146b60..00000000 --- a/packages/@dataform/core/index.ts +++ /dev/null @@ -1 +0,0 @@ -export { compiler, IDataformExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version } from "df/core"; diff --git a/packages/@dataform/BUILD b/packages/@sqlanvil/BUILD similarity index 100% rename from packages/@dataform/BUILD rename to packages/@sqlanvil/BUILD diff --git a/packages/@dataform/cli/BUILD b/packages/@sqlanvil/cli/BUILD similarity index 90% rename from packages/@dataform/cli/BUILD rename to packages/@sqlanvil/cli/BUILD index 7a1d7085..35b6f2ac 100644 --- a/packages/@dataform/cli/BUILD +++ b/packages/@sqlanvil/cli/BUILD @@ -53,13 +53,13 @@ externals = [ pkg_json( name = "json", - package_name = "@dataform/cli", - description = "Dataform command line interface.", + package_name = "@sqlanvil/cli", + description = "sqlanvil command line interface.", external_deps = externals, layers = [ "//:package.json", - "//packages/@dataform:package.layer.json", - "//packages/@dataform/cli:package.layer.json", + "//packages/@sqlanvil:package.layer.json", + "//packages/@sqlanvil/cli:package.layer.json", ], main = "bundle.js", version = DF_VERSION, diff --git a/packages/@sqlanvil/cli/index.ts b/packages/@sqlanvil/cli/index.ts new file mode 100644 index 00000000..09228d6f --- /dev/null +++ b/packages/@sqlanvil/cli/index.ts @@ -0,0 +1,2 @@ +import { runCli } from "sa/cli"; +runCli(); diff --git a/packages/@sqlanvil/cli/package.layer.json b/packages/@sqlanvil/cli/package.layer.json new file mode 100644 index 00000000..503898b2 --- /dev/null +++ b/packages/@sqlanvil/cli/package.layer.json @@ -0,0 +1,5 @@ +{ + "bin": { + "sqlanvil": "bundle.js" + } +} diff --git a/packages/@sqlanvil/cli/worker.ts b/packages/@sqlanvil/cli/worker.ts new file mode 100644 index 00000000..4a1b5e3c --- /dev/null +++ b/packages/@sqlanvil/cli/worker.ts @@ -0,0 +1,2 @@ +import { listenForCompileRequest } from "sa/cli/vm/compile"; +listenForCompileRequest(); diff --git a/packages/@dataform/core/BUILD b/packages/@sqlanvil/core/BUILD similarity index 95% rename from packages/@dataform/core/BUILD rename to packages/@sqlanvil/core/BUILD index 15edc71c..6709290e 100644 --- a/packages/@dataform/core/BUILD +++ b/packages/@sqlanvil/core/BUILD @@ -44,12 +44,12 @@ npm_package_bin( pkg_json( name = "json", - package_name = "@dataform/core", - description = "Dataform core API.", + package_name = "@sqlanvil/core", + description = "sqlanvil core API.", external_deps = [], layers = [ "//:package.json", - "//packages/@dataform:package.layer.json", + "//packages/@sqlanvil:package.layer.json", ], main = "bundle.js", version = DF_VERSION, diff --git a/packages/@sqlanvil/core/index.ts b/packages/@sqlanvil/core/index.ts new file mode 100644 index 00000000..1658df9e --- /dev/null +++ b/packages/@sqlanvil/core/index.ts @@ -0,0 +1 @@ +export { compiler, ISqlanvilExtension, indexFileGenerator, IJitCompiler, jitCompiler, main, session, supportedFeatures, version } from "sa/core"; diff --git a/packages/@dataform/core/webpack.config.js b/packages/@sqlanvil/core/webpack.config.js similarity index 79% rename from packages/@dataform/core/webpack.config.js rename to packages/@sqlanvil/core/webpack.config.js index 2e389a04..ec7f9e2e 100644 --- a/packages/@dataform/core/webpack.config.js +++ b/packages/@sqlanvil/core/webpack.config.js @@ -5,7 +5,7 @@ module.exports = (env, argv) => { const config = { mode: argv.mode || "development", target: 'node', - entry: [path.resolve(process.env.RUNFILES, "df/packages/@dataform/core/index")], + entry: [path.resolve(process.env.RUNFILES, "sa/packages/@sqlanvil/core/index")], output: { libraryTarget: "commonjs-module", }, @@ -18,7 +18,7 @@ module.exports = (env, argv) => { resolve: { extensions: [".ts", ".js", ".json"], alias: { - df: path.resolve(process.env.RUNFILES, "df") + sa: path.resolve(process.env.RUNFILES, "sa") } }, plugins: [ diff --git a/packages/@dataform/package.layer.json b/packages/@sqlanvil/package.layer.json similarity index 74% rename from packages/@dataform/package.layer.json rename to packages/@sqlanvil/package.layer.json index 74d87690..5021cff9 100644 --- a/packages/@dataform/package.layer.json +++ b/packages/@sqlanvil/package.layer.json @@ -1,8 +1,8 @@ { - "homepage": "https://github.com/dataform-co/dataform", + "homepage": "https://github.com/ihistand/sqlanvil", "license": "Apache-2.0", "keywords": [ - "dataform", + "sqlanvil", "etl", "data-pipeline", "big-data", diff --git a/packages/rollup.config.js b/packages/rollup.config.js index 3ac33cde..5d72077f 100644 --- a/packages/rollup.config.js +++ b/packages/rollup.config.js @@ -23,7 +23,7 @@ const knownNodeBuiltins = [ "net" ].map(moduleName => convertToRegex(moduleName)); -const importsToBundle = ["df", /df\/.*$/, /^bazel\-.*$/]; +const importsToBundle = ["sa", /sa\/.*$/, /^bazel\-.*$/]; const checkImports = imports => { const allowedImports = [...imports].map(pattern => convertToRegex(pattern)); diff --git a/packages/sample-extension/BUILD b/packages/sample-extension/BUILD index b99f7f1b..e5b6ce30 100644 --- a/packages/sample-extension/BUILD +++ b/packages/sample-extension/BUILD @@ -43,12 +43,12 @@ npm_package_bin( pkg_json( name = "json", - package_name = "@dataform/sample-extension", + package_name = "@sqlanvil/sample-extension", description = "Sample extension.", external_deps = [], layers = [ "//:package.json", - "//packages/@dataform:package.layer.json", + "//packages/@sqlanvil:package.layer.json", ], main = "bundle.js", version = DF_VERSION, diff --git a/packages/sample-extension/index.ts b/packages/sample-extension/index.ts index 4ace4404..64421f9a 100644 --- a/packages/sample-extension/index.ts +++ b/packages/sample-extension/index.ts @@ -1,9 +1,9 @@ -import type { IDataformExtension } from "df/core"; -import { Session } from "df/core/session"; -import { dataform } from "df/protos/ts"; +import type { ISqlanvilExtension } from "sa/core"; +import { Session } from "sa/core/session"; +import { sqlanvil } from "sa/protos/ts"; -class SampleExtension implements IDataformExtension { - public compile(request: dataform.ICompileExecutionRequest, session: Session): void { +class SampleExtension implements ISqlanvilExtension { + public compile(request: sqlanvil.ICompileExecutionRequest, session: Session): void { if (request.compileConfig?.projectConfigOverride?.vars["throw-error"] === "true") { throw new Error("throwing exception as requested!"); } @@ -16,6 +16,6 @@ class SampleExtension implements IDataformExtension { } } -export function extension(): IDataformExtension { +export function extension(): ISqlanvilExtension { return new SampleExtension(); } diff --git a/packages/sample-extension/webpack.config.js b/packages/sample-extension/webpack.config.js index c659abfb..e0ddca64 100644 --- a/packages/sample-extension/webpack.config.js +++ b/packages/sample-extension/webpack.config.js @@ -5,7 +5,7 @@ module.exports = (env, argv) => { const config = { mode: argv.mode || "development", target: 'node', - entry: [path.resolve(process.env.RUNFILES, "df/packages/sample-extension/index")], + entry: [path.resolve(process.env.RUNFILES, "sa/packages/sample-extension/index")], output: { libraryTarget: "commonjs-module", }, @@ -18,7 +18,7 @@ module.exports = (env, argv) => { resolve: { extensions: [".ts", ".js", ".json"], alias: { - df: path.resolve(process.env.RUNFILES, "df") + sa: path.resolve(process.env.RUNFILES, "sa") } }, plugins: [ diff --git a/protos/BUILD b/protos/BUILD index 4466c887..808fe19e 100644 --- a/protos/BUILD +++ b/protos/BUILD @@ -4,7 +4,7 @@ load("//tools:ts_proto_library.bzl", "ts_proto_library") package(default_visibility = ["//visibility:public"]) proto_library( - name = "dataform_proto", + name = "sqlanvil_proto", srcs = [ "configs.proto", "core.proto", @@ -15,15 +15,17 @@ proto_library( "profiles.proto", "extension.proto", ], + strip_import_prefix = "/protos", deps = [ "@com_google_protobuf//:empty_proto", "@com_google_protobuf//:struct_proto", + "@com_google_protobuf//:timestamp_proto", ], ) ts_proto_library( name = "ts", deps = [ - ":dataform_proto", + ":sqlanvil_proto", ], ) diff --git a/protos/configs.proto b/protos/configs.proto index ad8668c9..e50b38af 100644 --- a/protos/configs.proto +++ b/protos/configs.proto @@ -1,12 +1,12 @@ syntax = "proto3"; -package dataform; +package sqlanvil; -option java_package = "com.dataform.protos"; +option java_package = "com.sqlanvil.protos"; option java_outer_classname = "ConfigsMeta"; option java_multiple_files = true; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; import "google/protobuf/struct.proto"; import "extension.proto"; @@ -14,8 +14,8 @@ import "extension.proto"; // Workflow Settings defines the contents of the `workflow_settings.yaml` // configuration file. message WorkflowSettings { - // The desired dataform core version to compile against. - string dataform_core_version = 1; + // The desired sqlanvil core version to compile against. + string sqlanvil_core_version = 1; // Required. The default Google Cloud project (database). string default_project = 2; @@ -59,7 +59,7 @@ message WorkflowSettings { // Optional. The default BigQuery reservation to use for execution. // If unset, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string default_reservation = 14; // Optional. An external package that provides an extension. @@ -278,7 +278,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 24; } @@ -366,7 +366,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 21; } @@ -418,7 +418,7 @@ message ActionConfig { // If set, unique key represents a set of names of columns that will act as // a the unique key. To enforce this, when updating the incremental - // table, Dataform merges rows with `uniqueKey` instead of appending them. + // table, sqlanvil merges rows with `uniqueKey` instead of appending them. repeated string unique_key = 11; // Description of the incremental table. @@ -483,7 +483,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 27; } @@ -526,7 +526,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 11; } @@ -578,7 +578,7 @@ message ActionConfig { // Optional. The BigQuery reservation to use for execution. // If unset, the value from workflow_settings.yaml is used. If neither is set, default BigQuery behavior applies. - // Dataform CLI only (GCP Dataform support pending). + // sqlanvil CLI only (GCP sqlanvil support pending). string reservation = 13; } @@ -744,3 +744,188 @@ message NotebookRuntimeOptionsConfig { } } + +// ============================================================================= +// Postgres-first-class adapter — additions per +// docs/postgres_first_class_design.md. Phase 3c. +// +// These messages are declared here for wiring in subsequent phases: +// - PostgresOptions / SupabaseOptions → ActionConfig table-level blocks +// - PostgresConnection / SupabaseConnection / BigQueryConnection +// → WorkflowSettings.warehouse +// - WarehouseConfig → discriminated union over the +// connection variants +// +// Adding the messages alone does not change behavior — wiring happens in +// Phase 4 (CLI) and the parallel rewrite of ActionConfig's table/view/ +// incremental_table sub-messages. +// ============================================================================= + +// PostgresOptions — Postgres-native table-level options. Mirrors what +// BigQueryOptions-style fields do in TableConfig but in idiomatic Postgres. +// +// Used as a peer of the existing `bigquery: {...}` shape on action configs: +// publish("daily_orders", { postgres: { tablespace: "fast_ssd", ... } }) +message PostgresOptions { + // Physical storage placement (CREATE TABLE ... TABLESPACE ). + string tablespace = 1; + + // Storage parameter — fraction of each page to fill on insert (1-100). + uint32 fillfactor = 2; + + // CREATE UNLOGGED TABLE — faster writes, lost on crash. For staging/temp + // tables where durability isn't required. + bool unlogged = 3; + + // Native Postgres declarative partitioning. + message Partition { + enum Kind { + RANGE = 0; + LIST = 1; + HASH = 2; + } + Kind kind = 1; + repeated string columns = 2; + } + Partition partition = 4; + + // Indexes to create alongside the table. + message Index { + string name = 1; + repeated string columns = 2; + + enum Method { + BTREE = 0; + HASH = 1; + GIN = 2; + GIST = 3; + BRIN = 4; + } + Method method = 3; + + // Partial index predicate (WHERE ). + string where = 4; + + bool unique = 5; + + // INCLUDE non-key columns for covering indexes. + repeated string include = 6; + } + repeated Index indexes = 5; + + // Materialized view: WITH DATA vs WITH NO DATA on initial creation. + bool with_data = 6; + + // Materialized view refresh policy. "manual" | "on_dependency_change". + string refresh_policy = 7; +} + +// SupabaseOptions — Supabase-specific platform features layered on top of +// standard Postgres. Used as a peer of `postgres: {...}` for projects +// targeting `warehouse: { kind: supabase }`. +message SupabaseOptions { + // Standard Postgres options apply. Set these via `postgres:` directly or + // nest under `supabase.postgres:` — either is accepted. + PostgresOptions postgres = 1; + + // ALTER PUBLICATION supabase_realtime ADD TABLE . + // Implicitly sets REPLICA IDENTITY appropriately. + bool publish_to_realtime = 2; + + // ALTER TABLE ENABLE ROW LEVEL SECURITY. + // Note: only enables RLS — policies are declared via the `rlsPolicy` + // action type (see Phase 5). + bool enable_rls = 3; + + // OWNER TO . Typically "postgres" or "service_role". + string owner_role = 4; + + // pgvector convenience config. Equivalent to declaring a + // PostgresOptions.Index with method=HNSW or method=GIST + ivfflat ops, + // but more ergonomic for RAG pipelines. + message VectorConfig { + string column = 1; + uint32 dimensions = 2; + + enum IndexType { + IVFFLAT = 0; + HNSW = 1; + } + IndexType index_type = 3; + + // ivfflat: { lists }, hnsw: { m, ef_construction }. + map params = 4; + } + repeated VectorConfig vectors = 5; +} + +// BigQueryConnection — connection params for warehouse.kind = "bigquery". +// Mirrors the legacy flat fields on WorkflowSettings (default_project, +// default_location, default_dataset) but namespaced under warehouse. +message BigQueryConnection { + // The Google Cloud project (database). + string project = 1; + + // BigQuery location, e.g. "US", "EU", "europe-west4". + string location = 2; + + // Default dataset (schema). + string default_dataset = 3; +} + +// PostgresConnection — libpq-style connection params for +// warehouse.kind = "postgres". Standard Postgres host/port/database/user. +message PostgresConnection { + string host = 1; + uint32 port = 2; + string database = 3; + string user = 4; + string password = 5; + + // SSL mode: "disable" | "allow" | "prefer" | "require" | "verify-ca" + // | "verify-full". See https://www.postgresql.org/docs/current/libpq-ssl.html. + string ssl_mode = 6; + + string default_schema = 7; +} + +// SupabaseConnection — connection params for warehouse.kind = "supabase". +// Supabase projects expose a Postgres connection via project_ref + +// service_role_key, or a direct connection string for bypassing PostgREST. +message SupabaseConnection { + // From the Supabase dashboard (project URL host before .supabase.co). + string project_ref = 1; + + // Project service-role JWT. NEVER commit literally — use ${ENV_VAR} + // interpolation in workflow_settings.yaml. + string service_role_key = 2; + + string default_schema = 3; + + // Optional override — direct Postgres URL bypassing the PostgREST proxy. + // e.g. "postgresql://postgres:${PASSWORD}@db..supabase.co:5432/postgres". + // If set, takes precedence over project_ref + service_role_key for the + // direct DB connection. service_role_key is still used for RLS bypass. + string connection_string = 4; +} + +// WarehouseConfig — discriminated union over connection variants. The +// `kind:` YAML tag selects which `oneof` arm is unmarshalled. +// +// Example YAML: +// warehouse: +// kind: postgres +// host: db.example.com +// port: 5432 +// database: analytics +// user: sqlanvil_writer +// password: ${PG_PASSWORD} +// ssl_mode: require +// default_schema: public +message WarehouseConfig { + oneof connection { + BigQueryConnection bigquery = 1; + PostgresConnection postgres = 2; + SupabaseConnection supabase = 3; + } +} diff --git a/protos/core.proto b/protos/core.proto index 66333201..b1f6c7f0 100644 --- a/protos/core.proto +++ b/protos/core.proto @@ -1,16 +1,16 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "configs.proto"; import "extension.proto"; import "google/protobuf/struct.proto"; -option java_package = "com.dataform.protos"; +option java_package = "com.sqlanvil.protos"; option java_outer_classname = "CoreMeta"; option java_multiple_files = true; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message ProjectConfig { string warehouse = 1; @@ -111,7 +111,7 @@ message CompilationError { // Compilation mode, unspecified is interpreted to AoT. enum ActionCompilationMode { ACTION_COMPILATION_MODE_UNSPECIFIED = 0; - // Ahead-of-time compilation (regular Dataform compilation). + // Ahead-of-time compilation (regular sqlanvil compilation). ACTION_COMPILATION_MODE_AOT = 1; // Just-in-time compilation. // Will only populate jit_code fields in compiled graph, @@ -411,7 +411,7 @@ message CompiledGraph { GraphErrors graph_errors = 7; - string dataform_core_version = 10; + string sqlanvil_core_version = 10; repeated Target targets = 11; @@ -441,7 +441,7 @@ message CompileExecutionResponse { } // This feature list is added to when making potentilly backwards breaking -// changes. It lets the caller of Dataform Core know whether it supports the +// changes. It lets the caller of sqlanvil Core know whether it supports the // change. enum SupportedFeatures { UNKNOWN_FEATURE = 0; diff --git a/protos/db_adapter.proto b/protos/db_adapter.proto index 3760c672..c4cf49fc 100644 --- a/protos/db_adapter.proto +++ b/protos/db_adapter.proto @@ -1,6 +1,6 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "core.proto"; diff --git a/protos/evaluation.proto b/protos/evaluation.proto index 9b7dafe7..6748e34d 100644 --- a/protos/evaluation.proto +++ b/protos/evaluation.proto @@ -1,8 +1,8 @@ syntax="proto3"; -package dataform; +package sqlanvil; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message QueryEvaluationError { string message = 1; diff --git a/protos/execution.proto b/protos/execution.proto index b7578d17..d6e9c4d8 100644 --- a/protos/execution.proto +++ b/protos/execution.proto @@ -1,12 +1,12 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "core.proto"; import "db_adapter.proto"; import "google/protobuf/struct.proto"; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message RunConfig { repeated string actions = 1; diff --git a/protos/extension.proto b/protos/extension.proto index 8284bd95..b6a52042 100644 --- a/protos/extension.proto +++ b/protos/extension.proto @@ -1,20 +1,20 @@ syntax = "proto3"; -package dataform; +package sqlanvil; -option java_package = "com.dataform.protos"; +option java_package = "com.sqlanvil.protos"; option java_outer_classname = "ExtensionMeta"; option java_multiple_files = true; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; // Execution mode of compilation extension. enum ExtensionCompilationMode { // Unspecified compiled mode - extension will be skipped. COMPILATION_MODE_UNSPECIFIED = 0; - // Execute before regular Dataform compilation. + // Execute before regular sqlanvil compilation. PROLOGUE = 1; - // Replace regular Dataform compilation. + // Replace regular sqlanvil compilation. APPLICATION_CODE = 2; } diff --git a/protos/jit.proto b/protos/jit.proto index e86e6ba2..ad01ba24 100644 --- a/protos/jit.proto +++ b/protos/jit.proto @@ -1,6 +1,6 @@ syntax = "proto3"; -package dataform; +package sqlanvil; import "google/protobuf/empty.proto"; import "google/protobuf/struct.proto"; diff --git a/protos/profiles.proto b/protos/profiles.proto index 09eabcdb..f988dd43 100644 --- a/protos/profiles.proto +++ b/protos/profiles.proto @@ -1,8 +1,8 @@ syntax = "proto3"; -package dataform; +package sqlanvil; -option go_package = "github.com/dataform-co/dataform/protos/dataform"; +option go_package = "github.com/ihistand/sqlanvil/protos/sqlanvil"; message BigQuery { string project_id = 1; diff --git a/readme.md b/readme.md index 9c23579c..b1f0d0e3 100644 --- a/readme.md +++ b/readme.md @@ -1,60 +1,39 @@ -# Dataform Core +# sqlanvil -Dataform Core is an open source meta-language to create SQL tables and workflows in BigQuery. Dataform Core extends SQL by providing a dependency management system, automated data quality testing, and data documentation. +**sqlanvil** is an open-source SQL workflow tool. Define dependent tables, views, materialized views, incremental tables, assertions, and operations as code — then compile, test, and execute them against your warehouse. -Using Dataform Core, data teams can build scalable SQL data transformation pipelines following software engineering best practices, like version control and testing. +sqlanvil is a hard fork of [Dataform](https://github.com/dataform-co/dataform) (Apache 2.0), renamed and restructured to support **PostgreSQL/Supabase** as a first-class target alongside **BigQuery**. -For more details, see [how Dataform works](https://cloud.google.com/dataform/docs/overview). +## Status -![Data collections and integrations feed into Dataform, which exports this data to BI and analytics tools.](static/images/single-source-of-truth.png?raw=true) +Pre-alpha. Active reintegration of the Postgres adapter is in progress on the +`restore-postgres-adapter` branch. See `docs/postgres_first_class_design.md` +for the implementation spec and `docs/rename_checklist.md` for the +dataform → sqlanvil rename surface. -## Get started +## Supported warehouses -### In Google Cloud Platform +| Warehouse | Status | +| :--- | :--- | +| BigQuery | Working (inherited from upstream Dataform) | +| PostgreSQL | In progress — first-class native adapter | +| Supabase (PostgreSQL + RLS, Realtime, Wrappers, pgvector) | Planned | -Dataform in Google Cloud Platform provides a fully managed experience to build scalable data transformations pipelines in **BigQuery** using SQL. It includes: +## Quickstart (when published) -- A cloud development environment to develop data assets with SQL and Dataform Core and version control code with GitHub, GitLab, and other Git providers. -- A fully managed, serverless orchestration environment for data pipelines, fully integrated in Google Cloud Platform. - -Follow the [quickstart guide](https://cloud.google.com/dataform/docs/quickstart)! - -### With the CLI - -You can run Dataform locally using the Dataform CLI tool, which can be installed using the following command line. Follow the [CLI guide](https://cloud.google.com/dataform/docs/use-dataform-cli) to get started. - -``` -npm i -g @dataform/cli +```bash +npm i -g @sqlanvil/cli +sqlanvil init my-project +cd my-project +sqlanvil compile +sqlanvil run ``` -## Useful Links - -- [Documentation home page](https://cloud.google.com/dataform). -- [Create tables and views](https://cloud.google.com/dataform/docs/tables). -- [Configure dependencies](https://cloud.google.com/dataform/docs/define-table#define_table_structure_and_dependencies). -- Write [data quality checks](https://cloud.google.com/dataform/docs/assertions). -- Enable [scripting](https://cloud.google.com/dataform/docs/develop-workflows-js) and code re-use with a JavaScript API. -- Import [pre-defined packages](https://dataform-co.github.io/dataform/docs/packages), or create your own. -- View the [Dataform Core reference](https://cloud.google.com/dataform/docs/reference/dataform-core-reference). -- View the [Dataform configs reference](https://dataform-co.github.io/dataform/docs/configs-reference). - -_Note: this readme can also be viewed on https://dataform-co.github.io/dataform._ - -## Example Projects - -- [https://github.com/GoogleCloudPlatform/marketing-data-engine-dataform](https://github.com/GoogleCloudPlatform/marketing-data-engine-dataform). -- [https://github.com/wintermi/movielens-dataform](https://github.com/wintermi/movielens-dataform). -- [https://github.com/wintermi/bqe-dataform](https://github.com/wintermi/bqe-dataform). -- [https://github.com/wintermi/imdb-dataform](https://github.com/wintermi/imdb-dataform). -- [https://github.com/wintermi/fashion-dataform](https://github.com/wintermi/fashion-dataform). -- [https://github.com/G2H/dataform-stackoverflow](https://github.com/G2H/dataform-stackoverflow). -- [https://github.com/karcot1/dataform_deployment_sample](https://github.com/karcot1/dataform_deployment_sample). - -## Want to report a bug or request a feature? - -- For Dataform Core / open source requests, you can open an [issue](https://github.com/dataform-co/dataform/issues) in GitHub. -- For Dataform in Google Cloud Platform, you can file a bug [here](https://issuetracker.google.com/issues/new?component=1193995&template=1698201), and file feature requests [here](https://issuetracker.google.com/issues/new?component=1193995&template=1713836). +For now, building from source requires Bazel via Bazelisk +(`npm i -g @bazel/bazelisk`). See `contributing.md`. -## Want to contribute? +## Attribution -Check out our [contributors guide](https://github.com/dataform-co/dataform/blob/main/contributing.md) to get started with setting up the repo. +sqlanvil derives from Dataform OSS by Dataform Co (acquired by Google). +The original code remains under the Apache 2.0 license. See `NOTICE` and +`LICENSE` for required attribution. diff --git a/scripts/create_gh_pr b/scripts/create_gh_pr deleted file mode 100755 index c22c59de..00000000 --- a/scripts/create_gh_pr +++ /dev/null @@ -1,21 +0,0 @@ -set -e - -echo "Set the git config..." -git config user.name $_GITHUB_USER -git config user.email $_GITHUB_EMAIL -git remote set-url origin https://$_GITHUB_USER:$(cat token.txt)@github.com/dataform-co/dataform.git - -echo "Update version..." -./scripts/update_version - -export git_branch_name=npm_veriosn_$(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g") - -echo "Create new branch $git_branch_name..." -git checkout -b $git_branch_name - -git add version.bzl -git commit -m "Update the npm package version" - -echo "Push changes to remote..." -git push origin $git_branch_name -echo $git_branch_name > git_branch_name.txt diff --git a/scripts/decrypt_secret b/scripts/decrypt_secret deleted file mode 100755 index da7fe20b..00000000 --- a/scripts/decrypt_secret +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash - -gcloud kms decrypt \ - --ciphertext-file=$1.enc \ - --plaintext-file=$1 \ - --project=dataform-open-source \ - --keyring=dataform-builder-keyring \ - --key=dataform-builder-key \ - --location=global diff --git a/scripts/docker-bazel b/scripts/docker-bazel new file mode 100755 index 00000000..75adbc13 --- /dev/null +++ b/scripts/docker-bazel @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Run a bazel command inside the sqlanvil-dev Docker container. +# +# Usage: +# ./scripts/docker-bazel build //protos:sqlanvil_proto +# ./scripts/docker-bazel test //core/... +# ./scripts/docker-bazel build //... +# ./scripts/docker-bazel # drops into an interactive shell +# +# First run is slow: pulls the node:20 base image, installs deps, then +# downloads Bazel 5.4 + all toolchains/rules from cold. Subsequent runs +# reuse the named volumes for the Bazel cache. + +set -eo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +IMAGE="sqlanvil-dev" + +# Build the image if it doesn't exist locally yet. +if ! docker image inspect "$IMAGE" >/dev/null 2>&1; then + echo "Building $IMAGE image (one-time setup)..." >&2 + docker build -f "$REPO_ROOT/Dockerfile.dev" -t "$IMAGE" "$REPO_ROOT" +fi + +# tty + interactive flags only when STDIN is a terminal — keeps CI happy. +TTY_FLAGS=() +if [ -t 0 ] && [ -t 1 ]; then + TTY_FLAGS=(-it) +fi + +# If no args: drop into a shell. Otherwise run `bazel `. +if [ "$#" -eq 0 ]; then + CMD=(bash) +else + CMD=(bazel "$@") +fi + +docker run --rm "${TTY_FLAGS[@]}" \ + -v "$REPO_ROOT:/workspace" \ + -v sqlanvil-bazel-cache:/root/.cache/bazel \ + -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \ + "$IMAGE" "${CMD[@]}" diff --git a/scripts/publish b/scripts/publish deleted file mode 100755 index 8c585960..00000000 --- a/scripts/publish +++ /dev/null @@ -1,44 +0,0 @@ -#!/bin/bash -set -e - -# This can be uncommented once it no longer throws an error exit code. -# ./scripts/regenerate_docs - -# By default GCB uses master branch name. This code change it to main. -if [ "master" == "$(git branch --show-current)" ]; then - git branch -m main -fi - -if [ "$(git status --porcelain)" ]; then - echo "There are uncommitted changes; aborting." 1>&2 - exit 1 -fi - -# Run all the tests. -bazel run @nodejs//:yarn config set registry https://us-npm.pkg.dev/artifact-foundry-prod/ah-3p-staging-npm/ -bazel run //tools/registry-tools:switch_registry -- $(pwd)/yarn.lock https://us-npm.pkg.dev/artifact-foundry-prod/npm-3p-trusted/ -bazel run @nodejs//:yarn install -- --frozen-lockfile -bazel test //... --build_tests_only - -# After the code is build with Airlock dependencies, we change the registry to public npmjs -# to publish our npm package. -bazel run @nodejs//:npm config set registry https://registry.npmjs.org/ - -VERSION=$(cat version.bzl | grep DF_VERSION | awk '{ print $3 }' | sed "s/\"//g") - -TAG=next - -# If the version is a normal release (1.2.3) and not pre-release e.g (1.2.3-alpha.1) then publish with the 'latest' tag. -if [[ "$VERSION" =~ [0-9]+\.[0-9]+\.[0-9]+$ ]]; then - TAG=latest - - if [ "main" != "$(git branch --show-current)" ]; then - echo "Not on the 'main' branch; aborting." 1>&2 - exit 1 - fi -fi - -echo "Publishing as '$TAG' based on version: $VERSION" - -bazel run packages/@dataform/cli:package.publish -- --tag=$TAG -bazel run packages/@dataform/core:package.publish -- --tag=$TAG diff --git a/scripts/run b/scripts/run index aae89b8a..03006a0b 100755 --- a/scripts/run +++ b/scripts/run @@ -1,5 +1,5 @@ #!/bin/bash set -e -bazel build //packages/@dataform/cli:bin -./bazel-bin/packages/@dataform/cli/bin.sh "$@" +bazel build //packages/@sqlanvil/cli:bin +./bazel-bin/packages/@sqlanvil/cli/bin.sh "$@" diff --git a/scripts/update_test_credentials b/scripts/update_test_credentials deleted file mode 100755 index 81a61e23..00000000 --- a/scripts/update_test_credentials +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash - -# Generates new test credentials. -# -# The script must be executed from the repository root folder. -# -# Dependencies: -# - gcloud -# - jq - -SECRET_JSON_PATH=test_credentials/secret.json -BIGQUERY_JSON_PATH=test_credentials/bigquery.json -BIGQUERY_JSON_ENC_PATH=test_credentials/bigquery.json.enc - -# Generate a new key for dataform-testing service account and download it. -gcloud iam service-accounts keys create "${SECRET_JSON_PATH}" \ - --iam-account=dataform-testing@dataform-open-source.iam.gserviceaccount.com \ - --project=dataform-open-source - -# Create bigquery.json for encryption. Basically we do the same thing -# as `dataform init-creds` will do but without creating dataform project. -cat < "${BIGQUERY_JSON_PATH}" -{ - "projectId": "dataform-open-source", - "credentials": $(jq -Rsa < ${SECRET_JSON_PATH}), - "location": "US" -} -EOF - -# Create encrypted secret -gcloud kms encrypt \ - --ciphertext-file="${BIGQUERY_JSON_ENC_PATH}" \ - --plaintext-file="${BIGQUERY_JSON_PATH}" \ - --project=dataform-open-source \ - --keyring=dataform-builder-keyring \ - --key=dataform-builder-key \ - --location=global - -# Cleanup secrets -rm -f "${SECRET_JSON_PATH}" -rm -f "${BIGQUERY_JSON_PATH}" diff --git a/sqlx/format.ts b/sqlx/format.ts index 4fc3bf14..9d8f5a41 100644 --- a/sqlx/format.ts +++ b/sqlx/format.ts @@ -4,8 +4,8 @@ import * as jsBeautify from "js-beautify"; import { typeid } from "typeid-js"; import { promisify } from "util"; -import { ErrorWithCause } from "df/common/errors/errors"; -import { SyntaxTreeNode, SyntaxTreeNodeType } from "df/sqlx/lexer"; +import { ErrorWithCause } from "sa/common/errors/errors"; +import { SyntaxTreeNode, SyntaxTreeNodeType } from "sa/sqlx/lexer"; const JS_BEAUTIFY_OPTIONS = { indent_size: 2, diff --git a/sqlx/format_test.ts b/sqlx/format_test.ts index ccb2f38b..1a71fc40 100644 --- a/sqlx/format_test.ts +++ b/sqlx/format_test.ts @@ -2,10 +2,10 @@ import { expect } from "chai"; import * as fs from "fs"; import * as path from "path"; -import { format, formatFile } from "df/sqlx/format"; -import { suite, test } from "df/testing"; +import { format, formatFile } from "sa/sqlx/format"; +import { suite, test } from "sa/testing"; -suite("@dataform/sqlx", () => { +suite("@sqlanvil/sqlx", () => { suite("formatter", () => { test("correctly formats a simple SQLX file", async () => { const filePath = path.join(process.env.TEST_TMPDIR, "simple.sqlx"); diff --git a/sqlx/lexer_test.ts b/sqlx/lexer_test.ts index b5ea1fb1..19c8aec1 100644 --- a/sqlx/lexer_test.ts +++ b/sqlx/lexer_test.ts @@ -1,9 +1,9 @@ import { expect } from "chai"; -import { SyntaxTreeNode, SyntaxTreeNodeType } from "df/sqlx/lexer"; -import { suite, test } from "df/testing"; +import { SyntaxTreeNode, SyntaxTreeNodeType } from "sa/sqlx/lexer"; +import { suite, test } from "sa/testing"; -suite("@dataform/sqlx", () => { +suite("@sqlanvil/sqlx", () => { suite("syntax tree construction", () => { test("SQL strings don't affect the tree", () => { const actual = SyntaxTreeNode.create("SELECT SUM(IF(track.event = 'example', 1, 0)) js { }"); diff --git a/test_credentials/BUILD b/test_credentials/BUILD deleted file mode 100644 index bd9edfa9..00000000 --- a/test_credentials/BUILD +++ /dev/null @@ -1,15 +0,0 @@ -load("//tools/gcloud:secrets.bzl", "gcloud_secret") - -package(default_visibility = ["//visibility:public"]) - -# You can update the credentials for testing by: -# * Ensuring you have the required permissions (at time of writing, you need to request a grant). -# * Run the "scripts/update_test_credentials" script. -gcloud_secret( - name = "bigquery.json", - testonly = 1, - ciphertext_file = ":bigquery.json.enc", - project = "dataform-open-source", - key = "dataform-builder-key", - keyring = "dataform-builder-keyring", -) diff --git a/test_credentials/bigquery.json.enc b/test_credentials/bigquery.json.enc deleted file mode 100644 index 7aa4ba8f..00000000 Binary files a/test_credentials/bigquery.json.enc and /dev/null differ diff --git a/testing/child_process.ts b/testing/child_process.ts index 97815fb8..a6f5818c 100644 --- a/testing/child_process.ts +++ b/testing/child_process.ts @@ -2,7 +2,7 @@ import { ChildProcess, spawn } from "child_process"; import * as fs from "fs"; import * as path from "path"; -import { IHookHandler } from "df/testing"; +import { IHookHandler } from "sa/testing"; export class ChildProcessForBazelTestEnvironment { private childProcess: ChildProcess; diff --git a/testing/fixtures.ts b/testing/fixtures.ts index 70907f4d..761f0231 100644 --- a/testing/fixtures.ts +++ b/testing/fixtures.ts @@ -2,7 +2,7 @@ import * as fs from "fs"; import * as path from "path"; import * as rimraf from "rimraf"; -import { IHookHandler } from "df/testing"; +import { IHookHandler } from "sa/testing"; // TmpDirFixture can be used to create unique temporary directories which will be cleaned up // at the end of a test run. Intended for use within bazel tests. diff --git a/testing/hook.ts b/testing/hook.ts index cc342ffe..895cdc49 100644 --- a/testing/hook.ts +++ b/testing/hook.ts @@ -1,4 +1,4 @@ -import { IRunContext, IRunResult } from "df/testing"; +import { IRunContext, IRunResult } from "sa/testing"; export type IHookFunction = () => any; diff --git a/testing/index.ts b/testing/index.ts index 9779b639..4adb5d78 100644 --- a/testing/index.ts +++ b/testing/index.ts @@ -1,10 +1,10 @@ import { ChildProcess } from "child_process"; import * as os from "os"; -export * from "df/testing/hook"; -export * from "df/testing/suite"; -export * from "df/testing/test"; -export * from "df/testing/runner"; +export * from "sa/testing/hook"; +export * from "sa/testing/suite"; +export * from "sa/testing/test"; +export * from "sa/testing/runner"; export const platformPath = () => { if (os.platform() === "darwin") { @@ -25,7 +25,7 @@ export const platformPath = () => { // Note: it would be more correct for these to be injected by blaze at run time. export const nodePath = `external/${platformPath()}/bin/node`; export const npmPath = `external/${platformPath()}/bin/npm`; -export const corePackageTarPath = "packages/@dataform/core/package.tar.gz"; +export const corePackageTarPath = "packages/@sqlanvil/core/package.tar.gz"; export async function getProcessResult(childProcess: ChildProcess) { let stderr = ""; diff --git a/testing/index_test.ts b/testing/index_test.ts index 91b0fb87..6e88b9cd 100644 --- a/testing/index_test.ts +++ b/testing/index_test.ts @@ -1,6 +1,6 @@ import { expect } from "chai"; -import { ISuiteContext, Runner, suite, test } from "df/testing"; +import { ISuiteContext, Runner, suite, test } from "sa/testing"; Runner.setNoExit(true); diff --git a/testing/run_core.ts b/testing/run_core.ts index a5f23e42..5bf82115 100644 --- a/testing/run_core.ts +++ b/testing/run_core.ts @@ -3,17 +3,9 @@ import * as fs from "fs-extra"; import * as path from "path"; import { CompilerFunction, NodeVM } from "vm2"; -import { decode64, encode64 } from "df/common/protos"; -import { compile } from "df/core/compilers"; -import { dataform } from "df/protos/ts"; - -export const VALID_DATAFORM_JSON = ` -{ - "defaultDatabase": "defaultProject", - "defaultSchema": "defaultDataset", - "defaultLocation": "US" -} -`; +import { decode64, encode64 } from "sa/common/protos"; +import { compile } from "sa/core/compilers"; +import { sqlanvil } from "sa/protos/ts"; export const VALID_WORKFLOW_SETTINGS_YAML = ` defaultProject: defaultProject @@ -22,27 +14,27 @@ defaultLocation: US `; export class WorkflowSettingsTemplates { - public static bigquery = dataform.WorkflowSettings.create({ + public static bigquery = sqlanvil.WorkflowSettings.create({ defaultDataset: "defaultDataset", defaultLocation: "US" }); - public static bigqueryWithDefaultProject = dataform.WorkflowSettings.create({ + public static bigqueryWithDefaultProject = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigquery, defaultProject: "defaultProject" }); - public static bigqueryWithDatasetSuffix = dataform.WorkflowSettings.create({ + public static bigqueryWithDatasetSuffix = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigquery, datasetSuffix: "suffix" }); - public static bigqueryWithDefaultProjectAndDataset = dataform.WorkflowSettings.create({ + public static bigqueryWithDefaultProjectAndDataset = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigqueryWithDefaultProject, projectSuffix: "suffix" }); - public static bigqueryWithNamePrefix = dataform.WorkflowSettings.create({ + public static bigqueryWithNamePrefix = sqlanvil.WorkflowSettings.create({ ...WorkflowSettingsTemplates.bigquery, namePrefix: "prefix" }); @@ -52,10 +44,10 @@ const SOURCE_EXTENSIONS = ["js", "sql", "sqlx", "yaml", "ipynb"]; export function coreExecutionRequestFromPath( projectDir: string, - projectConfigOverride?: dataform.ProjectConfig -): dataform.CoreExecutionRequest { + projectConfigOverride?: sqlanvil.ProjectConfig +): sqlanvil.CoreExecutionRequest { const resolvedProjectDir = fs.realpathSync(path.resolve(projectDir)); - return dataform.CoreExecutionRequest.create({ + return sqlanvil.CoreExecutionRequest.create({ compile: { compileConfig: { projectDir: resolvedProjectDir, @@ -68,11 +60,11 @@ export function coreExecutionRequestFromPath( // A VM is needed when running main because Node functions like `require` are overridden. export function runMainInVm( - coreExecutionRequest: dataform.CoreExecutionRequest -): dataform.CoreExecutionResponse { + coreExecutionRequest: sqlanvil.CoreExecutionRequest +): sqlanvil.CoreExecutionResponse { const projectDir = coreExecutionRequest.compile.compileConfig.projectDir; - // Copy over the build Dataform Core that is set up as a node_modules directory. + // Copy over the build sqlanvil Core that is set up as a node_modules directory. fs.copySync(`${process.cwd()}/core/node_modules`, `${projectDir}/node_modules`); const compiler = compile as CompilerFunction; @@ -94,28 +86,27 @@ export function runMainInVm( compiler: (code, filePath) => { const compiledCode = compiler(code, filePath); return ` - var __old_file = global.__dataform_current_file; - global.__dataform_current_file = ${JSON.stringify(filePath)}; + var __old_file = global.__sqlanvil_current_file; + global.__sqlanvil_current_file = ${JSON.stringify(filePath)}; try { ${compiledCode} } finally { - global.__dataform_current_file = __old_file; + global.__sqlanvil_current_file = __old_file; } `; } }); - const encodedCoreExecutionRequest = encode64(dataform.CoreExecutionRequest, coreExecutionRequest); + const encodedCoreExecutionRequest = encode64(sqlanvil.CoreExecutionRequest, coreExecutionRequest); const vmIndexFileName = path.resolve(path.join(projectDir, "index.js")); const encodedCoreExecutionResponse = nodeVm.run( ` global.workflowSettingsYaml = (function() { try { return require("./workflow_settings.yaml"); } catch(e) { console.error("YAML require failed run_core:", e); } })(); - global.dataformJson = (function() { try { return require("./dataform.json"); } catch(e) {} })(); - return require("@dataform/core").main("${encodedCoreExecutionRequest}") + return require("@sqlanvil/core").main("${encodedCoreExecutionRequest}") `, vmIndexFileName ); - return decode64(dataform.CoreExecutionResponse, encodedCoreExecutionResponse); + return decode64(sqlanvil.CoreExecutionResponse, encodedCoreExecutionResponse); } function walkDirectoryForFilenames(projectDir: string, relativePath: string = ""): string[] { diff --git a/testing/runner.ts b/testing/runner.ts index 03771365..db2bfdb7 100644 --- a/testing/runner.ts +++ b/testing/runner.ts @@ -3,7 +3,7 @@ import * as Diff from "diff"; import DeterministicStringify from "json-stable-stringify"; import { promisify } from "util"; -import { Hook, Suite } from "df/testing"; +import { Hook, Suite } from "sa/testing"; export interface IRunResult { path: string[]; diff --git a/testing/suite.ts b/testing/suite.ts index c9b1d42e..b8b2f3bd 100644 --- a/testing/suite.ts +++ b/testing/suite.ts @@ -1,4 +1,4 @@ -import { Hook, hook, IRunContext, Runner, test, Test } from "df/testing"; +import { Hook, hook, IRunContext, Runner, test, Test } from "sa/testing"; export interface ISuiteOptions { name: string; diff --git a/testing/test.ts b/testing/test.ts index 6d3ea5a4..39dfae7f 100644 --- a/testing/test.ts +++ b/testing/test.ts @@ -1,4 +1,4 @@ -import { IRunContext, IRunResult, Runner, Suite } from "df/testing"; +import { IRunContext, IRunResult, Runner, Suite } from "sa/testing"; interface ITestOptions { name: string; diff --git a/tests/api/BUILD b/tests/api/BUILD index 79b72c30..a2f4198a 100644 --- a/tests/api/BUILD +++ b/tests/api/BUILD @@ -6,8 +6,6 @@ ts_test_suite( data = [ "//tests/api/projects/common_v2:files", "//tests/api/projects/common_v2:node_modules", - "//tests/api/projects/invalid_dataform_json:files", - "//tests/api/projects/invalid_dataform_json:node_modules", "//tests/api/projects/never_finishes_compiling:files", "//tests/api/projects/never_finishes_compiling:node_modules", ], diff --git a/tests/api/api.spec.ts b/tests/api/api.spec.ts index b2993614..e024dab2 100644 --- a/tests/api/api.spec.ts +++ b/tests/api/api.spec.ts @@ -4,24 +4,24 @@ import Long from "long"; import * as path from "path"; import { anyString, anything, instance, mock, verify, when } from "ts-mockito"; -import { Builder, credentials, prune, Runner } from "df/cli/api"; -import { IDbAdapter } from "df/cli/api/dbadapters"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { sleep, sleepUntil } from "df/common/promises"; -import { equals } from "df/common/protos"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { asPlainObject, cleanSql, suite, test } from "df/testing"; -import { TmpDirFixture } from "df/testing/fixtures"; +import { Builder, credentials, prune, Runner } from "sa/cli/api"; +import { IDbAdapter } from "sa/cli/api/dbadapters"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { sleep, sleepUntil } from "sa/common/promises"; +import { equals } from "sa/common/protos"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { asPlainObject, cleanSql, suite, test } from "sa/testing"; +import { TmpDirFixture } from "sa/testing/fixtures"; config.truncateThreshold = 0; -suite("@dataform/api", () => { +suite("@sqlanvil/api", () => { // c +-> b +-> a // ^ // d // Made with asciiflow.com - const TEST_GRAPH: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const TEST_GRAPH: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { @@ -66,7 +66,7 @@ suite("@dataform/api", () => { ] }); - const TEST_STATE = dataform.WarehouseState.create({ tables: [] }); + const TEST_STATE = sqlanvil.WarehouseState.create({ tables: [] }); suite("build", () => { test("exclude_disabled", () => { @@ -94,7 +94,7 @@ suite("@dataform/api", () => { test("build_with_errors", () => { expect(() => { - const graphWithErrors: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graphWithErrors: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, graphErrors: { compilationErrors: [{ message: "Some critical error" }] }, tables: [{ target: { schema: "schema", name: "a" } }] @@ -106,7 +106,7 @@ suite("@dataform/api", () => { }); test("action_types", () => { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { target: { schema: "schema", name: "a" }, type: "table" }, @@ -131,39 +131,39 @@ suite("@dataform/api", () => { expect(executedGraph.actions.length).greaterThan(0); - graph.tables.forEach((t: dataform.ITable) => { + graph.tables.forEach((t: sqlanvil.ITable) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, t.target) + equals(sqlanvil.Target, item.target, t.target) ); expect(action).to.include({ type: "table", target: t.target, tableType: t.type }); }); - graph.operations.forEach((o: dataform.IOperation) => { + graph.operations.forEach((o: sqlanvil.IOperation) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, o.target) + equals(sqlanvil.Target, item.target, o.target) ); expect(action).to.include({ type: "operation", target: o.target }); }); - graph.assertions.forEach((a: dataform.IAssertion) => { + graph.assertions.forEach((a: sqlanvil.IAssertion) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, a.target) + equals(sqlanvil.Target, item.target, a.target) ); expect(action).to.include({ type: "assertion" }); }); }); test("table_enum_types", () => { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ - { target: { schema: "schema", name: "a" }, enumType: dataform.TableType.TABLE }, + { target: { schema: "schema", name: "a" }, enumType: sqlanvil.TableType.TABLE }, { target: { schema: "schema", name: "b" }, - enumType: dataform.TableType.INCREMENTAL, + enumType: sqlanvil.TableType.INCREMENTAL, where: "test" }, - { target: { schema: "schema", name: "c" }, enumType: dataform.TableType.VIEW } + { target: { schema: "schema", name: "c" }, enumType: sqlanvil.TableType.VIEW } ] }); @@ -172,25 +172,25 @@ suite("@dataform/api", () => { expect(executedGraph.actions.length).greaterThan(0); - graph.tables.forEach((t: dataform.ITable) => { + graph.tables.forEach((t: sqlanvil.ITable) => { const action = executedGraph.actions.find(item => - equals(dataform.Target, item.target, t.target) + equals(sqlanvil.Target, item.target, t.target) ); expect(action).to.include({ type: "table", target: t.target, - tableType: dataform.TableType[t.enumType].toLowerCase() + tableType: sqlanvil.TableType[t.enumType].toLowerCase() }); }); }); test("table_enum_and_str_types_should_match", () => { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { target: { schema: "schema", name: "a" }, - enumType: dataform.TableType.TABLE, + enumType: sqlanvil.TableType.TABLE, type: "incremental" } ] @@ -203,7 +203,7 @@ suite("@dataform/api", () => { suite("pre and post ops", () => { for (const warehouse of ["bigquery"]) { - const graph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const graph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery" }, tables: [ { @@ -217,13 +217,13 @@ suite("@dataform/api", () => { incrementalPostOps: ["incremental postOp"] } ], - dataformCoreVersion: "1.4.9" + sqlanvilCoreVersion: "1.4.9" }); test(`${warehouse} when running non incrementally`, () => { const action = new Builder(graph, {}, TEST_STATE).build().actions[0]; expect(action.tasks).eql([ - dataform.ExecutionTask.create({ + sqlanvil.ExecutionTask.create({ type: "statement", statement: "preOp\n;\ncreate or replace table `schema.a` as foo\n;\npostOp" }) @@ -234,12 +234,12 @@ suite("@dataform/api", () => { const action = new Builder( graph, {}, - dataform.WarehouseState.create({ + sqlanvil.WarehouseState.create({ tables: [{ target: graph.tables[0].target, fields: [] }] }) ).build().actions[0]; expect(action.tasks).eql([ - dataform.ExecutionTask.create({ + sqlanvil.ExecutionTask.create({ type: "statement", statement: "incremental preOp\n;\ndrop view if exists `schema.a`\n;\ninsert into `schema.a`\t\n()\t\nselect \t\nfrom (incremental foo) as insertions\n;\nincremental postOp" @@ -256,7 +256,7 @@ suite("@dataform/api", () => { // +-> op_c // // op_d +---> tab_a - const TEST_GRAPH_WITH_TAGS: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const TEST_GRAPH_WITH_TAGS: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultLocation: "US" }, operations: [ { @@ -391,14 +391,14 @@ suite("@dataform/api", () => { query: "select 1 as test", where: "true" }; - const warehouseState = dataform.WarehouseState.create({ + const warehouseState = sqlanvil.WarehouseState.create({ tables: [ { target: { schema: "schema", name: "incremental" }, - type: dataform.TableMetadata.Type.TABLE, + type: sqlanvil.TableMetadata.Type.TABLE, fields: [ { name: "existing_field" @@ -409,7 +409,7 @@ suite("@dataform/api", () => { }); test("incremental_mode", () => { - const graph = dataform.CompiledGraph.create({ + const graph = sqlanvil.CompiledGraph.create({ projectConfig, tables: [ incrementalTable ] }); @@ -434,7 +434,7 @@ suite("@dataform/api", () => { }); test("full refresh", () => { - const graph = dataform.CompiledGraph.create({ + const graph = sqlanvil.CompiledGraph.create({ projectConfig, tables: [ incrementalTable ] }); @@ -457,7 +457,7 @@ suite("@dataform/api", () => { ...incrementalTable, protected: true, }; - const graph = dataform.CompiledGraph.create({ + const graph = sqlanvil.CompiledGraph.create({ projectConfig, tables: [ protectedIncrementalTable ] }); @@ -483,7 +483,7 @@ suite("@dataform/api", () => { }); test("bigquery_materialized", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -505,7 +505,7 @@ suite("@dataform/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "view", @@ -521,7 +521,7 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -537,17 +537,17 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_partitionby", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -572,7 +572,7 @@ suite("@dataform/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -588,7 +588,7 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -604,17 +604,17 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_options", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -641,7 +641,7 @@ suite("@dataform/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -657,7 +657,7 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -673,17 +673,17 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_clusterby", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -708,7 +708,7 @@ suite("@dataform/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -724,7 +724,7 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -740,17 +740,17 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); }); test("bigquery_additional_options", () => { - const testGraph: dataform.ICompiledGraph = dataform.CompiledGraph.create({ + const testGraph: sqlanvil.ICompiledGraph = sqlanvil.CompiledGraph.create({ projectConfig: { warehouse: "bigquery", defaultDatabase: "deeb", defaultLocation: "US" }, tables: [ { @@ -778,7 +778,7 @@ suite("@dataform/api", () => { } ] }); - const expectedExecutionActions: dataform.IExecutionAction[] = [ + const expectedExecutionActions: sqlanvil.IExecutionAction[] = [ { type: "table", tableType: "table", @@ -794,7 +794,7 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC }, { type: "table", @@ -810,10 +810,10 @@ suite("@dataform/api", () => { } ], dependencyTargets: [], - hermeticity: dataform.ActionHermeticity.HERMETIC + hermeticity: sqlanvil.ActionHermeticity.HERMETIC } ]; - const executionGraph = new Builder(testGraph, {}, dataform.WarehouseState.create({})).build(); + const executionGraph = new Builder(testGraph, {}, sqlanvil.WarehouseState.create({})).build(); expect(asPlainObject(executionGraph.actions)).deep.equals( asPlainObject(expectedExecutionActions) ); @@ -843,7 +843,7 @@ suite("@dataform/api", () => { }); suite("run", () => { - const RUN_TEST_GRAPH: dataform.IExecutionGraph = dataform.ExecutionGraph.create({ + const RUN_TEST_GRAPH: sqlanvil.IExecutionGraph = sqlanvil.ExecutionGraph.create({ projectConfig: { warehouse: "bigquery", defaultSchema: "foo", @@ -857,7 +857,7 @@ suite("@dataform/api", () => { warehouseState: { tables: [ { - type: dataform.TableMetadata.Type.TABLE, + type: sqlanvil.TableMetadata.Type.TABLE, target: { schema: "schema1", name: "target1" @@ -909,15 +909,15 @@ suite("@dataform/api", () => { ] }); - const EXPECTED_RUN_RESULT = dataform.RunResult.create({ - status: dataform.RunResult.ExecutionStatus.FAILED, + const EXPECTED_RUN_RESULT = sqlanvil.RunResult.create({ + status: sqlanvil.RunResult.ExecutionStatus.FAILED, actions: [ { target: RUN_TEST_GRAPH.actions[0].target, tasks: [ { - status: dataform.TaskResult.ExecutionStatus.SUCCESSFUL, + status: sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL, metadata: { bigquery: { jobId: "abc", @@ -927,22 +927,22 @@ suite("@dataform/api", () => { } }, { - status: dataform.TaskResult.ExecutionStatus.SUCCESSFUL, + status: sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL, metadata: {} } ], - status: dataform.ActionResult.ExecutionStatus.SUCCESSFUL + status: sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL }, { target: RUN_TEST_GRAPH.actions[1].target, tasks: [ { - status: dataform.TaskResult.ExecutionStatus.FAILED, + status: sqlanvil.TaskResult.ExecutionStatus.FAILED, metadata: {}, errorMessage: "bigquery error: bad statement" } ], - status: dataform.ActionResult.ExecutionStatus.FAILED + status: sqlanvil.ActionResult.ExecutionStatus.FAILED } ] }); @@ -979,7 +979,7 @@ suite("@dataform/api", () => { const runner = new Runner(mockDbAdapterInstance, RUN_TEST_GRAPH); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); verify(mockedDbAdapter.createSchema("database", "schema1")).once(); verify(mockedDbAdapter.createSchema("database2", "schema2")).once(); @@ -1028,13 +1028,13 @@ suite("@dataform/api", () => { stopWasCalled = true; const result = cleanTiming(await runner.result()); - expect(dataform.RunResult.create(result).toJSON()).to.deep.equal( - dataform.RunResult.create({ - status: dataform.RunResult.ExecutionStatus.RUNNING, + expect(sqlanvil.RunResult.create(result).toJSON()).to.deep.equal( + sqlanvil.RunResult.create({ + status: sqlanvil.RunResult.ExecutionStatus.RUNNING, actions: [ { target: EXPECTED_RUN_RESULT.actions[0].target, - status: dataform.ActionResult.ExecutionStatus.RUNNING, + status: sqlanvil.ActionResult.ExecutionStatus.RUNNING, tasks: [EXPECTED_RUN_RESULT.actions[0].tasks[0]] } ] @@ -1044,7 +1044,7 @@ suite("@dataform/api", () => { runner = new Runner(mockDbAdapterInstance, RUN_TEST_GRAPH, undefined, result); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); verify(mockedDbAdapter.createSchema("database", "schema1")).once(); verify(mockedDbAdapter.createSchema("database2", "schema2")).once(); @@ -1087,7 +1087,7 @@ suite("@dataform/api", () => { }); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); }); @@ -1127,21 +1127,21 @@ suite("@dataform/api", () => { }); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal( - dataform.RunResult.create({ - status: dataform.RunResult.ExecutionStatus.SUCCESSFUL, + sqlanvil.RunResult.create({ + status: sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL, actions: [ EXPECTED_RUN_RESULT.actions[0], { target: NEW_TEST_GRAPH.actions[1].target, tasks: [ { - status: dataform.TaskResult.ExecutionStatus.SUCCESSFUL, + status: sqlanvil.TaskResult.ExecutionStatus.SUCCESSFUL, metadata: {} } ], - status: dataform.ActionResult.ExecutionStatus.SUCCESSFUL + status: sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL } ] }).toJSON() @@ -1191,13 +1191,13 @@ suite("@dataform/api", () => { }); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal(EXPECTED_RUN_RESULT.toJSON()); }); }); test("execute_with_cancel", async () => { - const CANCEL_TEST_GRAPH: dataform.IExecutionGraph = dataform.ExecutionGraph.create({ + const CANCEL_TEST_GRAPH: sqlanvil.IExecutionGraph = sqlanvil.ExecutionGraph.create({ projectConfig: { warehouse: "bigquery", defaultSchema: "foo", @@ -1252,7 +1252,7 @@ suite("@dataform/api", () => { // Cancelling a run doesn't actually throw at the top level. // The action should fail, and have an appropriate error message. expect(result.actions[0].tasks[0].status).equal( - dataform.TaskResult.ExecutionStatus.CANCELLED + sqlanvil.TaskResult.ExecutionStatus.CANCELLED ); expect(result.actions[0].tasks[0].errorMessage).to.match(/cancelled/); }); @@ -1295,13 +1295,13 @@ suite("@dataform/api", () => { mockDbAdapterInstance.withClientLock = async callback => await callback(mockDbAdapterInstance); - const labels = { env: "testing", team: "dataform" }; + const labels = { env: "testing", team: "sqlanvil" }; const runner = new Runner(mockDbAdapterInstance, NEW_TEST_GRAPH, { bigquery: { labels } }); const result = await runner.execute().result(); - expect(result.status).to.equal(dataform.RunResult.ExecutionStatus.SUCCESSFUL); + expect(result.status).to.equal(sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL); // Verify that execute was called at least 3 times (for both tasks in first action and assertion) expect(executionOptions.length).to.equal(3); @@ -1310,7 +1310,7 @@ suite("@dataform/api", () => { const callsWithLabels = executionOptions.filter( opts => opts?.bigquery?.labels && opts.bigquery.labels.env === "testing" && - opts.bigquery.labels.team === "dataform" + opts.bigquery.labels.team === "sqlanvil" ); expect(callsWithLabels.length).to.equal(3, "Expected 3 execute calls to include the labels in options"); @@ -1358,13 +1358,13 @@ suite("@dataform/api", () => { mockDbAdapterInstance.withClientLock = async callback => await callback(mockDbAdapterInstance); - const globalLabels = { env: "testing", team: "dataform" }; + const globalLabels = { env: "testing", team: "sqlanvil" }; const runner = new Runner(mockDbAdapterInstance, NEW_TEST_GRAPH, { bigquery: { labels: globalLabels } }); const result = await runner.execute().result(); - expect(result.status).to.equal(dataform.RunResult.ExecutionStatus.SUCCESSFUL); + expect(result.status).to.equal(sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL); // Verify that execute was called 3 times expect(executionOptions.length).to.equal(3); @@ -1375,7 +1375,7 @@ suite("@dataform/api", () => { expect(opts?.bigquery?.labels).to.not.equal(undefined); // Should have global labels expect(opts.bigquery.labels.env).to.equal("testing", `Call ${index} should have global label 'env'`); - expect(opts.bigquery.labels.team).to.equal("dataform", `Call ${index} should have global label 'team'`); + expect(opts.bigquery.labels.team).to.equal("sqlanvil", `Call ${index} should have global label 'team'`); // Should have action-level label expect(opts.bigquery.labels.action_level).to.equal("specific_value", `Call ${index} should have action-level label 'action_level'`); @@ -1385,14 +1385,14 @@ suite("@dataform/api", () => { const assertionCall = executionOptions[2]; expect(assertionCall?.bigquery?.labels).to.not.equal(undefined); expect(assertionCall.bigquery.labels.env).to.equal("testing"); - expect(assertionCall.bigquery.labels.team).to.equal("dataform"); + expect(assertionCall.bigquery.labels.team).to.equal("sqlanvil"); // This action doesn't have action-level labels expect(assertionCall.bigquery.labels.action_level).to.equal(undefined); }); }); test("continues after setMetadata fails", async () => { - const METADATA_TEST_GRAPH: dataform.IExecutionGraph = dataform.ExecutionGraph.create({ + const METADATA_TEST_GRAPH: sqlanvil.IExecutionGraph = sqlanvil.ExecutionGraph.create({ projectConfig: { warehouse: "bigquery", defaultSchema: "foo", @@ -1440,7 +1440,7 @@ suite("@dataform/api", () => { const runner = new Runner(mockDbAdapterInstance, METADATA_TEST_GRAPH); expect( - dataform.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() + sqlanvil.RunResult.create(cleanTiming(await runner.execute().result())).toJSON() ).to.deep.equal({ actions: [ { @@ -1464,8 +1464,8 @@ suite("@dataform/api", () => { }); }); -function cleanTiming(runResult: dataform.IRunResult) { - const newRunResult = dataform.RunResult.create(runResult); +function cleanTiming(runResult: sqlanvil.IRunResult) { + const newRunResult = sqlanvil.RunResult.create(runResult); delete newRunResult.timing; newRunResult.actions.forEach(actionResult => { delete actionResult.timing; diff --git a/tests/api/projects.spec.ts b/tests/api/projects.spec.ts index 82c62f80..d0e6d629 100644 --- a/tests/api/projects.spec.ts +++ b/tests/api/projects.spec.ts @@ -2,10 +2,10 @@ import { fail } from "assert"; import { expect } from "chai"; import * as path from "path"; -import { compile } from "df/cli/api"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { cleanSql, suite, test } from "df/testing"; +import { compile } from "sa/cli/api"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { cleanSql, suite, test } from "sa/testing"; suite("examples", () => { suite("common_v2 bigquery", async () => { @@ -34,17 +34,17 @@ suite("examples", () => { { fileName: "definitions/has_compile_errors/assertion_with_bigquery.sqlx", message: - 'Unexpected property "bigquery", or property value type of "object" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "bigquery", or property value type of "object" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/assertion_with_materialized.sqlx", message: - 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/assertion_with_output.sqlx", message: - 'Unexpected property "hasOutput", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "hasOutput", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/assertion_with_postops.sqlx", @@ -62,12 +62,12 @@ suite("examples", () => { { fileName: "definitions/has_compile_errors/protected_assertion.sqlx", message: - 'Unexpected property "protected", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-AssertionConfig for allowed properties.' + 'Unexpected property "protected", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-AssertionConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/table_with_materialized.sqlx", message: - 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://dataform-co.github.io/dataform/docs/configs-reference#dataform-ActionConfig-TableConfig for allowed properties.' + 'Unexpected property "materialized", or property value type of "boolean" is incorrect. See https://github.com/ihistand/sqlanvil/blob/main/docs/reference/configs.md#sqlanvil-ActionConfig-TableConfig for allowed properties.' }, { fileName: "definitions/has_compile_errors/view_with_incremental.sqlx", @@ -87,7 +87,7 @@ suite("examples", () => { // Check JS blocks get processed. const exampleJsBlocks = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -96,12 +96,12 @@ suite("examples", () => { ) ); expect(exampleJsBlocks.type).equals("table"); - expect(exampleJsBlocks.enumType).equals(dataform.TableType.TABLE); + expect(exampleJsBlocks.enumType).equals(sqlanvil.TableType.TABLE); expect(exampleJsBlocks.query.trim()).equals("select 1 as foo"); // Check we can import and use an external package. const exampleIncremental = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -124,7 +124,7 @@ suite("examples", () => { ); const exampleIsIncremental = graph.tables.filter( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -155,11 +155,11 @@ suite("examples", () => { // Check tables defined in includes are not included. const exampleIgnore = graph.tables.find( - (t: dataform.ITable) => targetAsReadableString(t.target) === "example_ignore" + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === "example_ignore" ); expect(exampleIgnore).equal(undefined); const exampleIgnore2 = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -171,7 +171,7 @@ suite("examples", () => { // Check SQL files with raw back-ticks get escaped. const exampleBackticks = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -183,13 +183,13 @@ suite("examples", () => { "select * from `tada-analytics.df_integration_test.sample_data`" ); expect(exampleBackticks.preOps).to.eql([ - '\n GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@dataform.co"\n' + '\n GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@example.com"\n' ]); expect(exampleBackticks.postOps).to.eql([]); // Check deferred calls to table resolve to the correct definitions file. const exampleDeferred = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -201,7 +201,7 @@ suite("examples", () => { // Check view const exampleView = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -210,7 +210,7 @@ suite("examples", () => { ) ); expect(exampleView.type).equals("view"); - expect(exampleView.enumType).equals(dataform.TableType.VIEW); + expect(exampleView.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleView.query.trim()).equals( `select * from \`${dotJoined( databaseWithSuffix("tada-analytics"), @@ -229,31 +229,31 @@ suite("examples", () => { )}\`` ); expect(exampleView.target).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_view" }) ); expect(exampleView.canonicalTarget).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: "tada-analytics", schema: "df_integration_test", name: "example_view" }) ); expect(exampleView.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" }), - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("override_schema"), name: "override_schema_example" }), - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("override_database"), schema: schemaWithSuffix("df_integration_test"), name: "override_database_example" @@ -263,7 +263,7 @@ suite("examples", () => { // Check materialized view const exampleMaterializedView = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -272,7 +272,7 @@ suite("examples", () => { ) ); expect(exampleMaterializedView.type).equals("view"); - expect(exampleMaterializedView.enumType).equals(dataform.TableType.VIEW); + expect(exampleMaterializedView.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleMaterializedView.materialized).equals(true); expect(exampleMaterializedView.query.trim()).equals( `select * from \`${dotJoined( @@ -282,21 +282,21 @@ suite("examples", () => { )}\`\n` + `group by 1` ); expect(exampleMaterializedView.target).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_materialized_view" }) ); expect(exampleMaterializedView.canonicalTarget).deep.equals( - dataform.Target.create({ + sqlanvil.Target.create({ database: "tada-analytics", schema: "df_integration_test", name: "example_materialized_view" }) ); expect(exampleMaterializedView.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" @@ -306,7 +306,7 @@ suite("examples", () => { // Check table const exampleTable = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -315,7 +315,7 @@ suite("examples", () => { ) ); expect(exampleTable.type).equals("table"); - expect(exampleTable.enumType).equals(dataform.TableType.TABLE); + expect(exampleTable.enumType).equals(sqlanvil.TableType.TABLE); expect(exampleTable.query.trim()).equals( `select * from \`${dotJoined( databaseWithSuffix("tada-analytics"), @@ -324,7 +324,7 @@ suite("examples", () => { )}\`\n\n-- here \${"is"} a \`comment\n\n/* \${"another"} \` backtick \` containing \`\`\`comment */` ); expect(exampleTable.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" @@ -336,18 +336,18 @@ suite("examples", () => { databaseWithSuffix("tada-analytics"), schemaWithSuffix("df_integration_test"), "example_table" - )}\` TO GROUP "allusers@dataform.co"\n`, + )}\` TO GROUP "allusers@example.com"\n`, `\n GRANT SELECT ON \`${dotJoined( databaseWithSuffix("tada-analytics"), schemaWithSuffix("df_integration_test"), "example_table" - )}\` TO GROUP "otherusers@dataform.co"\n` + )}\` TO GROUP "otherusers@example.com"\n` ]); expect(exampleTable.tags).to.eql([]); // Check Table with tags const exampleTableWithTags = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -377,7 +377,7 @@ suite("examples", () => { )}\` group by sample) as data where index_row_count > 1` ); expect(exampleTableWithTagsUniqueKeyAssertion.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_table_with_tags" @@ -404,7 +404,7 @@ suite("examples", () => { )}\` where not (sample is not null)` ); expect(exampleTableWithTagsRowConditionsAssertion.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "example_table_with_tags" @@ -413,7 +413,7 @@ suite("examples", () => { // Check sample data const exampleSampleData = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -422,17 +422,17 @@ suite("examples", () => { ) ); expect(exampleSampleData.type).equals("view"); - expect(exampleSampleData.enumType).equals(dataform.TableType.VIEW); + expect(exampleSampleData.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleSampleData.query.trim()).equals( "select 1 as sample union all\nselect 2 as sample union all\nselect 3 as sample" ); expect(exampleSampleData.preOps).eql([]); expect(exampleSampleData.dependencyTargets).eql([]); expect(exampleSampleData.actionDescriptor).to.eql( - dataform.ActionDescriptor.create({ + sqlanvil.ActionDescriptor.create({ description: "This is some sample data.", columns: [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description: "Sample integers.", path: ["sample"] }) @@ -442,7 +442,7 @@ suite("examples", () => { // Check database override defined in "config {}". const exampleUsingOverriddenDatabase = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("override_database"), @@ -455,14 +455,14 @@ suite("examples", () => { databaseWithSuffix("override_database") ); expect(exampleUsingOverriddenDatabase.type).equals("view"); - expect(exampleUsingOverriddenDatabase.enumType).equals(dataform.TableType.VIEW); + expect(exampleUsingOverriddenDatabase.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleUsingOverriddenDatabase.query.trim()).equals( "select 1 as test_database_override" ); // Check schema overrides defined in "config {}" const exampleUsingOverriddenSchema = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -475,14 +475,14 @@ suite("examples", () => { schemaWithSuffix("override_schema") ); expect(exampleUsingOverriddenSchema.type).equals("view"); - expect(exampleUsingOverriddenSchema.enumType).equals(dataform.TableType.VIEW); + expect(exampleUsingOverriddenSchema.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleUsingOverriddenSchema.query.trim()).equals( "select 1 as test_schema_override" ); // Check schema overrides defined in "config {}" -- case with schema unchanged const exampleUsingOverriddenSchemaUnchanged = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -495,14 +495,14 @@ suite("examples", () => { schemaWithSuffix("df_integration_test") ); expect(exampleUsingOverriddenSchemaUnchanged.type).equals("view"); - expect(exampleUsingOverriddenSchemaUnchanged.enumType).equals(dataform.TableType.VIEW); + expect(exampleUsingOverriddenSchemaUnchanged.enumType).equals(sqlanvil.TableType.VIEW); expect(exampleUsingOverriddenSchemaUnchanged.query.trim()).equals( "select 1 as test_schema_override" ); // Check assertion const exampleAssertion = graph.assertions.find( - (a: dataform.IAssertion) => + (a: sqlanvil.IAssertion) => targetAsReadableString(a.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -519,7 +519,7 @@ suite("examples", () => { )}\` where sample = 100` ); expect(exampleAssertion.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("df_integration_test"), name: "sample_data" @@ -527,14 +527,14 @@ suite("examples", () => { ]); expect(exampleAssertion.tags).to.eql([]); expect(exampleAssertion.actionDescriptor).to.eql( - dataform.ActionDescriptor.create({ + sqlanvil.ActionDescriptor.create({ description: "An example assertion looking for incorrect 'sample' values." }) ); // Check Assertion with tags const exampleAssertionWithTags = graph.assertions.find( - (a: dataform.IAssertion) => + (a: sqlanvil.IAssertion) => targetAsReadableString(a.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -549,7 +549,7 @@ suite("examples", () => { // Check example operations file const exampleOperations = graph.operations.find( - (o: dataform.IOperation) => + (o: sqlanvil.IOperation) => targetAsReadableString(o.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -572,12 +572,12 @@ suite("examples", () => { )}\`\n` ]); expect(exampleOperations.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("tada-analytics"), schema: schemaWithSuffix("override_schema"), name: "override_schema_example" }), - dataform.Target.create({ + sqlanvil.Target.create({ database: databaseWithSuffix("override_database"), schema: schemaWithSuffix("df_integration_test"), name: "override_database_example" @@ -587,7 +587,7 @@ suite("examples", () => { // Check example operation with output. const exampleOperationWithOutput = graph.operations.find( - (o: dataform.IOperation) => + (o: sqlanvil.IOperation) => targetAsReadableString(o.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -607,17 +607,17 @@ suite("examples", () => { )}\` AS (SELECT * FROM \`some_database_name.some_external_schema_name.very_important_external_table\`)` ]); expect(exampleOperationWithOutput.dependencyTargets).eql([ - dataform.Target.create({ + sqlanvil.Target.create({ database: "some_database_name", schema: "some_external_schema_name", name: "very_important_external_table" }) ]); expect(exampleOperationWithOutput.actionDescriptor).to.eql( - dataform.ActionDescriptor.create({ + sqlanvil.ActionDescriptor.create({ description: "An example operations file which outputs a dataset.", columns: [ - dataform.ColumnDescriptor.create({ + sqlanvil.ColumnDescriptor.create({ description: "Just 1!", path: ["TEST"] }) @@ -627,7 +627,7 @@ suite("examples", () => { // Check Operation with tags const exampleOperationsWithTags = graph.operations.find( - (o: dataform.IOperation) => + (o: sqlanvil.IOperation) => targetAsReadableString(o.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -644,14 +644,14 @@ suite("examples", () => { "some_database_name.some_external_schema_name.very_important_external_table" ); expect(exampleDeclaration.target).eql( - dataform.Target.create({ + sqlanvil.Target.create({ database: "some_database_name", schema: "some_external_schema_name", name: "very_important_external_table" }) ); expect(exampleDeclaration.actionDescriptor.description).to.equal( - "This table is not generated by Dataform!" + "This table is not generated by sqlanvil!" ); // Check testcases. @@ -673,7 +673,7 @@ suite("examples", () => { // Check double backslashes don't get converted to singular. const exampleDoubleBackslash = graph.tables.find( - (t: dataform.ITable) => + (t: sqlanvil.ITable) => targetAsReadableString(t.target) === dotJoined( databaseWithSuffix("tada-analytics"), @@ -701,24 +701,13 @@ suite("examples", () => { } }); - test("invalid dataform json throws error", async () => { - try { - await compile({ - projectDir: path.resolve("tests/api/projects/invalid_dataform_json") - }); - fail("Should have failed."); - } catch (e) { - // OK - } - }); - test("version is correctly set", async () => { const graph = await compile({ projectDir: "tests/api/projects/common_v2", projectConfigOverride: { warehouse: "bigquery" } }); - const { version: expectedVersion } = require("df/core/version"); - expect(graph.dataformCoreVersion).equals(expectedVersion); + const { version: expectedVersion } = require("sa/core/version"); + expect(graph.sqlanvilCoreVersion).equals(expectedVersion); }); }); diff --git a/tests/api/projects/common_v2/BUILD b/tests/api/projects/common_v2/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/api/projects/common_v2/BUILD +++ b/tests/api/projects/common_v2/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/api/projects/common_v2/dataform.json b/tests/api/projects/common_v2/dataform.json deleted file mode 100644 index 88dabbe2..00000000 --- a/tests/api/projects/common_v2/dataform.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "warehouse": "bigquery", - "defaultDatabase": "tada-analytics", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "defaultLocation": "US" -} diff --git a/tests/api/projects/common_v2/definitions/example_backticks.sqlx b/tests/api/projects/common_v2/definitions/example_backticks.sqlx index d739a865..1d02a233 100644 --- a/tests/api/projects/common_v2/definitions/example_backticks.sqlx +++ b/tests/api/projects/common_v2/definitions/example_backticks.sqlx @@ -7,5 +7,5 @@ select * from `tada-analytics.df_integration_test.sample_data` pre_operations { - GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@dataform.co" + GRANT SELECT ON `tada-analytics.df_integration_test.sample_data` TO GROUP "allusers@example.com" } diff --git a/tests/api/projects/common_v2/definitions/example_declaration.sqlx b/tests/api/projects/common_v2/definitions/example_declaration.sqlx index 92ab0223..ee920e17 100644 --- a/tests/api/projects/common_v2/definitions/example_declaration.sqlx +++ b/tests/api/projects/common_v2/definitions/example_declaration.sqlx @@ -3,5 +3,5 @@ config { database: "some_database_name", schema: "some_external_schema_name", name: "very_important_external_table", - description: "This table is not generated by Dataform!" + description: "This table is not generated by sqlanvil!" } diff --git a/tests/api/projects/common_v2/definitions/example_table.sqlx b/tests/api/projects/common_v2/definitions/example_table.sqlx index af1fe7f2..06d2b6b4 100644 --- a/tests/api/projects/common_v2/definitions/example_table.sqlx +++ b/tests/api/projects/common_v2/definitions/example_table.sqlx @@ -8,5 +8,5 @@ select * from ${ref("df_integration_test", "sample_data")} post_operations { GRANT SELECT ON ${self()} TO GROUP "${constants.allUsersEmailAddress}" --- - GRANT SELECT ON ${self()} TO GROUP "otherusers@dataform.co" + GRANT SELECT ON ${self()} TO GROUP "otherusers@example.com" } diff --git a/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx b/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx index 6d572358..6e5a77d0 100644 --- a/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx +++ b/tests/api/projects/common_v2/definitions/example_table_with_tags.sqlx @@ -11,7 +11,7 @@ config { select * from ${ref("sample_data")} post_operations { - GRANT SELECT ON ${self()} TO GROUP "allusers@dataform.co" + GRANT SELECT ON ${self()} TO GROUP "allusers@example.com" --- - GRANT SELECT ON ${self()} TO GROUP "otherusers@dataform.co" + GRANT SELECT ON ${self()} TO GROUP "otherusers@example.com" } diff --git a/tests/api/projects/common_v2/includes/constants.js b/tests/api/projects/common_v2/includes/constants.js index d9b11471..6b3d5d16 100644 --- a/tests/api/projects/common_v2/includes/constants.js +++ b/tests/api/projects/common_v2/includes/constants.js @@ -1,3 +1,3 @@ module.exports = { - allUsersEmailAddress: "allusers@dataform.co" + allUsersEmailAddress: "allusers@example.com" }; diff --git a/tests/api/projects/common_v2/workflow_settings.yaml b/tests/api/projects/common_v2/workflow_settings.yaml new file mode 100644 index 00000000..471bb051 --- /dev/null +++ b/tests/api/projects/common_v2/workflow_settings.yaml @@ -0,0 +1,5 @@ +warehouse: bigquery +defaultProject: tada-analytics +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +defaultLocation: US diff --git a/tests/api/projects/invalid_dataform_json/BUILD b/tests/api/projects/invalid_dataform_json/BUILD deleted file mode 100644 index e1fafe2e..00000000 --- a/tests/api/projects/invalid_dataform_json/BUILD +++ /dev/null @@ -1,17 +0,0 @@ -package(default_visibility = ["//tests:__subpackages__"]) - -load("//tools:node_modules.bzl", "node_modules") - -filegroup( - name = "files", - srcs = glob([ - "**/*.*", - ]), -) - -node_modules( - name = "node_modules", - deps = [ - "//packages/@dataform/core:package_tar", - ], -) diff --git a/tests/api/projects/invalid_dataform_json/dataform.json b/tests/api/projects/invalid_dataform_json/dataform.json deleted file mode 100644 index 4d5f4197..00000000 --- a/tests/api/projects/invalid_dataform_json/dataform.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "warehouse": "bigquery", - "defaultDatabase": "tada-analytics", - "defaultSchema": "rock&roll", - "assertionSchema": "df_integration_test_assertions" -} diff --git a/tests/api/projects/invalid_dataform_json/package.json b/tests/api/projects/invalid_dataform_json/package.json deleted file mode 100644 index e69de29b..00000000 diff --git a/tests/api/projects/never_finishes_compiling/BUILD b/tests/api/projects/never_finishes_compiling/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/api/projects/never_finishes_compiling/BUILD +++ b/tests/api/projects/never_finishes_compiling/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/api/projects/never_finishes_compiling/dataform.json b/tests/api/projects/never_finishes_compiling/dataform.json deleted file mode 100644 index 23df7d64..00000000 --- a/tests/api/projects/never_finishes_compiling/dataform.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "warehouse": "bigquery", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "defaultLocation": "US" -} diff --git a/tests/api/projects/never_finishes_compiling/workflow_settings.yaml b/tests/api/projects/never_finishes_compiling/workflow_settings.yaml new file mode 100644 index 00000000..0c84a073 --- /dev/null +++ b/tests/api/projects/never_finishes_compiling/workflow_settings.yaml @@ -0,0 +1,4 @@ +warehouse: bigquery +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +defaultLocation: US diff --git a/tests/api/utils/cancellable_promise.spec.ts b/tests/api/utils/cancellable_promise.spec.ts index 0d6cb9cb..d9d6d2cb 100644 --- a/tests/api/utils/cancellable_promise.spec.ts +++ b/tests/api/utils/cancellable_promise.spec.ts @@ -1,7 +1,7 @@ import { expect } from "chai"; -import { CancellablePromise } from "df/cli/api/utils/cancellable_promise"; -import { suite, test } from "df/testing"; +import { CancellablePromise } from "sa/cli/api/utils/cancellable_promise"; +import { suite, test } from "sa/testing"; suite("cancellable_promise", () => { test("cancel is called", () => { diff --git a/tests/api/utils/error_parsing.spec.ts b/tests/api/utils/error_parsing.spec.ts index 0b77567e..a6bfd390 100644 --- a/tests/api/utils/error_parsing.spec.ts +++ b/tests/api/utils/error_parsing.spec.ts @@ -1,7 +1,7 @@ import { expect } from "chai"; -import { parseBigqueryEvalError } from "df/cli/api/utils/error_parsing"; -import { suite, test } from "df/testing"; +import { parseBigqueryEvalError } from "sa/cli/api/utils/error_parsing"; +import { suite, test } from "sa/testing"; suite("error_parsing", () => { suite("bigquery", () => { diff --git a/tests/integration/BUILD b/tests/integration/BUILD index 5f1ee674..c0164078 100644 --- a/tests/integration/BUILD +++ b/tests/integration/BUILD @@ -9,7 +9,6 @@ ts_test_suite( "utils.ts", ], data = [ - "//test_credentials:bigquery.json", "//tests/integration/bigquery_project:files", "//tests/integration/bigquery_project:node_modules", ], diff --git a/tests/integration/bigquery.spec.ts b/tests/integration/bigquery.spec.ts index 01b03ce2..4d5145ec 100644 --- a/tests/integration/bigquery.spec.ts +++ b/tests/integration/bigquery.spec.ts @@ -1,16 +1,16 @@ import { expect } from "chai"; import Long from "long"; -import * as dfapi from "df/cli/api"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { BigQueryDbAdapter } from "df/cli/api/dbadapters/bigquery"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; -import { compile, dropAllTables, getTableRows, keyBy } from "df/tests/integration/utils"; - -suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) => { +import * as dfapi from "sa/cli/api"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { BigQueryDbAdapter } from "sa/cli/api/dbadapters/bigquery"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; +import { compile, dropAllTables, getTableRows, keyBy } from "sa/tests/integration/utils"; + +suite("@sqlanvil/integration/bigquery", { parallel: true }, ({ before, after }) => { const credentials = dfapi.credentials.read("test_credentials/bigquery.json"); let dbadapter: BigQueryDbAdapter; @@ -27,7 +27,7 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) // Drop schemas to make sure schema creation works. await dbadapter.execute( - "drop schema if exists `dataform-open-source.df_integration_test_project_e2e` cascade" + "drop schema if exists `your-bigquery-project.df_integration_test_project_e2e` cascade" ); // Run the project. @@ -39,13 +39,13 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) // Check the status of action execution. const expectedFailedActions = [ - "dataform-open-source.df_integration_test_assertions_project_e2e.example_assertion_fail", - "dataform-open-source.df_integration_test_project_e2e.example_operation_partial_fail" + "your-bigquery-project.df_integration_test_assertions_project_e2e.example_assertion_fail", + "your-bigquery-project.df_integration_test_project_e2e.example_operation_partial_fail" ]; for (const actionName of Object.keys(actionMap)) { const expectedResult = expectedFailedActions.includes(actionName) - ? dataform.ActionResult.ExecutionStatus.FAILED - : dataform.ActionResult.ExecutionStatus.SUCCESSFUL; + ? sqlanvil.ActionResult.ExecutionStatus.FAILED + : sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL; expect(actionMap[actionName].status).equals( expectedResult, JSON.stringify(actionMap[actionName], null, 4) @@ -54,13 +54,13 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) expect( actionMap[ - "dataform-open-source.df_integration_test_assertions_project_e2e.example_assertion_fail" + "your-bigquery-project.df_integration_test_assertions_project_e2e.example_assertion_fail" ].tasks[1].errorMessage ).to.eql("bigquery error: Assertion failed: query returned 1 row(s)."); expect( actionMap[ - "dataform-open-source.df_integration_test_project_e2e.example_operation_partial_fail" + "your-bigquery-project.df_integration_test_project_e2e.example_operation_partial_fail" ].tasks[0].errorMessage ).to.eql("bigquery error: Query error: Unrecognized name: invalid_column at [3:8]"); }); @@ -77,7 +77,7 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) // Run two iterations of the project. const adapter = new ExecutionSql( compiledGraph.projectConfig, - compiledGraph.dataformCoreVersion + compiledGraph.sqlanvilCoreVersion ); for (const runIteration of [ { @@ -98,13 +98,13 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) ]) { const executionGraph = await dfapi.build(compiledGraph, runIteration.runConfig, dbadapter); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); const [incrementalRows, incrementalMergeRows] = await Promise.all([ getTableRows( { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_incremental_tables", name: "example_incremental" }, @@ -113,7 +113,7 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) ), getTableRows( { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_incremental_tables", name: "example_incremental_merge" }, @@ -142,44 +142,44 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); // Check expected metadata. for (const expectedMetadata of [ { target: { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_dataset_metadata", name: "example_incremental" }, expectedDescription: "An incremental table", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "the timestamp", name: "user_timestamp", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ description: "the id", name: "user_id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ name: "nested_data", description: "some nested data with duplicate fields", - struct: dataform.Fields.create({ + struct: sqlanvil.Fields.create({ fields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "nested timestamp", name: "user_timestamp", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ description: "nested id", name: "user_id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ] }) @@ -189,16 +189,16 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) }, { target: { - database: "dataform-open-source", + database: "your-bigquery-project", schema: "df_integration_test_dataset_metadata", name: "example_view" }, expectedDescription: "An example view", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "val doc", name: "val", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ], expectedLabels: { @@ -258,60 +258,60 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) await dfapi.run(dbadapter, executionGraph).result(); const view = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_view" + "your-bigquery-project.df_integration_test_evaluate.example_view" ]; - let evaluations = await dbadapter.evaluate(dataform.Table.create(view)); + let evaluations = await dbadapter.evaluate(sqlanvil.Table.create(view)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const materializedView = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_materialized_view" + "your-bigquery-project.df_integration_test_evaluate.example_materialized_view" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(materializedView)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(materializedView)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const table = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_table" + "your-bigquery-project.df_integration_test_evaluate.example_table" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(table)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(table)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const operation = keyBy(compiledGraph.operations, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_operation" + "your-bigquery-project.df_integration_test_evaluate.example_operation" ]; - evaluations = await dbadapter.evaluate(dataform.Operation.create(operation)); + evaluations = await dbadapter.evaluate(sqlanvil.Operation.create(operation)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const assertion = keyBy(compiledGraph.assertions, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_assertions_evaluate.example_assertion_pass" + "your-bigquery-project.df_integration_test_assertions_evaluate.example_assertion_pass" ]; - evaluations = await dbadapter.evaluate(dataform.Assertion.create(assertion)); + evaluations = await dbadapter.evaluate(sqlanvil.Assertion.create(assertion)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const incremental = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ - "dataform-open-source.df_integration_test_evaluate.example_incremental" + "your-bigquery-project.df_integration_test_evaluate.example_incremental" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(incremental)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(incremental)); expect(evaluations.length).to.equal(2); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); expect(evaluations[1].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); }); @@ -319,12 +319,12 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) const target = (name: string) => ({ schema: "df_integration_test", name, - database: "dataform-open-source" + database: "your-bigquery-project" }); let evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, preOps: ["declare var string; set var = 'val';"], query: "select var as col;", target: target("example_valid_variable") @@ -332,26 +332,26 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, query: "select var as col;", target: target("example_invalid_variable") }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE + sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE ); }); test("invalid table fails validation and error parsed correctly", async () => { const evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, query: "selects\n1 as x", target: { name: "EXAMPLE_ILLEGAL_TABLE", @@ -361,19 +361,19 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE + sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE ); expect( - dataform.QueryEvaluationError.ErrorLocation.create(evaluations[0].error.errorLocation) - ).eql(dataform.QueryEvaluationError.ErrorLocation.create({ line: 1, column: 1 })); + sqlanvil.QueryEvaluationError.ErrorLocation.create(evaluations[0].error.errorLocation) + ).eql(sqlanvil.QueryEvaluationError.ErrorLocation.create({ line: 1, column: 1 })); }); }); suite("publish tasks", { parallel: true }, async () => { test("incremental pre and post ops, core version <= 1.4.8", async () => { // 1.4.8 used `preOps` and `postOps` instead of `incrementalPreOps` and `incrementalPostOps`. - const table: dataform.ITable = { - enumType: dataform.TableType.INCREMENTAL, + const table: sqlanvil.ITable = { + enumType: sqlanvil.TableType.INCREMENTAL, query: "query", preOps: ["preop task1", "preop task2"], incrementalQuery: "", @@ -409,7 +409,7 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) const { bigquery: bqMetadata } = metadata; expect(bqMetadata).to.have.property("jobId"); expect(bqMetadata.jobId).to.match( - /^dataform-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ + /^sqlanvil-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ ); expect(bqMetadata).to.have.property("totalBytesBilled"); expect(bqMetadata.totalBytesBilled).to.eql(Long.fromNumber(0)); @@ -423,7 +423,7 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) const { bigquery: bqMetadata } = metadata; expect(bqMetadata).to.have.property("jobId"); expect(bqMetadata.jobId).to.match( - /^dataform-jobPrefix-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ + /^sqlanvil-jobPrefix-[0-9A-Fa-f]{8}(?:-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$/ ); }); @@ -472,8 +472,8 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); const [fullSearch, partialSearch, columnSearch] = await Promise.all([ @@ -489,12 +489,12 @@ suite("@dataform/integration/bigquery", { parallel: true }, ({ before, after }) }); async function cleanWarehouse( - compiledGraph: dataform.CompiledGraph, + compiledGraph: sqlanvil.CompiledGraph, dbadapter: dbadapters.IDbAdapter ) { await dropAllTables( (await dfapi.build(compiledGraph, {}, dbadapter)).warehouseState.tables, - new ExecutionSql(compiledGraph.projectConfig, compiledGraph.dataformCoreVersion), + new ExecutionSql(compiledGraph.projectConfig, compiledGraph.sqlanvilCoreVersion), dbadapter ); } diff --git a/tests/integration/bigquery_project/BUILD b/tests/integration/bigquery_project/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/integration/bigquery_project/BUILD +++ b/tests/integration/bigquery_project/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/integration/bigquery_project/definitions/sample_data.sqlx b/tests/integration/bigquery_project/definitions/sample_data.sqlx index fdd52e59..1b18ef69 100644 --- a/tests/integration/bigquery_project/definitions/sample_data.sqlx +++ b/tests/integration/bigquery_project/definitions/sample_data.sqlx @@ -3,6 +3,6 @@ config { hermetic: false } -select ${when(dataform.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all +select ${when(sqlanvil.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all select 2 as val union all -select ${when(dataform.projectConfig.warehouse === "bigquery", "3", "2")} as val +select ${when(sqlanvil.projectConfig.warehouse === "bigquery", "3", "2")} as val diff --git a/tests/integration/bigquery_project/workflow_settings.yaml b/tests/integration/bigquery_project/workflow_settings.yaml index c69b1cb7..f68bbf0b 100644 --- a/tests/integration/bigquery_project/workflow_settings.yaml +++ b/tests/integration/bigquery_project/workflow_settings.yaml @@ -1,4 +1,4 @@ -defaultProject: dataform-open-source +defaultProject: your-bigquery-project defaultLocation: us defaultDataset: df_integration_test defaultAssertionDataset: df_integration_test_assertions diff --git a/tests/integration/postgres.spec.ts b/tests/integration/postgres.spec.ts index 74e3ba52..90402b8c 100644 --- a/tests/integration/postgres.spec.ts +++ b/tests/integration/postgres.spec.ts @@ -1,16 +1,16 @@ import { expect } from "chai"; -import * as dfapi from "df/api"; -import * as dbadapters from "df/api/dbadapters"; -import * as adapters from "df/core/adapters"; -import { RedshiftAdapter } from "df/core/adapters/redshift"; -import { targetAsReadableString } from "df/core/targets"; -import { dataform } from "df/protos/ts"; -import { suite, test } from "df/testing"; -import { compile, getTableRows, keyBy } from "df/tests/integration/utils"; -import { PostgresFixture } from "df/tools/postgres/postgres_fixture"; - -suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) => { +import * as dfapi from "sa/api"; +import * as dbadapters from "sa/api/dbadapters"; +import * as adapters from "sa/core/adapters"; +import { RedshiftAdapter } from "sa/core/adapters/redshift"; +import { targetAsReadableString } from "sa/core/targets"; +import { sqlanvil } from "sa/protos/ts"; +import { suite, test } from "sa/testing"; +import { compile, getTableRows, keyBy } from "sa/tests/integration/utils"; +import { PostgresFixture } from "sa/tools/postgres/postgres_fixture"; + +suite("@sqlanvil/integration/postgres", { parallel: true }, ({ before, after }) => { let dbadapter: dbadapters.IDbAdapter; const postgres = new PostgresFixture(5432, before, after); @@ -45,8 +45,8 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) ]; for (const actionName of Object.keys(actionMap)) { const expectedResult = expectedFailedActions.includes(actionName) - ? dataform.ActionResult.ExecutionStatus.FAILED - : dataform.ActionResult.ExecutionStatus.SUCCESSFUL; + ? sqlanvil.ActionResult.ExecutionStatus.FAILED + : sqlanvil.ActionResult.ExecutionStatus.SUCCESSFUL; expect(actionMap[actionName].status).equals( expectedResult, actionMap[actionName].tasks.map(task => task.errorMessage).join("\n") @@ -59,7 +59,7 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) ).to.eql("postgres error: Assertion failed: query returned 1 row(s)."); // Check the data in the incremental table. - const adapter = adapters.create(compiledGraph.projectConfig, compiledGraph.dataformCoreVersion); + const adapter = adapters.create(compiledGraph.projectConfig, compiledGraph.sqlanvilCoreVersion); let incrementalTable = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_project_e2e.example_incremental" ]; @@ -88,7 +88,7 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) ); executedGraph = await dfapi.run(dbadapter, executionGraph).result(); expect(executedGraph.status).equals( - dataform.RunResult.ExecutionStatus.SUCCESSFUL, + sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL, executedGraph.actions .map(action => action.tasks.map(task => task.errorMessage).join("\n")) .join("\n") @@ -122,8 +122,8 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); // Check expected metadata. @@ -135,15 +135,15 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) }, expectedDescription: "An incremental 'table'", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ description: "the 'timestamp'", name: "user_timestamp", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }), - dataform.Field.create({ + sqlanvil.Field.create({ description: "the id", name: "user_id", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ] }, @@ -154,10 +154,10 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) }, expectedDescription: "An example view", expectedFields: [ - dataform.Field.create({ + sqlanvil.Field.create({ name: "val", description: "val doc", - primitive: dataform.Field.Primitive.INTEGER + primitive: sqlanvil.Field.Primitive.INTEGER }) ] } @@ -239,58 +239,58 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) const view = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_evaluate.example_view" ]; - let evaluations = await dbadapter.evaluate(dataform.Table.create(view)); + let evaluations = await dbadapter.evaluate(sqlanvil.Table.create(view)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const table = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_evaluate.example_table" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(table)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(table)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const assertion = keyBy(compiledGraph.assertions, t => targetAsReadableString(t.target))[ "df_integration_test_assertions_evaluate.example_assertion_pass" ]; - evaluations = await dbadapter.evaluate(dataform.Assertion.create(assertion)); + evaluations = await dbadapter.evaluate(sqlanvil.Assertion.create(assertion)); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); const incremental = keyBy(compiledGraph.tables, t => targetAsReadableString(t.target))[ "df_integration_test_evaluate.example_incremental" ]; - evaluations = await dbadapter.evaluate(dataform.Table.create(incremental)); + evaluations = await dbadapter.evaluate(sqlanvil.Table.create(incremental)); expect(evaluations.length).to.equal(2); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); expect(evaluations[1].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS + sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS ); }); test("invalid table fails validation", async () => { const evaluations = await dbadapter.evaluate( - dataform.Table.create({ - enumType: dataform.TableType.TABLE, + sqlanvil.Table.create({ + enumType: sqlanvil.TableType.TABLE, query: "thisisillegal", target: { schema: "df_integration_test", name: "example_illegal_table", - database: "dataform-integration-tests" + database: "sqlanvil-integration-tests" } }) ); expect(evaluations.length).to.equal(1); expect(evaluations[0].status).to.equal( - dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE + sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE ); }); }); @@ -298,8 +298,8 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) suite("publish tasks", async () => { test("incremental pre and post ops, core version <= 1.4.8", async () => { // 1.4.8 used `preOps` and `postOps` instead of `incrementalPreOps` and `incrementalPostOps`. - const table: dataform.ITable = { - enumType: dataform.TableType.INCREMENTAL, + const table: sqlanvil.ITable = { + enumType: sqlanvil.TableType.INCREMENTAL, query: "query", preOps: ["preop task1", "preop task2"], incrementalQuery: "", @@ -338,8 +338,8 @@ suite("@dataform/integration/postgres", { parallel: true }, ({ before, after }) dbadapter ); const runResult = await dfapi.run(dbadapter, executionGraph).result(); - expect(dataform.RunResult.ExecutionStatus[runResult.status]).eql( - dataform.RunResult.ExecutionStatus[dataform.RunResult.ExecutionStatus.SUCCESSFUL] + expect(sqlanvil.RunResult.ExecutionStatus[runResult.status]).eql( + sqlanvil.RunResult.ExecutionStatus[sqlanvil.RunResult.ExecutionStatus.SUCCESSFUL] ); const [fullSearch, partialSearch, columnSearch] = await Promise.all([ diff --git a/tests/integration/postgres_project/BUILD b/tests/integration/postgres_project/BUILD index e1fafe2e..e83056bb 100644 --- a/tests/integration/postgres_project/BUILD +++ b/tests/integration/postgres_project/BUILD @@ -12,6 +12,6 @@ filegroup( node_modules( name = "node_modules", deps = [ - "//packages/@dataform/core:package_tar", + "//packages/@sqlanvil/core:package_tar", ], ) diff --git a/tests/integration/postgres_project/dataform.json b/tests/integration/postgres_project/dataform.json deleted file mode 100644 index 71f60132..00000000 --- a/tests/integration/postgres_project/dataform.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "warehouse": "postgres", - "defaultSchema": "df_integration_test", - "assertionSchema": "df_integration_test_assertions", - "vars": { - "fooVar": "bar" - } -} diff --git a/tests/integration/postgres_project/definitions/sample_data.sqlx b/tests/integration/postgres_project/definitions/sample_data.sqlx index f008e44b..35dfc87e 100644 --- a/tests/integration/postgres_project/definitions/sample_data.sqlx +++ b/tests/integration/postgres_project/definitions/sample_data.sqlx @@ -2,6 +2,6 @@ config { type: "view" } -select ${when(dataform.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all +select ${when(sqlanvil.projectConfig.vars.fooVar === "bar", "1", "2")} as val union all select 2 as val union all -select ${when(dataform.projectConfig.warehouse === "postgres", "3", "2")} as val +select ${when(sqlanvil.projectConfig.warehouse === "postgres", "3", "2")} as val diff --git a/tests/integration/postgres_project/workflow_settings.yaml b/tests/integration/postgres_project/workflow_settings.yaml new file mode 100644 index 00000000..ee9a0ad7 --- /dev/null +++ b/tests/integration/postgres_project/workflow_settings.yaml @@ -0,0 +1,5 @@ +warehouse: postgres +defaultDataset: df_integration_test +defaultAssertionDataset: df_integration_test_assertions +vars: + fooVar: bar diff --git a/tests/integration/utils.ts b/tests/integration/utils.ts index 4d7a1d6f..cb33a9c4 100644 --- a/tests/integration/utils.ts +++ b/tests/integration/utils.ts @@ -1,9 +1,9 @@ import { expect } from "chai"; -import * as dfapi from "df/cli/api"; -import * as dbadapters from "df/cli/api/dbadapters"; -import { ExecutionSql } from "df/cli/api/dbadapters/execution_sql"; -import { dataform } from "df/protos/ts"; +import * as dfapi from "sa/cli/api"; +import * as dbadapters from "sa/cli/api/dbadapters"; +import { ExecutionSql } from "sa/cli/api/dbadapters/execution_sql"; +import { sqlanvil } from "sa/protos/ts"; export function keyBy(values: V[], keyFn: (value: V) => string): { [key: string]: V } { return values.reduce((map, value) => { @@ -13,7 +13,7 @@ export function keyBy(values: V[], keyFn: (value: V) => string): { [key: stri } export async function dropAllTables( - tables: dataform.ITableMetadata[], + tables: sqlanvil.ITableMetadata[], executionSql: ExecutionSql, dbadapter: dbadapters.IDbAdapter ) { @@ -23,7 +23,7 @@ export async function dropAllTables( } export async function getTableRows( - target: dataform.ITarget, + target: sqlanvil.ITarget, executionSql: ExecutionSql, dbadapter: dbadapters.IDbAdapter ) { @@ -33,7 +33,7 @@ export async function getTableRows( export async function compile( projectDir: string, schemaSuffixOverride: string, - projectConfigOverrides?: dataform.IProjectConfig + projectConfigOverrides?: sqlanvil.IProjectConfig ) { const compiledGraph = await dfapi.compile({ projectDir, diff --git a/tools/postgres/BUILD b/tools/postgres/BUILD index e57ec0f9..1b3770a3 100644 --- a/tools/postgres/BUILD +++ b/tools/postgres/BUILD @@ -1,13 +1,5 @@ package(default_visibility = ["//visibility:public"]) -load("@io_bazel_rules_docker//container:image.bzl", "container_image") - -# Exists purely to give a clean name to the postgres Docker image. -container_image( - name = "postgres_image", - base = "@postgres//image", -) - load("//tools:ts_library.bzl", "ts_library") ts_library( @@ -15,15 +7,11 @@ ts_library( srcs = glob( ["*.ts"], ), - data = [ - ":postgres_image", - ], deps = [ "//common/promises", "//testing", "@npm//@types/node", "@npm//@types/pg", - "@npm//@types/pg-query-stream", "@npm//@types/uuid", "@npm//pg", "@npm//promise-pool-executor", diff --git a/tools/postgres/postgres_fixture.ts b/tools/postgres/postgres_fixture.ts index 490a6fd2..db2c87f6 100644 --- a/tools/postgres/postgres_fixture.ts +++ b/tools/postgres/postgres_fixture.ts @@ -1,26 +1,19 @@ import * as pg from "pg"; import { execSync } from "child_process"; -import { sleepUntil } from "df/common/promises"; -import { IHookHandler } from "df/testing"; +import { sleepUntil } from "sa/common/promises"; +import { IHookHandler } from "sa/testing"; const USE_CLOUD_BUILD_NETWORK = !!process.env.USE_CLOUD_BUILD_NETWORK; -const DOCKER_CONTAINER_NAME = "postgres-df-integration-testing"; +const DOCKER_CONTAINER_NAME = "postgres-sa-integration-testing"; +const POSTGRES_IMAGE = "postgres:15-alpine"; const POSTGRES_SERVE_PORT = 5432; export class PostgresFixture { public static readonly host = USE_CLOUD_BUILD_NETWORK ? DOCKER_CONTAINER_NAME : "localhost"; - private static imageLoaded = false; - constructor(port: number, setUp: IHookHandler, tearDown: IHookHandler) { setUp("starting postgres", async () => { - if (!PostgresFixture.imageLoaded) { - // Load the postgres image into the local Docker daemon. - execSync("tools/postgres/postgres_image.executable"); - PostgresFixture.imageLoaded = true; - } - // Run the postgres Docker image. execSync( [ "docker run", @@ -30,7 +23,7 @@ export class PostgresFixture { "-d", `-p ${port}:${POSTGRES_SERVE_PORT}`, USE_CLOUD_BUILD_NETWORK ? "--network cloudbuild" : "", - "bazel/tools/postgres:postgres_image" + POSTGRES_IMAGE ].join(" ") ); diff --git a/tsconfig.json b/tsconfig.json index b04b1687..77a72946 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -2,7 +2,7 @@ "compilerOptions": { "baseUrl": ".", "paths": { - "df/*": [ + "sa/*": [ "*", "bazel-bin/*", "bazel-genfiles/*" diff --git a/typedoc.json b/typedoc.json index 78d767c5..a40b7de1 100644 --- a/typedoc.json +++ b/typedoc.json @@ -1,7 +1,7 @@ { "out": "newdocs", "plugin": ["typedoc-plugin-markdown"], - "name": "Dataform Javascript API Reference", + "name": "sqlanvil Javascript API Reference", "excludeExternals": true, "exclude": [ "*", "!core/**" diff --git a/vscode/BUILD b/vscode/BUILD index 64c402ca..f1e1b364 100644 --- a/vscode/BUILD +++ b/vscode/BUILD @@ -43,7 +43,7 @@ sh_binary( srcs = ["packager.sh"], data = [ ":language-configuration.json", - ":dataform_logo.png", + ":sqlanvil_logo.png", ":package.json", ":README.md", ":LICENSE", diff --git a/vscode/README.md b/vscode/README.md index c369386b..def4ac2f 100644 --- a/vscode/README.md +++ b/vscode/README.md @@ -1,14 +1,17 @@ +# sqlanvil VSCode Extension -> **❗IMPORTANT**: This extension has been deprecated and will no longer receive updates -> -> We recommend you migrate to the [open sourced alternative](https://marketplace.visualstudio.com/items?itemName=ashishalex.dataform-lsp-vscode) or to the [Google Cloud Dataform UI](https://cloud.google.com/dataform). +Syntax highlighting and project compilation for sqlanvil projects in VSCode. ## Usage -**To run this extension you will need the dataform cli installed globally: `npm i -g @dataform/cli`.** +This extension requires the sqlanvil CLI installed globally: + +```bash +npm i -g @sqlanvil/cli +``` Includes: - Syntax highlighting for `.sqlx` files - Realtime compilation of your project -- `cmd + click` on a `ref()` function to go to the file that it references +- `cmd + click` on a `ref()` function to navigate to the referenced file diff --git a/vscode/contributing.md b/vscode/contributing.md index 70d55c55..60d62c2d 100644 --- a/vscode/contributing.md +++ b/vscode/contributing.md @@ -2,12 +2,12 @@ There are two routes to publishing new versions: ### Web (actually working): -- In the main repo run `bazel run vscode:packager /tmp/dataform-package.vsix` -- Take the generated package and upload it [here](https://marketplace.visualstudio.com/manage/publishers/dataform) +- In the main repo run `bazel run vscode:packager /tmp/sqlanvil-package.vsix` +- Take the generated package and upload it [here](https://marketplace.visualstudio.com/manage/publishers/ihistand) ### CLI (not working right now) -- In the main repo run `bazel run vscode:packager /tmp/dataform-package.vsix` +- In the main repo run `bazel run vscode:packager /tmp/sqlanvil-package.vsix` - Then take the package and paste it into this directory - Then run `vsce publish` - Currently this doesn't work as `vsce publish` will try to publish typescript files diff --git a/vscode/extension.ts b/vscode/extension.ts index 19c51e93..abd30aa3 100644 --- a/vscode/extension.ts +++ b/vscode/extension.ts @@ -31,13 +31,13 @@ export async function activate(context: vscode.ExtensionContext) { }; client = new LanguageClient( - "dataformLanguageServer", - "Dataform Language Server", + "sqlanvilLanguageServer", + "sqlanvil Language Server", serverOptions, clientOptions ); - const compile = vscode.commands.registerCommand("dataform.compile", () => { + const compile = vscode.commands.registerCommand("sqlanvil.compile", () => { const _ = client.sendRequest("compile"); }); @@ -61,12 +61,12 @@ export async function activate(context: vscode.ExtensionContext) { // We also can add the extension to "extensionDependencies" in package.json, // but this way we can avoid forcing users to install the extension. // You can control this recommendation behavior through the setting. - if (workspace.getConfiguration("dataform").get("recommendYamlExtension")) { + if (workspace.getConfiguration("sqlanvil").get("recommendYamlExtension")) { const yamlExtension = vscode.extensions.getExtension("redhat.vscode-yaml"); if (!yamlExtension) { await vscode.window .showInformationMessage( - "The Dataform extension recommends installing the YAML extension for workflow_settings.yaml support.", + "The sqlanvil extension recommends installing the YAML extension for workflow_settings.yaml support.", "Install", "Don't show again" ) @@ -77,7 +77,7 @@ export async function activate(context: vscode.ExtensionContext) { } else if (selection === "Don't show again") { // Disable the recommendation workspace - .getConfiguration("dataform") + .getConfiguration("sqlanvil") .update("recommendYamlExtension", false, vscode.ConfigurationTarget.Global); } }); diff --git a/vscode/package-lock.json b/vscode/package-lock.json index 5841e9e1..574414a6 100644 --- a/vscode/package-lock.json +++ b/vscode/package-lock.json @@ -1,11 +1,11 @@ { - "name": "dataform", + "name": "sqlanvil", "version": "0.0.14", "lockfileVersion": 2, "requires": true, "packages": { "": { - "name": "dataform", + "name": "sqlanvil", "version": "0.0.14", "dependencies": { "vscode-languageclient": "^6.1.3", diff --git a/vscode/package.json b/vscode/package.json index f9a400cc..8e9f5850 100644 --- a/vscode/package.json +++ b/vscode/package.json @@ -1,15 +1,15 @@ { - "name": "dataform", + "name": "sqlanvil", "categories": [ "Programming Languages" ], - "description": "Syntax highlighting, compilation, and intellisense for Dataform and SQLX projects.", - "displayName": "Dataform", - "publisher": "dataform", - "version": "0.0.16", - "icon": "dataform_logo.png", + "description": "Syntax highlighting, compilation, and intellisense for sqlanvil and SQLX projects.", + "displayName": "sqlanvil", + "publisher": "ihistand", + "version": "0.0.1", + "icon": "sqlanvil_logo.png", "repository": { - "url": "https://github.com/dataform-co/dataform/tree/master/vscode" + "url": "https://github.com/ihistand/sqlanvil/tree/main/vscode" }, "engines": { "vscode": "^1.48.0" @@ -26,22 +26,22 @@ "contributes": { "configuration": [ { - "title": "Dataform", + "title": "sqlanvil", "properties": { - "dataform.compileOnSave": { + "sqlanvil.compileOnSave": { "type": "boolean", "default": true, "markdownDescription": "Whether to re-compile project each time an `.sqlx` file is saved. Will always re-compile on initialize and on config change." }, - "dataform.compilerOptions": { + "sqlanvil.compilerOptions": { "type": "array", "items": { "type": "string" }, "default": [], - "markdownDescription": "An array of additional arguments the extension should pass to the Dataform cli while executing `dataform compile --json`." + "markdownDescription": "An array of additional arguments the extension should pass to the sqlanvil cli while executing `sqlanvil compile --json`." }, - "dataform.recommendYamlExtension": { + "sqlanvil.recommendYamlExtension": { "type": "boolean", "default": true, "markdownDescription": "Whether to recommend the YAML extension for validating `workflow_settings.yaml`." @@ -51,9 +51,9 @@ ], "commands": [ { - "command": "dataform.compile", + "command": "sqlanvil.compile", "title": "Compile project", - "category": "Dataform" + "category": "sqlanvil" } ], "languages": [ @@ -79,4 +79,4 @@ } ] } -} \ No newline at end of file +} diff --git a/vscode/server.ts b/vscode/server.ts index ec404bd9..b830e260 100644 --- a/vscode/server.ts +++ b/vscode/server.ts @@ -10,11 +10,11 @@ import { } from "vscode-languageserver"; import { TextDocument } from "vscode-languageserver-textdocument"; -import { dataform } from "df/protos/ts"; +import { sqlanvil } from "sa/protos/ts"; const connection = createConnection(ProposedFeatures.all); const documents: TextDocuments = new TextDocuments(TextDocument); -let CACHED_COMPILE_GRAPH: dataform.ICompiledGraph = null; +let CACHED_COMPILE_GRAPH: sqlanvil.ICompiledGraph = null; let WORKSPACE_ROOT_FOLDER: string = null; let settings = { @@ -59,23 +59,23 @@ documents.onDidSave(change => { }); async function applySettings() { - settings = await connection.workspace.getConfiguration("dataform"); + settings = await connection.workspace.getConfiguration("sqlanvil"); } async function compileAndValidate() { let compilationFailed = false; - const spawnedProcess = spawn("dataform", ["compile", "--json", ...settings.compilerOptions], { + const spawnedProcess = spawn("sqlanvil", ["compile", "--json", ...settings.compilerOptions], { shell: true }); const compileResult = await getProcessResult(spawnedProcess); if (compileResult.exitCode !== 0) { // tslint:disable-next-line: no-console - console.error("Error running 'dataform compile':", compileResult); + console.error("Error running 'sqlanvil compile':", compileResult); if (compileResult.error?.code === "ENOENT") { connection.sendNotification( "error", - "Errors encountered when running 'dataform' CLI. Please ensure that the CLI is installed and up-to-date: 'npm i -g @dataform/cli'." + "Errors encountered when running 'sqlanvil' CLI. Please ensure that the CLI is installed and up-to-date: 'npm i -g @sqlanvil/cli'." ); return; } else { @@ -83,15 +83,15 @@ async function compileAndValidate() { } } - let parsedResult: dataform.ICompiledGraph = null; + let parsedResult: sqlanvil.ICompiledGraph = null; try { parsedResult = JSON.parse(compileResult.stdout); } catch (e) { // tslint:disable-next-line: no-console - console.error("Error parsing 'dataform compile' output", e); + console.error("Error parsing 'sqlanvil compile' output", e); connection.sendNotification( "error", - "Error parsing 'dataform compile' output. Please check the output for more information." + "Error parsing 'sqlanvil compile' output. Please check the output for more information." ); return; } @@ -106,7 +106,7 @@ async function compileAndValidate() { if (compilationFailed) { connection.sendNotification( "error", - "Errors encountered when running 'dataform' CLI. Please check the output for more information." + "Errors encountered when running 'sqlanvil' CLI. Please check the output for more information." ); return; } @@ -133,7 +133,7 @@ async function getProcessResult(childProcess: ChildProcess) { function gatherAllActions( graph = CACHED_COMPILE_GRAPH -): Array { +): Array { return [].concat( graph.tables ?? [], graph.operations ?? [], @@ -190,7 +190,7 @@ connection.onDefinition( })[0].refContent; // split to dataset, schema and name - const linkedTable: dataform.ITarget = { database: null, schema: null, name: null }; + const linkedTable: sqlanvil.ITarget = { database: null, schema: null, name: null }; const splitMatch = clickedRef.match( /^ref\s*\(\s*(["'](.+?)["'])\s*(,\s*["'](.+?)["']\s*)?(,\s*["'](.+?)["']\s*)?,?\s*\)$/ // tslint:disable-line ); diff --git a/vscode/dataform_logo.png b/vscode/sqlanvil_logo.png similarity index 100% rename from vscode/dataform_logo.png rename to vscode/sqlanvil_logo.png diff --git a/vscode/workflow_settings_yaml.schema.json b/vscode/workflow_settings_yaml.schema.json index 89e76da0..c2a1815d 100644 --- a/vscode/workflow_settings_yaml.schema.json +++ b/vscode/workflow_settings_yaml.schema.json @@ -1,12 +1,12 @@ { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", - "$comment": "Created from 'protos/configs.proto'. Even options are specified as 'required' in proto, they might be specified from the CLI options, so leave them optional except for the dataformCoreVersion which is required in compile.", + "$comment": "Created from 'protos/configs.proto'. Even options are specified as 'required' in proto, they might be specified from the CLI options, so leave them optional except for the sqlanvilCoreVersion which is required in compile.", "additionalProperties": false, "properties": { - "dataformCoreVersion": { + "sqlanvilCoreVersion": { "type": "string", - "description": "The desired dataform core version to compile against." + "description": "The desired sqlanvil core version to compile against." }, "defaultProject": { "type": "string", @@ -53,6 +53,6 @@ } }, "required": [ - "dataformCoreVersion" + "sqlanvilCoreVersion" ] } \ No newline at end of file diff --git a/yarn.lock b/yarn.lock index 5b0a95dd..a56c740e 100644 --- a/yarn.lock +++ b/yarn.lock @@ -454,6 +454,15 @@ resolved "https://registry.yarnpkg.com/@types/node/-/node-16.18.91.tgz#3e7b3b3d28f740e3e2d4ceb7ad9d16e6b9277c91" integrity sha512-h8Q4klc8xzc9kJKr7UYNtJde5TU2qEePVyH3WyzJaUC+3ptyc5kPQbWOIUcn8ZsG5+KSkq+P0py0kC0VqxgAXw== +"@types/pg@^8.11.0": + version "8.20.0" + resolved "https://registry.yarnpkg.com/@types/pg/-/pg-8.20.0.tgz#8bd03d3ac6b19143a8de7d66a9d13da32cd91526" + integrity sha512-bEPFOaMAHTEP1EzpvHTbmwR8UsFyHSKsRisLIHVMXnpNefSbGA1bD6CVy+qKjGSqmZqNqBDV2azOBo8TgkcVow== + dependencies: + "@types/node" "*" + pg-protocol "*" + pg-types "^2.2.0" + "@types/readline-sync@^1.4.3": version "1.4.3" resolved "https://registry.yarnpkg.com/@types/readline-sync/-/readline-sync-1.4.3.tgz#eac55a39d5a349912062c9e5216cd550c07fd9c8" @@ -509,6 +518,11 @@ resolved "https://registry.yarnpkg.com/@types/tough-cookie/-/tough-cookie-2.3.5.tgz#9da44ed75571999b65c37b60c9b2b88db54c585d" integrity "sha1-naRO11VxmZtlw3tgybK4jbVMWF0= sha512-SCcK7mvGi3+ZNz833RRjFIxrn4gI1PPR3NtuIS+6vMkvmsGjosqTJwRt5bAEFLRz+wtJMWv8+uOnZf2hi2QXTg==" +"@types/uuid@^9.0.0": + version "9.0.8" + resolved "https://registry.yarnpkg.com/@types/uuid/-/uuid-9.0.8.tgz#7545ba4fc3c003d6c756f651f3bf163d8f0f29ba" + integrity sha512-jg+97EGIcY9AGHJJRaaPVgetKDsrTgbRjQ5Msgjh/DQKEFl0DtyRr/VCOyD1T2R1MNeWPK/u7JoGhlDZnKBAfA== + "@types/vscode@^1.45.1": version "1.45.1" resolved "https://registry.yarnpkg.com/@types/vscode/-/vscode-1.45.1.tgz#672fb8c2cc33cf14cd4d3bdaa19bb294fe2b2706" @@ -3316,6 +3330,74 @@ performance-now@^2.1.0: resolved "https://registry.yarnpkg.com/performance-now/-/performance-now-2.1.0.tgz#6309f4e0e5fa913ec1c69307ae364b4b377c9e7b" integrity "sha1-Ywn04OX6kT7BxpMHrjZLSzd8nns= sha512-7EAHlyLHI56VEIdK57uwHdHKIaAGbnXPiw0yWbarQZOKaKpvUIgW0jWRVLiatnM+XXlSwsanIBH/hzGMJulMow==" +pg-cloudflare@^1.4.0: + version "1.4.0" + resolved "https://registry.yarnpkg.com/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz#4b4c20e6d8ae531d400730f4804571a8d62f1497" + integrity sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A== + +pg-connection-string@^2.13.0: + version "2.13.0" + resolved "https://registry.yarnpkg.com/pg-connection-string/-/pg-connection-string-2.13.0.tgz#8678113465a5af3cc977dcb51eadc847b27aa2de" + integrity sha512-EMnU9E2fSULdsbErBbMaXJvFeD9B4+nPcM3f+4lsiCR0BHLPrLVjv3DbyM2hgQQviKJaTWIRRTjKjWlHg3p2ig== + +pg-cursor@^2.20.0: + version "2.20.0" + resolved "https://registry.yarnpkg.com/pg-cursor/-/pg-cursor-2.20.0.tgz#c8f8c1d2bdf11c462e37aa595259bb520eb5668e" + integrity sha512-HP/EbUafheaUOs7DxlG6tda/rhmsX2hCTJJJ+gCnhljGyNEs6pBHddbNuomlW3DqEhP3zYD+GqBWkYnJPIZ4tA== + +pg-int8@1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/pg-int8/-/pg-int8-1.0.1.tgz#943bd463bf5b71b4170115f80f8efc9a0c0eb78c" + integrity sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw== + +pg-pool@^3.14.0: + version "3.14.0" + resolved "https://registry.yarnpkg.com/pg-pool/-/pg-pool-3.14.0.tgz#f35ae4eb846780cad71af24099b3edfa9781ad90" + integrity sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw== + +pg-protocol@*, pg-protocol@^1.14.0: + version "1.14.0" + resolved "https://registry.yarnpkg.com/pg-protocol/-/pg-protocol-1.14.0.tgz#c1f045b74274b007078c687147141f785f59b8de" + integrity sha512-n5taZ1kO3s9ngDTVxsEznOqCyToTgz0FLuPq0B33COy5pPpuWJpY3/2oRBVETuOgzdqRXfWpM9HIhp2LBBT1BA== + +pg-query-stream@^4.5.3: + version "4.15.0" + resolved "https://registry.yarnpkg.com/pg-query-stream/-/pg-query-stream-4.15.0.tgz#37ab9ed7eb36a15149d25d9a06b3e7fe7a3fbb6e" + integrity sha512-hyCs0PaOyCWqC90N9vyHL2wVNyR3OjnqFrLvgX74Pyh0JihTKUywpPyhKnuVp9TCCcGz7Fc9LdtxoBU+3nv10A== + dependencies: + pg-cursor "^2.20.0" + +pg-types@2.2.0, pg-types@^2.2.0: + version "2.2.0" + resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-2.2.0.tgz#2d0250d636454f7cfa3b6ae0382fdfa8063254a3" + integrity sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA== + dependencies: + pg-int8 "1.0.1" + postgres-array "~2.0.0" + postgres-bytea "~1.0.0" + postgres-date "~1.0.4" + postgres-interval "^1.1.0" + +pg@^8.11.3: + version "8.21.0" + resolved "https://registry.yarnpkg.com/pg/-/pg-8.21.0.tgz#d7fa2118d960cec5cc7d2b24525f9850dd5932b0" + integrity sha512-AUP1EYJuHraQGsVoCQVIcM7TEJVGtDzxWtGFZd8rds9d+CCXlU5Js1rYgfLNvxy9iJrpHjGrRjoi/3BT9fRyiA== + dependencies: + pg-connection-string "^2.13.0" + pg-pool "^3.14.0" + pg-protocol "^1.14.0" + pg-types "2.2.0" + pgpass "1.0.5" + optionalDependencies: + pg-cloudflare "^1.4.0" + +pgpass@1.0.5: + version "1.0.5" + resolved "https://registry.yarnpkg.com/pgpass/-/pgpass-1.0.5.tgz#9b873e4a564bb10fa7a7dbd55312728d422a223d" + integrity sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug== + dependencies: + split2 "^4.1.0" + picocolors@^1.1.1: version "1.1.1" resolved "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz#3d321af3eab939b083c8f929a1d12cda81c26b6b" @@ -3338,6 +3420,28 @@ posix-character-classes@^0.1.0: resolved "https://registry.yarnpkg.com/posix-character-classes/-/posix-character-classes-0.1.1.tgz#01eac0fe3b5af71a2a6c02feabb8c1fef7e00eab" integrity "sha1-AerA/jta9xoqbAL+q7jB/vfgDqs= sha512-xTgYBc3fuo7Yt7JbiuFxSYGToMoz8fLoE6TC9Wx1P/u+LfeThMOAqmuyECnlBaaJb+u1m9hHiXUEtwW4OzfUJg==" +postgres-array@~2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/postgres-array/-/postgres-array-2.0.0.tgz#48f8fce054fbc69671999329b8834b772652d82e" + integrity sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA== + +postgres-bytea@~1.0.0: + version "1.0.1" + resolved "https://registry.yarnpkg.com/postgres-bytea/-/postgres-bytea-1.0.1.tgz#c40b3da0222c500ff1e51c5d7014b60b79697c7a" + integrity sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ== + +postgres-date@~1.0.4: + version "1.0.7" + resolved "https://registry.yarnpkg.com/postgres-date/-/postgres-date-1.0.7.tgz#51bc086006005e5061c591cee727f2531bf641a8" + integrity sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q== + +postgres-interval@^1.1.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/postgres-interval/-/postgres-interval-1.2.0.tgz#b460c82cb1587507788819a06aa0fffdb3544695" + integrity sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ== + dependencies: + xtend "^4.0.0" + prelude-ls@~1.1.2: version "1.1.2" resolved "https://registry.yarnpkg.com/prelude-ls/-/prelude-ls-1.1.2.tgz#21932a549f5e52ffd9a827f570e04be62a97da54" @@ -3882,6 +3986,11 @@ split-string@^3.0.1, split-string@^3.0.2: dependencies: extend-shallow "^3.0.0" +split2@^4.1.0: + version "4.2.0" + resolved "https://registry.yarnpkg.com/split2/-/split2-4.2.0.tgz#c9c5920904d148bab0b9f67145f245a86aadbfa4" + integrity sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg== + sprintf-js@~1.0.2: version "1.0.3" resolved "https://registry.yarnpkg.com/sprintf-js/-/sprintf-js-1.0.3.tgz#04e6926f662895354f3dd015203633b857297e2c" @@ -4638,6 +4747,11 @@ xmlcreate@^2.0.4: resolved "https://registry.yarnpkg.com/xmlcreate/-/xmlcreate-2.0.4.tgz#0c5ab0f99cdd02a81065fa9cd8f8ae87624889be" integrity "sha1-DFqw+ZzdAqgQZfqc2Piuh2JIib4= sha512-nquOebG4sngPmGPICTS5EnxqhKbCmz5Ox5hsszI2T6U5qdrJizBc+0ilYSEjTSzU0yZcmvppztXe/5Al5fUwdg==" +xtend@^4.0.0: + version "4.0.2" + resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" + integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== + y18n@^5.0.5: version "5.0.8" resolved "https://registry.yarnpkg.com/y18n/-/y18n-5.0.8.tgz#7f4934d0f7ca8c56f95314939ddcd2dd91ce1d55"