refactor: rename dataform → sqlanvil - #1
Closed
ihistand wants to merge 13 commits into
Closed
Conversation
Add three planning docs for the dataform → sqlanvil work: - CLAUDE.md: project guidance for Claude Code sessions - docs/postgres_first_class_design.md: spec for Postgres + Supabase adapters (replaces antigravity assessment's Phase 3+ framing) - docs/rename_checklist.md: exhaustive rename surface map Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- All 8 .proto files: package dataform → package sqlanvil - Java options: com.dataform.protos → com.sqlanvil.protos - Go options: github.com/dataform-co/dataform → github.com/ihistand/sqlanvil - Field rename (wire-breaking): dataform_core_version → sqlanvil_core_version (in configs.proto + core.proto) - WORKSPACE name: df → sa - Bazel target //protos:dataform_proto → //protos:sqlanvil_proto cloud.google.com/dataform/docs/... URL comments retained — factually correct references to BigQuery partitioning docs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…te dead infra - git mv packages/@dataform → packages/@SQLAnvil (3 subdirs: cli, core, package.layer.json; all symbol-pinned package_name fields updated) - Sed @dataform/ → @sqlanvil/ across all BUILD files, .bzl, scripts, .ts/.js/.sqlx test fixtures (40+ files) - Root BUILD: gazelle:prefix → github.com/ihistand/sqlanvil - .github/CODEOWNERS: dataform-co/dataform-reviewers → ihistand - scripts/run: build target → //packages/@sqlanvil/cli:bin - Test fixture emails: allusers@dataform.co → allusers@example.com (consistent across constants.js, .sqlx fixtures, and matching assertions in tests/api/projects.spec.ts) Deletions (dead infra targeting upstream's GCP project): - cloudbuild-publish.yaml, cloudbuild-test.yaml, cloudbuild-version.yaml (Google Cloud Build configs for dataform-co's project) - scripts/publish, scripts/update_test_credentials (publishing to upstream npm + KMS key access against dataform-co's project) - test_credentials/{BUILD,bigquery.json.enc} + all //test_credentials:bigquery.json refs from 4 BUILD files (integration tests against BQ will need fresh credential wiring) NOT renamed (preserved historical/upstream references): - contributing.md upstream issue tracker links - readme.md links to Google Cloud Dataform docs - .bazelrc commented-out remote cache (already disabled) - cli/credentials.ts docs link (TODO: update once sqlanvil docs exist) - All `dataform.*` proto namespace usages in TS — deferred to phase C Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…efaults
Path mapping:
- tsconfig.json paths: df/* → sa/*
- All `from "df/..."` / require("df/...") / import("df/...") → sa/* (450+)
- Runtime path strings in jit_loader.js + webpack configs + cli/index_test_base.ts
Proto namespace cascade (proto package dataform → sqlanvil triggers TS
generation under `sqlanvil` namespace):
- import { dataform, google } → import { sqlanvil, google } (and reverse)
- dataform.IFoo / dataform.JFoo / dataform.DbAdapter / etc. → sqlanvil.*
- camelCase field name dataformCoreVersion → sqlanvilCoreVersion
- Internal function name dataformCompile → sqlanvilCompile
Framework runtime defaults:
- globalAny.dataform → globalAny.sqlanvil (the user-facing sqlx namespace)
- __dataform_current_file → __sqlanvil_current_file (private global)
- "_dataform" table folder root default → "_sqlanvil"
- "dataform_assertions" assertion schema default → "sqlanvil_assertions"
- defaultSchema/defaultDatabase/defaultProject "dataform" → "sqlanvil"
- BigQuery job ID prefix "dataform-" → "sqlanvil-" (for BQ telemetry attribution)
- Test fixture labels team:"dataform" → team:"sqlanvil"
Doc comment anchors: configs#dataform-ActionConfig-* → configs#sqlanvil-*
(reference the docs site anchor IDs which derive from proto package name)
Test fixture strings: dataform-iceberg-test/-partial, dataform-database,
dataform-conn_id, dataform-integration-tests, dataform-test-schema-suffix
all renamed for consistency.
Preserved (will address in later phases or kept as upstream attribution):
- dataform.json strings (Phase D deletes the code path)
- dataform-co/dataform URLs (legitimate upstream attribution)
- dataform-open-source GCP project (test_credentials already deleted;
these integration tests are broken regardless)
- CLI help text "dataform [command]" (Phase E renames yargs scriptName)
- "invalid dataform json throws error" test name (Phase D deletes test)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Upstream Dataform marked dataform.json deprecated in favor of
workflow_settings.yaml. sqlanvil completes the cleanup — no
sqlanvil.json parallel, no back-compat fallback.
Removed code:
- core/workflow_settings.ts: readWorkflowSettings() dataform.json fallback
and "deprecated alongside" error path (workflow_settings.yaml is the
only source of truth now)
- cli/index.ts: dataformJsonPath presence check in projectDirMustExistOption
("does not appear to be a dataform directory" → "sqlanvil directory")
- cli/vm/compile.ts: global.dataformJson VM-time require
- cli/api/commands/init.ts: dataform.json existence check + init error
message ("Cannot init dataform project" → "Cannot init sqlanvil project")
- testing/run_core.ts: VM dataformJson global injection +
VALID_DATAFORM_JSON constant
- core/main_test.ts: 5 test blocks exercising the legacy path
("a valid dataform.json", "fails when both...", "invalid json file",
"contains unknown fields", "variables must be strings")
- tests/api/projects.spec.ts: "invalid dataform json throws error" test
- cli/index_compile_test.ts: replaced dataform.json fixture write with
equivalent workflow_settings.yaml
Removed fixtures:
- tests/api/projects/invalid_dataform_json/ (entire directory)
- tests/api/BUILD: invalid_dataform_json deps
Migrated fixtures dataform.json → workflow_settings.yaml:
- tests/integration/postgres_project/
- tests/api/projects/never_finishes_compiling/
- tests/api/projects/common_v2/
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… sqlanvil
CLI (cli/index.ts):
- yargs scriptName: dataform → sqlanvil (sets program name in --help)
- All command descriptions: "Compile the dataform project" → "sqlanvil"
- Error messages: "dataform encountered an error" → "sqlanvil"
- "for Dataform to use" → "for sqlanvil to use"
- projectDirOption describe: "Dataform project directory" → "sqlanvil"
- jobPrefix template: dataform-${...}- → sqlanvil-${...}-
Help test (cli/index_help_test.ts): assertion strings match new help output.
Init test (cli/index_init_test.ts): defaultAssertionDataset literal + error
messages.
cli/credentials.ts: replaced broken docs link
(docs.sqlanvil.co/dataform-cli — which didn't exist on either domain) with
Google Cloud's official iam.docs.google.com link to service-account-key
creation.
VSCode extension (vscode/):
- extension.ts: language server ID + display name, command IDs
(dataform.compile → sqlanvil.compile), getConfiguration("sqlanvil")
- server.ts: spawn("sqlanvil"), all error message text
- package.json: rewrote manifest — name/publisher/displayName/repository/
icon/configuration namespace/command IDs all switch to sqlanvil. Version
reset 0.0.16 → 0.0.1 (fresh extension).
- Renamed vscode/dataform_logo.png → vscode/sqlanvil_logo.png (placeholder;
actual icon needs design before publishing)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- IDataformExtension → ISqlanvilExtension (and all imports/implementations: core/extension.ts, core/index.ts, core/main.ts, packages/sample-extension, packages/@sqlanvil/core) Docs: - readme.md: complete rewrite for sqlanvil positioning (was upstream Dataform copy referencing GCP-managed Dataform) - contributing.md: rewrite for sqlanvil — removed dataform-co/dataform upstream links, dropped GCP-credentials-team section (no longer accessible), added Postgres integration test prerequisite (Docker) - docs/configs-reference.md: placeholder until new proto schemas merge - docs/packages.md: light update (cloned base repo + framework name) - docs/postgres_reintegration_assessment.md: marked SUPERSEDED with pointer to postgres_first_class_design.md NOTICE: Apache 2.0 attribution file. Required by §4 of the license when distributing derivative works. Credits Dataform Co + Google LLC as upstream authors. Preserved upstream references (legitimate attribution / external packages): - readme/docs links to cloud.google.com/dataform (Google's product docs) - docs/packages.md sample package URLs (external community packages authored against upstream Dataform; not maintained by sqlanvil) - LICENSE (Apache 2.0 text, unchanged) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… dead infra
Comprehensive `Dataform` (capital D) → `sqlanvil` across all remaining
source code via brute-force sed on TS/proto files (BSD sed's word-boundary
regex didn't fire on first pass; explicit pattern caught the rest).
Auto-generated docs (docs/reference/*.md):
- Anchor IDs: <a name="dataform-..."> → <a name="sqlanvil-...">
- Cross-ref links: configs#dataform-ActionConfig-* → configs#sqlanvil-*
- Body text describing framework concepts: "Dataform" → "sqlanvil"
CLI test fixtures (cli/util_test.ts, cli/index_run_e2e_test.ts):
- jobId prefix "dataform-915a03fe1" → "sqlanvil-..."
- schema "dataform_test_schema_suffix" → "sqlanvil_test_schema_suffix"
- --job-labels team=dataform → team=sqlanvil
- uniqueDataset prefix dataform_e2e_osc_ → sqlanvil_e2e_osc_
cli/api/dbadapters/execution_sql.ts + matching goldens (*.sql):
- Generated DECLARE/SET variable name dataform_columns → sqlanvil_columns
(BQ on-schema-change synchronize/extend/fail paths)
cli/api/commands/init.ts: workflow_settings.yaml init writes
"defaultDataset: sqlanvil" (was "dataform")
cli/index_test_base.ts:
- DEFAULT_DATABASE: dataform-open-source → your-bigquery-project
- DEFAULT_RESERVATION reservation name dataform-test → sqlanvil-test
cli/vm/compile.ts + cli/vm/jit_worker.ts: node_modules/@dataform/core
bundle.js lookup → @sqlanvil/core (CLI tells users to run
`sqlanvil install`, not `dataform install`)
common/protos/index.ts: error messages now point at
github.com/ihistand/sqlanvil docs + issues, not dataform-co.
protos/configs.proto: "dataformCoreVersion" doc string + comments
updated.
tests/integration/bigquery_project + bigquery.spec.ts + examples/:
- defaultProject: dataform-open-source → your-bigquery-project
- 3-part BQ names rewritten with the placeholder
- Integration tests will need fresh credentials configured before they
can run (already noted in contributing.md)
- examples/.../snowy_repository_creation.ipynb: dataform_examples dataset
reference → your_examples_dataset
VSCode (vscode/{README.md,workflow_settings_yaml.schema.json,package-lock.json}):
- Deprecation notice referencing upstream extension removed
- JSON schema field dataformCoreVersion → sqlanvilCoreVersion
- package-lock.json name field
Deletions (dead infra):
- scripts/decrypt_secret (KMS keyring access to dataform-co's GCP project)
- scripts/create_gh_pr (sets origin URL to dataform-co/dataform.git)
Preserved legitimate references:
- protos/configs.proto: cloud.google.com/dataform/docs/partitions-clusters
URL comments (factual BigQuery partition/cluster reference docs)
- core/main.ts, core/main_test.ts, api/utils/postgres.ts: TODO comments
and upstream issue/PR links for historical context
- .bazelrc: commented-out remote cache configuration (already disabled)
- NOTICE: Apache 2.0 attribution to Dataform Co + Google LLC
- readme.md, CLAUDE.md, docs/*: attribution phrases like "upstream Dataform",
"Dataform OSS by Dataform Co"
- docs/reference/configs.md table rows referencing partition-clusters docs
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The pinned Bazel 5.4 + 2022-era protobuf chain inherited from upstream does not build on macOS Tahoe + Apple Silicon: - Bazel 5.4 ships wrapped_clang without LC_UUID load commands; rejected by current dyld. - Bumping to Bazel 6.x triggers @bazel_tools//platforms removal failures in the WORKSPACE. - Bypassing wrapped_clang via system clang gets further but hits Apple SDK <-> old protobuf preprocessor conflicts (_stdio.h sprintf redefinition). Linux is unaffected. Until a future Bazel-7 + Bzlmod modernization PR lands, macOS contributors should build inside the dev container: ./scripts/docker-bazel build //protos:sqlanvil_proto ./scripts/docker-bazel test //core/... ./scripts/docker-bazel # interactive shell Dockerfile.dev: - Base: node:20-bookworm (gives node + npm) - Adds JDK 17, python 3, build-essential, zlib1g-dev, git, zip/unzip - Installs Bazelisk globally via npm - Mounts /root/.cache/bazel + /root/.cache/bazel-disk as named volumes for persistent Bazel cache across runs scripts/docker-bazel: - Lazy-builds the sqlanvil-dev image on first run - Mounts the repo at /workspace - Mounts named volumes sqlanvil-bazel-cache and sqlanvil-bazel-disk - Auto-attaches a TTY when stdin is a terminal (CI-safe otherwise) - No args → drops into bash inside the container contributing.md: documents the macOS compatibility caveat and the Docker workflow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When running bazel build //protos:sqlanvil_proto directly on the proto_library target (rather than going through ts_proto_library which wraps protoc with different flags), two pre-existing upstream bugs surface: 1. protos/configs.proto and core.proto import "extension.proto" without the protos/ prefix. Without strip_import_prefix on the proto_library, protoc passes -Iprotos/extension.proto=protos/extension.proto and the bare-name import fails. Adding strip_import_prefix = "/protos" removes the protos/ prefix so file references match. 2. jit.proto imports google/protobuf/timestamp.proto but the proto_library only declared deps on empty_proto and struct_proto. Added timestamp_proto. Both are upstream Dataform bugs (same content in upstream/main); they weren't caught because the upstream CI path went through ts_proto_library which has its own protoc invocation. Direct proto_library builds always would have failed. Verified working via: ./scripts/docker-bazel build //protos:sqlanvil_proto → Build completed successfully, 4 total actions This proves the dataform → sqlanvil rename is structurally sound at the proto layer (workspace name `sa`, package `sqlanvil`, target `sqlanvil_proto`, java/go options all resolve and protoc accepts them). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Captures branch state, verification results, known pre-existing issues blocking fuller verification, and the recommended sequence of follow-up work (refresh node pin, then Postgres adapter PR, then Bazel modernization later). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rename sweep missed these 4 BUILD files that referenced the old `@df` workspace name in load() statements and dep labels. Without this fix, //core/... fails to analyze with: error loading package 'common/errors': Unable to find package for @df//testing:index.bzl: The repository '@df' could not be resolved Found while verifying //core/... in Docker.
Rename sweep missed: - packages/@sqlanvil/core/webpack.config.js — webpack resolve.alias - packages/sample-extension/webpack.config.js — webpack resolve.alias - packages/rollup.config.js — importsToBundle pattern - BUILD — ts_library `modules-fix` module_name Without these, //core/... fails to bundle: Module not found: Error: Can't resolve 'sa/core' in '...' Webpack alias still pointed `df` → RUNFILES/df, so any import of `sa/core` (post-rename TS path mapping) couldn't be resolved. Found while verifying //core/... in Docker.
This was referenced May 27, 2026
ihistand
added a commit
that referenced
this pull request
Jun 4, 2026
…FRESH, WITH NO DATA Closes the last #1 item. - Adapter now detects materialized views: information_schema excludes matviews, so table()/tables() union in pg_matviews (listing) + pg_attribute (columns, via format_type with modifiers stripped for convertFieldType). Add a MATERIALIZED_VIEW value to the TableMetadata.Type proto enum. The information_schema path for tables/views is untouched (dataset_metadata test unaffected). - REFRESH-on-rerun: when an existing matview is detected AND the user opts in with refresh_policy "on_dependency_change" (and not --full-refresh), emit REFRESH MATERIALIZED VIEW instead of drop+recreate. Default remains drop+recreate (safe — also picks up definition changes, which REFRESH skips). Skip the generic cross-type drop for materialized views. - WITH NO DATA: rename proto with_data -> no_data so proto3's false default means WITH DATA; generator appends WITH NO DATA when set. TDD: red/green unit tests (no_data, refresh-in-place); integration tests on live PG verify the adapter finds the matview, refreshes in place (oid stable + data updates), and creates unpopulated WITH NO DATA. Supabase specs still pass (they extend PostgresDbAdapter). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Superseded: the Postgres reintegration + Dataform→SQLAnvil rename landed on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comprehensive rename of the
dataformcodebase tosqlanvilto avoid trademark conflict with Google's Dataform product. ~150 files changed across 10 commits, mechanical rename touching proto packages, Bazel labels, TS imports, CLI, VSCode extension, root docs, and the Apache 2.0 NOTICE.Verification
Verified at proto layer via Docker:
./scripts/docker-bazel build //protos:sqlanvil_proto # → Build completed successfully, 4 total actionsProves the new
saworkspace name,//protos:sqlanvil_prototarget,package sqlanvil;in all 8.protofiles,java_package+go_packageupdates, and wire-format-breakingsqlanvil_core_versionfield rename all resolve through the C++ protoc chain.Blocked beyond proto layer by pre-existing upstream toolchain rot (not introduced by this PR):
//protos:ts,//core/..., full treewrapped_clangmissingLC_UUID→ blocks native macOS builds on Tahoeupstream/mainTracked for follow-up PR: bump Node pin to LTS, then separately migrate to Bazel 7 + Bzlmod.
Bug fixes included
Two pre-existing
proto_librarybugs fixed in commit63685541:strip_import_prefix = "/protos"(fixesextension.proto: File not found)@com_google_protobuf//:timestamp_protodep (fixes missing transitive dep)Commits
Test plan
🤖 Generated with Claude Code