Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
b4b24cd
docs: rename plan + postgres-first-class design
ihistand May 27, 2026
f084b0a
refactor: rename proto packages dataform → sqlanvil + WORKSPACE
ihistand May 27, 2026
2148bb3
refactor: rename packages/@dataform → @sqlanvil + Bazel labels + dele…
ihistand May 27, 2026
513e696
refactor: rename TS imports df/ → sa/ + proto namespace + framework d…
ihistand May 27, 2026
1399c23
refactor: drop dataform.json legacy config path (clean break)
ihistand May 27, 2026
9809e45
refactor: rename CLI binary + help text + VSCode extension dataform →…
ihistand May 27, 2026
31adafa
refactor: rename interface ID + write NOTICE + rewrite root docs
ihistand May 27, 2026
ce6af2e
refactor: final rename sweep — proto auto-docs + goldens + examples +…
ihistand May 27, 2026
a11d83c
build: Dockerfile.dev + scripts/docker-bazel for macOS users
ihistand May 27, 2026
6368554
fix(protos): strip_import_prefix + add timestamp_proto dep
ihistand May 27, 2026
b0a4d42
docs: rename handoff — snapshot for next session
ihistand May 27, 2026
ffea539
fix(rename): catch missed @df → @sa workspace refs in common/ BUILDs
ihistand May 27, 2026
569f77c
fix(rename): catch missed df → sa module aliases in bundler configs
ihistand May 27, 2026
b45a19f
fix(deps): refresh Node toolchain pin to v20.20.2 LTS
ihistand May 27, 2026
d8c28a1
fix(postgres-fixture): drop deprecated rules_docker; pull postgres im…
ihistand May 27, 2026
a25410d
feat(deps): add pg, pg-query-stream, uuid for Postgres adapter
ihistand May 27, 2026
430c88c
refactor(postgres-adapter): relocate api/ → cli/api/ + update imports…
ihistand May 27, 2026
f582416
feat(protos): add Postgres + Supabase config messages (Phase 3c)
ihistand May 27, 2026
4463a85
feat(postgres-adapter): implement IDbAdapter contract (Phase 3a)
ihistand May 27, 2026
7b97bf2
docs: docs site sourcing & build plan
ihistand May 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1 +1 @@
* @dataform-co/dataform-reviewers
* @ihistand
4 changes: 2 additions & 2 deletions BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ nodejs_binary(

load("@bazel_gazelle//:def.bzl", "gazelle")

# gazelle:prefix github.com/dataform-co/dataform
# gazelle:prefix github.com/ihistand/sqlanvil
# gazelle:proto package
# gazelle:proto_group go_package
gazelle(name = "gazelle")
Expand All @@ -64,5 +64,5 @@ load("//tools:ts_library.bzl", "ts_library")
ts_library(
name = "modules-fix",
srcs = [],
module_name = "df",
module_name = "sa",
)
104 changes: 104 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# CLAUDE.md

Guidance for Claude Code when working in the `sqlanvil/` project.

## What This Is

**sqlanvil** is Ivan's fork of [`dataform-co/dataform`](https://github.com/dataform-co/dataform), renamed and being repositioned as an open-source SQL workflow tool that runs against **both BigQuery and PostgreSQL/Supabase** (upstream Dataform OSS dropped Postgres support some time ago).

- **Upstream**: `git@github.com:dataform-co/dataform.git` (Google's Dataform OSS — low activity since GA in BigQuery)
- **Origin**: `git@github.com:ihistand/sqlanvil.git`
- **Marketing site**: sibling repo `../sqlanvil-com/` — static HTML on Vercel (project: `sqlanvil-com`, team `Zlu36JPJdwPqwMeAWAqISllx`)

## Stack

- **Language**: TypeScript
- **Build**: Bazel (via Bazelisk) — old-style `WORKSPACE`, not `MODULE.bazel`
- **Protos**: protobuf (`protos/*.proto`) for core/configs/db_adapter/etc.
- **Target warehouses**: BigQuery (working), PostgreSQL (being reintegrated)
- **No npm `scripts`** in `package.json` — everything runs through Bazel.

## Layout

```
core/ Compiler + action types (table/view/incremental/assertion/operation/notebook/declaration)
cli/ CLI entrypoint (cli/index.ts) and per-adapter glue (cli/api/dbadapters/)
protos/ Protobuf definitions for core/configs/execution/db_adapter
api/ Legacy directory — currently holds restored Postgres adapter files awaiting relocation
tools/ Bazel rules + the Postgres docker test fixture (tools/postgres/postgres_fixture.ts)
tests/ Integration specs (bigquery + postgres) against real warehouses
docs/ Reference + the two new Antigravity design docs (see below)
examples/ Sample Dataform projects
scripts/ `./scripts/run` is the CLI entrypoint wrapper
```

## Common Commands

```bash
# One-time
npm i -g @bazel/bazelisk
sudo sysctl -w kern.maxfiles=65536 # Mac only — Bazel hits the default fd limit

# Run the CLI (substitute for `dataform` from @dataform/cli)
./scripts/run help
./scripts/run compile path/to/project

# Tests
bazel test //... # everything
bazel test //core/... # core only
bazel test //cli:index_test # CLI integration (needs GCP creds — see contributing.md)
```

## Design Directive — Rename Is Mandatory

The fork must be fully renamed from `dataform` → `sqlanvil` before public-facing artifacts (npm packages, CLI binary, docs site, marketing) ship. Reason: avoid trademark conflict with Google's "Dataform" product. Scope of rename:

- Proto package names (`dataform.*` → `sqlanvil.*`)
- npm package names (`@dataform/*` → `@sqlanvil/*` or unscoped `sqlanvil-*`)
- CLI binary (`dataform` → `sqlanvil`)
- Config files (`dataform.json` → `sqlanvil.json`, `workflow_settings.yaml` keys)
- Internal class names referencing `Dataform`
- Docs site references

The rename is a hard prerequisite, not nice-to-have. Sequence it before — or in parallel with — the Postgres adapter work.

## Design Directive — Postgres Is First-Class

The Postgres adapter is **not** a BigQuery adapter with translated SQL. It generates idiomatic Postgres DDL/DML. Typical sqlanvil users may never have touched BigQuery; they should never see BigQuery quirks like `CREATE PRIMARY KEY mykey NOT ENFORCED`, `OPTIONS(...)` table options, `PARTITION BY`/`CLUSTER BY` clauses, or BigQuery's `MERGE` dialect.

Two adapter variants ship:

- **`postgres`** — standard Postgres. Idiomatic DDL, `INSERT ... ON CONFLICT` for upserts, native `CREATE INDEX`, tablespaces, fillfactor, partitioning via `PARTITION BY RANGE/LIST/HASH`.
- **`supabase`** — extends `postgres` with Supabase-specific surface area: RLS policies in actions, `auth.users` references, Realtime publications, `pgvector` indexes, `pg_cron` scheduling, Supabase Wrappers (FDW) declarations.

Implications for the Antigravity reintegration doc: Phase 3 ("Interface Alignment") is **under-scoped**. It frames the work as making the restored adapter implement BigQuery's `IDbAdapter` interface. Real work also includes:

- Postgres-specific action config blocks (e.g., `postgres: { tablespace, fillfactor, indexes, partition }` parallel to the existing `bigquery: { partitionBy, clusterBy, ... }`).
- A separate Postgres SQL generator path in `core/compilation_sql/` rather than reusing BigQuery's.
- Config schema additions in `protos/configs.proto` for both variants.

## Active Work — Postgres Reintegration

Current branch: `restore-postgres-adapter`. Recent commits (Ivan's, on top of upstream):

1. `a220e2ed` — restored the Postgres adapter files from git history into `api/dbadapters/postgres.ts` and `api/utils/postgres.ts` (won't compile as-is)
2. `fcca60c1` — added `docs/postgres_reintegration_assessment.md` (Antigravity)
3. `1636e275` — added `docs/hybrid_warehouses_supabase_bigquery.md` (Antigravity)

The two new design docs are Antigravity-authored and **load-bearing for the next sprint**:

- **`docs/postgres_reintegration_assessment.md`** — 5-phase, ~1-2 day plan to make the restored adapter compile and wire into the CLI. Phases: deps (`pg`, `pg-query-stream`) → relocate `api/` → `cli/api/` → align `IDbAdapter` interface (implement `executeRaw`, `deleteTable`, full `ITableMetadata`) → branch CLI on `projectConfig.warehouse === "postgres"` → Bazel/docker fixture verification.
- **`docs/hybrid_warehouses_supabase_bigquery.md`** — marketing/architecture doc: three patterns for combining Supabase + BigQuery (federated queries, sequential pipeline, Supabase Wrappers / FDW). Reference material, no implementation required.

When working on Postgres reintegration, **follow the assessment doc's phase order** — deps before relocation before interface work — because each phase's tests depend on the previous one passing through Bazel.

## Fork Hygiene

- Upstream changes still merge in cleanly today; the longer Ivan diverges (renames, Postgres adapter, future Supabase-specific features), the harder this gets. When pulling upstream, prefer rebasing feature branches onto `upstream/main` over merge commits to keep the history readable.
- Renaming Dataform → sqlanvil should be done in one sweep (package names, proto packages, CLI binary, docs) rather than incrementally — partial renames create grep ambiguity.

## Things To Know

- The `api/` directory at the repo root is **legacy**. The active CLI/adapter layout lives under `cli/api/`. Restored Postgres files are in the legacy location and need to move (see Phase 2 of the assessment doc).
- Integration tests need real warehouses: BigQuery creds in `test_credentials/bigquery.json`, Postgres via Docker container started by `tools/postgres/postgres_fixture.ts` inside the Bazel sandbox.
- The fork still references `@dataform/...` in many places — rename surface area is large.
55 changes: 55 additions & 0 deletions Dockerfile.dev
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Dev container for building sqlanvil on Linux.
#
# Why: the pinned Bazel 5.4 + 2022-era rules_proto/protobuf chain does not
# build on macOS Tahoe (wrapped_clang missing LC_UUID, Apple SDK header
# conflicts with old protobuf). On Linux those issues don't apply.
#
# Build the image once:
# docker build -f Dockerfile.dev -t sqlanvil-dev .
#
# Run a one-shot Bazel command (caches persist in a named volume):
# docker run --rm -it \
# -v "$PWD:/workspace" \
# -v sqlanvil-bazel-cache:/root/.cache/bazel \
# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \
# sqlanvil-dev bazel build //protos:sqlanvil_proto
#
# Drop into a shell:
# docker run --rm -it \
# -v "$PWD:/workspace" \
# -v sqlanvil-bazel-cache:/root/.cache/bazel \
# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \
# sqlanvil-dev
#
# Or use the wrapper: ./scripts/docker-bazel build //protos:sqlanvil_proto

FROM node:20-bookworm

# Bazel needs: JDK, python (for some rules), git, build-essential, unzip,
# zip, and the usual C/C++ toolchain (gcc, g++, make).
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
git \
gnupg \
openjdk-17-jdk-headless \
python3 \
python-is-python3 \
build-essential \
unzip \
zip \
zlib1g-dev \
&& rm -rf /var/lib/apt/lists/*

# Bazelisk — reads .bazelversion and downloads the matching Bazel.
RUN npm install -g @bazel/bazelisk

# Bazel writes a lot to ~/.cache/bazel; expose it as a volume target so
# the host docker volume sticks across runs.
ENV BAZEL_DISK_CACHE_DIR=/root/.cache/bazel-disk
RUN mkdir -p /root/.cache/bazel /root/.cache/bazel-disk

WORKDIR /workspace

# Default to an interactive shell. Override with the bazel command you want.
CMD ["bash"]
14 changes: 14 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
sqlanvil
Copyright 2026 Ivan Histand

This product includes software originally developed as Dataform
(https://github.com/dataform-co/dataform) by Dataform Co and contributed
to by Google LLC, licensed under the Apache License, Version 2.0.

The original Dataform copyright notice from upstream is preserved in
the LICENSE file. This derivative work (renamed sqlanvil) adds
PostgreSQL and Supabase warehouse adapter support, and is distributed
under the same Apache License, Version 2.0.

Apache License, Version 2.0
http://www.apache.org/licenses/LICENSE-2.0
14 changes: 7 additions & 7 deletions WORKSPACE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
workspace(name = "df")
workspace(name = "sa")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

Expand Down Expand Up @@ -43,13 +43,13 @@ load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "yarn_install

node_repositories(
node_repositories = {
"24.13.0-darwin_amd64": ("node-v24.13.0-darwin-x64.tar.xz", "node-v24.13.0-darwin-x64", "4ca0a48233f091a2a69ec28dd58e59f394a1b2d4f052b6c6b10f760377fe266f"),
"24.13.0-darwin_arm64": ("node-v24.13.0-darwin-arm64.tar.xz", "node-v24.13.0-darwin-arm64", "c59a517e9147f25c6167426875a571432f1478c1d7ee7ecc10baa46b0d0e8545"),
"24.13.0-linux_amd64": ("node-v24.13.0-linux-x64.tar.xz", "node-v24.13.0-linux-x64", "e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e"),
"24.13.0-linux_arm64": ("node-v24.13.0-linux-arm64.tar.xz", "node-v24.13.0-linux-arm64", "e798599612f4bb71333a3397ab0d095fd62214e115aea45aa858a145fc72d67e"),
"24.13.0-windows_amd64": ("node-v24.13.0-win-x64.zip", "node-v24.13.0-win-x64", "ca2742695be8de44027d71b3f53a4bdb36009b95575fe1ae6f7f0b5ce091cb88"),
"20.20.2-darwin_amd64": ("node-v20.20.2-darwin-x64.tar.xz", "node-v20.20.2-darwin-x64", "4d4c020eb534497e616de38f3733289ff33c615ddab38c048edec6547b7f76ea"),
"20.20.2-darwin_arm64": ("node-v20.20.2-darwin-arm64.tar.xz", "node-v20.20.2-darwin-arm64", "6375a1d4421bc04ab284ba89459df788a78c49c89e83c463d0eede47e2efc07b"),
"20.20.2-linux_amd64": ("node-v20.20.2-linux-x64.tar.xz", "node-v20.20.2-linux-x64", "df770b2a6f130ed8627c9782c988fda9669fa23898329a61a871e32f965e007d"),
"20.20.2-linux_arm64": ("node-v20.20.2-linux-arm64.tar.xz", "node-v20.20.2-linux-arm64", "73093db209e4e9e09dd7d15a47aeaab1b74833830df03efa5f942a1122c5fa71"),
"20.20.2-windows_amd64": ("node-v20.20.2-win-x64.zip", "node-v20.20.2-win-x64", "dc3700fdd57a63eedb8fd7e3c7baaa32e6a740a1b904167ff4204bc68ed8bf77"),
},
node_version = "24.13.0",
node_version = "20.20.2",
package_json = ["//:package.json"],
yarn_version = "1.13.0",
)
Expand Down
Loading