Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1 +1 @@
* @dataform-co/dataform-reviewers
* @ihistand
4 changes: 2 additions & 2 deletions BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ nodejs_binary(

load("@bazel_gazelle//:def.bzl", "gazelle")

# gazelle:prefix github.com/dataform-co/dataform
# gazelle:prefix github.com/ihistand/sqlanvil
# gazelle:proto package
# gazelle:proto_group go_package
gazelle(name = "gazelle")
Expand All @@ -64,5 +64,5 @@ load("//tools:ts_library.bzl", "ts_library")
ts_library(
name = "modules-fix",
srcs = [],
module_name = "df",
module_name = "sa",
)
104 changes: 104 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# CLAUDE.md

Guidance for Claude Code when working in the `sqlanvil/` project.

## What This Is

**sqlanvil** is Ivan's fork of [`dataform-co/dataform`](https://github.com/dataform-co/dataform), renamed and being repositioned as an open-source SQL workflow tool that runs against **both BigQuery and PostgreSQL/Supabase** (upstream Dataform OSS dropped Postgres support some time ago).

- **Upstream**: `git@github.com:dataform-co/dataform.git` (Google's Dataform OSS — low activity since GA in BigQuery)
- **Origin**: `git@github.com:ihistand/sqlanvil.git`
- **Marketing site**: sibling repo `../sqlanvil-com/` — static HTML on Vercel (project: `sqlanvil-com`, team `Zlu36JPJdwPqwMeAWAqISllx`)

## Stack

- **Language**: TypeScript
- **Build**: Bazel (via Bazelisk) — old-style `WORKSPACE`, not `MODULE.bazel`
- **Protos**: protobuf (`protos/*.proto`) for core/configs/db_adapter/etc.
- **Target warehouses**: BigQuery (working), PostgreSQL (being reintegrated)
- **No npm `scripts`** in `package.json` — everything runs through Bazel.

## Layout

```
core/ Compiler + action types (table/view/incremental/assertion/operation/notebook/declaration)
cli/ CLI entrypoint (cli/index.ts) and per-adapter glue (cli/api/dbadapters/)
protos/ Protobuf definitions for core/configs/execution/db_adapter
api/ Legacy directory — currently holds restored Postgres adapter files awaiting relocation
tools/ Bazel rules + the Postgres docker test fixture (tools/postgres/postgres_fixture.ts)
tests/ Integration specs (bigquery + postgres) against real warehouses
docs/ Reference + the two new Antigravity design docs (see below)
examples/ Sample Dataform projects
scripts/ `./scripts/run` is the CLI entrypoint wrapper
```

## Common Commands

```bash
# One-time
npm i -g @bazel/bazelisk
sudo sysctl -w kern.maxfiles=65536 # Mac only — Bazel hits the default fd limit

# Run the CLI (substitute for `dataform` from @dataform/cli)
./scripts/run help
./scripts/run compile path/to/project

# Tests
bazel test //... # everything
bazel test //core/... # core only
bazel test //cli:index_test # CLI integration (needs GCP creds — see contributing.md)
```

## Design Directive — Rename Is Mandatory

The fork must be fully renamed from `dataform` → `sqlanvil` before public-facing artifacts (npm packages, CLI binary, docs site, marketing) ship. Reason: avoid trademark conflict with Google's "Dataform" product. Scope of rename:

- Proto package names (`dataform.*` → `sqlanvil.*`)
- npm package names (`@dataform/*` → `@sqlanvil/*` or unscoped `sqlanvil-*`)
- CLI binary (`dataform` → `sqlanvil`)
- Config files (`dataform.json` → `sqlanvil.json`, `workflow_settings.yaml` keys)
- Internal class names referencing `Dataform`
- Docs site references

The rename is a hard prerequisite, not nice-to-have. Sequence it before — or in parallel with — the Postgres adapter work.

## Design Directive — Postgres Is First-Class

The Postgres adapter is **not** a BigQuery adapter with translated SQL. It generates idiomatic Postgres DDL/DML. Typical sqlanvil users may never have touched BigQuery; they should never see BigQuery quirks like `CREATE PRIMARY KEY mykey NOT ENFORCED`, `OPTIONS(...)` table options, `PARTITION BY`/`CLUSTER BY` clauses, or BigQuery's `MERGE` dialect.

Two adapter variants ship:

- **`postgres`** — standard Postgres. Idiomatic DDL, `INSERT ... ON CONFLICT` for upserts, native `CREATE INDEX`, tablespaces, fillfactor, partitioning via `PARTITION BY RANGE/LIST/HASH`.
- **`supabase`** — extends `postgres` with Supabase-specific surface area: RLS policies in actions, `auth.users` references, Realtime publications, `pgvector` indexes, `pg_cron` scheduling, Supabase Wrappers (FDW) declarations.

Implications for the Antigravity reintegration doc: Phase 3 ("Interface Alignment") is **under-scoped**. It frames the work as making the restored adapter implement BigQuery's `IDbAdapter` interface. Real work also includes:

- Postgres-specific action config blocks (e.g., `postgres: { tablespace, fillfactor, indexes, partition }` parallel to the existing `bigquery: { partitionBy, clusterBy, ... }`).
- A separate Postgres SQL generator path in `core/compilation_sql/` rather than reusing BigQuery's.
- Config schema additions in `protos/configs.proto` for both variants.

## Active Work — Postgres Reintegration

Current branch: `restore-postgres-adapter`. Recent commits (Ivan's, on top of upstream):

1. `a220e2ed` — restored the Postgres adapter files from git history into `api/dbadapters/postgres.ts` and `api/utils/postgres.ts` (won't compile as-is)
2. `fcca60c1` — added `docs/postgres_reintegration_assessment.md` (Antigravity)
3. `1636e275` — added `docs/hybrid_warehouses_supabase_bigquery.md` (Antigravity)

The two new design docs are Antigravity-authored and **load-bearing for the next sprint**:

- **`docs/postgres_reintegration_assessment.md`** — 5-phase, ~1-2 day plan to make the restored adapter compile and wire into the CLI. Phases: deps (`pg`, `pg-query-stream`) → relocate `api/` → `cli/api/` → align `IDbAdapter` interface (implement `executeRaw`, `deleteTable`, full `ITableMetadata`) → branch CLI on `projectConfig.warehouse === "postgres"` → Bazel/docker fixture verification.
- **`docs/hybrid_warehouses_supabase_bigquery.md`** — marketing/architecture doc: three patterns for combining Supabase + BigQuery (federated queries, sequential pipeline, Supabase Wrappers / FDW). Reference material, no implementation required.

When working on Postgres reintegration, **follow the assessment doc's phase order** — deps before relocation before interface work — because each phase's tests depend on the previous one passing through Bazel.

## Fork Hygiene

- Upstream changes still merge in cleanly today; the longer Ivan diverges (renames, Postgres adapter, future Supabase-specific features), the harder this gets. When pulling upstream, prefer rebasing feature branches onto `upstream/main` over merge commits to keep the history readable.
- Renaming Dataform → sqlanvil should be done in one sweep (package names, proto packages, CLI binary, docs) rather than incrementally — partial renames create grep ambiguity.

## Things To Know

- The `api/` directory at the repo root is **legacy**. The active CLI/adapter layout lives under `cli/api/`. Restored Postgres files are in the legacy location and need to move (see Phase 2 of the assessment doc).
- Integration tests need real warehouses: BigQuery creds in `test_credentials/bigquery.json`, Postgres via Docker container started by `tools/postgres/postgres_fixture.ts` inside the Bazel sandbox.
- The fork still references `@dataform/...` in many places — rename surface area is large.
55 changes: 55 additions & 0 deletions Dockerfile.dev
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Dev container for building sqlanvil on Linux.
#
# Why: the pinned Bazel 5.4 + 2022-era rules_proto/protobuf chain does not
# build on macOS Tahoe (wrapped_clang missing LC_UUID, Apple SDK header
# conflicts with old protobuf). On Linux those issues don't apply.
#
# Build the image once:
# docker build -f Dockerfile.dev -t sqlanvil-dev .
#
# Run a one-shot Bazel command (caches persist in a named volume):
# docker run --rm -it \
# -v "$PWD:/workspace" \
# -v sqlanvil-bazel-cache:/root/.cache/bazel \
# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \
# sqlanvil-dev bazel build //protos:sqlanvil_proto
#
# Drop into a shell:
# docker run --rm -it \
# -v "$PWD:/workspace" \
# -v sqlanvil-bazel-cache:/root/.cache/bazel \
# -v sqlanvil-bazel-disk:/root/.cache/bazel-disk \
# sqlanvil-dev
#
# Or use the wrapper: ./scripts/docker-bazel build //protos:sqlanvil_proto

FROM node:20-bookworm

# Bazel needs: JDK, python (for some rules), git, build-essential, unzip,
# zip, and the usual C/C++ toolchain (gcc, g++, make).
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
git \
gnupg \
openjdk-17-jdk-headless \
python3 \
python-is-python3 \
build-essential \
unzip \
zip \
zlib1g-dev \
&& rm -rf /var/lib/apt/lists/*

# Bazelisk — reads .bazelversion and downloads the matching Bazel.
RUN npm install -g @bazel/bazelisk

# Bazel writes a lot to ~/.cache/bazel; expose it as a volume target so
# the host docker volume sticks across runs.
ENV BAZEL_DISK_CACHE_DIR=/root/.cache/bazel-disk
RUN mkdir -p /root/.cache/bazel /root/.cache/bazel-disk

WORKDIR /workspace

# Default to an interactive shell. Override with the bazel command you want.
CMD ["bash"]
14 changes: 14 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
sqlanvil
Copyright 2026 Ivan Histand

This product includes software originally developed as Dataform
(https://github.com/dataform-co/dataform) by Dataform Co and contributed
to by Google LLC, licensed under the Apache License, Version 2.0.

The original Dataform copyright notice from upstream is preserved in
the LICENSE file. This derivative work (renamed sqlanvil) adds
PostgreSQL and Supabase warehouse adapter support, and is distributed
under the same Apache License, Version 2.0.

Apache License, Version 2.0
http://www.apache.org/licenses/LICENSE-2.0
2 changes: 1 addition & 1 deletion WORKSPACE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
workspace(name = "df")
workspace(name = "sa")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

Expand Down
46 changes: 23 additions & 23 deletions api/dbadapters/postgres.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
import * as pg from "pg";

import { Credentials } from "df/api/commands/credentials";
import { IDbAdapter, IDbClient } from "df/api/dbadapters/index";
import { SSHTunnelProxy } from "df/api/ssh_tunnel_proxy";
import { parseRedshiftEvalError } from "df/api/utils/error_parsing";
import { convertFieldType, PgPoolExecutor } from "df/api/utils/postgres";
import { ErrorWithCause } from "df/common/errors/errors";
import { collectEvaluationQueries, QueryOrAction } from "df/core/adapters";
import { dataform } from "df/protos/ts";
import { Credentials } from "sa/api/commands/credentials";
import { IDbAdapter, IDbClient } from "sa/api/dbadapters/index";
import { SSHTunnelProxy } from "sa/api/ssh_tunnel_proxy";
import { parseRedshiftEvalError } from "sa/api/utils/error_parsing";
import { convertFieldType, PgPoolExecutor } from "sa/api/utils/postgres";
import { ErrorWithCause } from "sa/common/errors/errors";
import { collectEvaluationQueries, QueryOrAction } from "sa/core/adapters";
import { sqlanvil } from "sa/protos/ts";

interface IPostgresAdapterOptions {
sshTunnel?: SSHTunnelProxy;
Expand All @@ -18,7 +18,7 @@ export class PostgresDbAdapter implements IDbAdapter {
credentials: Credentials,
options?: { concurrencyLimit?: number; disableSslForTestsOnly?: boolean }
) {
const jdbcCredentials = credentials as dataform.IJDBC;
const jdbcCredentials = credentials as sqlanvil.IJDBC;
const baseClientConfig: Partial<pg.ClientConfig> = {
user: jdbcCredentials.username,
password: jdbcCredentials.password,
Expand Down Expand Up @@ -107,21 +107,21 @@ export class PostgresDbAdapter implements IDbAdapter {
).map((validationQuery, index) => ({ index, validationQuery }));
const validationQueriesWithoutWrappers = collectEvaluationQueries(queryOrAction, false);

const queryEvaluations = new Array<dataform.IQueryEvaluation>();
const queryEvaluations = new Array<sqlanvil.IQueryEvaluation>();
for (const { index, validationQuery } of validationQueries) {
let evaluationResponse: dataform.IQueryEvaluation = {
status: dataform.QueryEvaluation.QueryEvaluationStatus.SUCCESS
let evaluationResponse: sqlanvil.IQueryEvaluation = {
status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.SUCCESS
};
try {
await this.execute(validationQuery.query);
} catch (e) {
evaluationResponse = {
status: dataform.QueryEvaluation.QueryEvaluationStatus.FAILURE,
status: sqlanvil.QueryEvaluation.QueryEvaluationStatus.FAILURE,
error: parseRedshiftEvalError(validationQuery.query, e)
};
}
queryEvaluations.push(
dataform.QueryEvaluation.create({
sqlanvil.QueryEvaluation.create({
...evaluationResponse,
incremental: validationQuery.incremental,
query: validationQueriesWithoutWrappers[index].query
Expand All @@ -131,7 +131,7 @@ export class PostgresDbAdapter implements IDbAdapter {
return queryEvaluations;
}

public async tables(): Promise<dataform.ITarget[]> {
public async tables(): Promise<sqlanvil.ITarget[]> {
const queryResult = await this.execute(
`select table_name, table_schema
from information_schema.tables
Expand All @@ -150,7 +150,7 @@ export class PostgresDbAdapter implements IDbAdapter {
public async search(
searchText: string,
options: { limit: number } = { limit: 1000 }
): Promise<dataform.ITableMetadata[]> {
): Promise<sqlanvil.ITableMetadata[]> {
// TODO: It would be nice to extend this to search through table/column descriptions. However, this involves
// a somewhat crazy 5-way join.
const results = await this.execute(
Expand All @@ -174,7 +174,7 @@ export class PostgresDbAdapter implements IDbAdapter {
);
}

public async table(target: dataform.ITarget): Promise<dataform.ITableMetadata> {
public async table(target: sqlanvil.ITarget): Promise<sqlanvil.ITableMetadata> {
const params = [target.schema, target.name];
const [tableResults, columnResults, descriptionResults] = await Promise.all([
this.execute(
Expand All @@ -201,14 +201,14 @@ export class PostgresDbAdapter implements IDbAdapter {
if (tableResults.rows.length === 0) {
return null;
}
return dataform.TableMetadata.create({
return sqlanvil.TableMetadata.create({
target,
type:
tableResults.rows[0].table_type === "VIEW"
? dataform.TableMetadata.Type.VIEW
: dataform.TableMetadata.Type.TABLE,
? sqlanvil.TableMetadata.Type.VIEW
: sqlanvil.TableMetadata.Type.TABLE,
fields: columnResults.rows.map(row =>
dataform.Field.create({
sqlanvil.Field.create({
name: row.column_name,
primitive: convertFieldType(row.data_type),
description: descriptionResults.rows.find(
Expand All @@ -222,7 +222,7 @@ export class PostgresDbAdapter implements IDbAdapter {
});
}

public async preview(target: dataform.ITarget, limitRows: number = 10): Promise<any[]> {
public async preview(target: sqlanvil.ITarget, limitRows: number = 10): Promise<any[]> {
const { rows } = await this.execute(
`SELECT * FROM "${target.schema}"."${target.name}" LIMIT ${limitRows}`
);
Expand All @@ -247,7 +247,7 @@ export class PostgresDbAdapter implements IDbAdapter {
}
}

public async setMetadata(action: dataform.IExecutionAction): Promise<void> {
public async setMetadata(action: sqlanvil.IExecutionAction): Promise<void> {
const { target, actionDescriptor, tableType } = action;

const actualMetadata = await this.table(target);
Expand Down
20 changes: 10 additions & 10 deletions api/utils/postgres.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import * as pg from "pg";
import QueryStream from "pg-query-stream";

import { LimitedResultSet } from "df/api/utils/results";
import { dataform } from "df/protos/ts";
import { LimitedResultSet } from "sa/api/utils/results";
import { sqlanvil } from "sa/protos/ts";

const maybeInitializePg = (() => {
let initialized = false;
Expand Down Expand Up @@ -144,21 +144,21 @@ export function convertFieldType(type: string) {
case "FLOAT8":
case "DOUBLE PRECISION":
case "REAL":
return dataform.Field.Primitive.FLOAT;
return sqlanvil.Field.Primitive.FLOAT;
case "INTEGER":
case "INT":
case "INT2":
case "INT4":
case "INT8":
case "BIGINT":
case "SMALLINT":
return dataform.Field.Primitive.INTEGER;
return sqlanvil.Field.Primitive.INTEGER;
case "DECIMAL":
case "NUMERIC":
return dataform.Field.Primitive.NUMERIC;
return sqlanvil.Field.Primitive.NUMERIC;
case "BOOLEAN":
case "BOOL":
return dataform.Field.Primitive.BOOLEAN;
return sqlanvil.Field.Primitive.BOOLEAN;
case "STRING":
case "VARCHAR":
case "CHAR":
Expand All @@ -168,15 +168,15 @@ export function convertFieldType(type: string) {
case "TEXT":
case "NCHAR":
case "BPCHAR":
return dataform.Field.Primitive.STRING;
return sqlanvil.Field.Primitive.STRING;
case "DATE":
return dataform.Field.Primitive.DATE;
return sqlanvil.Field.Primitive.DATE;
case "TIMESTAMP":
case "TIMESTAMPZ":
case "TIMESTAMP WITHOUT TIME ZONE":
case "TIMESTAMP WITH TIME ZONE":
return dataform.Field.Primitive.TIMESTAMP;
return sqlanvil.Field.Primitive.TIMESTAMP;
default:
return dataform.Field.Primitive.UNKNOWN;
return sqlanvil.Field.Primitive.UNKNOWN;
}
}
5 changes: 2 additions & 3 deletions cli/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ ts_library(
node_modules(
name = "node_modules",
deps = [
"//packages/@dataform/cli:package_tar",
"//packages/@sqlanvil/cli:package_tar",
],
)

Expand Down Expand Up @@ -69,8 +69,7 @@ ts_test_suite(
],
data = [
":node_modules",
"//packages/@dataform/core:package_tar",
"//test_credentials:bigquery.json",
"//packages/@sqlanvil/core:package_tar",
"@nodejs//:node",
"@nodejs//:npm",
],
Expand Down
Loading