Skip to content

Prevent .sqlx files in actions.yaml configs with a clear error - #2189

Merged
kolina merged 1 commit into
dataform-co:mainfrom
aicayzer:feat/validate-actions-yaml-sqlx
Jun 27, 2026
Merged

Prevent .sqlx files in actions.yaml configs with a clear error#2189
kolina merged 1 commit into
dataform-co:mainfrom
aicayzer:feat/validate-actions-yaml-sqlx

Conversation

@aicayzer

Copy link
Copy Markdown
Contributor

Closes #1785.

Problem

Referencing a .sqlx file from definitions/actions.yaml (e.g.

actions:
- table:
    filename: my_table.sqlx

) is unsupported, but the existing code path doesn't say so. The Table/View/etc. constructor reaches nativeRequire(config.filename).query, which fails with a Cannot find module 'definitions/my_table.sqlx' (or equivalently obscure) error far from the actual mistake. As Ekrekr noted on the issue, "this results in strange compilation behavior, and can cause cryptic errors."

Fix

Validate filenames at the loadActionConfigs entry point, before the action is constructed. For each action config sub-message that exposes a filename field — table, view, incrementalTable, assertion, operation, declaration, notebook — if filename ends in .sqlx (case-insensitive), emit a session.compileError naming the action type, the offending filename, and the remediation (use .sql, or rely on Dataform's automatic .sqlx discovery). The action is then skipped so a downstream nativeRequire error doesn't fire on top.

dataPreparation is intentionally excluded: it supports .dp.sqlx files via its own loader (core/actions/data_preparation.ts).

Test plan

Added one new test suite to core/main_test.ts (".sqlx filenames in actions.yaml are rejected with a clear error") covering:

  • One test per affected action type (7 in total), each verifying that a single compilation error fires with the expected message shape.
  • One case-insensitive test (filename: table.SQLX).
  • One regression test ensuring dataPreparation with a .dp.sqlx filename does not trigger this validation.

tslint --project . passes locally. I wasn't able to run bazel test //core/... locally (a @npm//@bazel/typescript:index.bzl loading error in tools/ts_library.bzl against bazel 5.4.0 on my machine), so the unit tests will be exercised by CI here.

Scope

No proto changes. No CLI wiring. No yargs wrapper edits. Scope mirrors PR #2154 (validation + clear error in core/, +16/-5).

Files touched

  • core/main.ts — new actionConfigFilenameIsInvalidSqlx helper called from loadActionConfigs.
  • core/main_test.ts — new test suite.

@aicayzer
aicayzer requested a review from a team as a code owner May 29, 2026 20:58
@aicayzer
aicayzer requested review from udim and removed request for a team May 29, 2026 20:58
@kolina

kolina commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

Comment thread core/main.ts Outdated
Comment thread core/main.ts

@kolina kolina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I didn't notice tests are failing with YAML require failed run_core: VMError: Cannot find module './workflow_settings.yaml' before approving.

Can you please rebase on the latest HEAD? I think this error is not related to your changes and was fixed in the latest version

@aicayzer
aicayzer force-pushed the feat/validate-actions-yaml-sqlx branch from 52f7337 to f604e1e Compare June 4, 2026 12:59
@aicayzer

aicayzer commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

No problem at all @kolina, just rebased onto the latest main. Lmk if it's still something.

@kolina

kolina commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@kolina kolina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, your new test is failing like this

Step #1:         at /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/sandbox/linux-sandbox/127/execroot/df/_tmp/e6baaed8819351a8a5cad263fdd5689f/tmp_dir_48/node_modules/@dataform/core/bundle.js:18:54579
Step #1:         at new u (/builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/sandbox/linux-sandbox/127/execroot/df/_tmp/e6baaed8819351a8a5cad263fdd5689f/tmp_dir_48/node_modules/@dataform/core/bundle.js:18:139849)
Step #1:         at u.configureSqlx (/builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/sandbox/linux-sandbox/127/execroot/df/_tmp/e6baaed8819351a8a5cad263fdd5689f/tmp_dir_48/node_modules/@dataform/core/bundle.js:18:144800)
Step #1:         at Object.require [as nativeRequire] (node_modules/vm2/lib/setup-node-sandbox.js:171:10)
Step #1:         at requireImpl (node_modules/vm2/lib/setup-node-sandbox.js:90:19)
Step #1:         at VM2 Wrapper.apply (node_modules/vm2/lib/bridge.js:1579:11)
Step #1:         at resolve (node_modules/vm2/lib/nodevm.js:383:21)
Step #1:         at CustomResolver.resolve (node_modules/vm2/lib/resolver.js:121:15)
Step #1:         at CustomResolver.resolveFull (node_modules/vm2/lib/resolver.js:316:16)
Step #1:         at CustomResolver.resolveFull (node_modules/vm2/lib/resolver.js:126:9)
Step #1:     VMError: Cannot find module 'prep.dp.sqlx'
Step #1: 
Step #1:                                                                          FAILED
Step #1: @dataform/core > action configs > .sqlx filenames in actions.yaml are rejected with a clear error > but data preparations still accept .dp.sqlx files

I can't say for sure but I suspect we may need to allowlist it here. I think it wasn't caught because GCP compilation doesn't use vm2 and data preparation is quite unlikely to be used by a lot of CLI users

@aicayzer

aicayzer commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @kolina. I'll look into it tomorrow.

@aicayzer
aicayzer force-pushed the feat/validate-actions-yaml-sqlx branch from f604e1e to 69ebc03 Compare June 7, 2026 23:09
@aicayzer

aicayzer commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @kolina. I dug into this and the data preparation case turned out to be a bit of a red herring.

Referencing a .dp.sqlx from actions.yaml doesnt actually work today: the filename isnt resolved relative to the config file (hence the Cannot find module), and even when it does load it ignores the file's own config {} block and double-registers the action, since .dp.sqlx files are already auto-compiled from definitions/. So rather than allowlisting it in the vm2 resolver, I dropped the data preparation exception entirely - actions.yaml now rejects .sqlx filenames for every action type, data preparations included, with the same clear error pointing people at auto-discovery. .dp.yaml data preparations are unaffected.

Updated the test to assert the rejection instead. Green locally (//core/... + tslint).

@kolina

kolina commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@kolina kolina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late response, LGTM.

I left a couple of comments. Feel free to address them if you want, otherwise I'll be happy to merge this PR

@kolina
kolina removed the request for review from udim June 23, 2026 20:16
Putting a .sqlx file under the ``filename`` field of an entry in
``definitions/actions.yaml`` used to fall through to ``nativeRequire``,
which then failed with a cryptic error far from the source of the
mistake. ``.sqlx`` files are loaded directly from the ``definitions/``
directory by the sqlx compiler, not referenced through ``actions.yaml``.

Add an explicit validation step in ``loadActionConfigs`` that runs before
each action is constructed. If the action's ``filename`` ends in
``.sqlx`` (case-insensitive), emit a ``compileError`` naming the action
type, the offending filename, and pointing at the fix. The action is
then skipped so we don't double-up with a downstream ``nativeRequire``
error.

The check covers every action type that accepts a ``filename``: ``table``,
``view``, ``incrementalTable``, ``assertion``, ``operation``,
``declaration``, ``notebook`` and ``dataPreparation``. Data preparations
are included because a ``.dp.sqlx`` file is also auto-compiled from
``definitions/`` - referencing one from ``actions.yaml`` doesn't resolve
its path, ignores its ``config {}`` block and double-registers the
action, so it's the same mistake. ``.dp.yaml`` data preparations are
unaffected.

Closes dataform-co#1785
@aicayzer
aicayzer force-pushed the feat/validate-actions-yaml-sqlx branch from 69ebc03 to 5cda602 Compare June 24, 2026 15:21
@aicayzer

Copy link
Copy Markdown
Contributor Author

Thanks @kolina. Dropped the dynamic as any cast for a static list of the typed sub-configs, following the L143 pattern. Green locally on //core/... (16/16) + tslint.

Left the helper name as actionConfigFilenameIsInvalidSqlx since it reads clearly, but happy to switch to actionConfigIncludesSqlxFiles if you'd prefer.

@kolina

kolina commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@kolina
kolina merged commit faed1b5 into dataform-co:main Jun 27, 2026
7 of 8 checks passed
spatel11 pushed a commit to spatel11/dataform that referenced this pull request Jul 8, 2026
…orm-co#2189)

Putting a .sqlx file under the ``filename`` field of an entry in
``definitions/actions.yaml`` used to fall through to ``nativeRequire``,
which then failed with a cryptic error far from the source of the
mistake. ``.sqlx`` files are loaded directly from the ``definitions/``
directory by the sqlx compiler, not referenced through ``actions.yaml``.

Add an explicit validation step in ``loadActionConfigs`` that runs before
each action is constructed. If the action's ``filename`` ends in
``.sqlx`` (case-insensitive), emit a ``compileError`` naming the action
type, the offending filename, and pointing at the fix. The action is
then skipped so we don't double-up with a downstream ``nativeRequire``
error.

The check covers every action type that accepts a ``filename``: ``table``,
``view``, ``incrementalTable``, ``assertion``, ``operation``,
``declaration``, ``notebook`` and ``dataPreparation``. Data preparations
are included because a ``.dp.sqlx`` file is also auto-compiled from
``definitions/`` - referencing one from ``actions.yaml`` doesn't resolve
its path, ignores its ``config {}`` block and double-registers the
action, so it's the same mistake. ``.dp.yaml`` data preparations are
unaffected.

Closes dataform-co#1785
apilaskowski pushed a commit that referenced this pull request Jul 13, 2026
* ISSUE-2103 Handle trailing and trailing statement comments

* Bump vm2 from 3.10.2 to 3.11.3 (#2168)

* Bump vm2 from 3.10.2 to 3.11.3

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.10.2 to 3.11.3.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md)
- [Commits](patriksimek/vm2@v3.10.2...v3.11.3)

---
updated-dependencies:
- dependency-name: vm2
  dependency-version: 3.11.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix vm2 sandbox execution errors and CallSite path stripping

The upgrade of vm2 to 3.11.3 tightened sandbox security, stripping file path information from V8 CallSite objects during Error.prepareStackTrace and breaking the resolution of symlinked modules. This caused critical test failures ("Unable to find valid caller file") during Dataform compilation.

This commit resolves these issues by:
* Resolving `projectDir` to its absolute realpath before sandbox compilation to circumvent symlink boundaries (especially important for bazel runfiles).
* Injecting `global.__dataform_current_file` into the NodeVM compiler configuration with a try-finally block to track cross-boundary macro scopes.
* Adding a fallback in `core/utils.ts:getCallerFile` to read the injected global file path when the V8 stack trace path is stripped or undefined.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rafal Hawrylak <hawrylak@google.com>

* Bump DF_VERSION from 3.0.56 to 3.0.57 (#2173)

* Support onSchemaChange for incremental tables (#2101)

* Support onSchemaChange for incremental tables

This change introduces support for the onSchemaChange option in incremental tables for the BigQuery adapter. It adds the incrementalSchemaChangeBody() strategy to handle schema changes.

End-to-end tests have been created to verify the new functionality.

* fix: Private functions moved after public functions

* - Split onSchemaChange queries to sub-tasks
- Split sql queries to sun-functions
- Fixed function names

* fix: Address feedback for PR #2101 and fix presubmit failures

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

* feat: implement .jitCode() support for Assertion actions (#2170)

- Add `jit_code` field to the `Assertion` message in `protos/core.proto`
- Fix missing protobuf imports and configuration in `protos/BUILD`
- Introduce `JitAssertionResult` type alias and `jitCode` method to `Assertion` builder (`core/actions/assertion.ts`)
- Add strictly typed `jitContextable` parameter to `Session.sqlxAction` in `core/session.ts`
- Update assertion `compile()` to handle `jitCode` execution bypassing standard string evaluation
- Parse and apply `jitContextable` safely for assertions in `core/session.ts`
- Add unit test coverage in `core/main_test.ts` to verify compilation output

* fix: Resolve presubmit failure introduced by PR #2101

- extend onSchemaChange e2e test timeout (#2175)

* Bump DF_VERSION from 3.0.57 to 3.0.58 (#2176)

* Guard workflow_settings.yaml and dataform.json require with existence check (#2178)

* Refactor CompileChildProcess to use BaseWorker (#2179)

* Rename worker timeout error to "Compilation timed out" (#2186)

* Extract task creation into executionSql helpers (#2180)

* Add JiT compilation support for assertion actions (#2193)

Adds JIT_COMPILATION_TARGET_TYPE_ASSERTION alongside the existing
TABLE / OPERATION / INCREMENTAL_TABLE target types, plus a matching
JitAssertionResult { string query } proto and a jitCompileAssertion()
dispatcher in core/jit_compiler.ts. Assertions reuse SqlActionJitContext
since AssertionContext implements IActionContext, and the result is a
single SELECT query string (matching the existing
JitAssertionResult = string alias in core/actions/assertion.ts).

* Fix caller-file error (#2177) and enforce Core/CLI version match (#2191)

vm2 3.11.3 strips file paths from V8 CallSite objects inside the sandbox,
so `getCallerFile()` in @dataform/core can no longer resolve the current
file from the stack. Track it via a host-side stack exposed through
`__df_enter`/`__df_exit`/`__df_current` sandbox helpers and read it through
a getter on `global.__dataform_current_file`.

For @dataform/core <= 3.0.56 (no global fallback in getCallerFile), patch
the bundle text at load time to consult the global. Gated on the installed
Core version so newer bundles are never touched.

Add a CLI/Core compatibility check: reject installed Core whose major
differs from the CLI's, or whose minor is below the CLI's minor. Resolve
the installed Core's version by running
`require("@dataform/core").version` inside the index-generator vm so any
install layout (package.json, workflow_settings.yaml stateless install,
JiT) reports the version actually loaded. The error tells the user the
exact `dataformCoreVersion:` line to set in `workflow_settings.yaml`.
Skipped when the CLI version can't be determined (unbundled local dev).

Mirror the same wrapper pattern in `testing/run_core.ts`.

Add `compile rejects @dataform/core with incompatible version` test in
`cli/index_compile_test.ts` that drives the workflow_settings.yaml
stateless install path with dataformCoreVersion: "2.9.0" (latest 2.x on
the registry) to exercise the real `npm i` install flow.

Add `compile succeeds with @dataform/core <= 3.0.56 via caller-file shim`
test that installs @dataform/core@3.0.50 and asserts the compiled action's
`fileName` matches the source path, end-to-end proving the bundle-text
patch and host-side file stack restore getCallerFile under vm2 3.11.3.

* Raise default compile timeout to 60s and improve timeout error message (#2192)

Default compilation timeout was 30s. Bump to 60s so larger projects
don't hit it on a single compile pass.

Extends the "Compilation timed out" error to point at --timeout with
concrete examples (--timeout=2m, --timeout=1h) so users can self-mitigate
without reading docs.

* Bump DF_VERSION from 3.0.58 to 3.0.59 (#2194)

* Bump vm2 from 3.11.3 to 3.11.4 (#2188)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.11.3 to 3.11.4.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md)
- [Commits](patriksimek/vm2@v3.11.3...v3.11.4)

---
updated-dependencies:
- dependency-name: vm2
  dependency-version: 3.11.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Extract BigQueryClientProvider for injectable BQ clients (#2181)

* Extract BigQueryClientProvider for injectable BQ clients

* Address review: collapse BigQueryClientProvider to a function type

* Bump protobufjs from 7.5.5 to 7.5.8 (#2172)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.5 to 7.5.8.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.5.8/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-v7.5.5...protobufjs-v7.5.8)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-version: 7.5.8
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add go_package option to db_adapter.proto and jit.proto (#2197)

* Migrate to bazel 7.3.2 (#2187)

* Migrate to bazel 7.3.2

* Use fixed ref for github action

* Update gcloud to 572.0.0 (#2207)

KMS has an org-policy enabled, requiring the usage of client certificate aware endpoints when performing the decryption of credentials.

Updating gcloud, so that it uses an updated compatible endpoint.

* Add metadata field to AssertionConfig (#2208)

Mirrors TableConfig.metadata / ViewConfig.metadata so callers can
attach key/value data that lands on
Assertion.action_descriptor.metadata.extra_properties.

* Bump DF_VERSION from 3.0.59 to 3.0.60 (#2209)

* Ignore trailing positional arguments in the global flag parser (#2199)

The flag parser in common/flags/index.ts runs in a static initialiser at
module load and parses process.argv. Once it had seen a --flag, any later
token that was neither another flag nor a flag value caused it to throw
"Arg neither flag name nor flag value", crashing the entire CLI before yargs
ran. yargs accepts positional arguments after flags (for example
`dataform run --some-flag value my_project`), so this lightweight parser was
rejecting valid argument orderings.

Ignore any token that is neither a flag nor a flag value instead of throwing,
exactly as already happened for positional arguments before the first flag;
yargs remains responsible for parsing positionals. Extract the parsing loop
into an exported parseArgv function so it can be unit-tested in isolation, and
add common/flags/index_test.ts covering the regression and related orderings.

Fixes #2198.

* Bump protobufjs-cli from 1.0.0 to 1.3.3 (#2202)

* Bump protobufjs-cli from 1.0.0 to 1.3.3

Bumps [protobufjs-cli](https://github.com/protobufjs/protobuf.js) from 1.0.0 to 1.3.3.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-cli-v1.0.0...protobufjs-cli-v1.3.3)

---
updated-dependencies:
- dependency-name: protobufjs-cli
  dependency-version: 1.3.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(tools): patch generated proto typings to use ES6 import for 'long'

This avoids syntax errors in older rollup-plugin-dts which does not support
'import Long = require("long")' syntax. We patch the generated ts.d.ts file
to use 'import Long from "long"' instead.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nick Nalivaika <ikolina@google.com>

* Bump js-yaml from 4.1.1 to 4.2.0 (#2205)

Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.1 to 4.2.0.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/commits)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 4.2.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add getContents() for reading external markdown files into table descriptions pr fix (#2164)

* initial attempt at md reader

* adding relevant tests and better path resolving for md

* cleaner get contents path handling

* escaping newlines for md

* adding documentation

* add check if file in rootdir for get contents and update documentation on getContents to include limits

* cleaner and filesystem agnostic check if getContents target is in the root  dir

* making sure root dir has seperator on check if getcontents file is in dir

* removing node module usage

* removing redundant normalize and root variable

* adding seperator when checking if file is in rootDir

* Bump protobufjs from 7.5.8 to 7.6.3 (#2204)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.8 to 7.6.3.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.3/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-v7.5.8...protobufjs-v7.6.3)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-version: 7.6.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(gcloud): support client certificate environment in gcloud actions and tests (#2213)

- Enable `use_default_shell_env` in `gcloud_secret` rule to pass host environment variables (like `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE`) to the gcloud tool.
- Pass `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE` to bazel test in `scripts/publish`.

* Prevent .sqlx files in actions.yaml configs with a clear error (#2189)

Putting a .sqlx file under the ``filename`` field of an entry in
``definitions/actions.yaml`` used to fall through to ``nativeRequire``,
which then failed with a cryptic error far from the source of the
mistake. ``.sqlx`` files are loaded directly from the ``definitions/``
directory by the sqlx compiler, not referenced through ``actions.yaml``.

Add an explicit validation step in ``loadActionConfigs`` that runs before
each action is constructed. If the action's ``filename`` ends in
``.sqlx`` (case-insensitive), emit a ``compileError`` naming the action
type, the offending filename, and pointing at the fix. The action is
then skipped so we don't double-up with a downstream ``nativeRequire``
error.

The check covers every action type that accepts a ``filename``: ``table``,
``view``, ``incrementalTable``, ``assertion``, ``operation``,
``declaration``, ``notebook`` and ``dataPreparation``. Data preparations
are included because a ``.dp.sqlx`` file is also auto-compiled from
``definitions/`` - referencing one from ``actions.yaml`` doesn't resolve
its path, ignores its ``config {}`` block and double-registers the
action, so it's the same mistake. ``.dp.yaml`` data preparations are
unaffected.

Closes #1785

* Bump tmp from 0.2.4 to 0.2.7 (#2183)

Bumps [tmp](https://github.com/raszi/node-tmp) from 0.2.4 to 0.2.7.
- [Changelog](https://github.com/raszi/node-tmp/blob/master/CHANGELOG.md)
- [Commits](raszi/node-tmp@v0.2.4...v0.2.7)

---
updated-dependencies:
- dependency-name: tmp
  dependency-version: 0.2.6
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @tootallnate/once from 2.0.0 to 2.0.1 (#2171)

Bumps [@tootallnate/once](https://github.com/TooTallNate/once) from 2.0.0 to 2.0.1.
- [Release notes](https://github.com/TooTallNate/once/releases)
- [Changelog](https://github.com/TooTallNate/once/blob/v2.0.1/CHANGELOG.md)
- [Commits](TooTallNate/once@2.0.0...v2.0.1)

---
updated-dependencies:
- dependency-name: "@tootallnate/once"
  dependency-version: 2.0.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fast-uri from 3.0.6 to 3.1.2 (#2161)

Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.0.6 to 3.1.2.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](fastify/fast-uri@v3.0.6...v3.1.2)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* remove extra utils_test.ts from api build

* allow pbts command to run on mac

* refactor concatenateQueries to be a bit more readable

* update on_schema_change tests to reflect the new format

* update format for api spec

* replace drop table in statement comment test because it was causing a test failure

* add new tests for comments within a query

* update test count in bigquery spec

* multi line comment tests

* update new golden tests to updated format

* fix bigquery spec test to add ending semi colons

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Stephen <spatel11@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rafal Hawrylak <hawrylak@google.com>
Co-authored-by: rafal-hawrylak <33635872+rafal-hawrylak@users.noreply.github.com>
Co-authored-by: Edvin <56413069+SuchodolskiEdvin@users.noreply.github.com>
Co-authored-by: Nick Nalivayka <ikolina@google.com>
Co-authored-by: ikholopov-omni <73560138+ikholopov-omni@users.noreply.github.com>
Co-authored-by: August Cayzer <august@cayzer.me>
Co-authored-by: Amman <87317538+AmmanuelT@users.noreply.github.com>
Co-authored-by: August Cayzer <a@cyzr.me>
mohamed-saad pushed a commit that referenced this pull request Jul 24, 2026
Putting a .sqlx file under the ``filename`` field of an entry in
``definitions/actions.yaml`` used to fall through to ``nativeRequire``,
which then failed with a cryptic error far from the source of the
mistake. ``.sqlx`` files are loaded directly from the ``definitions/``
directory by the sqlx compiler, not referenced through ``actions.yaml``.

Add an explicit validation step in ``loadActionConfigs`` that runs before
each action is constructed. If the action's ``filename`` ends in
``.sqlx`` (case-insensitive), emit a ``compileError`` naming the action
type, the offending filename, and pointing at the fix. The action is
then skipped so we don't double-up with a downstream ``nativeRequire``
error.

The check covers every action type that accepts a ``filename``: ``table``,
``view``, ``incrementalTable``, ``assertion``, ``operation``,
``declaration``, ``notebook`` and ``dataPreparation``. Data preparations
are included because a ``.dp.sqlx`` file is also auto-compiled from
``definitions/`` - referencing one from ``actions.yaml`` doesn't resolve
its path, ignores its ``config {}`` block and double-registers the
action, so it's the same mistake. ``.dp.yaml`` data preparations are
unaffected.

Closes #1785
mohamed-saad pushed a commit that referenced this pull request Jul 24, 2026
* ISSUE-2103 Handle trailing and trailing statement comments

* Bump vm2 from 3.10.2 to 3.11.3 (#2168)

* Bump vm2 from 3.10.2 to 3.11.3

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.10.2 to 3.11.3.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md)
- [Commits](patriksimek/vm2@v3.10.2...v3.11.3)

---
updated-dependencies:
- dependency-name: vm2
  dependency-version: 3.11.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix vm2 sandbox execution errors and CallSite path stripping

The upgrade of vm2 to 3.11.3 tightened sandbox security, stripping file path information from V8 CallSite objects during Error.prepareStackTrace and breaking the resolution of symlinked modules. This caused critical test failures ("Unable to find valid caller file") during Dataform compilation.

This commit resolves these issues by:
* Resolving `projectDir` to its absolute realpath before sandbox compilation to circumvent symlink boundaries (especially important for bazel runfiles).
* Injecting `global.__dataform_current_file` into the NodeVM compiler configuration with a try-finally block to track cross-boundary macro scopes.
* Adding a fallback in `core/utils.ts:getCallerFile` to read the injected global file path when the V8 stack trace path is stripped or undefined.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rafal Hawrylak <hawrylak@google.com>

* Bump DF_VERSION from 3.0.56 to 3.0.57 (#2173)

* Support onSchemaChange for incremental tables (#2101)

* Support onSchemaChange for incremental tables

This change introduces support for the onSchemaChange option in incremental tables for the BigQuery adapter. It adds the incrementalSchemaChangeBody() strategy to handle schema changes.

End-to-end tests have been created to verify the new functionality.

* fix: Private functions moved after public functions

* - Split onSchemaChange queries to sub-tasks
- Split sql queries to sun-functions
- Fixed function names

* fix: Address feedback for PR #2101 and fix presubmit failures

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

* feat: implement .jitCode() support for Assertion actions (#2170)

- Add `jit_code` field to the `Assertion` message in `protos/core.proto`
- Fix missing protobuf imports and configuration in `protos/BUILD`
- Introduce `JitAssertionResult` type alias and `jitCode` method to `Assertion` builder (`core/actions/assertion.ts`)
- Add strictly typed `jitContextable` parameter to `Session.sqlxAction` in `core/session.ts`
- Update assertion `compile()` to handle `jitCode` execution bypassing standard string evaluation
- Parse and apply `jitContextable` safely for assertions in `core/session.ts`
- Add unit test coverage in `core/main_test.ts` to verify compilation output

* fix: Resolve presubmit failure introduced by PR #2101

- extend onSchemaChange e2e test timeout (#2175)

* Bump DF_VERSION from 3.0.57 to 3.0.58 (#2176)

* Guard workflow_settings.yaml and dataform.json require with existence check (#2178)

* Refactor CompileChildProcess to use BaseWorker (#2179)

* Rename worker timeout error to "Compilation timed out" (#2186)

* Extract task creation into executionSql helpers (#2180)

* Add JiT compilation support for assertion actions (#2193)

Adds JIT_COMPILATION_TARGET_TYPE_ASSERTION alongside the existing
TABLE / OPERATION / INCREMENTAL_TABLE target types, plus a matching
JitAssertionResult { string query } proto and a jitCompileAssertion()
dispatcher in core/jit_compiler.ts. Assertions reuse SqlActionJitContext
since AssertionContext implements IActionContext, and the result is a
single SELECT query string (matching the existing
JitAssertionResult = string alias in core/actions/assertion.ts).

* Fix caller-file error (#2177) and enforce Core/CLI version match (#2191)

vm2 3.11.3 strips file paths from V8 CallSite objects inside the sandbox,
so `getCallerFile()` in @dataform/core can no longer resolve the current
file from the stack. Track it via a host-side stack exposed through
`__df_enter`/`__df_exit`/`__df_current` sandbox helpers and read it through
a getter on `global.__dataform_current_file`.

For @dataform/core <= 3.0.56 (no global fallback in getCallerFile), patch
the bundle text at load time to consult the global. Gated on the installed
Core version so newer bundles are never touched.

Add a CLI/Core compatibility check: reject installed Core whose major
differs from the CLI's, or whose minor is below the CLI's minor. Resolve
the installed Core's version by running
`require("@dataform/core").version` inside the index-generator vm so any
install layout (package.json, workflow_settings.yaml stateless install,
JiT) reports the version actually loaded. The error tells the user the
exact `dataformCoreVersion:` line to set in `workflow_settings.yaml`.
Skipped when the CLI version can't be determined (unbundled local dev).

Mirror the same wrapper pattern in `testing/run_core.ts`.

Add `compile rejects @dataform/core with incompatible version` test in
`cli/index_compile_test.ts` that drives the workflow_settings.yaml
stateless install path with dataformCoreVersion: "2.9.0" (latest 2.x on
the registry) to exercise the real `npm i` install flow.

Add `compile succeeds with @dataform/core <= 3.0.56 via caller-file shim`
test that installs @dataform/core@3.0.50 and asserts the compiled action's
`fileName` matches the source path, end-to-end proving the bundle-text
patch and host-side file stack restore getCallerFile under vm2 3.11.3.

* Raise default compile timeout to 60s and improve timeout error message (#2192)

Default compilation timeout was 30s. Bump to 60s so larger projects
don't hit it on a single compile pass.

Extends the "Compilation timed out" error to point at --timeout with
concrete examples (--timeout=2m, --timeout=1h) so users can self-mitigate
without reading docs.

* Bump DF_VERSION from 3.0.58 to 3.0.59 (#2194)

* Bump vm2 from 3.11.3 to 3.11.4 (#2188)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.11.3 to 3.11.4.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md)
- [Commits](patriksimek/vm2@v3.11.3...v3.11.4)

---
updated-dependencies:
- dependency-name: vm2
  dependency-version: 3.11.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Extract BigQueryClientProvider for injectable BQ clients (#2181)

* Extract BigQueryClientProvider for injectable BQ clients

* Address review: collapse BigQueryClientProvider to a function type

* Bump protobufjs from 7.5.5 to 7.5.8 (#2172)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.5 to 7.5.8.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.5.8/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-v7.5.5...protobufjs-v7.5.8)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-version: 7.5.8
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add go_package option to db_adapter.proto and jit.proto (#2197)

* Migrate to bazel 7.3.2 (#2187)

* Migrate to bazel 7.3.2

* Use fixed ref for github action

* Update gcloud to 572.0.0 (#2207)

KMS has an org-policy enabled, requiring the usage of client certificate aware endpoints when performing the decryption of credentials.

Updating gcloud, so that it uses an updated compatible endpoint.

* Add metadata field to AssertionConfig (#2208)

Mirrors TableConfig.metadata / ViewConfig.metadata so callers can
attach key/value data that lands on
Assertion.action_descriptor.metadata.extra_properties.

* Bump DF_VERSION from 3.0.59 to 3.0.60 (#2209)

* Ignore trailing positional arguments in the global flag parser (#2199)

The flag parser in common/flags/index.ts runs in a static initialiser at
module load and parses process.argv. Once it had seen a --flag, any later
token that was neither another flag nor a flag value caused it to throw
"Arg neither flag name nor flag value", crashing the entire CLI before yargs
ran. yargs accepts positional arguments after flags (for example
`dataform run --some-flag value my_project`), so this lightweight parser was
rejecting valid argument orderings.

Ignore any token that is neither a flag nor a flag value instead of throwing,
exactly as already happened for positional arguments before the first flag;
yargs remains responsible for parsing positionals. Extract the parsing loop
into an exported parseArgv function so it can be unit-tested in isolation, and
add common/flags/index_test.ts covering the regression and related orderings.

Fixes #2198.

* Bump protobufjs-cli from 1.0.0 to 1.3.3 (#2202)

* Bump protobufjs-cli from 1.0.0 to 1.3.3

Bumps [protobufjs-cli](https://github.com/protobufjs/protobuf.js) from 1.0.0 to 1.3.3.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-cli-v1.0.0...protobufjs-cli-v1.3.3)

---
updated-dependencies:
- dependency-name: protobufjs-cli
  dependency-version: 1.3.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(tools): patch generated proto typings to use ES6 import for 'long'

This avoids syntax errors in older rollup-plugin-dts which does not support
'import Long = require("long")' syntax. We patch the generated ts.d.ts file
to use 'import Long from "long"' instead.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nick Nalivaika <ikolina@google.com>

* Bump js-yaml from 4.1.1 to 4.2.0 (#2205)

Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.1 to 4.2.0.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/commits)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 4.2.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add getContents() for reading external markdown files into table descriptions pr fix (#2164)

* initial attempt at md reader

* adding relevant tests and better path resolving for md

* cleaner get contents path handling

* escaping newlines for md

* adding documentation

* add check if file in rootdir for get contents and update documentation on getContents to include limits

* cleaner and filesystem agnostic check if getContents target is in the root  dir

* making sure root dir has seperator on check if getcontents file is in dir

* removing node module usage

* removing redundant normalize and root variable

* adding seperator when checking if file is in rootDir

* Bump protobufjs from 7.5.8 to 7.6.3 (#2204)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.8 to 7.6.3.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.3/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-v7.5.8...protobufjs-v7.6.3)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-version: 7.6.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(gcloud): support client certificate environment in gcloud actions and tests (#2213)

- Enable `use_default_shell_env` in `gcloud_secret` rule to pass host environment variables (like `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE`) to the gcloud tool.
- Pass `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE` to bazel test in `scripts/publish`.

* Prevent .sqlx files in actions.yaml configs with a clear error (#2189)

Putting a .sqlx file under the ``filename`` field of an entry in
``definitions/actions.yaml`` used to fall through to ``nativeRequire``,
which then failed with a cryptic error far from the source of the
mistake. ``.sqlx`` files are loaded directly from the ``definitions/``
directory by the sqlx compiler, not referenced through ``actions.yaml``.

Add an explicit validation step in ``loadActionConfigs`` that runs before
each action is constructed. If the action's ``filename`` ends in
``.sqlx`` (case-insensitive), emit a ``compileError`` naming the action
type, the offending filename, and pointing at the fix. The action is
then skipped so we don't double-up with a downstream ``nativeRequire``
error.

The check covers every action type that accepts a ``filename``: ``table``,
``view``, ``incrementalTable``, ``assertion``, ``operation``,
``declaration``, ``notebook`` and ``dataPreparation``. Data preparations
are included because a ``.dp.sqlx`` file is also auto-compiled from
``definitions/`` - referencing one from ``actions.yaml`` doesn't resolve
its path, ignores its ``config {}`` block and double-registers the
action, so it's the same mistake. ``.dp.yaml`` data preparations are
unaffected.

Closes #1785

* Bump tmp from 0.2.4 to 0.2.7 (#2183)

Bumps [tmp](https://github.com/raszi/node-tmp) from 0.2.4 to 0.2.7.
- [Changelog](https://github.com/raszi/node-tmp/blob/master/CHANGELOG.md)
- [Commits](raszi/node-tmp@v0.2.4...v0.2.7)

---
updated-dependencies:
- dependency-name: tmp
  dependency-version: 0.2.6
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @tootallnate/once from 2.0.0 to 2.0.1 (#2171)

Bumps [@tootallnate/once](https://github.com/TooTallNate/once) from 2.0.0 to 2.0.1.
- [Release notes](https://github.com/TooTallNate/once/releases)
- [Changelog](https://github.com/TooTallNate/once/blob/v2.0.1/CHANGELOG.md)
- [Commits](TooTallNate/once@2.0.0...v2.0.1)

---
updated-dependencies:
- dependency-name: "@tootallnate/once"
  dependency-version: 2.0.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fast-uri from 3.0.6 to 3.1.2 (#2161)

Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.0.6 to 3.1.2.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](fastify/fast-uri@v3.0.6...v3.1.2)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* remove extra utils_test.ts from api build

* allow pbts command to run on mac

* refactor concatenateQueries to be a bit more readable

* update on_schema_change tests to reflect the new format

* update format for api spec

* replace drop table in statement comment test because it was causing a test failure

* add new tests for comments within a query

* update test count in bigquery spec

* multi line comment tests

* update new golden tests to updated format

* fix bigquery spec test to add ending semi colons

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Stephen <spatel11@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rafal Hawrylak <hawrylak@google.com>
Co-authored-by: rafal-hawrylak <33635872+rafal-hawrylak@users.noreply.github.com>
Co-authored-by: Edvin <56413069+SuchodolskiEdvin@users.noreply.github.com>
Co-authored-by: Nick Nalivayka <ikolina@google.com>
Co-authored-by: ikholopov-omni <73560138+ikholopov-omni@users.noreply.github.com>
Co-authored-by: August Cayzer <august@cayzer.me>
Co-authored-by: Amman <87317538+AmmanuelT@users.noreply.github.com>
Co-authored-by: August Cayzer <a@cyzr.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prevent putting .sqlx files in actions.yaml configs

2 participants