From 69c3603d817ab0bcc01501c6da100d9a220e8eea Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Thu, 10 Jul 2025 11:54:05 +0200 Subject: [PATCH 01/10] Convert warning about using fallback paths to error --- acceptance/bundle/paths/fallback/output.txt | 43 +++++++++++++------ .../paths/fallback_metric/databricks.yml | 18 -------- .../paths/fallback_metric/out.test.toml | 5 --- .../paths/fallback_metric/resources/job.yml | 18 -------- .../bundle/paths/fallback_metric/script | 5 --- .../paths/fallback_metric/src/notebook.py | 2 - .../bundle/paths/fallback_metric/test.toml | 10 ----- bundle/config/mutator/translate_paths_jobs.go | 11 +++-- .../mutator/translate_paths_pipelines.go | 11 +++-- 9 files changed, 46 insertions(+), 77 deletions(-) delete mode 100644 acceptance/bundle/paths/fallback_metric/databricks.yml delete mode 100644 acceptance/bundle/paths/fallback_metric/out.test.toml delete mode 100644 acceptance/bundle/paths/fallback_metric/resources/job.yml delete mode 100644 acceptance/bundle/paths/fallback_metric/script delete mode 100644 acceptance/bundle/paths/fallback_metric/src/notebook.py delete mode 100644 acceptance/bundle/paths/fallback_metric/test.toml diff --git a/acceptance/bundle/paths/fallback/output.txt b/acceptance/bundle/paths/fallback/output.txt index 6b819fe9c0..db40a2bc8c 100644 --- a/acceptance/bundle/paths/fallback/output.txt +++ b/acceptance/bundle/paths/fallback/output.txt @@ -1,18 +1,35 @@ >>> [CLI] bundle validate -t development -o json -Warning: required field "entry_point" is not set - at resources.jobs.my_job.tasks[2].python_wheel_task - in resources/my_job.yml:33:13 - override_job.yml:29:17 - -Warn: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. -Warn: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. -Warn: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. -Warn: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. -Warn: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. -Warn: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. -Warn: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. -Warn: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_job.yml:9:32 + resources/my_job.yml:9:28 + +Error: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_job.yml:13:30 + resources/my_job.yml:14:26 + +Error: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_job.yml:17:36 + resources/my_job.yml:19:32 + +Error: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_job.yml:24:25 + resources/my_job.yml:27:21 + +Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_job.yml:33:24 + +Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_job.yml:41:24 + +Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_pipeline.yml:13:23 + +Error: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. + in override_pipeline.yml:10:23 + + +Exit code: 1 >>> [CLI] bundle validate -t error Warning: required field "entry_point" is not set diff --git a/acceptance/bundle/paths/fallback_metric/databricks.yml b/acceptance/bundle/paths/fallback_metric/databricks.yml deleted file mode 100644 index 3655dc1d2d..0000000000 --- a/acceptance/bundle/paths/fallback_metric/databricks.yml +++ /dev/null @@ -1,18 +0,0 @@ -bundle: - name: fallback_metric - -include: - - "resources/*.yml" - -targets: - fallback: - resources: - jobs: - my_job: - tasks: - - task_key: notebook_example - notebook_task: - notebook_path: ../src/notebook.py - - default: - default: true diff --git a/acceptance/bundle/paths/fallback_metric/out.test.toml b/acceptance/bundle/paths/fallback_metric/out.test.toml deleted file mode 100644 index 8f3575be7b..0000000000 --- a/acceptance/bundle/paths/fallback_metric/out.test.toml +++ /dev/null @@ -1,5 +0,0 @@ -Local = true -Cloud = false - -[EnvMatrix] - DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"] diff --git a/acceptance/bundle/paths/fallback_metric/resources/job.yml b/acceptance/bundle/paths/fallback_metric/resources/job.yml deleted file mode 100644 index a345674ab6..0000000000 --- a/acceptance/bundle/paths/fallback_metric/resources/job.yml +++ /dev/null @@ -1,18 +0,0 @@ -resources: - jobs: - my_job: - name: "placeholder" - tasks: - - task_key: notebook_example - new_cluster: - spark_version: 15.4.x-scala2.12 - node_type_id: i3.xlarge - data_security_mode: SINGLE_USER - num_workers: 0 - spark_conf: - spark.master: "local[*, 4]" - spark.databricks.cluster.profile: singleNode - custom_tags: - ResourceClass: SingleNode - notebook_task: - notebook_path: "../src/notebook.py" diff --git a/acceptance/bundle/paths/fallback_metric/script b/acceptance/bundle/paths/fallback_metric/script deleted file mode 100644 index 6fbe46ab57..0000000000 --- a/acceptance/bundle/paths/fallback_metric/script +++ /dev/null @@ -1,5 +0,0 @@ -errcode trace $CLI bundle deploy -errcode trace $CLI bundle deploy -t fallback -trace cat out.requests.txt | jq 'select(has("path") and .path == "/telemetry-ext") | .body.protoLogs[] | fromjson | .entry.databricks_cli_log.bundle_deploy_event.experimental.bool_values' - -rm -rf out.requests.txt diff --git a/acceptance/bundle/paths/fallback_metric/src/notebook.py b/acceptance/bundle/paths/fallback_metric/src/notebook.py deleted file mode 100644 index 38d86b79c7..0000000000 --- a/acceptance/bundle/paths/fallback_metric/src/notebook.py +++ /dev/null @@ -1,2 +0,0 @@ -# Databricks notebook source -print("Hello, World!") diff --git a/acceptance/bundle/paths/fallback_metric/test.toml b/acceptance/bundle/paths/fallback_metric/test.toml deleted file mode 100644 index cfd0d852bc..0000000000 --- a/acceptance/bundle/paths/fallback_metric/test.toml +++ /dev/null @@ -1,10 +0,0 @@ -Cloud = false -RecordRequests = true - -Ignore = [ - ".databricks" -] - -[[Repls]] -Old = '\\' -New = '/' diff --git a/bundle/config/mutator/translate_paths_jobs.go b/bundle/config/mutator/translate_paths_jobs.go index 854c13fbca..cb0098f59b 100644 --- a/bundle/config/mutator/translate_paths_jobs.go +++ b/bundle/config/mutator/translate_paths_jobs.go @@ -2,13 +2,15 @@ package mutator import ( "context" + "fmt" "path/filepath" "slices" "github.com/databricks/cli/bundle/config/mutator/paths" + "github.com/databricks/cli/libs/diag" "github.com/databricks/cli/libs/dyn" - "github.com/databricks/cli/libs/log" + "github.com/databricks/cli/libs/logdiag" ) func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value) (dyn.Value, error) { @@ -66,8 +68,11 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value originalValue := dyn.NewValue(originalPath, v.Locations()) nv, nerr := t.rewriteValue(ctx, p, originalValue, fallback[key], opts) if nerr == nil { - t.b.Metrics.AddBoolValue("is_job_path_fallback", true) - log.Warnf(ctx, "path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location()) + logdiag.LogDiag(ctx, diag.Diagnostic{ + Severity: diag.Error, + Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location()), + Locations: v.Locations(), + }) return nv, nil } } diff --git a/bundle/config/mutator/translate_paths_pipelines.go b/bundle/config/mutator/translate_paths_pipelines.go index b65146f796..5136091032 100644 --- a/bundle/config/mutator/translate_paths_pipelines.go +++ b/bundle/config/mutator/translate_paths_pipelines.go @@ -2,12 +2,14 @@ package mutator import ( "context" + "fmt" "path/filepath" "github.com/databricks/cli/bundle/config/mutator/paths" + "github.com/databricks/cli/libs/diag" "github.com/databricks/cli/libs/dyn" - "github.com/databricks/cli/libs/log" + "github.com/databricks/cli/libs/logdiag" ) func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn.Value) (dyn.Value, error) { @@ -51,8 +53,11 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn. originalValue := dyn.NewValue(originalPath, v.Locations()) nv, nerr := t.rewriteValue(ctx, p, originalValue, fallback[key], opts) if nerr == nil { - t.b.Metrics.AddBoolValue("is_pipeline_path_fallback", true) - log.Warnf(ctx, "path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location()) + logdiag.LogDiag(ctx, diag.Diagnostic{ + Severity: diag.Error, + Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location()), + Locations: v.Locations(), + }) return nv, nil } } From 0b3a3126273441b1920f0fff64f1f80f482cc825 Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Fri, 25 Jul 2025 13:15:38 +0200 Subject: [PATCH 02/10] fixes --- acceptance/bundle/paths/fallback/output.txt | 16 ++++++++-------- bundle/config/mutator/translate_paths_jobs.go | 5 ++--- .../config/mutator/translate_paths_pipelines.go | 5 ++--- 3 files changed, 12 insertions(+), 14 deletions(-) diff --git a/acceptance/bundle/paths/fallback/output.txt b/acceptance/bundle/paths/fallback/output.txt index db40a2bc8c..f59e221d28 100644 --- a/acceptance/bundle/paths/fallback/output.txt +++ b/acceptance/bundle/paths/fallback/output.txt @@ -1,31 +1,31 @@ >>> [CLI] bundle validate -t development -o json -Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. in override_job.yml:9:32 resources/my_job.yml:9:28 -Error: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. in override_job.yml:13:30 resources/my_job.yml:14:26 -Error: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. in override_job.yml:17:36 resources/my_job.yml:19:32 -Error: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined. in override_job.yml:24:25 resources/my_job.yml:27:21 -Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined. in override_job.yml:33:24 -Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined. in override_job.yml:41:24 -Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. in override_pipeline.yml:13:23 -Error: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release. +Error: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. in override_pipeline.yml:10:23 diff --git a/bundle/config/mutator/translate_paths_jobs.go b/bundle/config/mutator/translate_paths_jobs.go index cb0098f59b..f331fa6224 100644 --- a/bundle/config/mutator/translate_paths_jobs.go +++ b/bundle/config/mutator/translate_paths_jobs.go @@ -47,8 +47,7 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value return nv, nil } - // If we failed to rewrite the path, try to rewrite it relative to the fallback directory. - // We only do this for jobs and pipelines because of the comment in [gatherFallbackPaths]. + // If we failed to rewrite the path, it uses an old path format which relied on fallback. if fallback[key] != "" { dir, nerr := locationDirectory(v.Location()) if nerr != nil { @@ -70,7 +69,7 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value if nerr == nil { logdiag.LogDiag(ctx, diag.Diagnostic{ Severity: diag.Error, - Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location()), + Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined.", originalPath, fallback[key], v.Location()), Locations: v.Locations(), }) return nv, nil diff --git a/bundle/config/mutator/translate_paths_pipelines.go b/bundle/config/mutator/translate_paths_pipelines.go index 5136091032..5f8b13b411 100644 --- a/bundle/config/mutator/translate_paths_pipelines.go +++ b/bundle/config/mutator/translate_paths_pipelines.go @@ -32,8 +32,7 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn. return nv, nil } - // If we failed to rewrite the path, try to rewrite it relative to the fallback directory. - // We only do this for jobs and pipelines because of the comment in [gatherFallbackPaths]. + // If we failed to rewrite the path, it uses an old path format which relied on fallback. if fallback[key] != "" { dir, nerr := locationDirectory(v.Location()) if nerr != nil { @@ -55,7 +54,7 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn. if nerr == nil { logdiag.LogDiag(ctx, diag.Diagnostic{ Severity: diag.Error, - Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location()), + Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined.", originalPath, fallback[key], v.Location()), Locations: v.Locations(), }) return nv, nil From a96fd9d72f9a20160ed3372a903695a90f3c5b0c Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Fri, 25 Jul 2025 13:20:58 +0200 Subject: [PATCH 03/10] changelog --- NEXT_CHANGELOG.md | 7 +++++++ acceptance/bundle/paths/fallback/output.txt | 5 +++++ 2 files changed, 12 insertions(+) diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md index d53d3a2816..48fd8e4781 100644 --- a/NEXT_CHANGELOG.md +++ b/NEXT_CHANGELOG.md @@ -3,12 +3,19 @@ ## Release v0.266.0 ### Notable Changes +* Breaking change: if the relative paths to the bundle resources are defined relatively to where the job or pipeline +is defined rather than the configuration file where this path is defined, DABs will return an error. +Previously, it would fallback to older path resolution. This fallback path resolution is deprecated now. +Please update the path to be relative to the configuration file where this path is defined. * Add support volumes in Python support ([#3383])(https://github.com/databricks/cli/pull/3383)) +### Dependency updates + ### CLI ### Dependency updates ### Bundles +* [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](https://github.com/databricks/cli/pull/3225)) ### API Changes diff --git a/acceptance/bundle/paths/fallback/output.txt b/acceptance/bundle/paths/fallback/output.txt index f59e221d28..1425c156e7 100644 --- a/acceptance/bundle/paths/fallback/output.txt +++ b/acceptance/bundle/paths/fallback/output.txt @@ -1,5 +1,10 @@ >>> [CLI] bundle validate -t development -o json +Warning: required field "entry_point" is not set + at resources.jobs.my_job.tasks[2].python_wheel_task + in resources/my_job.yml:33:13 + override_job.yml:29:17 + Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. in override_job.yml:9:32 resources/my_job.yml:9:28 From aa3ad3d7ea8ad8a2a402938a36a197eb6d333d52 Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Mon, 28 Jul 2025 10:46:21 +0200 Subject: [PATCH 04/10] removed trailing slashes --- NEXT_CHANGELOG.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md index 48fd8e4781..bfed2b9870 100644 --- a/NEXT_CHANGELOG.md +++ b/NEXT_CHANGELOG.md @@ -3,11 +3,10 @@ ## Release v0.266.0 ### Notable Changes -* Breaking change: if the relative paths to the bundle resources are defined relatively to where the job or pipeline -is defined rather than the configuration file where this path is defined, DABs will return an error. -Previously, it would fallback to older path resolution. This fallback path resolution is deprecated now. -Please update the path to be relative to the configuration file where this path is defined. -* Add support volumes in Python support ([#3383])(https://github.com/databricks/cli/pull/3383)) +* Breaking change: DABs now returns an error when paths are incorrectly defined relative to the job or +pipeline definition location instead of the configuration file location. Previously, the CLI would show a +warning and fallback to resolving the path relative to the resource location. Users must update their bundle +configurations to define all relative paths relative to the configuration file where the path is specified. ### Dependency updates From 02daa25236d34fbafdde24f13f07768b59e3218a Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Mon, 28 Jul 2025 13:00:03 +0200 Subject: [PATCH 05/10] Update NEXT_CHANGELOG.md --- NEXT_CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md index bfed2b9870..bf44d6f3e0 100644 --- a/NEXT_CHANGELOG.md +++ b/NEXT_CHANGELOG.md @@ -3,10 +3,11 @@ ## Release v0.266.0 ### Notable Changes -* Breaking change: DABs now returns an error when paths are incorrectly defined relative to the job or +* Breaking change: DABs now return an error when paths are incorrectly defined relative to the job or pipeline definition location instead of the configuration file location. Previously, the CLI would show a warning and fallback to resolving the path relative to the resource location. Users must update their bundle configurations to define all relative paths relative to the configuration file where the path is specified. +See more details here: ([#3225](https://github.com/databricks/cli/pull/3225)) ### Dependency updates @@ -16,5 +17,6 @@ configurations to define all relative paths relative to the configuration file w ### Bundles * [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](https://github.com/databricks/cli/pull/3225)) +* Modified run flag validation to allow `--refresh` and `--full-refresh` flags to be used together ([#3300](https://github.com/databricks/cli/pull/3300)) ### API Changes From 04fd8af5c7401aa2056787cef5e9fb394cdfbd7a Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Mon, 28 Jul 2025 13:25:37 +0200 Subject: [PATCH 06/10] include option to use earlier version --- acceptance/bundle/paths/fallback/output.txt | 12 ++++++------ bundle/config/mutator/translate_paths_jobs.go | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/acceptance/bundle/paths/fallback/output.txt b/acceptance/bundle/paths/fallback/output.txt index 1425c156e7..8b4af4d97b 100644 --- a/acceptance/bundle/paths/fallback/output.txt +++ b/acceptance/bundle/paths/fallback/output.txt @@ -5,26 +5,26 @@ Warning: required field "entry_point" is not set in resources/my_job.yml:33:13 override_job.yml:29:17 -Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. +Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_job.yml:9:32 resources/my_job.yml:9:28 -Error: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. +Error: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_job.yml:13:30 resources/my_job.yml:14:26 -Error: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. +Error: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_job.yml:17:36 resources/my_job.yml:19:32 -Error: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined. +Error: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_job.yml:24:25 resources/my_job.yml:27:21 -Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined. +Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_job.yml:33:24 -Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined. +Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_job.yml:41:24 Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. diff --git a/bundle/config/mutator/translate_paths_jobs.go b/bundle/config/mutator/translate_paths_jobs.go index f331fa6224..9a3e492a89 100644 --- a/bundle/config/mutator/translate_paths_jobs.go +++ b/bundle/config/mutator/translate_paths_jobs.go @@ -69,7 +69,7 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value if nerr == nil { logdiag.LogDiag(ctx, diag.Diagnostic{ Severity: diag.Error, - Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined.", originalPath, fallback[key], v.Location()), + Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).", originalPath, fallback[key], v.Location()), Locations: v.Locations(), }) return nv, nil From de6f8b1c152723fe8f580acccb15001e47818b99 Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Mon, 28 Jul 2025 13:27:20 +0200 Subject: [PATCH 07/10] also for pipelines --- acceptance/bundle/paths/fallback/output.txt | 4 ++-- bundle/config/mutator/translate_paths_pipelines.go | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/acceptance/bundle/paths/fallback/output.txt b/acceptance/bundle/paths/fallback/output.txt index 8b4af4d97b..5bdbb9942c 100644 --- a/acceptance/bundle/paths/fallback/output.txt +++ b/acceptance/bundle/paths/fallback/output.txt @@ -27,10 +27,10 @@ Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resourc Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_job.yml:41:24 -Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. +Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_pipeline.yml:13:23 -Error: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. +Error: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier). in override_pipeline.yml:10:23 diff --git a/bundle/config/mutator/translate_paths_pipelines.go b/bundle/config/mutator/translate_paths_pipelines.go index 5f8b13b411..828981248a 100644 --- a/bundle/config/mutator/translate_paths_pipelines.go +++ b/bundle/config/mutator/translate_paths_pipelines.go @@ -54,7 +54,7 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn. if nerr == nil { logdiag.LogDiag(ctx, diag.Diagnostic{ Severity: diag.Error, - Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined.", originalPath, fallback[key], v.Location()), + Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).", originalPath, fallback[key], v.Location()), Locations: v.Locations(), }) return nv, nil From ce5a4caa9fa02cd86e892cfd4328310cc23fe703 Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Mon, 25 Aug 2025 13:14:52 +0200 Subject: [PATCH 08/10] merge changelog --- NEXT_CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md index bf44d6f3e0..745c155db1 100644 --- a/NEXT_CHANGELOG.md +++ b/NEXT_CHANGELOG.md @@ -8,6 +8,7 @@ pipeline definition location instead of the configuration file location. Previou warning and fallback to resolving the path relative to the resource location. Users must update their bundle configurations to define all relative paths relative to the configuration file where the path is specified. See more details here: ([#3225](https://github.com/databricks/cli/pull/3225)) +* Add support volumes in Python support ([#3383])(https://github.com/databricks/cli/pull/3383)) ### Dependency updates From dff6707cfb1d93a9db952465a48314edbaf63b7a Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Mon, 25 Aug 2025 13:16:07 +0200 Subject: [PATCH 09/10] - --- NEXT_CHANGELOG.md | 1 - 1 file changed, 1 deletion(-) diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md index 745c155db1..afab20c774 100644 --- a/NEXT_CHANGELOG.md +++ b/NEXT_CHANGELOG.md @@ -18,6 +18,5 @@ See more details here: ([#3225](https://github.com/databricks/cli/pull/3225)) ### Bundles * [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](https://github.com/databricks/cli/pull/3225)) -* Modified run flag validation to allow `--refresh` and `--full-refresh` flags to be used together ([#3300](https://github.com/databricks/cli/pull/3300)) ### API Changes From 8c509cfda4ac0be3d58b2aee51de98f218f538d8 Mon Sep 17 00:00:00 2001 From: Andrew Nester Date: Mon, 25 Aug 2025 13:16:34 +0200 Subject: [PATCH 10/10] - --- NEXT_CHANGELOG.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md index afab20c774..54e0ad8e8d 100644 --- a/NEXT_CHANGELOG.md +++ b/NEXT_CHANGELOG.md @@ -10,8 +10,6 @@ configurations to define all relative paths relative to the configuration file w See more details here: ([#3225](https://github.com/databricks/cli/pull/3225)) * Add support volumes in Python support ([#3383])(https://github.com/databricks/cli/pull/3383)) -### Dependency updates - ### CLI ### Dependency updates