Skip to content

[Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines #3225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewnester merged 10 commits into from
Aug 25, 2025
Merged

[Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines #3225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
69c3603
Convert warning about using fallback paths to error
andrewnester Jul 10, 2025
0b3a312
fixes
andrewnester Jul 25, 2025
a96fd9d
changelog
andrewnester Jul 25, 2025
aa3ad3d
removed trailing slashes
andrewnester Jul 28, 2025
02daa25
Update NEXT_CHANGELOG.md
andrewnester Jul 28, 2025
04fd8af
include option to use earlier version
andrewnester Jul 28, 2025
de6f8b1
also for pipelines
andrewnester Jul 28, 2025
ce5a4ca
merge changelog
andrewnester Aug 25, 2025
dff6707
-
andrewnester Aug 25, 2025
8c509cf
-
andrewnester Aug 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions NEXT_CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,18 @@
## Release v0.266.0

### Notable Changes
* Breaking change: DABs now return an error when paths are incorrectly defined relative to the job or
pipeline definition location instead of the configuration file location. Previously, the CLI would show a
warning and fallback to resolving the path relative to the resource location. Users must update their bundle
configurations to define all relative paths relative to the configuration file where the path is specified.
See more details here: ([#3225](https://github.com/databricks/cli/pull/3225))
Copy link
Copy Markdown
Contributor

@pietern pietern Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds scary, but it only affects users with a configuration that already displays warnings. It removes behavior that was already considered deprecated. Is it possible to make it sound less scary?

* Add support volumes in Python support ([#3383])(https://github.com/databricks/cli/pull/3383))

### CLI

### Dependency updates

### Bundles
* [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](https://github.com/databricks/cli/pull/3225))

### API Changes
38 changes: 30 additions & 8 deletions acceptance/bundle/paths/fallback/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,36 @@ Warning: required field "entry_point" is not set
in resources/my_job.yml:33:13
override_job.yml:29:17

Warn: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_job.yml:9:32
resources/my_job.yml:9:28

Error: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_job.yml:13:30
resources/my_job.yml:14:26

Error: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_job.yml:17:36
resources/my_job.yml:19:32

Error: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_job.yml:24:25
resources/my_job.yml:27:21

Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_job.yml:33:24

Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_job.yml:41:24

Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_pipeline.yml:13:23

Error: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
in override_pipeline.yml:10:23


Exit code: 1

>>> [CLI] bundle validate -t error
Warning: required field "entry_point" is not set
Expand Down
18 changes: 0 additions & 18 deletions acceptance/bundle/paths/fallback_metric/databricks.yml
View file
Open in desktop

This file was deleted.

5 changes: 0 additions & 5 deletions acceptance/bundle/paths/fallback_metric/out.test.toml
View file
Open in desktop

This file was deleted.

18 changes: 0 additions & 18 deletions acceptance/bundle/paths/fallback_metric/resources/job.yml
View file
Open in desktop

This file was deleted.

5 changes: 0 additions & 5 deletions acceptance/bundle/paths/fallback_metric/script
View file
Open in desktop

This file was deleted.

2 changes: 0 additions & 2 deletions acceptance/bundle/paths/fallback_metric/src/notebook.py
View file
Open in desktop

This file was deleted.

10 changes: 0 additions & 10 deletions acceptance/bundle/paths/fallback_metric/test.toml
View file
Open in desktop

This file was deleted.

14 changes: 9 additions & 5 deletions bundle/config/mutator/translate_paths_jobs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@ package mutator

import (
"context"
"fmt"
"path/filepath"
"slices"

"github.com/databricks/cli/bundle/config/mutator/paths"

"github.com/databricks/cli/libs/diag"
"github.com/databricks/cli/libs/dyn"
"github.com/databricks/cli/libs/log"
"github.com/databricks/cli/libs/logdiag"
)

func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value) (dyn.Value, error) {
Expand Down Expand Up @@ -45,8 +47,7 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value
return nv, nil
}

// If we failed to rewrite the path, try to rewrite it relative to the fallback directory.
// We only do this for jobs and pipelines because of the comment in [gatherFallbackPaths].
// If we failed to rewrite the path, it uses an old path format which relied on fallback.
if fallback[key] != "" {
dir, nerr := locationDirectory(v.Location())
if nerr != nil {
Expand All @@ -66,8 +67,11 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value
originalValue := dyn.NewValue(originalPath, v.Locations())
nv, nerr := t.rewriteValue(ctx, p, originalValue, fallback[key], opts)
if nerr == nil {
t.b.Metrics.AddBoolValue("is_job_path_fallback", true)
log.Warnf(ctx, "path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location())
logdiag.LogDiag(ctx, diag.Diagnostic{
Severity: diag.Error,
Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).", originalPath, fallback[key], v.Location()),
Locations: v.Locations(),
})
return nv, nil
}
}
Expand Down
14 changes: 9 additions & 5 deletions bundle/config/mutator/translate_paths_pipelines.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@ package mutator

import (
"context"
"fmt"
"path/filepath"

"github.com/databricks/cli/bundle/config/mutator/paths"

"github.com/databricks/cli/libs/diag"
"github.com/databricks/cli/libs/dyn"
"github.com/databricks/cli/libs/log"
"github.com/databricks/cli/libs/logdiag"
)

func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn.Value) (dyn.Value, error) {
Expand All @@ -30,8 +32,7 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn.
return nv, nil
}

// If we failed to rewrite the path, try to rewrite it relative to the fallback directory.
// We only do this for jobs and pipelines because of the comment in [gatherFallbackPaths].
// If we failed to rewrite the path, it uses an old path format which relied on fallback.
if fallback[key] != "" {
dir, nerr := locationDirectory(v.Location())
if nerr != nil {
Expand All @@ -51,8 +52,11 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn.
originalValue := dyn.NewValue(originalPath, v.Locations())
nv, nerr := t.rewriteValue(ctx, p, originalValue, fallback[key], opts)
if nerr == nil {
t.b.Metrics.AddBoolValue("is_pipeline_path_fallback", true)
log.Warnf(ctx, "path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location())
logdiag.LogDiag(ctx, diag.Diagnostic{
Severity: diag.Error,
Summary: fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).", originalPath, fallback[key], v.Location()),
Locations: v.Locations(),
})
return nv, nil
}
}
Expand Down
Loading