Skip to content

Fixed normalising requirements file path in dependencies section #2861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewnester merged 4 commits into from
May 13, 2025
Merged

Fixed normalising requirements file path in dependencies section #2861

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cff6306
Fixed normalising requirements file path in dependencies section
andrewnester May 13, 2025
f588eed
added changelog
andrewnester May 13, 2025
9c72b26
Merge branch 'main' into fix/normalise-requirements-path
andrewnester May 13, 2025
280e1a2
trim spaces
andrewnester May 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions NEXT_CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,6 @@

### Bundles
* Removed unused fields from resources.models schema: creation\_timestamp, last\_updated\_timestamp, latest\_versions and user\_id. Using them now raises a warning.
* Fixed normalising requirements file path in dependencies section ([#2861](https://github.com/databricks/cli/pull/2861))

### API Changes
2 changes: 2 additions & 0 deletions acceptance/bundle/environments/dependencies/databricks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
bundle:
name: "dependencies"

include:
- resources/*.yml

resources:
jobs:
Expand Down
44 changes: 35 additions & 9 deletions acceptance/bundle/environments/dependencies/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,44 @@ Deployment complete!
"requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip"
]
}
},
{
"environment_key": "test_env_2",
"spec": {
"client": "1",
"dependencies": [
"-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
]
}
}
]

>>> [CLI] bundle validate -o json
[
"-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
"test_package",
"test_package==2.0.1",
"test_package>=2.0.1",
"./dist/*.whl",
"/Workspace/Users/test@databricks.com/test-package.whl",
"beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
"beautifulsoup4[security, tests] ~= 4.12.3",
"requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip"
{
"environment_key": "test_env",
"spec": {
"client": "1",
"dependencies": [
"-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
"test_package",
"test_package==2.0.1",
"test_package>=2.0.1",
"./dist/*.whl",
"/Workspace/Users/test@databricks.com/test-package.whl",
"beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
"beautifulsoup4[security, tests] ~= 4.12.3",
"requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip"
]
}
},
{
"environment_key": "test_env_2",
"spec": {
"client": "1",
"dependencies": [
"-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
]
}
}
]
13 changes: 13 additions & 0 deletions acceptance/bundle/environments/dependencies/resources/job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
targets:
default:
default: true
resources:
jobs:
test_job:
name: "Test Job"
environments:
- environment_key: "test_env_2"
spec:
client: "1"
dependencies:
- "-r ../requirements.txt"
2 changes: 1 addition & 1 deletion acceptance/bundle/environments/dependencies/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ trace $CLI bundle validate

trace $CLI bundle deploy
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body.environments' out.requests.txt
trace $CLI bundle validate -o json | jq '.resources.jobs.test_job.environments[0].spec.dependencies'
trace $CLI bundle validate -o json | jq '.resources.jobs.test_job.environments'
rm out.requests.txt
15 changes: 15 additions & 0 deletions bundle/config/mutator/normalize_paths.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"net/url"
pathlib "path"
"path/filepath"
"strings"

"github.com/databricks/cli/bundle"
"github.com/databricks/cli/bundle/config/mutator/paths"
Expand Down Expand Up @@ -84,6 +85,20 @@ func collectGitSourcePaths(b *bundle.Bundle) []dyn.Path {
}

func normalizePath(path string, location dyn.Location, bundleRootPath string) (string, error) {
// Handle requirements file paths with -r flag
reqPath, ok := strings.CutPrefix(path, "-r ")
Copy link
Copy Markdown
Contributor

@denik denik May 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also TrimSpace in case there multiple spaces?

Copy link
Copy Markdown
Contributor Author

@andrewnester andrewnester May 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, fixed + added test

Copy link
Copy Markdown
Collaborator

@kanterov kanterov May 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about --requirement

if ok {
// Normalize the path part
reqPath = strings.TrimSpace(reqPath)
normalizedPath, err := normalizePath(reqPath, location, bundleRootPath)
if err != nil {
return "", err
}

// Reconstruct the path with -r flag
return "-r " + normalizedPath, nil
}

pathAsUrl, err := url.Parse(path)
if err != nil {
return "", err
Expand Down
12 changes: 12 additions & 0 deletions bundle/config/mutator/normalize_paths_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,18 @@ func TestNormalizePath_url(t *testing.T) {
assert.Equal(t, "s3:///path/to/notebook.py", value)
}

func TestNormalizePath_requirementsFile(t *testing.T) {
tmpDir := t.TempDir()
location := dyn.Location{File: filepath.Join(tmpDir, "resources", "job_1.yml")}
value, err := normalizePath("-r ../requirements.txt", location, tmpDir)
assert.NoError(t, err)
assert.Equal(t, "-r requirements.txt", value)

value, err = normalizePath("-r ../requirements.txt", location, tmpDir)
assert.NoError(t, err)
assert.Equal(t, "-r requirements.txt", value)
}

func TestLocationDirectory(t *testing.T) {
loc := dyn.Location{File: "file", Line: 1, Column: 2}
dir, err := locationDirectory(loc)
Expand Down
Loading