Skip to content

Add pydabs template #3806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kanterov merged 1 commit into from
Oct 23, 2025
Merged

Add pydabs template #3806

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions acceptance/bundle/templates/pydabs/check-consistency/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
== Collecting resources into ./enable_pydabs=yes

>>> [CLI] bundle init default-python --config-file ./input.json
>>> [CLI] bundle init pydabs --config-file ./input.json

Welcome to the default Python template for Databricks Asset Bundles!
Workspace to use (auto-detected, edit in 'my_pydabs/databricks.yml'): [DATABRICKS_URL]
Expand All @@ -17,7 +17,7 @@ See also the documentation at https://docs.databricks.com/dev-tools/bundles/inde
>>> [CLI] bundle validate --output json
== Collecting resources into ./enable_pydabs=no

>>> [CLI] bundle init default-python --config-file ./input.json
>>> [CLI] bundle init pydabs --config-file ./input.json

Welcome to the default Python template for Databricks Asset Bundles!
Workspace to use (auto-detected, edit in 'my_pydabs/databricks.yml'): [DATABRICKS_URL]
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/check-consistency/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ for enable_pydabs in yes no; do

enable_pydabs=$enable_pydabs envsubst < input.json.tmpl > input.json

trace $CLI bundle init default-python --config-file ./input.json
trace $CLI bundle init pydabs --config-file ./input.json

(
cd my_pydabs
Expand Down
1 change: 0 additions & 1 deletion acceptance/bundle/templates/pydabs/check-formatting/input.json.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,5 @@
"include_notebook": "$INCLUDE_NOTEBOOK",
"include_dlt": "$INCLUDE_DLT",
"include_python": "$INCLUDE_PYTHON",
"enable_pydabs": "yes",
"serverless": "$SERVERLESS"
}
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/check-formatting/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

>>> [CLI] bundle init default-python --config-file ./input.json
>>> [CLI] bundle init pydabs --config-file ./input.json

Welcome to the default Python template for Databricks Asset Bundles!
Workspace to use (auto-detected, edit in 'my_pydabs/databricks.yml'): [DATABRICKS_URL]
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/check-formatting/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
envsubst < input.json.tmpl > input.json

trace $CLI bundle init default-python --config-file ./input.json
trace $CLI bundle init pydabs --config-file ./input.json

# only keep relevant files for snapshots
trace find my_pydabs -mindepth 1 ! -name 'pyproject.toml' ! -regex '.*/resources.*' -delete
Expand Down
3 changes: 1 addition & 2 deletions acceptance/bundle/templates/pydabs/deploy-classic/input.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,5 @@
"include_notebook": "yes",
"include_dlt": "yes",
"include_python": "yes",
"serverless": "no",
"enable_pydabs": "yes"
"serverless": "no"
}
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/deploy-classic/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

>>> [CLI] bundle init default-python --config-file ./input.json
>>> [CLI] bundle init pydabs --config-file ./input.json

Welcome to the default Python template for Databricks Asset Bundles!
Workspace to use (auto-detected, edit in 'my_pydabs/databricks.yml'): [DATABRICKS_URL]
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/deploy-classic/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
trace $CLI bundle init default-python --config-file ./input.json
trace $CLI bundle init pydabs --config-file ./input.json

(
cd my_pydabs
Expand Down
3 changes: 1 addition & 2 deletions acceptance/bundle/templates/pydabs/init-classic/input.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,5 @@
"include_notebook": "yes",
"include_dlt": "yes",
"include_python": "yes",
"serverless": "no",
"enable_pydabs": "yes"
"serverless": "no"
}
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/init-classic/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

>>> [CLI] bundle init default-python --config-file ./input.json --output-dir output
>>> [CLI] bundle init pydabs --config-file ./input.json --output-dir output

Welcome to the default Python template for Databricks Asset Bundles!
Workspace to use (auto-detected, edit in 'my_pydabs/databricks.yml'): [DATABRICKS_URL]
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/init-classic/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
trace $CLI bundle init default-python --config-file ./input.json --output-dir output
trace $CLI bundle init pydabs --config-file ./input.json --output-dir output

# only keep relevant files for snapshots
trace find output/my_pydabs -mindepth 1 ! -name 'pyproject.toml' ! -regex '.*/resources.*' -delete
27 changes: 27 additions & 0 deletions libs/template/reader.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,33 @@ func (r *builtinReader) LoadSchemaAndTemplateFS(ctx context.Context) (*jsonschem

func (r *builtinReader) Cleanup(ctx context.Context) {}

// overridingReader wraps another Reader and overrides their properties
type overridingReader struct {
underlying Reader
propertyDefaultOverrides map[string]any
}

func (r *overridingReader) LoadSchemaAndTemplateFS(ctx context.Context) (*jsonschema.Schema, fs.FS, error) {
schema, schemaFS, err := r.underlying.LoadSchemaAndTemplateFS(ctx)
if err != nil {
return nil, nil, err
}

for propName, newDefault := range r.propertyDefaultOverrides {
if propSchema, ok := schema.Properties[propName]; ok {
propSchema.Default = newDefault
} else {
return nil, nil, fmt.Errorf("cannot override default for non-existing property %s", propName)
}
}

return schema, schemaFS, nil
}

func (r *overridingReader) Cleanup(ctx context.Context) {
r.underlying.Cleanup(ctx)
}

Copy link
Copy Markdown
Contributor

@pietern pietern Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this usable for other templates?

@lennartkats-db added something to reuse templates from another directory and pre-populate values from the template inputs definition. Seems related.

Copy link
Copy Markdown
Collaborator Author

@kanterov kanterov Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's consolidate the approach with Lennart once he is available

// gitReader reads a template from a git repository.
type gitReader struct {
gitUrl string
Expand Down
38 changes: 38 additions & 0 deletions libs/template/reader_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import (
"path/filepath"
"testing"

"github.com/databricks/cli/libs/jsonschema"

"github.com/databricks/cli/internal/testutil"
"github.com/databricks/cli/libs/cmdio"
"github.com/stretchr/testify/assert"
Expand Down Expand Up @@ -215,3 +217,39 @@ func TestGitReaderWithTemplateDir(t *testing.T) {
// Cleanup
r.Cleanup(ctx)
}

func TestOverridingReader_LoadSchemaAndTemplateFS(t *testing.T) {
tmpDir := t.TempDir()
content := `{
"welcome_message": "test",
"properties": {
"enable_feature_x": {
"type": "string",
"default": "no"
}
}
}`

testutil.WriteFile(t, filepath.Join(tmpDir, "databricks_template_schema.json"), content)
ctx := context.Background()

r := overridingReader{
underlying: &localReader{path: tmpDir},
propertyDefaultOverrides: map[string]any{
"enable_feature_x": "yes",
},
}

schema, fsys, err := r.LoadSchemaAndTemplateFS(ctx)

require.NoError(t, err)
require.NotNil(t, schema)
require.NotNil(t, fsys)

assert.Equal(t, map[string]*jsonschema.Schema{
"enable_feature_x": {
Type: "string",
Default: "yes",
},
}, schema.Properties)
}
18 changes: 13 additions & 5 deletions libs/template/template.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ const (
CLIPipelines TemplateName = "cli-pipelines"
DbtSql TemplateName = "dbt-sql"
MlopsStacks TemplateName = "mlops-stacks"
DefaultPydabs TemplateName = "default-pydabs"
Pydabs TemplateName = "pydabs"
Custom TemplateName = "custom"
ExperimentalJobsAsCode TemplateName = "experimental-jobs-as-code"
Default TemplateName = "default"
Expand Down Expand Up @@ -85,11 +85,19 @@ var databricksTemplates = []Template{
Writer: &writerWithFullTelemetry{defaultWriter: defaultWriter{name: MlopsStacks}},
},
{
name: DefaultPydabs,
name: Pydabs,
hidden: true,
description: "The default PyDABs template",
Reader: &gitReader{gitUrl: "https://databricks.github.io/workflows-authoring-toolkit/pydabs-template.git", cloneFunc: git.Clone},
Writer: &writerWithFullTelemetry{defaultWriter: defaultWriter{name: DefaultPydabs}},
description: "A variant of 'default-python' template that defines resources in Python instead of YAML",
Reader: &overridingReader{
underlying: &builtinReader{name: string(DefaultPython)},
propertyDefaultOverrides: map[string]any{
// In 'default-python' template 'enable_pydabs' prompt is hidden with default of 'no'.
// We want a separate template called 'pydabs' to reuse 'default-python' but have
// the 'enable_pydabs' property enabled by default that switches from YAML to Python.
"enable_pydabs": "yes",
},
},
Writer: &writerWithFullTelemetry{defaultWriter: defaultWriter{name: Pydabs}},
},
{
name: ExperimentalJobsAsCode,
Expand Down
2 changes: 1 addition & 1 deletion libs/template/template_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ func TestTemplateGetDatabricksTemplate(t *testing.T) {
DefaultSql,
DbtSql,
MlopsStacks,
DefaultPydabs,
Pydabs,
}

for _, name := range names {
Expand Down
Loading