Update the lakeflow-pipelines template according to the latest Lakeflow conventions by lennartkats-db · Pull Request #3558 · databricks/cli

lennartkats-db · 2025-09-08T07:36:31Z

This adds a new version of the default-python template. To enable the first integration and evaluation of this template, this first pull request replaces the internal lakeflow-pipelines template. Replacing default-python will be the next milestone.

Changes

Update default-python to follow the Lakeflow conventions (per-pipeline sources go into resources/; shared sources go into (lib/)
Adapt this template so it can first be used in place of the lakeflow-pipelines template (this required disabling some of the template questions and components)

Why

Lakeflow comes with new conventions that emphasize modularity. This PR represents an initial milestone towards updating the DABs default-python template. default-sql will follow in a later milestone.

Tests

Existing tests were updated.

…he lakeflow-pipelines template

libs/template/templates/lakeflow-pipelines/databricks_template_schema.json

libs/template/templates/lakeflow-pipelines/library/versions.tmpl

libs/template/templates/lakeflow-pipelines/library/variables.tmpl

...s/lakeflow-pipelines/template/{{.project_name}}/resources/sample_job/sample_job.job.yml.tmpl

libs/template/templates/lakeflow-pipelines/template/{{.project_name}}/tests/conftest.py

eng-dev-ecosystem-bot · 2025-09-08T07:50:46Z

Run: 17893981483

	Env	✅pass	🔄flaky	🙈skip
✅	aws linux	311		528
✅	aws windows	312		527
🔄	aws-ucws linux	414	9	426
🔄	aws-ucws windows	415	9	425
✅	azure linux	311		527
✅	azure windows	312		526
🔄	azure-ucws linux	419	4	425
🔄	azure-ucws windows	416	8	424
✅	gcp linux	310		529
✅	gcp windows	311		528

10 failing tests:

Test Name	aws-ucws linux	aws-ucws windows	azure-ucws linux	azure-ucws windows
TestAccept	✅pass	✅pass	🔄flaky	✅pass
TestAccept/bundle/deploy/dashboard/detect-change	🔄flaky	🔄flaky	✅pass	🔄flaky
TestAccept/bundle/deploy/dashboard/generate_inplace	🔄flaky	🔄flaky	✅pass	🔄flaky
TestAccept/bundle/deploy/dashboard/nested-folders	🔄flaky	🔄flaky	✅pass	🔄flaky
TestAccept/bundle/deploy/dashboard/simple	🔄flaky	🔄flaky	✅pass	🔄flaky
TestAccept/bundle/deploy/dashboard/simple_outside_bundle_root	🔄flaky	🔄flaky	✅pass	🔄flaky
TestAccept/bundle/deploy/dashboard/simple_syncroot	🔄flaky	🔄flaky	✅pass	🔄flaky
TestAccept/bundle/deployment/bind/dashboard	🔄flaky	🔄flaky	🔄flaky	🔄flaky
TestAccept/bundle/deployment/bind/dashboard/recreation	🔄flaky	🔄flaky	🔄flaky	✅pass
TestDashboardAssumptions_WorkspaceImport	🔄flaky	🔄flaky	🔄flaky	🔄flaky

...undle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/.vscode/settings.json

...t/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_etl.pipeline.yml

...output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_job.job.yml

...lines/resources/lakeflow_pipelines_etl/transformations/sample_trips_my_lakeflow_pipelines.py

libs/template/templates/lakeflow-pipelines/databricks_template_schema.json

libs/template/templates/lakeflow-pipelines/databricks_template_schema.vnext.json

pietern

I can't test this end-to-end yet because of the missing pyproject.toml.

libs/template/templates/lakeflow-pipelines/template/__preamble.tmpl

libs/template/templates/lakeflow-pipelines/databricks_template_schema.vnext.json

libs/template/templates/lakeflow-pipelines/databricks_template_schema.json

...undle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/.vscode/settings.json

...ines/python/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/utilities/utils.py

…on-v0

Though this is unfortunately inconsistent with the one from Black

...t/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_etl.pipeline.yml

fjakobs

I'm OK with the proposed changes but let's make sure to address @andersrexdb 's comment.

libs/template/templates/lakeflow-pipelines/databricks_template_schema.vnext.json

Add a Lakeflow-friendly default-python template, adapted to replace t…

181627d

…he lakeflow-pipelines template

lennartkats-db requested review from andrewnester, anton-107, denik, pietern and shreyas-goenka as code owners September 8, 2025 07:36

lennartkats-db temporarily deployed to test-trigger-is September 8, 2025 07:36 — with GitHub Actions Inactive

lennartkats-db changed the title ~~Add a Lakeflow-friendly default-python template~~ Add a Lakeflow-friendly DABs template Sep 8, 2025

lennartkats-db commented Sep 8, 2025

View reviewed changes

lennartkats-db requested a review from fjakobs September 8, 2025 09:31

Fix .gitignore issue

a7f1447

lennartkats-db temporarily deployed to test-trigger-is September 8, 2025 09:36 — with GitHub Actions Inactive

fjakobs reviewed Sep 8, 2025

View reviewed changes

pietern reviewed Sep 8, 2025

View reviewed changes

lennartkats-db added 2 commits September 14, 2025 15:44

Process reviewer feedback

a7ff123

Merge remote-tracking branch 'origin/main' into lakeflow-default-pyth…

ea97b0c

…on-v0

lennartkats-db temporarily deployed to test-trigger-is September 14, 2025 13:47 — with GitHub Actions Inactive

lennartkats-db requested review from fjakobs and pietern September 14, 2025 13:57

Apply repo formatting

d3acf61

Though this is unfortunately inconsistent with the one from Black

lennartkats-db temporarily deployed to test-trigger-is September 14, 2025 14:08 — with GitHub Actions Inactive

lennartkats-db mentioned this pull request Sep 15, 2025

Added new Lakeflow Pipelines support for bundle generate #3568

Merged

lennartkats-db added 2 commits September 15, 2025 14:10

REmove spurious newline

75bd17f

Formatting

d708592

lennartkats-db requested review from jefferycheng1 and kanterov as code owners September 15, 2025 12:36

lennartkats-db temporarily deployed to test-trigger-is September 15, 2025 12:36 — with GitHub Actions Inactive

lennartkats-db added 2 commits September 16, 2025 11:32

Minor fixes

fbb1617

Use pyspark.pipelines

e588d1f

lennartkats-db temporarily deployed to test-trigger-is September 16, 2025 11:26 — with GitHub Actions Inactive

lennartkats-db changed the title ~~Add a Lakeflow-friendly DABs template~~ Update the lakeflow-pipelines template according to the latest Lakeflow conventions Sep 16, 2025

andersrexdb reviewed Sep 19, 2025

View reviewed changes

...t/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_etl.pipeline.yml Show resolved Hide resolved

Include the right version of Python

bc9a90f

lennartkats-db temporarily deployed to test-trigger-is September 21, 2025 13:00 — with GitHub Actions Inactive

fjakobs approved these changes Sep 22, 2025

View reviewed changes

lennartkats-db commented Sep 22, 2025

View reviewed changes

libs/template/templates/lakeflow-pipelines/databricks_template_schema.vnext.json Show resolved Hide resolved

lennartkats-db added this pull request to the merge queue Sep 23, 2025

Merged via the queue into main with commit a6398a3 Sep 23, 2025
13 checks passed

lennartkats-db deleted the lakeflow-default-python-v0 branch September 23, 2025 09:11

Conversation

lennartkats-db commented Sep 8, 2025

Changes

Why

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fjakobs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

eng-dev-ecosystem-bot commented Sep 8, 2025 •

edited

Loading