Skip to content

New pipelines init command#3114

Merged
alyssa-db merged 20 commits intomainfrom
pipelines-init
Jul 1, 2025
Merged

New pipelines init command#3114
alyssa-db merged 20 commits intomainfrom
pipelines-init

Conversation

@alyssa-db
Copy link
Copy Markdown
Contributor

@alyssa-db alyssa-db commented Jun 24, 2025

Changes

pipelines init new command, with new cli-pipelines template based off lakeflow-pipelines template

While most files are kept the same, differences between the model lakeflow-pipelines and cli-pipelines templates are viewable with diff -r libs/template/templates/cli-pipelines libs/template/templates/lakeflow-pipelines.
Differences

  • removing {{.project_name}}.job.yml.tmpl file (only for jobs, which are removing, not pipelines)
  • removing bundle mentions
  • changed directories {{.project_name}}/resources/{{.project_name}}_pipeline/ to be just {{.project_name}}/{{.project_name}}_pipeline/
  • renamed project folder from my_lakeflow_project to be my_project by default
  • removed previous formatting that caused issues with the linter

Added root package to pipelines, copying over functionality from cmd/root to also apply to new pipelines CLI
Modified linter to exclude template files

Why

pipelines CLI needs a way to initialize specifically templates for Databricks pipelines

Tests

Added acceptance testing for pipelines init

3 folders, each with an output folder showing a sample output template from the script with various input.json

  • python
  • sql
  • error-cases

@alyssa-db
Copy link
Copy Markdown
Contributor Author

@jefferycheng1

@alyssa-db alyssa-db temporarily deployed to test-trigger-is June 24, 2025 21:13 — with GitHub Actions Inactive
@alyssa-db alyssa-db temporarily deployed to test-trigger-is June 24, 2025 22:59 — with GitHub Actions Inactive
@alyssa-db alyssa-db temporarily deployed to test-trigger-is June 25, 2025 09:04 — with GitHub Actions Inactive
@alyssa-db alyssa-db temporarily deployed to test-trigger-is June 30, 2025 17:24 — with GitHub Actions Inactive
@alyssa-db alyssa-db temporarily deployed to test-trigger-is June 30, 2025 19:04 — with GitHub Actions Inactive
@alyssa-db alyssa-db requested a review from pietern July 1, 2025 09:12
@alyssa-db alyssa-db temporarily deployed to test-trigger-is July 1, 2025 09:13 — with GitHub Actions Inactive
Copy link
Copy Markdown
Collaborator

@jefferycheng1 jefferycheng1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alyssa-db since we don't have a good way of keeping these helper methods in sync for the time being, let's make sure we leave a comment for them noting this down, otherwise lgtm!

@alyssa-db alyssa-db temporarily deployed to test-trigger-is July 1, 2025 10:46 — with GitHub Actions Inactive
@alyssa-db alyssa-db requested a review from kanterov July 1, 2025 12:44
@alyssa-db alyssa-db temporarily deployed to test-trigger-is July 1, 2025 14:23 — with GitHub Actions Inactive
@alyssa-db alyssa-db requested a review from pietern July 1, 2025 14:30
@alyssa-db alyssa-db requested a review from lennartkats-db as a code owner July 1, 2025 17:23
@alyssa-db alyssa-db temporarily deployed to test-trigger-is July 1, 2025 17:23 — with GitHub Actions Inactive
@alyssa-db alyssa-db enabled auto-merge July 1, 2025 18:21
@alyssa-db alyssa-db temporarily deployed to test-trigger-is July 1, 2025 19:56 — with GitHub Actions Inactive
@alyssa-db alyssa-db added this pull request to the merge queue Jul 1, 2025
Merged via the queue into main with commit 7c9ba16 Jul 1, 2025
13 checks passed
@alyssa-db alyssa-db deleted the pipelines-init branch July 1, 2025 20:25
denik pushed a commit that referenced this pull request Jan 21, 2026
## Changes

This PR removes the unused standalone pipelines CLI entrypoint
infrastructure. The ability to invoke the CLI binary as `pipelines`
(instead of `databricks`) was never released and is no longer needed.
All pipelines functionality remains available through `databricks
pipelines` commands.

PRs that added this:
- #3009
- #3114
- #3170
- #3177
- #3221

## Tests

Updated acceptance tests.

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants