Duplicate workflows being created with databricks bundle deploy

[NodeJSmith]

Describe the issue

Databricks bundle deploy sometimes generates a new workflow instead of updating the existing workflow. This seems to be inconsistent, so unfortunately I can't provide much more detail. I know that this happens if the person/service principal running the deploy command is different, which we account for, but sometimes it happens when the same service principal is running the same command.

This may be user error, but it is hard to determine since I don't know what method the Databricks CLI uses to determine if an existing workflow should be updated vs a new one created. For example, when migrating from DBX to DABs, the first deployment of a DAB would create a new workflow. A different user deploying a bundle creates a new workflow. Is is possible to provide more insight or detail into how this is determined?

Configuration

Please provide a minimal reproducible configuration for the issue

This is done in an azure devops pipeline, the basic configuration is

1. Install the databricks cli
   • curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
2. Set DATABRICKS_HOST and DATABRICKS_TOKEN environment variables
3. Download a Python package from the build pipeline
4. Create a databricks bundle with an internal script and run deploy

• create_deployment --environment $ENVIRONMENT && databricks bundle deploy

Steps to reproduce the behavior

The above steps plus

1. Download the Python package from the build pipeline
2. Create a databricks bundle with an internal script and run deploy

• create_deployment --environment $ENVIRONMENT && databricks bundle deploy

Expected Behavior

The existing Databricks workflow that has been previously created/updated with databricks bundle deploy is updated again with the new changes.

Actual Behavior

The existing Databricks workflow remains unchanged and a brand new workflow is created - this is intermittent, most of the time the existing workflow is updated

OS and CLI version

Please provide the version of the CLI (eg: v0.1.2) and the operating system (eg: windows). You can run databricks --version to get the version of your Databricks CLI

CLI version: Databricks CLI v0.209.1
OS: Ubuntu-22.04.3

[pietern]

Thanks for reporting this issue. This is definitely not expected.

Do you ever see error messages pop up during these deployments? I would expect every deployment to say "Resource deployment completed!" and then proceed with synchronizing the deployment state to the workspace file system.

Every bundle deployment has a stable path in the workspace file system (workspace.root_path in the bundle validate output) where it stores the deployment state. When you run a deploy, it first pulls the existing state from that path (if it exists), the performs a deploy, and then replaces the existing state with the new state. Since you see duplicate jobs from the same bundle deployment, something must be off in this flow.

This stable path defaults to /Users/<username>/.bundle/<bundle name>/<bundle target>. If any of these variables change, you end up with a new deployment, but if they're all constant, it should update an existing deployment.

Could you instrument your automation to capture debug logs during these deploys? You can set DATABRICKS_LOG_FILE and DATABRICKS_LOG_LEVEL=debug to capture a log file. Then when this happens again, I'd love to see those logs for more information. If the log contains confidential information, you can send it over to dabs-preview@databricks.com and we can take it from there. Thanks in advance!

[NodeJSmith]

@pietern I think this is due to my configuration - I had the root path set to /Shared/dab/dbx_pipeline_legacy_dev instead of anything with dynamic values. I will update my configuration and I think we can close this. I will open a new issue in the unlikely event that something is actually broken in this process.

Question - if the suggested default is to use /Shared/.bundle/${bundle.name}/${bundle.target}, why can I not just leave the workspace root path blank and allow that to be used? When I leave it blank I get this unclear error message, which I believe is what led me to setting the value in the first place.

Error: root_path must not contain the current username when using 'mode: production'
  tip: set workspace.root_path to a shared path such as /Shared/.bundle/${bundle.name}/${bundle.target}

[lennartkats-db]

@NodeJSmith

I had the root path set to /Shared/dab/dbx_pipeline_legacy_dev instead of anything with dynamic values. I will update my configuration and I think we can close this. I will open a new issue in the unlikely event that something is actually broken in this process.

That should be fine. You don't really need to use dynamic values. Maybe one thing you could check is whether the duplicate jobs that were deployed all point to the same path? Do they all point to /Shared/dab/dbx_pipeline_legacy_dev or are different paths referenced?

[NodeJSmith]

Fair enough, I see your point. Yes, I can confirm they all point to the same path, we have a library that builds the DAB yaml for us and the path is set based on a single parameter, so it is always /Shared/dab/<package_name>

[andrewnester]

The fix was implemented and released in 0.210.0 version of CLI
#1008