Add "Secrets" to DAB Resources

[mniehoff]

Describe the issue

Support creating Secrets using the Asset Bundles under a given secret scope.

Use Case

We have autonomous teams that provide data products on our platform. Each team uses DAB for this and creates its own schema, volume for raw data and so on, using the DAB.
The team also has their own secrets they use to connect to data sources, that are not cloud storages (using volumes for this). It would be great if the team could manage the secrets completely in the asset bundles (secret values of course not stored in the yml files, but provided as an environment variable).

Databricks Terraform, CLI and Rest API already provide the functionality, would be nice to have it in yml/Bundles as well.

[johalnes]

Second this! Would love it to be available.

And @andrewnester, couldn't this also be a good temporary fix for #3679? db connection variable could be added to secret and then referenced from:secret in the apps?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[khushbookaul]

This feature would be super useful for teams using Asset bundles without the need to include terraform in their code base.

[josepmartinez-vista]

Currently, Databricks Asset Bundles (DABs) support the creation and permission management of Secret Scopes through the secret_scopes resource. However, there is no native way to manage the Secrets within those scopes directly inside the bundle configuration.
This creates a "configuration gap" where developers must use external shell scripts or manual CLI commands (databricks secrets put-secret) to populate the secrets after the bundle has deployed the scope.

The core requirements for this feature would be:

• Declarative Secret Management: The ability to define secret keys as part of the bundle resource.
• Secure Value Injection: Support for environment variable (e.g., ${env.SECRET_VALUE}) to ensure sensitive credentials are never hardcoded in the YAML files.
• Lifecycle Integration: Secrets should be created or updated during the databricks bundle deploy phase in CI/CD.

Proposed solution

We propose extending the existing secret_scopes resource to include a secrets mapping. This would allow users to define secret keys and their corresponding values (sourced from environment variables that are available in CI/CD) directly in databricks.yml or included resource files.

Example Configuration:

resources:
  secret_scopes:
    data_product_scope:
      name: ${var.scope_name}
      # Existing permission support
      permissions:
        - level: MANAGE
          service_principal_name: ${var.sp_application_id}
      
      # PROPOSED: New secrets attribute
      secrets:
        - key: api_client_id
          value: ${env.API_CLIENT_ID}
        - key: api_client_secret
          value: ${env.API_CLIENT_SECRET}

Implementation Details:

• Interpolation: The CLI should resolve ${env.VARIABLE} at deployment time.
• Security: If a value is provided as a plain string (not an environment variable), the CLI should ideally surface an error to enforce the use of environment variables for security best practices.

This addition would make DABs a truly complete solution for workspace initialization and application deployment, removing the need for auxiliary "bootstrap" scripts.

Currently we implemented this approach successfully. As we cannot add the secrets, we defined in another file and we have a shell script that read it and runs the databricks secrets put-secret.

[andrewnester]

Quick update: adding secrets as a separate resources is on our roadmap for the current 2 quarters. Currently we're exploring the details of implementation, in particularly focussed on security and credentials leaking to make sure secrets are not leaked in logs or other places when used

We'll keep this issue updated