Added secrets resource support to DABs

[varundeepsaini]

Changes

Add secrets and secret_scopes as bundle resources

resources:
  secret_scopes:
    my_scope:
      name: my-secret-scope
      permissions:
        - user_name: user@example.com
          level: READ

  secrets:
    api_key:
      scope: my-secret-scope
      key: api-key
      string_value: ${var.api_key_value}

Why

Closes #3689

Users need to manage secrets alongside their jobs/pipelines in bundles rather than creating them manually.

Tests

• Unit tests for resource definitions and Terraform converters
• Acceptance tests for both Terraform and direct deployment modes
• Integration tests for deploy/update/destroy workflows
• Manual verification against the actual Databricks workspace

[github-actions]

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 4034
• Commit SHA: ef49ed716752ed92d88ac110103cb823a68d4a2e

Checks will be approved automatically on success.

[varundeepsaini]

hey @andrewnester
can you please review this pr

[denik]

should just return nil there, the framework will call DoRead() if it's needed. Same for DoCreate().

[denik]

Pr in progress for secret scopes: #3886

[varundeepsaini]

Oh, didnt realise it is being worked on, better that I don't work on this feature

do you guys have anything else that i can pick up that you are not actively working on

[johalnes]

@varundeepsaini and @denik - Was this closed to soon? Ref #3689 - I would really love to be able to put a secret within a secret scope, as mentioned in the description of this PR. Can't see that #3886 solves this.

[shreyas-goenka]

@johalnes Can you please consider creating a new issue for adding support for secrets. We are unlikely to add it anytime soon because secret values are sensitive and need to be modelled appropriately. But a trackable issue can help us guage interest and then appropriately prioritize this.

[johalnes]

@johalnes Can you please consider creating a new issue for adding support for secrets. We are unlikely to add it anytime soon because secret values are sensitive and need to be modelled appropriately. But a trackable issue can help us guage interest and then appropriately prioritize this.

@shreyas-goenka Isn't the issue I mentioned above enough? That is #3689 ?