Add top level run_as support for DLT pipelines

[shreyas-goenka]

Changes

This PR adds support for top-level run_as to DABs. Before DABs would error if pipelines were used with the run_as field set to a value.

Now, we'll transparently read the run_as value and set it for all pipelines in the bundle.

Tests

New acceptance tests. Also manually tested that the run_as feature works as expected.

[eng-dev-ecosystem-bot]

Run: 17406750859

	Env	✅​pass	🔄​flaky	🙈​skip
✅​	aws linux	308		512
✅​	aws windows	309		511
✅​	aws-ucws linux	420		410
✅​	aws-ucws windows	421		409
✅​	azure linux	308		511
✅​	azure windows	309		510
✅​	azure-ucws linux	420		409
✅​	azure-ucws windows	421		408
🔄​	gcp linux	304	3	513
✅​	gcp windows	308		512

Test Name	gcp linux
TestAccept	🔄​flaky
TestAccept/bundle/templates/default-python/integration_classic	🔄​flaky
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_CLI_DEPLOYMENT=direct-exp/UV_PYTHON=3.10	🔄​flaky

[shreyas-goenka]

Recording discussion from slack.

Pieter Noordhuis:
Since this is net-new, it seems good.
I have questions:

1. What happens if you change from no run_as to setting run_as , if a pipeline has already created tables? I recall a thread with the team saying they would perform a blocking "chown", which can be error prone.
2. What happens in the reverse, if you try to go from having a run_as to not having one?
3. Is there a path to get rid of the experimental flag and get existing users to use this?

Shreyas Goenka:

I recall a thread with the team saying they would perform a blocking "chown"

That's exactly what happens. The table's ownership is changed to the run_as identity. I only tried this once but I did not observe any errors during that run.

What happens in the reverse, if you try to go from having a run_as to not having one?

That is not allowed by the backend.

Is there a path to get rid of the experimental flag and get existing users to use this?

There should be options:

1. Transparently onboard legacy users. Legacy user pipelines have owners set to the run_as identity. Transparently onboarding them would mean changing the owner of the deployment identity and then setting run_as for the pipeline. Denys Kuznietsov agreed that this should be safe but I'd need to validate whether there are failure modes in this.
2. Make it explicit, have a migration guide for them. As long as we can decouple the tables' lifecycle from the DLT pipeline (DPM mode), we should be able to do this safely. This needs investigation.
   For me, it's also a question of prioritization. Right now, removing the legacy option does not seem high enough priority compared to the other things we have going on.