DAB deploy different resources to different targets

[iam-meghana]

Describe the issue

We have defined resources in individual yml files in the bundle that has all the jobs in one folder. However lets say we have two targets a and b, and want to include only resources that end with *_a.yml for target a and *_b.yml for target b plus any common resources, there is no way to include/exclude only relevant resources when deploying the targets.

└── some_sample_proj/
└── bundle_name/
|── resources/
│ ├── jobs/
│ │ ├── job_a.yml
│ │ ├── job_b.yml
│ │ └── job_common.yml
│ ├── pipelines/
│ │ ├── pipeline_a.yml
│ │ └── pipeline_b.yml
│ └── clusters/
│ └── shared_cluster.yml
├── databricks.yml
└── ...

In the databricks.yml file, it would be great to have some way of referencing resources in the targets to be included or excluded (similar to sync include/exclude) that also supports wildcards, like:

bundle:
   name: bundle_name

include:
  - resources/jobs/*.yml

targets:
  a:
     ....
     resources: 
        exclude: 
            - resources/jobs/*_b.yml

 b:
     ....
     resources: 
        exclude: 
            - resources/jobs/*_a.yml

[andrewnester]

there is no way to include/exclude only relevant resources when deploying the targets.

If you define resources in file _a.yml under targets a sections like

targets:
  a:
    resources:
      ...

They will only be deployed to target a and not target b.

[iam-meghana]

@andrewnester Thanks for getting back.

Yes I understand we can define resources under a and b target sections.. The issue is it gets unreadable very fast defining resources under targets. In our project the targets share about 50 something jobs and defining them repeatedly under both a and b targets is neither good for readability or code reuse. So we want to write the actual definition in seperate files that we then want to include/exclude into the targets. This would be a good feature enhacement imo.

The capability of being able to write resource definitions in seperate yml files and including them in databricks.yml file already exists. We just want to be able to apply this to individual targets rather than all targets.

[andrewnester]

In our project the targets share about 50 something jobs and defining them repeatedly under both a and b targets is neither good for readability or code reuse

If there jobs are shared between targets you can put them outside of the target block. If you have jobs which only needs to be deployed in a certain target you put them under this target. You can separate this configuration by separate files anyway you want.
You can have something like this for example

databricks.yml

bundle:
  name: test

include:
  - common.yml
  - target_a.yml
  - target_b.yml

targets:
  a:
   ... some target config
  b:
   ... some target config

common.yml

resources:
  jobs:
   job1:
   ...
   job2:
   ...

target_a.yml

targets:
  a:
    resources:
      jobs:
       job100
       ...

[pietern]

@iam-meghana If the two sets of jobs are logically independent, could you split the definition into 2 separate bundles?

You can still include common resources with includes with relative paths (e.g. ../common/common_job.yml) and include the same source code through the sync.paths setting.