Asset bundle run_job_task fails

[meretri]

I get an error when trying to deploy a workflow job wich contains a "Run Job" Task:

databricks.yml

 # yaml-language-server: $schema=bundle-settings-schema.json 
bundle:
  name: run_job_example
targets:
  development:
    workspace:
      host: https://xxx.azuredatabricks.net
      profile: xyz
resources:
  jobs:
    job1:
      name: job1
      tasks:       
      - task_key: STEP_1
        run_job_task:
          job_id: 12345

The validation step is successful, but when trying to deploy, I get the following error:

Starting resource deployment
Error: terraform apply: exit status 1

Error: cannot create job: Job 0 does not exist.

  with databricks_job.job1,
  on bundle.tf.json line 25, in resource.databricks_job.job1:
  25:       }

The job ID is missing in the bundle.tf.json:

{
  "terraform": {
    "required_providers": {
      "databricks": {
        "source": "databricks/databricks",
        "version": "1.23.0"
      }
    }
  },
  "provider": {
    "databricks": {}
  },
  "resource": {
    "databricks_job": {
      "job1": {
        "name": "job1",
        "task": [
          {
            "task_key": "STEP_1",
            "run_job_task": {
              "job_id": ""
            }
          }
        ]
      }
    }
  }
}

I assume the reason for this is, that terraform expects a string as job ID and not a int. But quoting the ID results in a different error:

Error: failed to load databricks.yml: error unmarshaling JSON: json: cannot unmarshal string into Go struct field RunJobTask.resources.jobs.tasks.run_job_task.job_id of type int

[ckelly]

same here - we're hoping to ideally:

• reference another bundle/ job by name as a job_id here (via resources.jobs... or some variation)
• reference by a defined job id (this seems to be how we more or less can do it via the GUI currently)
• define the job in the same bundle and reference by resources.jobs...

[andrewnester]

Thank for reporting the issue, We're aware of the problem and are working on a fix. We will keep this thread updated

[kyleries]

We are in the exact same position, wanted to drop a note here in case you needed anyone else to test the solution. Happy to run a specific branch/commit if you'd like!

[timreddick-8451]

Same issue. Posted a description of the issue on the Databricks Community forum.

https://community.databricks.com/t5/data-engineering/using-run-job-task-in-databricks-asset-bundles/m-p/44807/highlight/true#M27716

[kyleries]

Thanks, Tim - I just posted this issue to the community forum and hadn't got back here yet!

[andrewnester]

We're working on a fix for the issue and will keep this thread updated. Unfortunately, it's not as simple as a mere type change, due to where the type information for the bundle schema comes from. We expect this to take ~2 weeks. We'll keep you posted.

[BMeyn]

@andrewnester, any updates on this?