Docs: clarify Databricks retry_delay uses exponential backoff

[sduvvuri26]

Summary

Clarifies databricks_retry_delay / retry_delay documentation for Databricks operators, hooks, and triggers that use BaseDatabricksHook.

The previous docs described a fixed number of seconds between retries. The default implementation uses exponential backoff with retry_delay as the minimum wait (wait_exponential in BaseDatabricksHook).

Note: databricks_retry_args can override the default wait strategy; this documents the default behavior only.

Test plan

• Doc-only change (docstrings + repos operator RST tables)
• Verified retry behavior manually with DatabricksSubmitRunOperator and a broken connection (retry_delay=1 and retry_delay=20)

Manual verification

Observed retry timing with DatabricksSubmitRunOperator, broken connection, databricks_retry_limit=10:

retry_delay	Observed waits between attempts
1	1s, 2s, 4s, 8s, 16s, 32s, …
20	20s × 5, then 32s, 64s, 128s, …

This matches wait_exponential(min=retry_delay, max=2**retry_limit) in BaseDatabricksHook.

---

Was generative AI tooling used to co-author this PR?

• Yes — Cursor

Generated-by: Cursor following the guidelines

---

Important

🛠️ Maintainer triage note for @sduvvuri26 · by @potiuk · 2026-06-17 18:53 UTC

Some review feedback from jroachgolf84 is waiting on you:

• Please respond to the open review thread(s) — reply or push a fix in each, then mark them resolved.
• See the Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[SameerMesiah97]

Just did a quick search in the databricks provider and it appears you have covered all relevant docstring entries. I would check once more before merge just in case any were missed.

[sduvvuri26]

Just did a quick search in the databricks provider and it appears you have covered all relevant docstring entries. I would check once more before merge just in case any were missed.

@SameerMesiah97 Thanks for the review. Re-searched the databricks provider for the old retry_delay wording ("Number of seconds to wait between retries") and found no remaining entries. All relevant docstrings/RST tables appear covered.

[jroachgolf84]

Re-reviewed all files, LGTM.

[potiuk]

@sduvvuri26 — you've pushed commits since the changes-requested review, but the review threads are still open. If you've addressed the feedback, reply to confirm and nudge the reviewer for another look; otherwise let us know what's still outstanding. Thanks!

See the PR quality criteria.

_{Automated first-pass triage note drafted by an AI-assisted tool — may get things wrong; once addressed, a real Apache Airflow maintainer takes the next look. (why automated)}

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[sduvvuri26]

@potiuk, I’ve addressed the feedback, resolved the review threads, and re-requested review. @jroachgolf84 has reviewed and approved the PR on Jun 20.
Please let me know if anything else is needed from my side.

[eladkal]

cc @moomindani for Databricks team review

[moomindani]

LGTM. Verified the claim rather than just the wording.

BaseDatabricksHook builds its default retry with wait_exponential(min=self.retry_delay, max=(2 ** retry_limit)), so the old "number of seconds between retries" was simply wrong. I reproduced your table against tenacity directly and it matches, including the counter-intuitive retry_delay=20 shape (20, 20, 20, 20, 20, 32, 64, 128 — the 2**(n-1) series clipped at the floor until it overtakes it) and the 2 ** retry_limit cap.

On completeness, which was the open question earlier: current main has 17 occurrences of the old phrasing across 8 files, and this PR updates all 8 with matching per-file counts. Nothing left behind — the remaining databricks_retry_delay: decimal lines are the type column, not stale prose.

Worth a rebase before merge; the branch is ~1.5 months behind main, though I confirmed no new retry_delay docstrings landed in the meantime, so nothing is newly stale.

---

Drafted-by: Claude Code (Opus 5)

[potiuk]

Docs and docstrings only, and the claim matches the implementation — databricks_base.py builds wait_exponential(min=self.retry_delay, max=2 ** retry_limit), so "minimum wait, exponential backoff, capped at 2 ** retry_limit" is accurate. Thanks.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.