Respect retries when a deferrable trigger ends a task with TaskFailedEvent

[hkc-8010]

Summary

When a deferrable operator's trigger yields a terminal TaskFailedEvent, the task was always marked failed and its on_failure_callback ran, even when the task still had retries remaining. A worker-side failure with retries left instead goes up_for_retry and runs on_retry_callback. This makes trigger-driven failures behave the same way.

The BaseTaskEndEvent handler in airflow-core/src/airflow/models/trigger.py now, for a FAILED event, loads the serialized task and routes through TaskInstance.handle_failure(), the same path the scheduler executor-event handling has used since #56586. handle_failure sets up_for_retry or failed based on retry-eligibility and fires the on_task_instance_failed listener, failure metrics, and a Log entry. The callback request now carries the resolved task_callback_type so the DAG processor runs on_retry_callback vs on_failure_callback correctly. TaskSuccessEvent and TaskSkippedEvent are unchanged.

closes: #69819

Changes

• airflow-core/src/airflow/models/trigger.py: route trigger-emitted failures through handle_failure, thread the retry-aware task_callback_type into the callback, and skip the xcom push on the retry path.
• airflow-core/tests/unit/models/test_trigger.py: new parametrized test covering the retry-eligible case (up_for_retry + on_retry_callback + on_task_instance_failed listener) and the exhausted case (failed + on_failure_callback); the existing end-event test now sets a realistic start_date (a deferred task has already run).
• airflow-core/newsfragments/69821.bugfix.rst.

Testing

• pytest airflow-core/tests/unit/models/test_trigger.py (39 passed).
• Confirmed the new test fails on main (the task stays failed and the callback is routed as a failure) and passes with this change.

This change was written with AI assistance (Claude); the author reviewed the diff and reproduction.

[amoghrajesh]

Code looks ok, but I have a qn, seems to be spillover from past version of the PR

[github-actions]

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker dd1f84a v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.