Airflow 3.2: fix Slack failure callbacks (companion to dataeng-airflow #250)#70
Draft
mrhallak wants to merge 1 commit into
Draft
Airflow 3.2: fix Slack failure callbacks (companion to dataeng-airflow #250)#70mrhallak wants to merge 1 commit into
mrhallak wants to merge 1 commit into
Conversation
Two callback paths were broken on the live Airflow 3.2.1 deploy:
1. SlackWebhookOperator signature changed in providers-slack 9.x.
- `webhook_token` and `http_conn_id` kwargs are gone.
- Replaced with `slack_webhook_conn_id`.
- The webhook URL now lives in the connection's password field;
the operator's hook reads it. The explicit
`BaseHook.get_connection(...).password` lookup is removed.
2. DagRun.external_trigger was removed in Airflow 3.
- Replaced with `dag_run.run_type == DagRunType.MANUAL`, which
preserves the original intent ("don't alert for manually
triggered runs").
Also: `context["dag"].is_paused` is not exposed on the SDK DAG object
in Airflow 3 callback context. Guarded with `getattr` so the check
no-ops cleanly if the attribute is missing.
Why this was invisible until now: imports still succeed (the operator
class still exists, just its `__init__` signature differs), so DAG
parsing is fine. The callback only runs on task failure, at which
point it raises TypeError, the original task is already failed, and
the only sign is a callback error in scheduler/dag-processor logs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Companion to dataeng-airflow#250 (Airflow 3.2.1 retry). Fixes the runtime breakages in dagger's Slack callback paths that surfaced after the upgrade.
The earlier Airflow 3.x prep (PR #69 — imports migration +
schedule_interval→schedule) is already on daggermasterand pinned in dataeng-airflow. This PR closes the remaining gaps.What broke
1.
SlackWebhookOperatorsignature change (apache-airflow-providers-slack==9.10.0, live container verified):webhook_tokenandhttp_conn_idkwargs are gone.slack_webhook_conn_idis now required.passwordfield directly, so the explicitBaseHook.get_connection(...).passwordlookup is removed.2.
DagRun.external_triggerremoved in Airflow 3. The live container reports only['run_type', 'triggered_by', 'triggering_user_name']. Userun_type == DagRunType.MANUALto preserve the original intent.3.
context["dag"].is_pausednot exposed on the SDK DAG class in Airflow 3 callback context. Guarded withgetattr(..., False)so the check no-ops cleanly when the attribute is missing.Why this was invisible until now
Imports still succeed (
SlackWebhookOperatorclass exists, just its__init__differs), so DAG parsing passes. The callback only runs on task failure, at which point it raisesTypeError; the original task is already in failed state, and the only signal is a callback error buried in scheduler/dag-processor logs. We hit it during the 3.2.1 redeploy validation on datatst.Files
dagger/dag_creator/airflow/utils/slack_alerts.py— bothtask_success_slack_alertandtask_fail_slack_alert. Extracted the three skip conditions into_should_skip_alertto deduplicate.dagger/alerts/alert.py—airflow_task_fail_alertshad the sameexternal_trigger/is_pausedchecks; fixed identically.Connection setup (operational note)
For the operator to send messages, the
slackconnection in Airflow must have the webhook URL stored in the password field (no other fields needed). The existingdataeng-airflow-slack_tokensecret loads into theSLACK_TOKENenv var, which docker-entrypoint converts into a connection — verify that conversion still puts the value inpassword.Test plan
origin/master(which already has the Airflow 3 import migration merged)pytest)docker/requirements.txtand redeploy datatstENV == "datatst"early-return, or testing on datastg) and confirm a Slack message lands in the alerts channel