Skip to content

Update airflow config update to show breaking config changes by default#49223

Merged
uranusjr merged 6 commits into
apache:mainfrom
astronomer:breaking-changes-airflow-config-update
Apr 15, 2025
Merged

Update airflow config update to show breaking config changes by default#49223
uranusjr merged 6 commits into
apache:mainfrom
astronomer:breaking-changes-airflow-config-update

Conversation

@sunank200

@sunank200 sunank200 commented Apr 14, 2025

Copy link
Copy Markdown
Collaborator

airflow.cfg in Airflow 2. x usually has 2000+ lines of configs. Users typically want to change only the configurations causing issues in airflow migrations rather than dealing with a long list of config changes they may not fully understand.

This PR refines the airflow config update to better serve users upgrading to Airflow 3.0. Key changes include:

  • Dry-Run by default (Breaking Changes Only):
    The airflow config update command now defaults to a dry run (same as --dry-run) and displays only the breaking configuration changes. This avoids overwhelming users with recommended but non-critical changes. The breaking changes here are the configs causing errors/ warnings / deprecations.

  • New CLI Arguments:

    • --fix: Applies the breaking changes instead of a dry run.
    • --all-recommendations: When used with either --dry-run or --fix, it includes all recommended changes (i.e., non-breaking ones) in addition to breaking ones.

CLI Command Argument:

  • Dry-run mode (showing only breaking changes):
    airflow config update
    

I tested it default airflow.cfg from Airflow 2.10.5:
Screenshot 2025-04-14 at 8 42 12 PM

  • Dry-run mode including all recommendations (breaking + non-breaking):
    airflow config update --all-recommendations
    

I tested it default airflow.cfg from Airflow 2.10.5:
Screenshot 2025-04-14 at 8 42 41 PM

  • Apply only the breaking changes:
    airflow config update --fix
    

I tested it default airflow.cfg from Airflow 2.10.5:
Screenshot 2025-04-14 at 8 42 54 PM

  • Apply all recommended changes (both breaking and non-breaking):

    airflow config update --fix --all-recommendations
    
  • Update only specific sections (e.g. core and database):

    airflow config update --section core,database
    
  • Update only specific options (e.g. sql_alchemy_conn and dag_concurrency):

    airflow config update --option sql_alchemy_conn,dag_concurrency
    
  • Ignore a specific section (e.g. webserver):

    airflow config update --ignore-section webserver
    
  • Ignore specific options (e.g. smtp_user and session_lifetime_days):

    airflow config update --ignore-option smtp_user,session_lifetime_days
    

Breaking config changes


Section Option Change Type Old Value / Condition New Value / New Config Name
core executor Default Value Change SequentialExecutor LocalExecutor
logging log_filename_template Removal {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log The actual replacement value will be updated after defaults are loaded from config.yml
logging log_filename_template Removal dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log The actual replacement value will be updated after defaults are loaded from config.yml
webserver web_server_host Rename web_server_host host (in [api])
webserver web_server_port Rename web_server_port port (in [api])
webserver workers Rename workers workers (in [api])
webserver web_server_ssl_cert Rename web_server_ssl_cert ssl_cert (in [api])
webserver web_server_ssl_key Rename web_server_ssl_key ssl_key (in [api])
webserver access_logfile Rename access_logfile access_logfile (in [api])
scheduler catchup_by_default Default Value Change True False
scheduler dag_dir_list_interval Rename dag_dir_list_interval refresh_interval (in [dag_processor])
triggerer default_capacity Rename default_capacity capacity (in [triggerer])
elasticsearch log_id_template Removal {dag_id}-{task_id}-{logical_date}-{try_number} Removed

closes: #49202


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch 6 times, most recently from dc93d38 to 29968a7 Compare April 14, 2025 15:00
@sunank200 sunank200 marked this pull request as ready for review April 14, 2025 15:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the Airflow configuration update command to show only breaking configuration changes by default in dry-run mode and introduces new CLI arguments to optionally apply and include all recommendations.

  • Change default behavior of airflow config update to display only breaking changes in dry-run mode.
  • Introduce new CLI arguments (--fix and --all-recommendations) to control application of configuration changes.
  • Update tests and documentation strings to reflect the revised behavior.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
airflow-core/tests/unit/cli/commands/test_config_command.py Updates tests to check for dry-run messages and configuration backup, but uses an incorrect CLI flag.
airflow-core/src/airflow/cli/commands/config_command.py Refactors configuration change message formatting and enhances the update_config command to include new flags.
airflow-core/src/airflow/cli/cli_config.py Adds definitions for new CLI parameters (--fix and --all-recommendations) for the config update command.

Comment thread airflow-core/tests/unit/cli/commands/test_config_command.py Outdated
@sunank200 sunank200 changed the title Update airflow config lint to show breaking changes by default Update airflow config lint to show breaking config changes by default Apr 14, 2025
@sunank200 sunank200 requested a review from Copilot April 14, 2025 15:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/cli/commands/config_command.py:837

  • The flag '--all-recommendation' is inconsistent with the '--all-recommendations' defined in cli_config.py. Consider standardizing the flag name to avoid confusion.
        --all-recommendation: flag (optional)

@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch from 04a1a07 to 1e1cda6 Compare April 14, 2025 15:20
@sunank200 sunank200 requested a review from Copilot April 14, 2025 15:21
@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch from 1e1cda6 to ee12908 Compare April 14, 2025 15:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/cli/commands/config_command.py:983

  • [nitpick] Consider revising the message 'Dry-run is mode enabled.' to 'Dry-run mode enabled.' to improve clarity.
console.print("[blue]Dry-run is mode enabled. To apply above airflow.cfg run the command with `--fix`.[/blue]")

Comment thread airflow-core/src/airflow/cli/cli_config.py Outdated
@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch from ee12908 to 21fe3e7 Compare April 14, 2025 15:33
@sunank200 sunank200 requested a review from uranusjr April 14, 2025 15:38
@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch 3 times, most recently from 00a7ace to 1bb9c12 Compare April 14, 2025 16:26
@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch from 217f681 to 80b6009 Compare April 14, 2025 18:03

@Lee-W Lee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also cover some of the rules in newsfragments in this PR. Could you please also update the corresponding newsfragments?

You can list the rule todos through uv run scripts/ci/pre_commit/significant_newsfragments_checker.py

Thanks!

@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch from 80b6009 to 3d50ac8 Compare April 15, 2025 06:31
@uranusjr uranusjr requested a review from Lee-W April 15, 2025 06:57
Comment thread airflow-core/newsfragments/49223.significant.rst Outdated
@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch 2 times, most recently from db8e993 to 3c15b64 Compare April 15, 2025 08:18
@sunank200 sunank200 requested a review from Lee-W April 15, 2025 08:18
Comment thread airflow-core/newsfragments/49223.significant.rst Outdated
Comment thread airflow-core/newsfragments/49223.significant.rst Outdated
@sunank200 sunank200 requested a review from Lee-W April 15, 2025 08:24
@sunank200 sunank200 force-pushed the breaking-changes-airflow-config-update branch from 8ad27e7 to ef9f94c Compare April 15, 2025 08:48
@uranusjr uranusjr merged commit 6b7c0f7 into apache:main Apr 15, 2025
@uranusjr uranusjr deleted the breaking-changes-airflow-config-update branch April 15, 2025 09:53
@sunank200 sunank200 added the _eol_backport-to-v2-11-test Mark PR with this label to backport to v2-11-test branch label May 8, 2025
@sunank200 sunank200 changed the title Update airflow config lint to show breaking config changes by default Update airflow config update to show breaking config changes by default May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:CLI _eol_backport-to-v2-11-test Mark PR with this label to backport to v2-11-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix breaking changes by default in airflow config update

6 participants