Skip to content

Fix bug in SchedulerJobRunner._process_executor_events#40563

Merged
RNHTTR merged 1 commit into
apache:mainfrom
aws-mwaa:vincbeck/bug_scheduler
Jul 3, 2024
Merged

Fix bug in SchedulerJobRunner._process_executor_events#40563
RNHTTR merged 1 commit into
apache:mainfrom
aws-mwaa:vincbeck/bug_scheduler

Conversation

@vincbeck

@vincbeck vincbeck commented Jul 2, 2024

Copy link
Copy Markdown
Contributor

#40334 introduced a small bug where the error message is already formatted, hence there is no need to pass parameters to self._task_context_logger.error. That results in this error below:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/usr/local/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
  File "/usr/local/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/opt/airflow/airflow/__main__.py", line 58, in main
    args.func(args)
  File "/opt/airflow/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/opt/airflow/airflow/utils/cli.py", line 115, in wrapper
    return f(*args, **kwargs)
  File "/opt/airflow/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
  File "/opt/airflow/airflow/cli/commands/scheduler_command.py", line 61, in scheduler
    run_command_with_daemon_option(
  File "/opt/airflow/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
    callback()
  File "/opt/airflow/airflow/cli/commands/scheduler_command.py", line 64, in <lambda>
    callback=lambda: _run_scheduler_job(args),
  File "/opt/airflow/airflow/cli/commands/scheduler_command.py", line 50, in _run_scheduler_job
    run_job(job=job_runner.job, execute_callable=job_runner._execute)
  File "/opt/airflow/airflow/utils/session.py", line 84, in wrapper
    return func(*args, session=session, **kwargs)
  File "/opt/airflow/airflow/jobs/job.py", line 410, in run_job
    return execute_job(job, execute_callable=execute_callable)
  File "/opt/airflow/airflow/jobs/job.py", line 439, in execute_job
    ret = execute_callable()
  File "/opt/airflow/airflow/jobs/scheduler_job_runner.py", line 859, in _execute
    self._run_scheduler_loop()
  File "/opt/airflow/airflow/jobs/scheduler_job_runner.py", line 996, in _run_scheduler_loop
    num_finished_events = self._process_executor_events(session=session)
  File "/opt/airflow/airflow/jobs/scheduler_job_runner.py", line 783, in _process_executor_events
    self._task_context_logger.error(msg, ti, state, ti.state, info, ti=ti)
  File "/opt/airflow/airflow/utils/log/task_context_logger.py", line 170, in error
    self._log(logging.ERROR, msg, *args, ti=ti)
  File "/opt/airflow/airflow/utils/log/task_context_logger.py", line 141, in _log
    task_handler.emit(record)
  File "/opt/airflow/airflow/utils/log/file_task_handler.py", line 229, in emit
    self.handler.emit(record)
Message: "The executor reported that the task instance <TaskInstance: example_hello.hello manual__2024-07-02T19:54:10.890354+00:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally"
Arguments: (<TaskInstance: example_hello.hello manual__2024-07-02T19:54:10.890354+00:00 [queued]>, <TaskInstanceState.FAILED: 'failed'>, 'queued', None)

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@vincbeck vincbeck requested review from XD-DENG, ashb and kaxil as code owners July 2, 2024 20:57
@boring-cyborg boring-cyborg Bot added the area:Scheduler including HA (high availability) scheduler label Jul 2, 2024

@RNHTTR RNHTTR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing!

@RNHTTR RNHTTR merged commit a8c4830 into apache:main Jul 3, 2024
@utkarsharma2 utkarsharma2 added the type:misc/internal Changelog: Misc changes that should appear in change log label Jul 12, 2024
@utkarsharma2 utkarsharma2 added this to the Airflow 2.9.3 milestone Jul 12, 2024
utkarsharma2 pushed a commit that referenced this pull request Jul 12, 2024
@vincbeck vincbeck deleted the vincbeck/bug_scheduler branch July 17, 2024 17:30
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler type:misc/internal Changelog: Misc changes that should appear in change log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants