Skip to content

Reduce verbosity of provider import failure logs & make configurable #22539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dstandish wants to merge 1 commit into from
Closed

Reduce verbosity of provider import failure logs & make configurable #22539

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions airflow/config_templates/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,15 @@
type: string
example: ~
default: "INFO"
- name: provider_import_error_level
description: |
Logging level for provider import failures. Intended mainly for Airflow developers.

Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
version_added: 2.3.0
type: string
example: ~
default: ~
- name: celery_logging_level
description: |
Logging level for celery. If not set, it uses the value of logging_level
Expand Down
5 changes: 5 additions & 0 deletions airflow/config_templates/default_airflow.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,11 @@ encrypt_s3_logs = False
# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
logging_level = INFO

# Logging level for provider import failures. Intended mainly for Airflow developers.
#
# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
# provider_import_error_level =

# Logging level for celery. If not set, it uses the value of logging_level
#
# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
Expand Down
15 changes: 13 additions & 2 deletions airflow/providers_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@

from airflow.exceptions import AirflowOptionalProviderFeatureException
from airflow.hooks.base import BaseHook
from airflow.settings import conf
from airflow.utils import yaml
from airflow.utils.entry_points import entry_points_with_dist
from airflow.utils.log.logging_mixin import LoggingMixin
Expand All @@ -62,6 +63,10 @@
"apache-airflow-providers-celery": "2.1.0",
}

PROVIDER_IMPORT_ERROR_LEVEL = logging.getLevelName(
conf.get('logging', 'provider_import_error_level', fallback='WARNING').upper()
)


class LazyDictWithCache(MutableMapping):
"""
Expand Down Expand Up @@ -168,8 +173,14 @@ def _sanity_check(provider_package: str, class_name: str) -> Optional[Type[BaseH
except ImportError as e:
# When there is an ImportError we turn it into debug warnings as this is
# an expected case when only some providers are installed
log.warning(
"Exception when importing '%s' from '%s' package",
log.log(
PROVIDER_IMPORT_ERROR_LEVEL,
"Exception when importing '%s' from '%s' package. Set log level to DEBUG for traceback.",
class_name,
provider_package,
)
log.debug(
"Traceback from failed import of class %r from package %r",
class_name,
provider_package,
exc_info=e,
Expand Down