Make sqlalchemy optional dependency for openlineage provider#59921
Conversation
sunank200
force-pushed
the
make-sqlalchemy-optional-in-openlineage-provider
branch
from
af00c19 to
2a69289
Compare
|
Nice - few questions - but more to openlineage Airflow 3 integration :) |
|
One more note, in utils.py we're also using some db methods like |
Not yet. This will come later when we will split (some) providers into two types of providers (maybe they won't be named like that) - we've been disussing it but deferred decision on that to after we complete task isolation and separated core and task-sdk. We iwll have to separate some distributions in two:
This falls in the same part - part of the current openlineage provider should be a distribution installed with "task-sdk" and part should be another distribution installed with "airflow-core". Later we can even split "airflow-ui" as separate distribution and have 3 types of "providers" |
inner check for AIRFLOW_V_3_0_PLUS was unreachable dead code.
sunank200
force-pushed
the
make-sqlalchemy-optional-in-openlineage-provider
branch
from
e6bccd9 to
72bd9cc
Compare
|
Nice! |
…59921) * Make sqlalchemy optional dependency for openlineage provider * Refactor sqlalchemy imports to be lazy in openlineage utils * The function is already guarded by 'if not AIRFLOW_V_3_0_PLUS:' so the inner check for AIRFLOW_V_3_0_PLUS was unreachable dead code. * Remove sqlalchemy-specific ImportError handling in get_openlineage_facets_with_sql
…59921) * Make sqlalchemy optional dependency for openlineage provider * Refactor sqlalchemy imports to be lazy in openlineage utils * The function is already guarded by 'if not AIRFLOW_V_3_0_PLUS:' so the inner check for AIRFLOW_V_3_0_PLUS was unreachable dead code. * Remove sqlalchemy-specific ImportError handling in get_openlineage_facets_with_sql
…59921) * Make sqlalchemy optional dependency for openlineage provider * Refactor sqlalchemy imports to be lazy in openlineage utils * The function is already guarded by 'if not AIRFLOW_V_3_0_PLUS:' so the inner check for AIRFLOW_V_3_0_PLUS was unreachable dead code. * Remove sqlalchemy-specific ImportError handling in get_openlineage_facets_with_sql
…59921) * Make sqlalchemy optional dependency for openlineage provider * Refactor sqlalchemy imports to be lazy in openlineage utils * The function is already guarded by 'if not AIRFLOW_V_3_0_PLUS:' so the inner check for AIRFLOW_V_3_0_PLUS was unreachable dead code. * Remove sqlalchemy-specific ImportError handling in get_openlineage_facets_with_sql
…59921) * Make sqlalchemy optional dependency for openlineage provider * Refactor sqlalchemy imports to be lazy in openlineage utils * The function is already guarded by 'if not AIRFLOW_V_3_0_PLUS:' so the inner check for AIRFLOW_V_3_0_PLUS was unreachable dead code. * Remove sqlalchemy-specific ImportError handling in get_openlineage_facets_with_sql
3 participants
Makes SQLAlchemy an optional dependency for the OpenLineage provider by adding it as an optional extra.
pip install 'apache-airflow-providers-openlineage[sqlalchemy]'closes: #59904
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.