Skip to content

[v3-3-test] Limit pandas to < 3 for DataFrame XComs (#70791) #70912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Aug 1, 2026
Merged

[v3-3-test] Limit pandas to < 3 for DataFrame XComs (#70791) #70912

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions airflow-core/newsfragments/70791.significant.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
pandas is limited to ``< 3``; DataFrame XComs are refused on pandas 3

A ``pandas.DataFrame`` does not survive an XCom round trip under pandas 3 the way it does
under pandas 2, so this version of Airflow constrains pandas to ``< 3`` and refuses DataFrame
XComs at runtime when a newer pandas is installed anyway.

Two things change under pandas 3. Its public classes are exposed from the ``pandas``
namespace, so a DataFrame qualifies as ``pandas.DataFrame`` rather than
``pandas.core.frame.DataFrame`` and the XCom serializer registry -- keyed on the latter --
never dispatches to the pandas serializer. And the dtypes differ: an ``object`` column reads
back as ``str``, and missing values as ``nan`` rather than ``None``, so the *reader's* pandas
version decides what a task receives rather than the writer's.

Because pandas is an optional dependency, the ``< 3`` constraint alone cannot be relied on --
a deployment may install pandas 3 directly, or pull it in through another package. It is
therefore backed by a runtime check.

**Behaviour changes:**

- With pandas 3 or newer installed, pushing **or** pulling a ``DataFrame`` through XCom now
raises ``RuntimeError`` naming the installed pandas version and the supported range.
Reads are refused as well as writes, because a payload written under pandas 2 reads back
with pandas 3 dtypes and would otherwise hand the task different data than was pushed,
with no signal.
- Previously the same situation produced an opaque
``TypeError: cannot serialize object of type <class 'pandas.DataFrame'>`` from the
serializer registry, with nothing identifying pandas as the cause.
- Deployments that pass DataFrames through XCom must pin ``pandas < 3`` in the environment
that runs their tasks. Deployments that do not use DataFrame XComs are unaffected at
runtime.
- Nothing changes for pandas 2 users.

**This defers pandas 3 support rather than dropping it.** Moving from pandas 2 to pandas 3
is a deliberate migration for users to make, given the dtype changes it brings to data that
has already been written. Support for pandas 3 will be enabled in a future Airflow release.
15 changes: 12 additions & 3 deletions providers/amazon/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,9 +130,18 @@ dependencies = [
"apache-airflow-providers-mongo"
]
"pandas" = [
'pandas>=2.1.2; python_version <"3.13"',
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.1.2,<3; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
]
"openlineage" = [
"apache-airflow-providers-openlineage>=2.3.0"
Expand Down
12 changes: 6 additions & 6 deletions providers/apache/hdfs/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,19 +51,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
Requirements
------------

========================================== ==================================================================
========================================== ====================================================================
PIP package Version required
========================================== ==================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``hdfs[avro,dataframe,kerberos]`` ``>=2.5.4; python_version < "3.12"``
``hdfs[avro,dataframe,kerberos]`` ``>=2.7.3; python_version >= "3.12"``
``fastavro`` ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"``
``fastavro`` ``>=1.12.1; python_version >= "3.14"``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
========================================== ==================================================================
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
========================================== ====================================================================

Cross provider package dependencies
-----------------------------------
Expand Down
12 changes: 6 additions & 6 deletions providers/apache/hdfs/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,19 @@ Requirements

The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.

========================================== ==================================================================
========================================== ====================================================================
PIP package Version required
========================================== ==================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``hdfs[avro,dataframe,kerberos]`` ``>=2.5.4; python_version < "3.12"``
``hdfs[avro,dataframe,kerberos]`` ``>=2.7.3; python_version >= "3.12"``
``fastavro`` ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"``
``fastavro`` ``>=1.12.1; python_version >= "3.14"``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
========================================== ==================================================================
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
========================================== ====================================================================

Cross provider package dependencies
-----------------------------------
Expand Down
15 changes: 12 additions & 3 deletions providers/apache/hdfs/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,18 @@ dependencies = [
'hdfs[avro,dataframe,kerberos]>=2.7.3;python_version>="3.12"',
'fastavro>=1.10.0; python_version>="3.13" and python_version<"3.14"',
'fastavro>=1.12.1; python_version>="3.14"',
'pandas>=2.1.2; python_version <"3.13"',
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.1.2,<3; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
]

[dependency-groups]
Expand Down
12 changes: 6 additions & 6 deletions providers/apache/hive/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,19 +50,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
Requirements
------------

========================================== =================================================================
========================================== ====================================================================
PIP package Version required
========================================== =================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``apache-airflow-providers-common-sql`` ``>=1.32.0``
``hmsclient`` ``>=0.1.0``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
``pyhive[hive_pure_sasl]`` ``>=0.7.0``
``jmespath`` ``>=0.7.0``
========================================== =================================================================
========================================== ====================================================================

Cross provider package dependencies
-----------------------------------
Expand Down
12 changes: 6 additions & 6 deletions providers/apache/hive/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,19 +99,19 @@ Requirements

The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.

========================================== =================================================================
========================================== ====================================================================
PIP package Version required
========================================== =================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
``apache-airflow-providers-common-sql`` ``>=1.32.0``
``hmsclient`` ``>=0.1.0``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
``pyhive[hive_pure_sasl]`` ``>=0.7.0``
``jmespath`` ``>=0.7.0``
========================================== =================================================================
========================================== ====================================================================

Cross provider package dependencies
-----------------------------------
Expand Down
15 changes: 12 additions & 3 deletions providers/apache/hive/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,18 @@ dependencies = [
"apache-airflow-providers-common-compat>=1.12.0",
"apache-airflow-providers-common-sql>=1.32.0",
"hmsclient>=0.1.0",
'pandas>=2.1.2; python_version <"3.13"',
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.1.2,<3; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
"pyhive[hive_pure_sasl]>=0.7.0",
"jmespath>=0.7.0",
]
Expand Down
15 changes: 12 additions & 3 deletions providers/common/sql/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,20 @@ dependencies = [
# Any change in the dependencies is preserved when the file is regenerated
[project.optional-dependencies]
"pandas" = [
'pandas[sql-other]>=2.1.2; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas[sql-other]>=2.1.2,<3; python_version <"3.13"',
# Technically - we should add "sql-other" here as well, but this will only be possible when we move
# to sqlalchemy 2+ TODO(potiuk): do so.
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
]
"openlineage" = [
"apache-airflow-providers-openlineage"
Expand Down
12 changes: 6 additions & 6 deletions providers/databricks/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,23 +50,23 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
Requirements
------------

========================================== ==================================================================
========================================== ====================================================================
PIP package Version required
========================================== ==================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.13.0``
``apache-airflow-providers-common-sql`` ``>=1.32.0``
``requests`` ``>=2.32.0,<3``
``databricks-sql-connector`` ``>=4.4.0``
``aiohttp`` ``>=3.14.0,<4``
``mergedeep`` ``>=1.3.4``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
``pyarrow`` ``>=16.1.0; python_version < "3.13"``
``pyarrow`` ``>=18.0.0; python_version >= "3.13" and python_version < "3.14"``
``pyarrow`` ``>=22.0.0; python_version >= "3.14"``
========================================== ==================================================================
========================================== ====================================================================

Cross provider package dependencies
-----------------------------------
Expand Down
12 changes: 6 additions & 6 deletions providers/databricks/docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,23 +98,23 @@ Requirements

The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.

========================================== ==================================================================
========================================== ====================================================================
PIP package Version required
========================================== ==================================================================
========================================== ====================================================================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.13.0``
``apache-airflow-providers-common-sql`` ``>=1.32.0``
``requests`` ``>=2.32.0,<3``
``databricks-sql-connector`` ``>=4.4.0``
``aiohttp`` ``>=3.14.0,<4``
``mergedeep`` ``>=1.3.4``
``pandas`` ``>=2.1.2; python_version < "3.13"``
``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3; python_version >= "3.14"``
``pandas`` ``>=2.1.2,<3; python_version < "3.13"``
``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
``pandas`` ``>=2.3.3,<3; python_version >= "3.14"``
``pyarrow`` ``>=16.1.0; python_version < "3.13"``
``pyarrow`` ``>=18.0.0; python_version >= "3.13" and python_version < "3.14"``
``pyarrow`` ``>=22.0.0; python_version >= "3.14"``
========================================== ==================================================================
========================================== ====================================================================

Cross provider package dependencies
-----------------------------------
Expand Down
15 changes: 12 additions & 3 deletions providers/databricks/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,18 @@ dependencies = [
"databricks-sql-connector>=4.4.0",
"aiohttp>=3.14.0, <4",
"mergedeep>=1.3.4",
'pandas>=2.1.2; python_version <"3.13"',
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
'pandas>=2.3.3; python_version >="3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.1.2,<3; python_version <"3.13"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
# pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
# round trip; capped until pandas 3 support lands.
# Tracked at https://github.com/apache/airflow/pull/70558
'pandas>=2.3.3,<3; python_version >="3.14"',
"pyarrow>=16.1.0; python_version < '3.13'",
"pyarrow>=18.0.0; python_version >= '3.13' and python_version < '3.14'",
"pyarrow>=22.0.0; python_version >= '3.14'",
Expand Down
Loading