Airflow database upgrade fails with "psycopg2.errors.NotNullViolation: column "map_index" of relation "task_instance" contains null value"s

[dcardinha]

Apache Airflow version

main (development)

What happened

I currently have Airflow 2.2.3 and due to this issue I have tried to upgrade Airflow to this commit.

When I run airflow db upgrade I get the following error:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 587bdf053233 -> e655c0453f75, Add TaskMap and map_index on TaskInstance.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: column "map_index" of relation "task_instance" contains null values

The map_index column was introduced with this PR.

Could you please advise?

What you expected to happen

No response

How to reproduce

No response

Operating System

Ubuntu 18.04.6 LTS

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

Kubernetes

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kaxil]

Map Index hasn't been introduced in Airflow 2.2.3 and nor intended to, it is currently available in the main branch and will be released in Airflow 2.3. So it is very likely you somehow used some code from the main branch.

Check https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html -- that migration isn't available in 2.2.3

Double verify by running the following:

AIRFLOW_VERSION=2.2.3
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install "apache-airflow[async,postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

[dcardinha]

Thank you for your answer.

Yes, I was using the code from the main branch.

Is there an estimate as to when Airflow 2.3 will be released?

[ashb]

@dcardinha Nothing firm, but our aim is "in about a month"