ResolutionImpossible conflict when installing `devel-all` on head of main branch using `conststraints-main`

[hterik]

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

1. git clone airflow
2. Create and enter new virtualenv
3. pip install -e .[devel-all] --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt
4. Output:

    ERROR: Cannot install apache-airflow[cncf-kubernetes]==2.9.0.dev0 because these package versions have conflicting dependencies.
    
    The conflict is caused by:
        apache-airflow[cncf-kubernetes] 2.9.0.dev0 depends on kubernetes-asyncio<=29.0.0 and >=28.1.0; extra == "cncf-kubernetes"
        The user requested (constraint) kubernetes-asyncio==24.2.3

• Looking in pyproject.toml, it requires "kubernetes_asyncio>=28.1.0,<=29.0.0", (It was updated few weeks ago in Kubernetes version bump #37040)
• Whereas in downloaded constraints-3.11.txt it is pinned as kubernetes-asyncio==24.2.3

There is also a difference in dash vs underscore, don't know if this is significant.

What you think should happen instead?

Constraints listed under https://raw.githubusercontent.com/apache/airflow/constraints-main/ should be compatible with head of main branch.

How to reproduce

see above

Operating System

Ubuntu 23.10

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hterik]

Manually bypassing kubernetes-asyncio shows there are more inconsistencies:

ERROR: Cannot install apache-airflow, jsonschema and openapi-spec-validator because these package versions have conflicting dependencies.

The conflict is caused by:
    jsonschema 4.21.1 depends on referencing>=0.28.4
    jsonschema-specifications 2023.12.1 depends on referencing>=0.31.0
    jsonschema-path 0.3.2 depends on referencing<0.32.0 and >=0.28.0
    The user requested (constraint) referencing==0.33.0

All those versions are currently the same in https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt,

# This constraints file was automatically generated on 2024-02-20T10:14:41.212613
# via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.

jsonschema-path==0.3.2
jsonschema-specifications==2023.12.1
jsonschema==4.21.1
referencing==0.33.0

Reaching this constraints-file should not have been possible.

[Taragolis]

Just wondering where did you find information that you have to use https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt ?

[hterik]

Just wondering where did you find information that you have to use https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt ?

https://airflow.apache.org/docs/apache-airflow/2.8.1/installation/installing-from-pypi.html#reproducible-airflow-installation

[Taragolis]

Your link about released version of airflow and providers but you tried to install dev version of airflow. For create venv for development we have Local Virtual Environment (virtualenv) instruction.

k8s provider with this changes not released yet, rc3 at this moment, as soon as it released and CI step pass it will be reflected in this constraint file

[hterik]

Hi, the link you provided give exactly the same instructions:
https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst#installing-recommended-version-of-dependencies

pip install -e ".[devel,google]"" \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"

[Taragolis]

Nice catch! Never scroll down below the hatch instruction since it introduced.

Guess it was some outdated information, need to track all changes in the original file (this one created by splitting previous instruction). That is also section about constraints and extras for development: https://github.com/apache/airflow/blob/main/contributing-docs/12_airflow_dependencies_and_extras.rst

[Taragolis]

I reopen for visibility that it should be changed into the contributors documentation