If a branch uses a git+https://... dependency in order to integration-test some pending changes to a dependency, that version of the dependency may end up used in the CI checks for other branches, which could produce incorrect test successes or failures. We first observed this two weeks ago, but it is likely that it has been happening for longer.
These appear to be the necessary conditions:
- The same virtualenv is used for multiple branches. This can happen on a developer's machine, or when using the 2U-provided CI runners (which re-use virtualenvs).
- On one branch (such as
master), a dependency is installed from PyPI, but on another branch the dependency is changed to install from git (example) and requirements are re-compiled (example)
- The git-dependency's version is the same as the one on PyPI (that is, its setup.py version value has not changed on the feature branch that is pointed to by the requirements file)
- CI checks are run for the git-using branch and then for the PyPI-using branch
This is in spite of the CI checks using make test-requirements, which calls pip-sync. Normally, this clears out all stale packages, but it fails to in this situation.
Theory
When pip-sync sees a git-dependency, it always installs it (after uninstalling any current version.) When it sees a PyPI dependency, however, it first checks to see if the version is already satisfied. So after the switch back to the base branch, pip-sync has no idea that the wrong dependency is still installed.
Unknowns
- Does this still happen if the git-dependency uses a commit hash reference rather than a branch?
- Is this a bug in pip-tools? (Should it always uninstall when there's a current git-dependency?)
- Does this still happen with newer versions of pip and pip-tools?
Reproduce
This can be reproduced by using the release-2023-02-27-10.29 tag as the base commit (standing in for master) and timmc/repro-pip-stale-git-other as the integration test branch (compare). The "other branch" uses a branch of edx-drf-extensions that has a new dependency on cryptojwt (with no version change) and code changes that depend on it.
In a completely fresh Python 3.8 virtualenv, run the following in order to install requirements for the base branch, then the other branch, then the base branch again—and check the consistency of the environment after each requirements sync. It will end up using pip 23.0.1 and pip-sync 6.12.2.
git checkout release-2023-02-27-10.29
make test-requirements
pip check
# "No broken requirements found."
python -c 'import edx_rest_framework_extensions.auth.jwt.tests.utils'
# no error
git checkout timmc/repro-pip-stale-git-other
make test-requirements
# ...
# Found existing installation: edx-drf-extensions 8.4.1
# Uninstalling edx-drf-extensions-8.4.1:
# Successfully uninstalled edx-drf-extensions-8.4.1
# ...
# Collecting edx-drf-extensions@ git+https://github.com/openedx/edx-drf-extensions@timmc/repro-pip-stale-git
# ...
# Resolved https://github.com/openedx/edx-drf-extensions to commit 8719107b131cef344ae36288896b565b2a632321
# ...
# Collecting cryptojwt==1.8.3
# Using cached cryptojwt-1.8.3-py3-none-any.whl (85 kB)
# ...
# Successfully installed cryptojwt-1.8.3 edx-drf-extensions-8.4.1 mongodbproxy-0.1.0+edx.2 olxcleaner-0.1.3
# ...
pip check
# "No broken requirements found."
python -c 'import edx_rest_framework_extensions.auth.jwt.tests.utils'
# no error
git checkout release-2023-02-27-10.29
make test-requirements
# Found existing installation: cryptojwt 1.8.3
# Uninstalling cryptojwt-1.8.3:
# Successfully uninstalled cryptojwt-1.8.3
# ...
# [does not install/uninstall edx-drf-extensions]
pip check
# "edx-drf-extensions 8.4.1 requires cryptojwt, which is not installed."
python -c 'import edx_rest_framework_extensions.auth.jwt.tests.utils'
# Traceback (most recent call last):
# File "<string>", line 1, in <module>
# File "/home/timmc/edx-repos/edx-platform/venv-3.8/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/tests/utils.py", line 6, in <module>
# from cryptojwt.jws.jws import JWS
# ModuleNotFoundError: No module named 'cryptojwt'
If a branch uses a
git+https://...dependency in order to integration-test some pending changes to a dependency, that version of the dependency may end up used in the CI checks for other branches, which could produce incorrect test successes or failures. We first observed this two weeks ago, but it is likely that it has been happening for longer.These appear to be the necessary conditions:
master), a dependency is installed from PyPI, but on another branch the dependency is changed to install from git (example) and requirements are re-compiled (example)This is in spite of the CI checks using
make test-requirements, which callspip-sync. Normally, this clears out all stale packages, but it fails to in this situation.Theory
When pip-sync sees a git-dependency, it always installs it (after uninstalling any current version.) When it sees a PyPI dependency, however, it first checks to see if the version is already satisfied. So after the switch back to the base branch, pip-sync has no idea that the wrong dependency is still installed.
Unknowns
Reproduce
This can be reproduced by using the
release-2023-02-27-10.29tag as the base commit (standing in for master) andtimmc/repro-pip-stale-git-otheras the integration test branch (compare). The "other branch" uses a branch of edx-drf-extensions that has a new dependency on cryptojwt (with no version change) and code changes that depend on it.In a completely fresh Python 3.8 virtualenv, run the following in order to install requirements for the base branch, then the other branch, then the base branch again—and check the consistency of the environment after each requirements sync. It will end up using pip 23.0.1 and pip-sync 6.12.2.