edx-platform's requirements files pretty closely follow the recommendations in OEP-18. But as Kyle McCormick pointed out recently, the pip-tools README has since been updated to recommend something a little different. Try making the following changes:
- Benchmark the performance of make upgrade and make requirements before making any changes.
- Instead of basing each .in file on the previous fully pinned output file via -r, use -c instead. This should result in a requirements file that has just the dependencies needed for the packages listed in the file currently being processed, but at versions compatible with the previous layer of dependencies.
- Rename the output files generated from an input file that builds on top of a previous layer with a _layer suffix, like testing_layer.txt.
- Introduce new files with the original output names that include all the dependencies the previous ones did; for example, the new testing.txt would consist of the lines -r base.txt, -r coverage.txt, and -r testing_layer.txt.
- Benchmark the performance of make upgrade and make requirements after the changes.
What we hope to gain from this change are the following:
- Faster
make upgrade performance by not having to process as many dependencies for each layer
- Better ease of reviewing dependency updates, since most dependency versions will only appear in a single requirements file, whereas many currently appear in 3 or more.
In the process, we will lose having a single file (development.txt) that lists essentially all of the packages used by edx-platform. But we rarely use this, and we could easily reproduce it by running pip-compile on the new development.txt.
If the results are good, write new tickets for the following:
- An update to OEP-18 to recommend this new structure for requirements files
- Updating all repos owned by arch-bom to comply with the new structure (these could be either BOM or ARCHBOM tickets)
- Communicating the change to the rest of the organization
Copied description from this private JIRA issue: https://2u-internal.atlassian.net/browse/BOM-2483
edx-platform's requirements files pretty closely follow the recommendations in OEP-18. But as Kyle McCormick pointed out recently, the pip-tools README has since been updated to recommend something a little different. Try making the following changes:
What we hope to gain from this change are the following:
make upgradeperformance by not having to process as many dependencies for each layerIn the process, we will lose having a single file (development.txt) that lists essentially all of the packages used by edx-platform. But we rarely use this, and we could easily reproduce it by running pip-compile on the new development.txt.
If the results are good, write new tickets for the following:
Copied description from this private JIRA issue: https://2u-internal.atlassian.net/browse/BOM-2483