[DEPR]: Paver

[kdmccormick]

ℹ️ This DEPR ticket considers the Paver commands in two groups:

• Operator-facing commands, which were accepted for deprecation back in May 2023, and then rolled into this general Paver DEPR ticket. These are the commands that operators need to migrate away from.
• Internal CI commands, which will be replaced at the leisure of the edx-platform maintenance team. They should not affect operators in any way.

Timeline

Communicated

Operator-facing commands: 2023-03-08 (thread)
Internal CI commands: 2024-05-01 (thread)

Acceptance

Operator-facing commands: 2023-05-03
Internal CI commands: 2024-05-21

Replacement available

Operator-facing commands: 2024-05-07, Redwood
Internal CI commands: ~2024-11-09, early Teak (Target)

Earliest removal

Operator-facing commands: 2024-11-09, early Teak
Internal CI commands: as soon as replacement is available

Rationale

edx-platform historically handled its build scripts with paver:

Paver is a Python-based software project scripting tool along the lines of Make or Rake. It is not designed to handle the dependency tracking requirements of, for example, a C program. It is designed to help out with all of your other repetitive tasks (run documentation generators, moving files about, downloading things), all with the convenience of Python’s syntax and massive library of code.

Using paver has a few problems:

• Paver adds dependencies to edx-platform: Paver follows the Python packaging model, so anything that's a dependency of edx-platform's pavelib suite becomes a dependency of edx-platform itself. In addition to the paver package, this means that we install libsass-python in production, which takes 60+ seconds (!) to compile and install.
• Paver makes Python a requirement for all build actions: When your build tooling runs on Python, then you need a Python environment in order to do anything, even things that don't involve Python (including building static assets, running JS tests, etc.). This harms the cachability of edx-platform Dockerfiles, because it means that any change to the Python requirements list will invalidate any build steps used by Paver.
• Paver is indirect: Paver basically just wraps shell commands that you could have run directly, adding a layer of complexity and potential bugs. Build scripts are ideally very simple--unlike application code, repetition is preferable to abstraction; they should be easy to write, easy to read, and easy to throw away and replace if necessary. Standard tools like Makefiles, Bash scripts, and Dockerfiles all encourage and excel at this sort of simplicity; Paver does not.
• Paver is idiosyncratic: Unlike Make and Bash, Paver is not at all a standard part of a programmer's toolset, so it makes edx-platform less approachable for new contributors. Even for programmers who want to script in Python, Click is significantly more popular, well-documented, and user-friendly.

For the Paver Asset commands in particular, more depth is provided in the 'Reimplement edx-platform static asset processing' ADR.

Replacement

Module	Replacement
assets	New static assets guide
quality	Directly invoking tools (pylint, etc.) when possible, Makefile targets otherwise
test_js	Makefile target
prereqs	pip
docs	Makefile target
i18n	Makefile targets for common tasks; i18n_tool for advanced tasks
servers	Tutor (or the deprecated Docker-based Devstack, or bare-metal development

Migration

To ease migration for Paver users, in Redwood, each Paver Asset command will simply proxy to its replacement command, and will raise a deprecation warning explaining the new command that it is running.

By Nov 9th 2024, operators will need to have switched to the new commands, taking the steps indicated below by ACTION REQUIRED.

Requirements (Operators: ACTION REQUIRED)

In Redwood and earlier, Paver and its dependencies were included in requirements/edx/base.txt

Starting in Sumac, these dependencies will be removed from requirements/edx/base.txt. Instead, operators will need to install:

• requirements/edx/assets.txt to build static assets
• requirements/edx/testing.txt to run Python tests and linting
• requirements/edx/base.txt download translations and collect static assets

Operator-facing commands with known users (Operators: ACTION REQUIRED)

These replacements are production-ready as of May 7th, for Redwood.

Module	Known Users	Before	After
assets	Ansible, Devstack	paver update_assets	npm run build && ./manage.py lms collectstatic --noinput && ./manage.py cms collectstatic
assets	None	paver process_xmodule_assets	No longer needed - no replacement
assets	Tutor, pavelib	paver compile_sass	npm run compile-sass
assets	pavelib	./manage.py [lms/cms] compile_sass	npm run compile-sass
assets	pavelib	paver webpack	npm run webpack
assets	Devstack	paver watch_assets	npm run watch
prereqs	Ansible, Devstack, pavelib	paver install_prereqs	pip install -r requirements/edx/base.txt -r requirements/edx/assets.txt && npm clean-install
prereqs	pavelib	paver install_node_prereqs	npm clean-install
prereqs	pavelib	paver uninstall_python_packages	Will not be replaced
prereqs	pavelib	paver install_coverage_prereqs	pip install -r requirements/edx/coverage.txt
prereqs	pavelib	paver install_python_prereqs	pip install -r requirements/edx/base.txt

Operator-facing commands without known users

These commands have been replaced for a long time. We don't know of any users of them, except the old Vagrant Devstack, which was deprecated 8 years ago.

Module	Before	After
docs	paver build_docs	make docs
i18n	paver i18n_validate_gettext	which xgettext
i18n	paver i18n_extract	make extract_translations
i18n	paver i18n_dummy	i18n_tool dummy && i18n_tool generate
i18n	paver i18n_generate	i18n_tool generate
i18n	paver i18n_generate_strict	i18n_tool generate --strict
i18n	paver i18n_clean	make clean_translations
servers	paver check_settings	None
servers	paver lms	./manage.py lms runserver
servers	paver studio	./manage.py cms runserver
servers	paver run_all_servers	tutor local ...
servers	paver devstack	tutor dev ...

Internal CI commands

These commands are only used by the upstream openedx project for CI. They will replaced before Sumac. Operators do not need to take action.

Module	Replacement Status	Before	After
quality	This is a no-op	paver find_fixme	Remove
quality	Need to implement	paver run_eslint	eslint
quality	Need to implement	paver run_stylelint	make stylelint_js
quality	Need to implement	paver run_xsslint	make xsslint
quality	Need to implement	paver run_pii_check	make pii_check
quality	Need to implement	paver check_keywords	make check_keywords
quality	Ready to use	paver run_quality	Remove
quality	Ready to use	paver run_pylint	pylint
quality	Need to implement	paver run_pep8	pycodestyle
js_test	Need to implement	paver diff_coverage	make diff_coverage_js
js_test	Need to implement	paver test_js	make test_js
js_test	Need to implement	paver test_js_run	make test_js
js_test	Need to implement	paver test_js_dev	make test_js MODE=browser

Django settings for Asset commands (Operators: ACTION REQUIRED)

In order to reimplement the Paver Asset commands without Python/Django, we are changing how several settings are configured.

The following Django settings are becoming read-only mirrors. If you override them now, remove your overrides:

• LMS:
  • STATIC_ROOT (a string, loaded from STATIC_ROOT_LMS env var)
  • COMPREHENSIVE_THEME_DIRS (loaded from the env var, parsed into a list of strings)
• CMS:
  • STATIC_ROOT (a string, loaded from STATIC_ROOT_CMS env var)
  • COMPREHENSIVE_THEME_DIRS (loaded from the env var, parsed into a list of strings)

The following Django settings are being removed. If you override them now, remove your overrides:

• LMS:
  • STATIC_ROOT_BASE
  • WEBPACK_CONFIG_PATH
  • JS_ENV_EXTRA_CONFIG
• CMS:
  • STATIC_ROOT_BASE
  • WEBPACK_CONFIG_PATH
  • JS_ENV_EXTRA_CONFIG

The following new environment variables are available. Set these in your environment, in place of the overrides you removed above:

• STATIC_ROOT_LMS (path)
• STATIC_ROOT_CMS (path)
• COMPREHENSIVE_THEME_DIRS (colon-separated paths)
• WEBPACK_CONFIG_PATH (path)
• JS_ENV_EXTRA_CONFIG (serialized as json)

Note: If you previously set the STATIC_ROOT_BASE Django setting to /blah, then you should now set the LMS and CMS environment variables as so:

• STATIC_ROOT_LMS=/blah
• STATIC_ROOT_CMS=/blah/studio

Deprecation

In time for Redwood, deprecation warnings will be added to all edx-platform Paver Asset commands.

Removal

• In edx-platform:

  • rm -rf ./pavelib/
  • rm ./requirements/edx/paver.in
  • rm ./requirements/edx/paver.txt
  • Search docs for 'paver' and replace with new commands

• Elsewhere in the openedx GitHub org:

  • Search docs for 'paver' and replace with new commands

• In Tutor:

  • Replace paver Asset references with new commands

### Tasks/PRs
- [ ] https://github.com/openedx/edx-platform/issues/31624
- [ ] https://github.com/openedx/edx-platform/pull/31810
- [ ] https://github.com/openedx/edx-platform/pull/31934
- [ ] https://github.com/openedx/edx-platform/pull/31790
- [ ] https://github.com/openedx/edx-platform/pull/32717
- [ ] https://github.com/openedx/edx-platform/pull/32767
- [ ] https://github.com/openedx/edx-platform/pull/32804
- [ ] https://github.com/openedx/edx-platform/pull/32823
- [ ] https://github.com/openedx/edx-platform/pull/32866
- [ ] https://github.com/openedx/edx-platform/pull/34318
- [ ] https://github.com/openedx/edx-platform/pull/34468
- [ ] https://github.com/openedx/edx-platform/pull/34554
- [ ] https://github.com/overhangio/tutor/pull/1042
- [ ] https://github.com/openedx/edx-platform/pull/34829
- [ ] https://github.com/openedx/edx-platform/pull/34830
- [ ] https://github.com/openedx/edx-platform/issues/34845
- [ ] https://github.com/overhangio/tutor/pull/1178
- [ ] https://github.com/openedx/edx-platform/issues/36025
- [ ] https://github.com/openedx/edx-platform/pull/36046
- [ ] https://github.com/openedx/edx-platform/pull/35741
- [ ] https://github.com/openedx/edx-platform/pull/34832
- [ ] https://github.com/openedx/studio-frontend/pull/399
- [ ] https://github.com/openedx/edx-documentation/pull/2314