Support environment variables in PipInstall plugin

[hendrikmakait]

Adds support for environment variables to the PipInstall plugin by using a requirement file under the hood (https://pip.pypa.io/en/stable/reference/requirements-file-format/#using-environment-variables). We should document this plugin better, but I would like to leave this to a dedicated PR.

Blocked by (and includes) #8343

• Tests added / passed
• Passes pre-commit run --all-files

[hendrikmakait]

I don't really have an idea how to test this since we offload the functionality. When trying this manually on Coiled, it works.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       27 files  ±    0         27 suites  ±0   12h 6m 45s ⏱️ ±0s
  3 934 tests  -     2    3 822 ✔️ +    5     110 💤  -   5  2 ❌  - 2 
49 488 runs   - 203  47 196 ✔️  - 180  2 290 💤  - 21  2 ❌  - 2 

For more details on these failures, see this check.

Results for commit dd6dc63. ± Comparison against base commit 412a0a9.

This pull request removes 44 and adds 42 tests. Note that renamed tests count towards both.

distributed.comm.tests.test_comms ‑ test_comm_closed_on_read_error[asyncio]
distributed.comm.tests.test_comms ‑ test_comm_closed_on_write_error[asyncio-BufferError]
distributed.comm.tests.test_comms ‑ test_comm_closed_on_write_error[asyncio-CustomBase]
distributed.comm.tests.test_comms ‑ test_comm_failure_threading[asyncio]
distributed.comm.tests.test_comms ‑ test_default_client_server_ipv4[asyncio]
distributed.comm.tests.test_comms ‑ test_default_client_server_ipv6[asyncio]
distributed.comm.tests.test_comms ‑ test_do_not_share_buffers[asyncio-OpaqueList]
distributed.comm.tests.test_comms ‑ test_do_not_share_buffers[asyncio-list]
distributed.comm.tests.test_comms ‑ test_get_address_host[asyncio]
distributed.comm.tests.test_comms ‑ test_get_local_address_for[asyncio]
…

distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page["]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[']
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[(]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[10]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[11]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[123]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[[]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[a b]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[a+b]
distributed.http.scheduler.tests.test_scheduler_http ‑ test_call_stack_page[key14]
…

This pull request removes 7 skipped tests and adds 2 skipped tests. Note that renamed tests count towards both.

distributed.comm.tests.test_comms ‑ test_comm_closed_on_read_error[asyncio]
distributed.comm.tests.test_comms ‑ test_comm_closed_on_write_error[asyncio-BufferError]
distributed.comm.tests.test_comms ‑ test_comm_closed_on_write_error[asyncio-CustomBase]
distributed.comm.tests.test_comms ‑ test_comm_failure_threading[asyncio]
distributed.comm.tests.test_comms ‑ test_default_client_server_ipv6[asyncio]
distributed.comm.tests.test_comms ‑ test_tcp_client_server_ipv6[asyncio]
distributed.comm.tests.test_comms ‑ test_tls_client_server_ipv6[asyncio]

distributed.protocol.tests.test_numpy
distributed.shuffle.tests.test_rechunk

♻️ This comment has been updated with latest results.

[fjetter]

I get that it's difficult to test but I would add a comment here why the -r is important. From first principle, this test is otherwise pretty useless. We test that pip install something but we also care that it is using the requirements file mechanism since this comes with additional features.
Adding a comment elevates this from a too detailed unit test that asserts on an implementation detail to an actual intended feature

[hendrikmakait]

Fair point!

[hendrikmakait]

Done

[hendrikmakait]

I've also added an example using an environment variable to the docs.