-
Notifications
You must be signed in to change notification settings - Fork 645
[WIP] add: container test plugin for pytest #17372
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 4.0
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,13 @@ | ||
| # Azure Linux Image Tests | ||
|
|
||
| Static validation framework for built Azure Linux images (VM and container). | ||
| Mounts images read-only and runs pytest tests against the filesystem | ||
| without booting. | ||
| Validation framework for built Azure Linux images (VM and container). | ||
| Includes both static (offline filesystem) and runtime (live container) | ||
| tests, all driven by pytest. | ||
|
|
||
| ## How it gets invoked | ||
|
|
||
| These tests are wired into `azldev` via the `[test-suites.static-image-checks]` | ||
| table in `base/images/images.toml`, and referenced by each image's | ||
| These tests are wired into `azldev` via the `[test-suites.*]` tables | ||
| in `base/images/images.toml`, and referenced by each image's | ||
| `tests.test-suites`. The standard entry point is: | ||
|
|
||
| ```bash | ||
|
|
@@ -30,22 +30,41 @@ where present — to validate the dev variant.) | |
| `pyproject.toml`, and invokes pytest with the right `--image-path`, | ||
| `--image-name`, and `--capabilities` arguments. | ||
|
|
||
| ## Test suites | ||
|
|
||
| | Suite | Description | Runs for | | ||
| |-------|-------------|----------| | ||
| | `static-image-checks` | Offline filesystem validation — mounts images read-only | All images | | ||
| | `runtime-container-tests` | Live container tests via `podman exec` | Container images | | ||
|
|
||
| ## Direct (manual) invocation | ||
|
|
||
| ```bash | ||
| cd base/images/tests | ||
|
|
||
| # VM image — shared + VM-specific tests | ||
| uv run pytest cases/ \ | ||
| # Static tests — VM image | ||
| uv run pytest cases/static/ \ | ||
| --image-path /path/to/image.raw \ | ||
| --image-name vm-base \ | ||
| --capabilities machine-bootable,systemd,runtime-package-management | ||
|
|
||
| # Container image — shared + container-specific tests | ||
| uv run pytest cases/ \ | ||
| # Static tests — Container image | ||
| uv run pytest cases/static/ \ | ||
| --image-path /path/to/image.oci.tar.xz \ | ||
| --image-name container-base \ | ||
| --capabilities container,runtime-package-management | ||
|
|
||
| # Runtime tests — Container image (requires podman socket) | ||
| uv run pytest cases/runtime/ \ | ||
| --image-path /path/to/image.oci.tar.xz \ | ||
| --image-name container-base \ | ||
| --capabilities container,runtime-package-management | ||
|
|
||
| # Runtime tests — from a registry reference | ||
| uv run pytest cases/runtime/ \ | ||
| --image-ref mcr.microsoft.com/azurelinux/base/core:4.0 \ | ||
| --image-name container-base \ | ||
| --capabilities container,runtime-package-management | ||
| ``` | ||
|
|
||
| Test selection follows standard pytest positional arguments. Tests | ||
|
|
@@ -67,12 +86,36 @@ System packages (not pip-installable): | |
|
|
||
| - **`libguestfs-tools`** + **`guestfs-tools`** — `guestmount`, | ||
| `guestunmount`, `virt-inspector` (VM images) | ||
| - **`skopeo`** — OCI archive conversion (container images) | ||
| - **`umoci`** — OCI image unpacking (container images) | ||
| - **`buildah`** — cleanup of rootless umoci extracts (container images) | ||
| - **`skopeo`** — OCI archive conversion (container images, static tests) | ||
| - **`umoci`** — OCI image unpacking (container images, static tests) | ||
| - **`buildah`** — cleanup of rootless umoci extracts (container images, static tests) | ||
| - **`podman`** — container runtime for live tests (container images, runtime tests) | ||
| - **`rpm`** — for `rpm --root` package queries | ||
| - **`uv`** — Python project/package manager | ||
|
|
||
| For runtime tests, the **podman socket** must be active: | ||
| (ASK: Do we need this? podman-py is quite basic, maybe it's better without REST API and directly invoking podman CLI? The socket requirement is a bit of a pain, especially for WSL users. Then would need to have our own wrapper around podman CLI to handle the container orchestration logic currently in `container_runtime.py`. Or even docker SDK with moby might be more stable than podman-py?) | ||
| ```bash | ||
| # With systemd (standard Linux) | ||
| systemctl --user start podman.socket | ||
|
|
||
| # Without systemd (WSL, containers, CI) | ||
| podman system service --timeout=0 & | ||
| ``` | ||
|
|
||
| > **WSL users:** Rootless podman defaults to the systemd cgroup | ||
| > manager, but the REST API (used by podman-py) does not auto-fallback | ||
| > to cgroupfs like the CLI does. If container starts fail with | ||
| > `sd-bus call: Permission denied`, create | ||
| > `~/.config/containers/containers.conf`: | ||
| > | ||
| > ```toml | ||
| > [engine] | ||
| > cgroup_manager = "cgroupfs" | ||
| > ``` | ||
| > | ||
| > Then restart the podman socket. | ||
|
|
||
| `pytest_configure` does a preflight check and fails fast if any tool | ||
| needed for the current `--image-type` is missing. | ||
|
|
||
|
|
@@ -82,30 +125,40 @@ needed for the current `--image-type` is missing. | |
| base/images/ | ||
| ├── images.toml # Image registry + test-suite wiring | ||
| └── tests/ | ||
| ├── pyproject.toml # uv project: pytest, plugin entry point | ||
| ├── conftest.py # Session fixtures | ||
| ├── pyproject.toml # uv project: pytest + podman deps | ||
| ├── conftest.py # Session fixtures (static + runtime) | ||
| ├── utils/ # Helper package (not test-collected) | ||
| │ ├── pytest_plugin.py # CLI options, markers, tool preflight | ||
| │ ├── container_runtime.py # Podman-py based container orchestration | ||
| │ ├── extract.py # Image mounting / extraction | ||
| │ ├── disk.py # virt-inspector → DiskInfo | ||
| │ ├── parsers.py # File content parsers | ||
| │ ├── types.py # Dataclasses | ||
| │ └── tools.py # Native-tool registry | ||
| └── cases/ # Test cases | ||
| ├── test_os_release.py # Shared: /etc/os-release | ||
| ├── test_packages.py # Shared: rpm-db checks (capability-gated) | ||
| ├── vm-base/ # VM-specific tests (auto-restricted to the vm-base family — vm-base, vm-base-dev) | ||
| │ ├── test_kernel.py | ||
| │ └── test_partitions.py | ||
| └── container-base/ # Container-specific tests (auto-restricted to the container-base family) | ||
| └── test_container.py | ||
| ├── static/ # Offline filesystem tests | ||
| │ ├── test_os_release.py # Shared: /etc/os-release | ||
| │ ├── test_packages.py # Shared: rpm-db checks (capability-gated) | ||
| │ ├── vm-base/ # VM-specific static tests | ||
| │ │ ├── test_kernel.py | ||
| │ │ └── test_partitions.py | ||
| │ └── container-base/ # Container-specific static tests | ||
| │ └── test_container.py | ||
| └── runtime/ # Live container tests (via podman exec) | ||
| └── container-base/ | ||
| ├── test_basic.py # Basic: shell access, DNS resolution | ||
| └── test_nginx/ # Dockerfile test example | ||
| ├── test_nginx.py # Test logic | ||
| ├── Dockerfile # Custom image (ARG BASE_IMAGE) | ||
| └── nginx.conf # Supporting files | ||
| ``` | ||
|
|
||
| ## Available fixtures | ||
|
|
||
| | Fixture | Scope | Type | Description | | ||
| |---------|-------|------|-------------| | ||
| | `image_path` | session | `Path` | From `--image-path` | | ||
| | `image_path` | session | `Path \| None` | From `--image-path` (None when `--image-ref` used) | | ||
| | `image_ref` | session | `str \| None` | From `--image-ref` (None when `--image-path` used) | | ||
| | `image_name` | session | `str \| None` | From `--image-name` | | ||
| | `image_type` | session | `str` | `"vm"` or `"container"` (explicit / capabilities / extension) | | ||
| | `capabilities` | session | `set[str]` | Parsed `--capabilities` | | ||
|
|
@@ -115,20 +168,77 @@ base/images/ | |
| | `installed_packages` | session | `set[str]` | Installed RPM names (`rpm --root`) | | ||
| | `disk_info` | session | `DiskInfo \| None` | VM only | | ||
| | `partition_table` | session | `list[PartitionInfo]` | VM only — auto-skips on containers | | ||
| | `podman_client` | session | `PodmanClient \| None` | Podman API client; None for non-container images | | ||
| | `container_image_ref` | session | `str \| None` | Loaded image ID (cached); None for non-container | | ||
| | `running_container` | function | `ContainerInstance` | Fresh container per test — auto-skips on VMs | | ||
| | `container_exec` | function | callable | `(cmd) → ContainerExecResult` | | ||
|
|
||
| ## Adding tests | ||
|
|
||
| - **Shared (every image):** add a `cases/test_<topic>.py`. Use | ||
| - **Shared static (every image):** add a `cases/static/test_<topic>.py`. Use | ||
| `@pytest.mark.require_capability("…")` if the test only applies to | ||
| images with a given capability. | ||
| - **Image-specific:** add `cases/<image-family>/test_<topic>.py`. Tests | ||
| in such subdirectories are **automatically** restricted to that | ||
| - **Image-specific static:** add `cases/static/<image-family>/test_<topic>.py`. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have odd feeling about these directory-based tests discovery, markers might be more useful. As you add more and more tests, directories and subdirectories just for categorization might become too complex. |
||
| Tests in such subdirectories are **automatically** restricted to that | ||
| image family (the plugin applies `@pytest.mark.image("<dir>")` | ||
| during collection — no boilerplate per file or per subdir). The | ||
| directory name is treated as a *family*: an `--image-name` matches | ||
| the family if it equals the family exactly OR has the form | ||
| `<family>-<variant>` (so `cases/vm-base/` runs for both `vm-base` | ||
| and `vm-base-dev`). | ||
| `<family>-<variant>` (so `cases/static/vm-base/` runs for both | ||
| `vm-base` and `vm-base-dev`). | ||
| - **Shared runtime (every container):** add a `cases/runtime/test_<topic>.py`. | ||
| Use the `container_exec` fixture to run commands in a live container. | ||
| Tests under `cases/runtime/` are auto-marked with | ||
| `@pytest.mark.runtime_container_tests`. | ||
| - **Image-specific runtime:** add `cases/runtime/<image-family>/test_<topic>.py`. | ||
|
|
||
| ### Dockerfile-based runtime tests | ||
|
|
||
| When a runtime test needs packages or config beyond what the base image | ||
| ships, give it its own directory with a `Dockerfile`: | ||
|
|
||
| ``` | ||
| cases/runtime/container-base/test_nginx/ | ||
| test_nginx.py # test logic | ||
| Dockerfile # builds on top of the image-under-test | ||
| nginx.conf # supporting files (COPY'd in Dockerfile) | ||
| ``` | ||
|
|
||
| The Dockerfile must use `ARG BASE_IMAGE` / `FROM ${BASE_IMAGE}` — the | ||
| framework injects the image-under-test automatically: | ||
|
|
||
| ```dockerfile | ||
| ARG BASE_IMAGE | ||
| FROM ${BASE_IMAGE} | ||
| RUN dnf install -y nginx && dnf clean all | ||
| COPY nginx.conf /etc/nginx/nginx.conf | ||
| ``` | ||
|
|
||
| Mark tests with `@pytest.mark.dockerfile()` to trigger the build. The | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I like this! |
||
| marker optionally accepts a path relative to the test file's directory | ||
| (defaults to `Dockerfile` in the same directory): | ||
|
|
||
| ```python | ||
| # Auto-discovers Dockerfile in the same directory | ||
| @pytest.mark.dockerfile() | ||
| def test_nginx_config(container_exec): | ||
| result = container_exec("nginx -t 2>&1") | ||
| assert result.exit_code == 0 | ||
|
|
||
| # Explicit path to a different Dockerfile | ||
| @pytest.mark.dockerfile("alt/Dockerfile.debug") | ||
| def test_debug_variant(container_exec): | ||
| ... | ||
| ``` | ||
|
|
||
| Built images are cached per session — multiple tests sharing the same | ||
| Dockerfile only trigger one build. | ||
|
|
||
| > **Note:** All containers (plain and Dockerfile-based) run with | ||
| > `sleep infinity` as PID 1 — the Dockerfile's `CMD`/`ENTRYPOINT` is | ||
| > overridden. Tests that need a service should start it explicitly via | ||
| > `container_exec("nginx")`. This keeps behaviour predictable and | ||
| > ensures each test controls exactly what runs. | ||
|
|
||
| ## Adding a native-tool dependency | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # SPDX-License-Identifier: MIT | ||
| """Basic runtime tests for container-base images. | ||
|
|
||
| Validate fundamental container behavior by exec-ing commands into a | ||
| running container. Each test gets a fresh container instance via the | ||
| ``container_exec`` fixture (see conftest.py). | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
|
|
||
| def test_shell_accessible(container_exec) -> None: | ||
| """Container shell must be functional via exec.""" | ||
| result = container_exec("echo hello-from-container") | ||
| assert result.exit_code == 0, ( | ||
| f"Shell exec failed (exit_code={result.exit_code}): {result.output}" | ||
| ) | ||
| assert "hello-from-container" in result.output | ||
|
|
||
|
|
||
| def test_dns_resolution(container_exec) -> None: | ||
| """Container must be able to resolve localhost via DNS.""" | ||
| result = container_exec("getent hosts localhost") | ||
| assert result.exit_code == 0, ( | ||
| f"DNS resolution failed (exit_code={result.exit_code}): {result.output}" | ||
| ) | ||
| assert "localhost" in result.output |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| ARG BASE_IMAGE | ||
| FROM ${BASE_IMAGE} | ||
| RUN dnf install -y nginx && dnf clean all | ||
| COPY nginx.conf /etc/nginx/nginx.conf | ||
| EXPOSE 80 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| worker_processes auto; | ||
| error_log /var/log/nginx/error.log warn; | ||
| pid /run/nginx.pid; | ||
|
|
||
| events { | ||
| worker_connections 128; | ||
| } | ||
|
|
||
| http { | ||
| include /etc/nginx/mime.types; | ||
| default_type application/octet-stream; | ||
|
|
||
| server { | ||
| listen 80; | ||
| server_name localhost; | ||
|
|
||
| location / { | ||
| return 200 "azl-nginx-ok\n"; | ||
| add_header Content-Type text/plain; | ||
| } | ||
|
|
||
| location /health { | ||
| return 200 "healthy\n"; | ||
| add_header Content-Type text/plain; | ||
| } | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # SPDX-License-Identifier: MIT | ||
| """Validate nginx works on the container-base image. | ||
|
|
||
| Uses ``@pytest.mark.dockerfile()`` to build a custom image with | ||
| nginx installed on top of the image-under-test. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import pytest | ||
|
|
||
|
|
||
| @pytest.mark.dockerfile() | ||
| def test_nginx_config_valid(container_exec) -> None: | ||
| """nginx configuration must pass validation.""" | ||
| result = container_exec("nginx -t 2>&1") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why |
||
| assert result.exit_code == 0, f"nginx -t failed: {result.output}" | ||
| assert "syntax is ok" in result.output | ||
| assert "test is successful" in result.output | ||
|
|
||
|
|
||
| @pytest.mark.dockerfile() | ||
| def test_nginx_serves_response(container_exec) -> None: | ||
| """nginx must start and serve HTTP responses.""" | ||
| # Start nginx in the background. | ||
| start = container_exec("nginx") | ||
| assert start.exit_code == 0, f"nginx failed to start: {start.output}" | ||
|
|
||
| # Verify it responds on port 80. | ||
| result = container_exec("curl -sf http://localhost:80/") | ||
| assert result.exit_code == 0, f"curl failed: {result.output}" | ||
| assert "azl-nginx-ok" in result.output | ||
|
|
||
|
|
||
| @pytest.mark.dockerfile() | ||
| def test_nginx_health_endpoint(container_exec) -> None: | ||
| """nginx /health endpoint must return 200.""" | ||
| start = container_exec("nginx") | ||
| assert start.exit_code == 0, f"nginx failed to start: {start.output}" | ||
|
|
||
| result = container_exec("curl -sf http://localhost:80/health") | ||
| assert result.exit_code == 0, f"health check failed: {result.output}" | ||
| assert "healthy" in result.output | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Definitely open to other options here -- but we should leverage existing OSS if we can.
What about https://github.com/gabrieldemarmiesse/python-on-whales ? That seems to work with podman in rootless, daemonless mode too.