Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions base/images/images.toml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,10 @@ runtime-package-management = true
[images.container-base]
description = "Container Base Image"
definition = { type = "kiwi", path = "container-base/container-base.kiwi", profile = "core" }
tests.test-suites = [{ name = "static-image-checks" }]
tests.test-suites = [
{ name = "static-image-checks" },
{ name = "runtime-container-tests" },
]

[images.container-base.capabilities]
machine-bootable = false
Expand All @@ -67,7 +70,10 @@ runtime-package-management = true
[images.container-base-dev]
description = "Container Base Image (dev)"
definition = { type = "kiwi", path = "container-base/container-base.kiwi", profile = "core-dev" }
tests.test-suites = [{ name = "static-image-checks" }]
tests.test-suites = [
{ name = "static-image-checks" },
{ name = "runtime-container-tests" },
]

[images.container-base-dev.capabilities]
machine-bootable = false
Expand Down Expand Up @@ -123,7 +129,7 @@ description = "Offline image validation (shared + image-specific tests)"
[test-suites.static-image-checks.pytest]
working-dir = "tests"
install = "pyproject"
test-paths = ["cases/"]
test-paths = ["cases/static/"]
# {capabilities} is substituted by azldev as a comma-separated list of
# the names of capabilities set to `true` for the image (e.g.
# "machine-bootable,systemd,runtime-package-management"). Names use
Expand All @@ -135,3 +141,18 @@ extra-args = [
"--image-name", "{image-name}",
"--capabilities", "{capabilities}",
]

# Runtime container tests — validate live container behavior via podman.
[test-suites.runtime-container-tests]
type = "pytest"
description = "Runtime container validation (exec into live containers)"

[test-suites.runtime-container-tests.pytest]
working-dir = "tests"
install = "pyproject"
test-paths = ["cases/runtime/"]
extra-args = [
"--image-path", "{image-path}",
"--image-name", "{image-name}",
"--capabilities", "{capabilities}",
]
164 changes: 137 additions & 27 deletions base/images/tests/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Azure Linux Image Tests

Static validation framework for built Azure Linux images (VM and container).
Mounts images read-only and runs pytest tests against the filesystem
without booting.
Validation framework for built Azure Linux images (VM and container).
Includes both static (offline filesystem) and runtime (live container)
tests, all driven by pytest.

## How it gets invoked

These tests are wired into `azldev` via the `[test-suites.static-image-checks]`
table in `base/images/images.toml`, and referenced by each image's
These tests are wired into `azldev` via the `[test-suites.*]` tables
in `base/images/images.toml`, and referenced by each image's
`tests.test-suites`. The standard entry point is:

```bash
Expand All @@ -30,22 +30,41 @@ where present — to validate the dev variant.)
`pyproject.toml`, and invokes pytest with the right `--image-path`,
`--image-name`, and `--capabilities` arguments.

## Test suites

| Suite | Description | Runs for |
|-------|-------------|----------|
| `static-image-checks` | Offline filesystem validation — mounts images read-only | All images |
| `runtime-container-tests` | Live container tests via `podman exec` | Container images |

## Direct (manual) invocation

```bash
cd base/images/tests

# VM imageshared + VM-specific tests
uv run pytest cases/ \
# Static testsVM image
uv run pytest cases/static/ \
--image-path /path/to/image.raw \
--image-name vm-base \
--capabilities machine-bootable,systemd,runtime-package-management

# Container image — shared + container-specific tests
uv run pytest cases/ \
# Static tests — Container image
uv run pytest cases/static/ \
--image-path /path/to/image.oci.tar.xz \
--image-name container-base \
--capabilities container,runtime-package-management

# Runtime tests — Container image (requires podman socket)
uv run pytest cases/runtime/ \
--image-path /path/to/image.oci.tar.xz \
--image-name container-base \
--capabilities container,runtime-package-management

# Runtime tests — from a registry reference
uv run pytest cases/runtime/ \
--image-ref mcr.microsoft.com/azurelinux/base/core:4.0 \
--image-name container-base \
--capabilities container,runtime-package-management
```

Test selection follows standard pytest positional arguments. Tests
Expand All @@ -67,12 +86,36 @@ System packages (not pip-installable):

- **`libguestfs-tools`** + **`guestfs-tools`** — `guestmount`,
`guestunmount`, `virt-inspector` (VM images)
- **`skopeo`** — OCI archive conversion (container images)
- **`umoci`** — OCI image unpacking (container images)
- **`buildah`** — cleanup of rootless umoci extracts (container images)
- **`skopeo`** — OCI archive conversion (container images, static tests)
- **`umoci`** — OCI image unpacking (container images, static tests)
- **`buildah`** — cleanup of rootless umoci extracts (container images, static tests)
- **`podman`** — container runtime for live tests (container images, runtime tests)
- **`rpm`** — for `rpm --root` package queries
- **`uv`** — Python project/package manager

For runtime tests, the **podman socket** must be active:
(ASK: Do we need this? podman-py is quite basic, maybe it's better without REST API and directly invoking podman CLI? The socket requirement is a bit of a pain, especially for WSL users. Then would need to have our own wrapper around podman CLI to handle the container orchestration logic currently in `container_runtime.py`. Or even docker SDK with moby might be more stable than podman-py?)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely open to other options here -- but we should leverage existing OSS if we can.

What about https://github.com/gabrieldemarmiesse/python-on-whales ? That seems to work with podman in rootless, daemonless mode too.

```bash
# With systemd (standard Linux)
systemctl --user start podman.socket

# Without systemd (WSL, containers, CI)
podman system service --timeout=0 &
```

> **WSL users:** Rootless podman defaults to the systemd cgroup
> manager, but the REST API (used by podman-py) does not auto-fallback
> to cgroupfs like the CLI does. If container starts fail with
> `sd-bus call: Permission denied`, create
> `~/.config/containers/containers.conf`:
>
> ```toml
> [engine]
> cgroup_manager = "cgroupfs"
> ```
>
> Then restart the podman socket.

`pytest_configure` does a preflight check and fails fast if any tool
needed for the current `--image-type` is missing.

Expand All @@ -82,30 +125,40 @@ needed for the current `--image-type` is missing.
base/images/
├── images.toml # Image registry + test-suite wiring
└── tests/
├── pyproject.toml # uv project: pytest, plugin entry point
├── conftest.py # Session fixtures
├── pyproject.toml # uv project: pytest + podman deps
├── conftest.py # Session fixtures (static + runtime)
├── utils/ # Helper package (not test-collected)
│ ├── pytest_plugin.py # CLI options, markers, tool preflight
│ ├── container_runtime.py # Podman-py based container orchestration
│ ├── extract.py # Image mounting / extraction
│ ├── disk.py # virt-inspector → DiskInfo
│ ├── parsers.py # File content parsers
│ ├── types.py # Dataclasses
│ └── tools.py # Native-tool registry
└── cases/ # Test cases
├── test_os_release.py # Shared: /etc/os-release
├── test_packages.py # Shared: rpm-db checks (capability-gated)
├── vm-base/ # VM-specific tests (auto-restricted to the vm-base family — vm-base, vm-base-dev)
│ ├── test_kernel.py
│ └── test_partitions.py
└── container-base/ # Container-specific tests (auto-restricted to the container-base family)
└── test_container.py
├── static/ # Offline filesystem tests
│ ├── test_os_release.py # Shared: /etc/os-release
│ ├── test_packages.py # Shared: rpm-db checks (capability-gated)
│ ├── vm-base/ # VM-specific static tests
│ │ ├── test_kernel.py
│ │ └── test_partitions.py
│ └── container-base/ # Container-specific static tests
│ └── test_container.py
└── runtime/ # Live container tests (via podman exec)
└── container-base/
├── test_basic.py # Basic: shell access, DNS resolution
└── test_nginx/ # Dockerfile test example
├── test_nginx.py # Test logic
├── Dockerfile # Custom image (ARG BASE_IMAGE)
└── nginx.conf # Supporting files
```

## Available fixtures

| Fixture | Scope | Type | Description |
|---------|-------|------|-------------|
| `image_path` | session | `Path` | From `--image-path` |
| `image_path` | session | `Path \| None` | From `--image-path` (None when `--image-ref` used) |
| `image_ref` | session | `str \| None` | From `--image-ref` (None when `--image-path` used) |
| `image_name` | session | `str \| None` | From `--image-name` |
| `image_type` | session | `str` | `"vm"` or `"container"` (explicit / capabilities / extension) |
| `capabilities` | session | `set[str]` | Parsed `--capabilities` |
Expand All @@ -115,20 +168,77 @@ base/images/
| `installed_packages` | session | `set[str]` | Installed RPM names (`rpm --root`) |
| `disk_info` | session | `DiskInfo \| None` | VM only |
| `partition_table` | session | `list[PartitionInfo]` | VM only — auto-skips on containers |
| `podman_client` | session | `PodmanClient \| None` | Podman API client; None for non-container images |
| `container_image_ref` | session | `str \| None` | Loaded image ID (cached); None for non-container |
| `running_container` | function | `ContainerInstance` | Fresh container per test — auto-skips on VMs |
| `container_exec` | function | callable | `(cmd) → ContainerExecResult` |

## Adding tests

- **Shared (every image):** add a `cases/test_<topic>.py`. Use
- **Shared static (every image):** add a `cases/static/test_<topic>.py`. Use
`@pytest.mark.require_capability("…")` if the test only applies to
images with a given capability.
- **Image-specific:** add `cases/<image-family>/test_<topic>.py`. Tests
in such subdirectories are **automatically** restricted to that
- **Image-specific static:** add `cases/static/<image-family>/test_<topic>.py`.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have odd feeling about these directory-based tests discovery, markers might be more useful. As you add more and more tests, directories and subdirectories just for categorization might become too complex.

Tests in such subdirectories are **automatically** restricted to that
image family (the plugin applies `@pytest.mark.image("<dir>")`
during collection — no boilerplate per file or per subdir). The
directory name is treated as a *family*: an `--image-name` matches
the family if it equals the family exactly OR has the form
`<family>-<variant>` (so `cases/vm-base/` runs for both `vm-base`
and `vm-base-dev`).
`<family>-<variant>` (so `cases/static/vm-base/` runs for both
`vm-base` and `vm-base-dev`).
- **Shared runtime (every container):** add a `cases/runtime/test_<topic>.py`.
Use the `container_exec` fixture to run commands in a live container.
Tests under `cases/runtime/` are auto-marked with
`@pytest.mark.runtime_container_tests`.
- **Image-specific runtime:** add `cases/runtime/<image-family>/test_<topic>.py`.

### Dockerfile-based runtime tests

When a runtime test needs packages or config beyond what the base image
ships, give it its own directory with a `Dockerfile`:

```
cases/runtime/container-base/test_nginx/
test_nginx.py # test logic
Dockerfile # builds on top of the image-under-test
nginx.conf # supporting files (COPY'd in Dockerfile)
```

The Dockerfile must use `ARG BASE_IMAGE` / `FROM ${BASE_IMAGE}` — the
framework injects the image-under-test automatically:

```dockerfile
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
RUN dnf install -y nginx && dnf clean all
COPY nginx.conf /etc/nginx/nginx.conf
```

Mark tests with `@pytest.mark.dockerfile()` to trigger the build. The
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this!

marker optionally accepts a path relative to the test file's directory
(defaults to `Dockerfile` in the same directory):

```python
# Auto-discovers Dockerfile in the same directory
@pytest.mark.dockerfile()
def test_nginx_config(container_exec):
result = container_exec("nginx -t 2>&1")
assert result.exit_code == 0

# Explicit path to a different Dockerfile
@pytest.mark.dockerfile("alt/Dockerfile.debug")
def test_debug_variant(container_exec):
...
```

Built images are cached per session — multiple tests sharing the same
Dockerfile only trigger one build.

> **Note:** All containers (plain and Dockerfile-based) run with
> `sleep infinity` as PID 1 — the Dockerfile's `CMD`/`ENTRYPOINT` is
> overridden. Tests that need a service should start it explicitly via
> `container_exec("nginx")`. This keeps behaviour predictable and
> ensures each test controls exactly what runs.

## Adding a native-tool dependency

Expand Down
Empty file.
Empty file.
27 changes: 27 additions & 0 deletions base/images/tests/cases/runtime/container-base/test_basic.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# SPDX-License-Identifier: MIT
"""Basic runtime tests for container-base images.

Validate fundamental container behavior by exec-ing commands into a
running container. Each test gets a fresh container instance via the
``container_exec`` fixture (see conftest.py).
"""

from __future__ import annotations


def test_shell_accessible(container_exec) -> None:
"""Container shell must be functional via exec."""
result = container_exec("echo hello-from-container")
assert result.exit_code == 0, (
f"Shell exec failed (exit_code={result.exit_code}): {result.output}"
)
assert "hello-from-container" in result.output


def test_dns_resolution(container_exec) -> None:
"""Container must be able to resolve localhost via DNS."""
result = container_exec("getent hosts localhost")
assert result.exit_code == 0, (
f"DNS resolution failed (exit_code={result.exit_code}): {result.output}"
)
assert "localhost" in result.output
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
RUN dnf install -y nginx && dnf clean all
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 80
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /run/nginx.pid;

events {
worker_connections 128;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

server {
listen 80;
server_name localhost;

location / {
return 200 "azl-nginx-ok\n";
add_header Content-Type text/plain;
}

location /health {
return 200 "healthy\n";
add_header Content-Type text/plain;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# SPDX-License-Identifier: MIT
"""Validate nginx works on the container-base image.

Uses ``@pytest.mark.dockerfile()`` to build a custom image with
nginx installed on top of the image-under-test.
"""

from __future__ import annotations

import pytest


@pytest.mark.dockerfile()
def test_nginx_config_valid(container_exec) -> None:
"""nginx configuration must pass validation."""
result = container_exec("nginx -t 2>&1")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 2>&1?

assert result.exit_code == 0, f"nginx -t failed: {result.output}"
assert "syntax is ok" in result.output
assert "test is successful" in result.output


@pytest.mark.dockerfile()
def test_nginx_serves_response(container_exec) -> None:
"""nginx must start and serve HTTP responses."""
# Start nginx in the background.
start = container_exec("nginx")
assert start.exit_code == 0, f"nginx failed to start: {start.output}"

# Verify it responds on port 80.
result = container_exec("curl -sf http://localhost:80/")
assert result.exit_code == 0, f"curl failed: {result.output}"
assert "azl-nginx-ok" in result.output


@pytest.mark.dockerfile()
def test_nginx_health_endpoint(container_exec) -> None:
"""nginx /health endpoint must return 200."""
start = container_exec("nginx")
assert start.exit_code == 0, f"nginx failed to start: {start.output}"

result = container_exec("curl -sf http://localhost:80/health")
assert result.exit_code == 0, f"health check failed: {result.output}"
assert "healthy" in result.output
Empty file.
Loading
Loading