feat(driver-native): add native compute driver

[ericcurtin]

Summary

Adds openshell-driver-native, an OCI/runc-based compute driver that builds sandboxes from Linux namespaces and cgroups v2. A system-provided containerd is used only for image pull, unpack, and snapshot management: this driver, not containerd, drives the configured low-level OCI runtime (runc/crun) directly, and containerd never creates a Container or Task for these sandboxes.

This is a draft PR opened directly at the requester's instruction, ahead of the normal review gate this repo's agent workflow uses (posting a plan and waiting for state:agent-ready). That plan is posted on the issue for reference.

Related Issue

Related to #2255

Changes

• crates/openshell-driver-native/ (new crate): image pull/chain-ID resolution/snapshot management against containerd, protected from GC via a per-sandbox lease (image.rs); OCI runtime-spec generation (spec.rs); direct create/start/state/kill/delete invocation of the configured runtime (runtime.rs); per-sandbox network namespace + veth + nftables (network.rs); GPU kernel device-node passthrough (gpu.rs); the ComputeDriver gRPC service (driver.rs/grpc.rs).
• crates/openshell-nft-ruleset/ (new crate): the VM driver's nftables ruleset generator extracted so both drivers share the same NAT + default-deny shape.
• crates/openshell-core, crates/openshell-server: ComputeDriverKind::Native wired through config, telemetry, and every compute-driver selection path.
• Docs: new Native Driver section in docs/reference/sandbox-compute-drivers.mdx, [openshell.drivers.native] example in docs/reference/gateway-config.mdx, architecture/ mentions, crates/openshell-driver-native/README.md.

Hardening from @elezar's review: sandbox names are validated against path traversal; the runtime's --root is scoped under the driver's own state directory instead of the shared global default; the sandbox endpoint no longer defaults to an unreachable loopback address; per-sandbox subnet allocation detects and avoids collisions instead of a stateless hash; cleanup after a failed create no longer leaves a mounted rootfs and bundle directory behind; sandbox token files are written with owner-only permissions.

Testing

• cargo fmt --all -- --check passes
• cargo clippy --workspace --all-targets -- -D warnings passes
• cargo test --workspace --lib passes
• License header check passes
• markdownlint-cli2 and the mermaid linter pass on changed docs
• Verified end to end against a real containerd 2.x + runc/crun install: pull, chain-ID resolve, lease-protected snapshot prepare, bundle mount, create/start/state/delete against both runtimes, stop, delete (tests/containerd_integration.rs, #[ignore]d since CI has no containerd); confirmed a snapshot without a lease is reaped by containerd's GC while one with a lease survives; confirmed the full openshell-gateway binary connects and selects this driver correctly.

Known gaps (documented in the driver README and docs/reference/sandbox-compute-drivers.mdx): rootless mode is implemented but not yet functional (containerd's prepared snapshot isn't remapped to match, so it defaults off); the supervisor binary is bind-mounted from a host path rather than sourced from an image like the other drivers; GPU support is kernel-device-node passthrough only, not full CDI injection; WatchSandboxes polls rather than subscribing to containerd's event stream; no dedicated E2E lane yet.

Not addressed in this PR: @elezar's architectural proposal to split this into a RootfsProvider/SandboxProvisioner structure and share more of the VM driver's pipeline (subnet allocation, nftables lifecycle, supervisor env construction, transactional cleanup). That's a larger restructuring worth its own discussion; see the review thread.

Checklist

• Follows Conventional Commits
• Commit is signed off (DCO)
• Architecture docs updated

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ericcurtin]

If we want to go full daemonless, containerd daemon dependency can eventually be removed, it just keeps the scope of this PR smaller for this iteration.

[cgwalters]

openshell-driver-native

Shouldn't it be named openshell-driver-containerd ?

This driver — not containerd — spawns the low-level runtime. containerd is used only for image pull/unpack and snapshot management (Transfer/Images/Content/Snapshots services); it never creates a containerd Container or Task object, and its io.containerd.runc.v2 shim is never involved. The driver builds a standard OCI bundle (config.json + a mounted rootfs/) from the snapshot containerd prepared and drives the configured runtime directly through its own create/start/state/kill/delete CLI contract — the same integration pattern containerd's shim,

Oh. That seems like it needs a really strong justification.

This relates to #981 - I still would really like to move away from openshell driving lots of low level custom sandboxing itself towards one where it just reuses existing container framework tooling - especially where e.g. the proxy is a sidecar or other associated process etc.

[ericcurtin]

openshell-driver-native

Shouldn't it be named openshell-driver-containerd ?

This driver — not containerd — spawns the low-level runtime. containerd is used only for image pull/unpack and snapshot management (Transfer/Images/Content/Snapshots services); it never creates a containerd Container or Task object, and its io.containerd.runc.v2 shim is never involved. The driver builds a standard OCI bundle (config.json + a mounted rootfs/) from the snapshot containerd prepared and drives the configured runtime directly through its own create/start/state/kill/delete CLI contract — the same integration pattern containerd's shim,

Oh. That seems like it needs a really strong justification.

This relates to #981 - I still would really like to move away from openshell driving lots of low level custom sandboxing itself towards one where it just reuses existing container framework tooling - especially where e.g. the proxy is a sidecar or other associated process etc.

containerd-daemon dependency might eventually go away, it's undecided, it just keeps the implementation smaller for now, it only handles transport, runc or crun is orchestrated directly by OpenShell.

This is sorta linked to that:

#2313

oci-client crate in the Rust ecosystem I think, is not the right call.

docker, podman, k8s remain separate drivers.

[ericcurtin]

But thanks @cgwalters I did rename:

feat(driver-native): add containerd-backed native compute driver

to

feat(driver-native): add native compute driver

the "containerd" part of the name does distract from the purpose here. It's OpenShell that kicks off the runc's or crun's. containerd has nothing to do with the isolation.

[krishicks]

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

[ericcurtin]

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

@krishicks see here, it's described in more detail:

#2313

[krishicks]

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

@krishicks see here, it's described in more detail:

#2313

It's not really explained there either:

Using oci-client crate is a mistake.

What's the problem that's being solved in #2313, and why is oci-client a mistake or not the right call?

[ericcurtin]

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

@krishicks see here, it's described in more detail:
#2313

It's not really explained there either:

Using oci-client crate is a mistake.

What's the problem that's being solved in #2313, and why is oci-client a mistake or not the right call?

The OCI protocol is ossified... This new Rust crate is used in projects like oras, which is known to be unreliable compared to implementations based on containerd or containers/image

[ericcurtin]

It can be reliable on OCI registries like Docker Hub, but it's the internal ones where you get the bug reports, hand-rolled OCI clients reliably break on obscure auth schemes, manifest quirks, and layer edge cases

[cgwalters]

I'm a bit skeptical of the oci-client crate being problematic as far as talking to registries, I think the larger picture thing is ecosystem stuff like finding and using pull secrets, which is outside the scope of that crate. And container storage.

[elezar]

Thanks for putting this together. As discussed in our sync, the goal of an
OCI-based driver is to provide a smooth out-of-the-box, single-node
experience where the VM driver is unavailable -- for example, when nested
virtualisation is not supported.

Using containerd for reliable registry access, content storage, unpacking, and
snapshots seems reasonable while we assess feasibility. I do not, however,
consider a required system containerd daemon to be a viable long-term design.
It introduces another installed and running service—including its configuration,
socket, namespaces, snapshotter, leases, compatibility, and operational
lifecycle—solely to construct root filesystems before OpenShell invokes the OCI
runtime itself. Image management does not inherently require the containerd
daemon: the relevant registry, content, unpacking, and archive functionality can
be provided through containerd libraries, as #2313 explores, or through another
OCI image implementation.

That makes it especially important for the current implementation to make
containerd's role explicit and temporary. Containerd is not responsible for
sandbox process lifecycle or provisioning here: OpenShell constructs the OCI
bundle and directly manages runc/crun, while containerd acts as one rootfs
provider. Its API and lifecycle concepts should remain behind that provider
boundary rather than becoming part of the compute-driver contract. Structuring
the code this way would make the design easier to understand and leave room for
replacing the daemon-backed provider without redesigning the OCI provisioner.

Conceptually, I would like to see something along these lines:

Local Compute Driver
├── RootfsProvider
│   ├── ContainerdRootfsProvider
│   ├── VmRootfsProvider
│   └── Future daemonless OCI provider
└── SandboxProvisioner
    ├── OCI/runc
    └── VM/libkrun

Each provider may produce a backend-specific artifact—for example, OCI mounts
for runc or an ext4 block image for libkrun — while sharing the preparation and
lifecycle contract.

I also think we should explicitly evaluate how much of the existing VM rootfs
pipeline can be adapted for the OCI provisioner. The final VM artifact is an
ext4 block image and is not directly useful to runc, but the earlier stages—
image resolution, acquisition, verification, unpacking, cache identity, and
progress reporting—are not inherently VM-specific. A useful internal split may
therefore be an ImageSource, which produces a verified OCI layout or unpacked
rootfs, followed by a backend-specific RootfsMaterializer, which produces
either OCI mounts or a VM disk image.

There are also parts of the VM driver that we should consider extracting instead
of implementing them again for OCI:

• Collision-aware subnet allocation
• Host networking and nftables lifecycle
• Translation of the configured gateway endpoint into an address visible from the sandbox
• Supervisor environment and credential construction
• Transactional allocation and cleanup of rootfs, network, and runtime resources

The VM-specific ext4, boot, libkrun, and VFIO code would remain in the VM
provisioner. Likewise, OCI specs, namespaces, capabilities, cgroups, and runc
state management would remain in the OCI provisioner. In both cases, the
provisioner creates the outer sandbox environment and starts the supervisor;
the supervisor then applies the RFC-defined isolation backend to the untrusted
agent workload.

While reviewing the implementation, I also found several issues that appear to
block the current version:

• The default supervisor endpoint is 127.0.0.1, but the supervisor runs in a separate network namespace. It will therefore connect to its own loopback rather than the gateway.
• Sandbox names are used directly as path components. The standalone gRPC service only checks that they are nonempty, which permits path traversal into operations such as remove_dir_all.
• Errors after mounting the rootfs can leave the mount and bundle behind. Snapshot cleanup can then fail because the snapshot remains mounted, and subsequent creation reports AlreadyExists.
• Network allocation hashes sandboxes into only 64 slots without collision detection. Different sandboxes can receive identical IP addresses and routes.
• Direct runtime invocations use the default global runc state root instead of an OpenShell-specific runtime root.
• Sandbox token files are not created with the restricted directory and file permissions used by the existing drivers.

The unit tests cover the individual helpers well, but the real containerd/runtime tests are ignored and there is no native gateway end-to-end lane. I think an end-to-end test using the real supervisor is important before merging because it would catch the endpoint and lifecycle issues above.

My preference would be to treat this PR as the start of an OCI sandbox
provisioner and a temporary containerd rootfs provider, rather than introducing
another monolithic compute driver or making the containerd daemon a lasting
architectural dependency. That would clarify containerd's limited role, allow
useful VM-driver infrastructure to be shared, and better align the
implementation with the architecture we discussed and the terminology in RFC
0001.

Given the architectural questions and correctness issues above, I do not think
the PR should merge in its current form. I would prefer that we first establish
these boundaries either by restructuring this PR or agreeing on smaller
staged changes.

[ericcurtin]

Thanks for putting this together. As discussed in our sync, the goal of an OCI-based driver is to provide a smooth out-of-the-box, single-node experience where the VM driver is unavailable -- for example, when nested virtualisation is not supported.

Using containerd for reliable registry access, content storage, unpacking, and snapshots seems reasonable while we assess feasibility. I do not, however, consider a required system containerd daemon to be a viable long-term design. It introduces another installed and running service—including its configuration, socket, namespaces, snapshotter, leases, compatibility, and operational lifecycle—solely to construct root filesystems before OpenShell invokes the OCI runtime itself. Image management does not inherently require the containerd daemon: the relevant registry, content, unpacking, and archive functionality can be provided through containerd libraries, as #2313 explores, or through another OCI image implementation.

"I do not, however, consider a required system containerd daemon to be a viable long-term design.". I agree here.

I've been waiting for opinions and decisions on:

#2313

before changing that here. I might not have communicated that great here. The three daemonless, out-of-the-box, single-node options, I can think of are:

1. Use containerd as a libloading lib
2. Use https://github.com/podman-container-tools/container-libs as libloading lib/libs sort of what @cgwalters is suggesting, but I think he may be more leaning towards, just use docker/podman in this case, we have an alternate driver for that fatter option already though. I'm not sure this hits the goals of people like @jmabry who's been looking at bubblewrap and apptainer though. Also, containerd has more functionality multi-platform accross macOS/windows/linux. I've used it multiple times for multi-platform transport.
3. Reimplement everything in Rust.

I think 3. carries the most risk FWIW. And would likely end up in a multi-year effort.

That makes it especially important for the current implementation to make containerd's role explicit and temporary. Containerd is not responsible for sandbox process lifecycle or provisioning here: OpenShell constructs the OCI bundle and directly manages runc/crun, while containerd acts as one rootfs provider. Its API and lifecycle concepts should remain behind that provider boundary rather than becoming part of the compute-driver contract. Structuring the code this way would make the design easier to understand and leave room for replacing the daemon-backed provider without redesigning the OCI provisioner.

Conceptually, I would like to see something along these lines:

Local Compute Driver
├── RootfsProvider
│   ├── ContainerdRootfsProvider
│   ├── VmRootfsProvider
│   └── Future daemonless OCI provider
└── SandboxProvisioner
    ├── OCI/runc
    └── VM/libkrun

Each provider may produce a backend-specific artifact—for example, OCI mounts for runc or an ext4 block image for libkrun — while sharing the preparation and lifecycle contract.

I also think we should explicitly evaluate how much of the existing VM rootfs pipeline can be adapted for the OCI provisioner. The final VM artifact is an ext4 block image and is not directly useful to runc, but the earlier stages— image resolution, acquisition, verification, unpacking, cache identity, and progress reporting—are not inherently VM-specific. A useful internal split may therefore be an ImageSource, which produces a verified OCI layout or unpacked rootfs, followed by a backend-specific RootfsMaterializer, which produces either OCI mounts or a VM disk image.

There are also parts of the VM driver that we should consider extracting instead of implementing them again for OCI:

• Collision-aware subnet allocation
• Host networking and nftables lifecycle
• Translation of the configured gateway endpoint into an address visible from the sandbox
• Supervisor environment and credential construction
• Transactional allocation and cleanup of rootfs, network, and runtime resources

The VM-specific ext4, boot, libkrun, and VFIO code would remain in the VM provisioner. Likewise, OCI specs, namespaces, capabilities, cgroups, and runc state management would remain in the OCI provisioner. In both cases, the provisioner creates the outer sandbox environment and starts the supervisor; the supervisor then applies the RFC-defined isolation backend to the untrusted agent workload.

While reviewing the implementation, I also found several issues that appear to block the current version:

• The default supervisor endpoint is 127.0.0.1, but the supervisor runs in a separate network namespace. It will therefore connect to its own loopback rather than the gateway.
• Sandbox names are used directly as path components. The standalone gRPC service only checks that they are nonempty, which permits path traversal into operations such as remove_dir_all.
• Errors after mounting the rootfs can leave the mount and bundle behind. Snapshot cleanup can then fail because the snapshot remains mounted, and subsequent creation reports AlreadyExists.
• Network allocation hashes sandboxes into only 64 slots without collision detection. Different sandboxes can receive identical IP addresses and routes.
• Direct runtime invocations use the default global runc state root instead of an OpenShell-specific runtime root.
• Sandbox token files are not created with the restricted directory and file permissions used by the existing drivers.

The unit tests cover the individual helpers well, but the real containerd/runtime tests are ignored and there is no native gateway end-to-end lane. I think an end-to-end test using the real supervisor is important before merging because it would catch the endpoint and lifecycle issues above.

My preference would be to treat this PR as the start of an OCI sandbox provisioner and a temporary containerd rootfs provider, rather than introducing another monolithic compute driver or making the containerd daemon a lasting architectural dependency. That would clarify containerd's limited role, allow useful VM-driver infrastructure to be shared, and better align the implementation with the architecture we discussed and the terminology in RFC 0001.

Given the architectural questions and correctness issues above, I do not think the PR should merge in its current form. I would prefer that we first establish these boundaries either by restructuring this PR or agreeing on smaller staged changes.

The rest sounds ok, I think we can figure it out.

[ericcurtin]

@elezar pushed a round of fixes for the correctness issues you flagged (loopback endpoint, path traversal, runtime --root, subnet collisions, cleanup-on-error, token permissions), all with new unit tests. Left the RootfsProvider/SandboxProvisioner restructuring out of this push since it's a bigger design conversation. Want to spin that off as a follow-up issue/RFC before I touch the driver boundary?

[elezar]

before changing that here. I might not have communicated that great here. The three daemonless, out-of-the-box, single-node options, I can think of are:

Is reusing / adapting the implementation the VM uses an option? While I understand that #2313 raises questions about the robustness of the implementation and the long-term maintenance costs, it may be useful to separate the discussion around what we use for rootfs construction (home-rolled vs containerd lib) from what we're using to provision the sandboxes (libkrun vs OCI).

[ericcurtin]

before changing that here. I might not have communicated that great here. The three daemonless, out-of-the-box, single-node options, I can think of are:

Is reusing / adapting the implementation the VM uses an option? While I understand that #2313 raises questions about the robustness of the implementation and the long-term maintenance costs, it may be useful to separate the discussion around what we use for rootfs construction (home-rolled vs containerd lib) from what we're using to provision the sandboxes (libkrun vs OCI).

Yes I agree, that's an option and should be the end goal. Consolidate on whatever the transport is.