certgen: std::fs::rename fails with EXDEV on WSL/container cross-device mounts

[rsandbach]

Agent Diagnostic

Can't use an agent if I can't setup a gateway. :-D (I had GitHub copilot start drafting this issue, after working out why the cross device failure was happening)

When generating cert files, crates/openshell-server/src/certgen.rs uses std::fs::rename(...) to move the newly generated certificates to the specified output directory. In some cases, the source and destination can land on different mounts/devices. In that case, rename(2) returns EXDEV (Rust: std::io::ErrorKind::CrossesDevices), which causes cert generation to fail.

Due to the location the temporary files are created, ${output_dir}.certgen.tmp, they are actually being created on the container's tempfs, not the attached volume. This can lead to the cross devices error.

Workaround (set --output-dir to a new folder within the mounted volume)

PS C:\> container run --rm -it -u 0:0 -v ogc:/extvol ghcr.io/nvidia/openshell/gateway:latest generate-certs --output-dir "/extvol/z"
2026-07-08T03:31:06.530744Z  INFO openshell_server::certgen: PKI files created. dir=/extvol/z

Expected

Cert generation should succeed regardless of whether source/destination are on the same device.
The temp certs probably should get created IN the --output-dir, not next to it.

Actual

Move step fails with cross-device rename error (EXDEV / CrossesDevices).

Proposed fix

Use a cross-device-safe move:

1. Attempt std::fs::rename(src, dst) first.
2. On ErrorKind::CrossesDevices, fallback to:
   • std::fs::copy(src, dst)
   • preserve permissions if needed
   • std::fs::remove_file(src)
3. Propagate all other errors unchanged.

OR

Generate the temporary certificates within a folder of the output dir (/extvol/certgen.tmp/ca.crt), not in it.

Why this change

This preserves same-device fast path/behavior while handling common WSL/container mount layouts where cross-device moves are expected.

Description

generate-certs fails with --output-dir args.
generate-certs to be successful.

Reproduction Steps

Environment

• Windows host + WSLC

PS C:\> container run --rm -it -u 0:0 -v ogc:/extvol ghcr.io/nvidia/openshell/gateway:latest generate-certs --output-dir "/extvol"
Error:   x failed to move /extvol.certgen.tmp/ca.crt -> /extvol/ca.crt
  `-> Invalid cross-device link (os error 18)

Environment

Windows 11 25H2
wslc 2.9.3.0

Logs

Error:   x failed to move /extvol.certgen.tmp/ca.crt -> /extvol/ca.crt
  `-> Invalid cross-device link (os error 18)

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why

[drew]

📋 triage-agent

Triage Assessment

Classification: bug-confirmed

Summary

The reported cross-device failure follows directly from the cert-generation staging layout on current main (420a855d). Local generation places temporary files beside the requested output directory, which is not guaranteed to share a filesystem when that directory is itself a container or WSL volume mount.

Investigation

write_local_bundle stages under ${output_dir}.certgen.tmp and then renames each file into the output directory (crates/openshell-server/src/certgen.rs:626-686). sibling_temp_dir constructs that path with dir.with_file_name(...) at lines 769-776. For a mounted /extvol, the stage is /extvol.certgen.tmp on the parent filesystem while the destination is inside the mounted filesystem, so the first std::fs::rename returns EXDEV exactly as reported.

The TLS-only and JWT-only repair writers use the same pattern at lines 689-755. Existing tests stage and rename beneath one temporary filesystem, so they do not exercise this boundary. No existing issue duplicates this certgen failure; #1467 has the same errno in an unrelated uv sandbox path.

Recommendation

Stage on the destination filesystem and retain same-filesystem atomic replacement, stale-stage cleanup, and restricted key permissions. A copy fallback is acceptable only if it preserves those guarantees and cleans up partial output.

[gracesmith6504]

Im keen to work on this. Planning to move the temp dir inside --output-dir so the rename stays on the same filesystem.