[Bug] Unable to download the latest devcontainer `sha256:7292cb2...` on Apple Silicon

[tschaffter]

Is there an existing issue for this?

• I have searched the existing issues

What product(s) are you seeing the problem on?

Sage Monorepo

Current behavior

VS Code shows the following error when trying to build and start the latest dev container (sha256:7292cb2c571e5e4fab839d4dbfeefbd0129a0b9cf045d8e6626216c8b4d5e627) on Apple Silicon.

[2026-02-27T14:58:16.697Z] Error fetching image details: No manifest found for ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107.
[2026-02-27T14:58:16.698Z] Start: Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:17.751Z] Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found
[2026-02-27T14:58:17.754Z] Stop (1056 ms): Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:17.754Z] Retrying (Attempt 0) with error 
			  'Command failed: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
'
[2026-02-27T14:58:18.755Z] Start: Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:19.574Z] Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found
[2026-02-27T14:58:19.577Z] Stop (822 ms): Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:19.577Z] Retrying (Attempt 1) with error 
			  'Command failed: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
'
[2026-02-27T14:58:20.578Z] Start: Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:21.272Z] Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found
[2026-02-27T14:58:21.274Z] Stop (696 ms): Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:21.275Z] Retrying (Attempt 2) with error 
			  'Command failed: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
'
[2026-02-27T14:58:22.276Z] Start: Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:23.250Z] Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found
[2026-02-27T14:58:23.252Z] Stop (976 ms): Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:23.252Z] Retrying (Attempt 3) with error 
			  'Command failed: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
'
[2026-02-27T14:58:24.254Z] Start: Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:25.041Z] Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found
[2026-02-27T14:58:25.043Z] Stop (789 ms): Run: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:25.043Z] Retrying (Attempt 4) with error 
			  'Command failed: docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
'
[2026-02-27T14:58:26.044Z] []
[2026-02-27T14:58:26.044Z] Error response from daemon: No such image: ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107

[2026-02-27T14:58:26.045Z] Error: Command failed: docker inspect --type image ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
[2026-02-27T14:58:26.045Z]     at b9 (/Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js:467:1277)
[2026-02-27T14:58:26.046Z]     at mL (/Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js:467:1021)
[2026-02-27T14:58:26.046Z]     at async Z9 (/Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js:485:3870)
[2026-02-27T14:58:26.046Z]     at async EC (/Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js:485:4989)
[2026-02-27T14:58:26.046Z]     at async M5 (/Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js:666:205)
[2026-02-27T14:58:26.046Z]     at async k5 (/Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js:665:15084)
[2026-02-27T14:58:26.046Z]     at async /Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js:485:1188
[2026-02-27T14:58:26.058Z] Stop (11246 ms): Run: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin) /Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/hyterion/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --container-session-data-folder /tmp/devcontainers-c713ff94-dc8e-4f1d-852c-6862adf583671772204294193 --workspace-folder /Users/hyterion/dev/tschaffter --workspace-mount-consistency cached --gpu-availability detect --id-label devcontainer.local_folder=/Users/hyterion/dev/tschaffter --id-label devcontainer.config_file=/Users/hyterion/dev/tschaffter/.devcontainer/devcontainer.json --log-level debug --log-format json --config /Users/hyterion/dev/tschaffter/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --include-configuration --include-merged-configuration
[2026-02-27T14:58:26.059Z] Exit code 1
[2026-02-27T14:58:26.065Z] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin) /Users/hyterion/.vscode/extensions/ms-vscode-remote.remote-containers-0.442.0/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/hyterion/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --container-session-data-folder /tmp/devcontainers-c713ff94-dc8e-4f1d-852c-6862adf583671772204294193 --workspace-folder /Users/hyterion/dev/tschaffter --workspace-mount-consistency cached --gpu-availability detect --id-label devcontainer.local_folder=/Users/hyterion/dev/tschaffter --id-label devcontainer.config_file=/Users/hyterion/dev/tschaffter/.devcontainer/devcontainer.json --log-level debug --log-format json --config /Users/hyterion/dev/tschaffter/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --include-configuration --include-merged-configuration
[2026-02-27T14:58:26.065Z] Exit code 1

Expected behavior

The new dev container image should download and start successfully.

Anything else?

Note that the latest dev container image I created at Sage is for a single architecture: https://github.com/Sage-Bionetworks/sage-monorepo/pkgs/container/sage-monorepo-devcontainer/529947788?tag=sha-db91d6849b048b6ef69472cdeb80caad262b3e68

But the latest image has been created for two architectures (see the tab OS / Arch): https://github.com/Sage-Bionetworks/sage-monorepo/pkgs/container/sage-monorepo-devcontainer/694030290?tag=edge

Commit ID

7292cb2

Are you developing inside the dev container?

• I am using the dev container

Code of Conduct

• I agree to follow this project's Code of Conduct

[tschaffter]

Cc: @hallieswan

[hallieswan]

@tschaffter the latest dev container image only updated the dev container CLI from 0.72.0 to 0.83.1, see this PR. We activated the dev container in this PR. Are there other steps we should have taken to support Apple Silicon / ensure the dev container image was only built for a single architecture?

[tschaffter]

@hallieswan Not sure why the Docker image you published has the extra architecture unknown/unknown but that's the source of the issue I experience. Until now the Docker image was only published for linux/amd64, which Apple Silicon can download successfully using emulation with Rosetta. I can enforce this behavior for the new image:

docker pull --platform linux/amd64 ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107

Without specifying the platform, I have the following error, which I don't have for images that I published for a single arch linux/amd64.

docker pull ghcr.io/sage-bionetworks/sage-monorepo-devcontainer:sha-b308a4e280519bfefb1095b9cf5f2fe38d558107
Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found

I recommend checking if the updated dev container CLI has changed its behavior. Maybe specifying the arch for the devcontainer build instead of letting Docker inferring it/them may solve the issue.

[hallieswan]

Good call -- the unknown/unknown entries are Buildkit Attestations. When we updated to the newer dev container CLI, a newer version of BuildX is used that enables these attestations by default. They are explicitly labeled unknown/unknown so container runtimes don't mistake metadata for executable layers.

Previously, the image was a single manifest, so when the image was pulled on a Mac, Docker saw exactly one option (amd64) and triggered the Rosetta fallback automatically. By adding attestations, the image is now a manifest list (OCI index). When the Docker client sees an index, it searchers for a native arm64 match. If it doesn't find one, it sees a mix of amd64 and unknown blobs and fails to elect the amd64 version for emulation, resulting in the no matching manifest error you're seeing.

We could attempt to fix by either:

1. Adding --platform linux/amd64 to the build command to explicitly pin the platform and see if that structures the index in a way that helps Docker Desktop's resolution logic. However, this may not solve the problem -- we'd have to test it
2. Opting out of attestations by passing --custom-flags "--provenance=false --sbom=false" (or setting BUILDX_NO_DEFAULT_ATTESTATIONS=1) to revert the image to a legacy single manifest

Do you have a preference on the approach?

[tschaffter]

Unless attaching the attestation as unknown/unknown is widely recommended (this looks like a hack to me), I would suggest Option 2. Note that the GitHub workflow I created already generates attestation and SBOM and pushes them to GHCR.

[hallieswan]

@tschaffter PR with the fix is here, if you'd like to take a look.

[hallieswan]

Re-opening as this will not be fully closed until we activate the new devcontainer image. However, there was an error while creating the SBOM attestation with Cosign when publishing the new image. @tschaffter is this something you've encountered before?

[tschaffter]

This may be a transient issue with Cosign.

[hallieswan]

This may be a transient issue with Cosign.

@tschaffter I've tried rerunning the job without success -- I'll investigate further next week.