Image pull failed precondition due to sha256 digest mismatch

[lixiv]

Bug report:

$ crictl pull aaa.service.cloud.com/repd:123
E0205 12:14:58.427953 148988 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = FailedPrecondition desc = failed to pull and unpack image "aaa.service.cloud.com/repd:123": failed commit on ref "layer-sha256:ad7a11201ef64e11a6dbef61b018687b4d4653e1b94bc2166f1e4cce702b5072": unexpected commit digest sha256:0cd58cd550968e0296f86d854171830dff486223adcc19af00ebf1c129dd1eeb, expected sha256:ad7a11201ef64e11a6dbef61b018687b4d4653e1b94bc2166f1e4cce702b5072: failed precondition" image="aaa.service.cloud.com/repd:123"
FATA[0003] pulling image: rpc error: code = FailedPrecondition desc = failed to pull and unpack image "aaa.service.cloud.com/repd:123": failed commit on ref "layer-sha256:ad7a11201ef64e11a6dbef61b018687b4d4653e1b94bc2166f1e4cce702b5072": unexpected commit digest sha256:0cd58cd550968e0296f86d854171830dff486223adcc19af00ebf1c129dd1eeb, expected sha256:ad7a11201ef64e11a6dbef61b018687b4d4653e1b94bc2166f1e4cce702b5072: failed precondition

Expected behavior:

pull image success without the error

How to reproduce it:

The image is about 25GB, when many pods pulling the image from repo simultaneously, the image repository experiences rate limiting.

repo server logs:

[2026-02-05 10:36:01] [ERROR] Download speed is too low! (1.23 MB/s < 20 MB/s)

some dfdaemon error logs:

  2026-02-05T04:10:13.404818222+00:00 ERROR  failed to serve connection from 127.0.0.1:59778: connection closed before message completed
    at dragonfly-client/src/proxy/mod.rs:219

  2026-02-05T04:10:13.404836312+00:00 ERROR  failed to serve connection from 127.0.0.1:59766: connection closed before message completed
    at dragonfly-client/src/proxy/mod.rs:219

  2026-02-05T04:10:13.502845484+00:00 ERROR  download piece reader error: broken pipe
    at dragonfly-client/src/proxy/mod.rs:827
    in proxy_via_dfdaemon with host_id: "xxx", task_id: "4d02a4274ce087f926eacbdd60f80b9d2315c7afb3aab03ae9ac28a79db978f7", peer_id: "xxx"
    in http_handler
    in registry_mirror_http_handler
    in handler

  2026-02-05T04:10:13.502865085+00:00 ERROR  writer shutdown error: broken pipe
    at dragonfly-client/src/proxy/mod.rs:829
    in proxy_via_dfdaemon with host_id: "xxx", task_id: "4d02a4274ce087f926eacbdd60f80b9d2315c7afb3aab03ae9ac28a79db978f7", peer_id: "xxx"
    in http_handler
    in registry_mirror_http_handler
    in handler

  2026-02-05T04:10:13.551254503+00:00 ERROR  send DownloadPieceFinishedResponse for piece def12a3cb7f35e828d08714881486547f3ce295e65316ad96cc0c59bacfe33e3-4 failed: "Closed(..)"
    at dragonfly-client/src/resource/task.rs:1421
    in download_partial_with_scheduler_from_source
    in download_partial_with_scheduler
    in download
    in download_task with host_id: "xxx", task_id: "def12a3cb7f35e828d08714881486547f3ce295e65316ad96cc0c59bacfe33e3", peer_id: "xxx", url: "xxx/v2/ccc/aaa/blobs/sha256:035202c98163062751b4cd0fc7439da80b98d2bfd6fb4318031e0419cf4a03ef", content_length: 23361798

  2026-02-05T04:10:27.626976494+00:00 ERROR  failed to serve connection from 127.0.0.1:26148: error writing a body to connection
    at dragonfly-client/src/proxy/mod.rs:219

  2026-02-05T04:10:27.627002656+00:00 ERROR  download piece reader error: broken pipe
    at dragonfly-client/src/proxy/mod.rs:827
    in proxy_via_dfdaemon with host_id: "xxx", task_id: "def12a3cb7f35e828d08714881486547f3ce295e65316ad96cc0c59bacfe33e3", peer_id: "xxx"
    in http_handler
    in registry_mirror_http_handler
    in handler

  2026-02-05T04:10:27.627021268+00:00 ERROR  writer shutdown error: broken pipe
    at dragonfly-client/src/proxy/mod.rs:829
    in proxy_via_dfdaemon with host_id: "xxx", task_id: "def12a3cb7f35e828d08714881486547f3ce295e65316ad96cc0c59bacfe33e3", peer_id: "xxx"
    in http_handler
    in registry_mirror_http_handler
    in handler

logs filterd by task_id:
kubectl -n dragonfly logs dfdaemon-4d | grep def12a3cb7f35e828d08714881486547f3ce295e65316ad96cc0c59bacfe33e3 -C10

dfdaemon.log

Environment:

• Dragonfly version: dfdaemon: v0.2.36; manager: v2.2.2

[gaius-qi]

@lixiv Please provide the log files in the /var/log/dragonfly/dfdaemon/*.log.

[lixiv]

@lixiv Please provide the log files in the /var/log/dragonfly/dfdaemon/*.log.

@gaius-qi To recover from image pulling failure, the dfdaemon pod was deleted and recreated, and the logs /var/log/dragonfly/dfdaemon/*.log weren't saved. I'm trying to see if I can reproduce the problem.
The pod's standard output logs and the logs in /var/log/dragonfly/dfdaemon/*.log seems similar in content, except differences in format. Won't this help in locating the issue?

[lixiv]

@gaius-qi There's a digest check in handle_downloaded_piece_from_parent_finished ： https://github.com/dragonflyoss/client/blob/cd2847b03aae1646d348d73533104b3b1d0196b8/dragonfly-client-storage/src/lib.rs#L830-L841
when download from source or in handle_downloaded_from_source_finished , Is there a way to also check the expected digest? https://github.com/dragonflyoss/client/blob/cd2847b03aae1646d348d73533104b3b1d0196b8/dragonfly-client-storage/src/lib.rs#L775

[lixiv]

@gaius-qi #2818 (comment) indicates the download action on the peer node may have unexpected dirty cache. This issue can be one kind of dirty cache. And I tried the clean job, it worked, but there are lots of tasks needed to be cleaned. Are there any mechanisms to prevent or clear it automatically?