Skip to content

feat: support enableTaskIDBasedBlobDigest for preheat job in Manager#4603

Open
dask-58 wants to merge 2 commits intodragonflyoss:mainfrom
dask-58:feat/preheat-taskid-digest
Open

feat: support enableTaskIDBasedBlobDigest for preheat job in Manager#4603
dask-58 wants to merge 2 commits intodragonflyoss:mainfrom
dask-58:feat/preheat-taskid-digest

Conversation

@dask-58
Copy link

@dask-58 dask-58 commented Feb 4, 2026

Description

Add enable_task_id_based_blob_digest field to Manager REST API for preheat jobs.
When enabled for OCI blob URLs (e.g., /v2/<name>/blobs/sha256:<digest>), the task ID is derived from the blob digest rather than the full URL. This enables deduplication across registries - the same blob from different registries shares one task ID, eliminating redundant downloads and storage.

Related Issue

Fixes #4590

Motivation and Context

The client.download_task gRPC API now supports enableTaskIDBasedBlobDigest. This PR adds the field to Manager REST API so the frontend Console can set it when creating preheat jobs. The default value is false (TEMPORARILY) to ensure backward compatibility making it an opt-in feature.

Screenshots (if appropriate)

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation Update (if none of the other choices apply)

Checklist

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

@gaius-qi gaius-qi added the enhancement New feature or request label Feb 4, 2026
@gaius-qi gaius-qi added this to the v2.5.0 milestone Feb 4, 2026
Signed-off-by: dask-58 <140686560+dask-58@users.noreply.github.com>
@dask-58 dask-58 force-pushed the feat/preheat-taskid-digest branch from f38550e to abb5cca Compare February 4, 2026 09:24
@codecov
Copy link

codecov bot commented Feb 4, 2026

Codecov Report

❌ Patch coverage is 20.00000% with 80 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.71%. Comparing base (ce173de) to head (36a99fc).

Files with missing lines Patch % Lines
scheduler/job/job.go 8.16% 45 Missing ⚠️
manager/job/preheat.go 0.00% 35 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #4603   +/-   ##
=======================================
  Coverage   28.70%   28.71%           
=======================================
  Files         231      231           
  Lines       23026    23036   +10     
=======================================
+ Hits         6609     6614    +5     
- Misses      15955    15960    +5     
  Partials      462      462           
Flag Coverage Δ
unittests 28.71% <20.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
internal/job/image.go 72.41% <100.00%> (+0.15%) ⬆️
manager/job/preheat.go 0.00% <0.00%> (ø)
scheduler/job/job.go 0.59% <8.16%> (+0.59%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: dask-58 <140686560+dask-58@users.noreply.github.com>
@dask-58
Copy link
Author

dask-58 commented Feb 5, 2026

The issue originally requested the default value of enable_task_id_based_blob_digest to be true. When I set the default to true, the E2E tests failed in the CI pipeline.

Specifically, the test:
[FAIL] Preheat with Manager ghcr.io/dragonflyoss/busybox:v1.35.0 image [It] preheat image should be ok

The cause of test failure:
The E2E tests in test/e2e/manager/preheat.go have hardcoded task IDs that were calculated with the old behavior (without enableTaskIDBasedBlobDigest). When the feature is enabled by default, task IDs are calculated differently (based on blob digest rather than full URL), causing a mismatch:

// test/e2e/manager/preheat.go:237-244
taskMetadatas := []util.TaskMetadata{
    {
        ID:     "a5040eb77de7f771cb3ce3ecb2ebb61af124d3341e0b5c6854b7e220eb0dc680",  // OLD task ID
        Sha256: "a711f05d33845e2e9deffcfcc5adf082d7c6e97e3e3a881d193d9aae38f092a8",
    },
    // ...
}

I confirmed this to be the correct cause by setting the default value to false and the pipeline checks succeeded.

Currently, the default is set to false as of the latest commit

What I think could be some potential solutions to enable true as default:

  • Calculate the new task IDs with enableTaskIDBasedBlobDigest=true and update the hardcoded values
  • Keep existing tests (with enableTaskIDBasedBlobDigest=false) and add new tests that explicitly use enableTaskIDBasedBlobDigest=true with the correct task IDs.
  • Make the E2E tests dynamically compute expected taskID ?

@gaius-qi Would appreciate any feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support enableTaskIDBasedBlobDigest for preheat job in Manager

5 participants