Skip to content

feat(chunk-upload): Assemble DIFs exclusively to Objectstore - #120184

Closed
lcian wants to merge 30 commits into
ref/objectstore-debugfiles-dif-storagefrom
feat/objectstore-debugfiles-assemble
Closed

feat(chunk-upload): Assemble DIFs exclusively to Objectstore#120184
lcian wants to merge 30 commits into
ref/objectstore-debugfiles-dif-storagefrom
feat/objectstore-debugfiles-assemble

Conversation

@lcian

@lcian lcian commented Jul 21, 2026

Copy link
Copy Markdown
Member

Follow-up PR, and stacked on #120183.

This updates the chunk-upload/assemble flow to stop relying on FileBlob/File when organizations:objectstore-debugfiles-exclusive-write, paving the way for a complete migration of DIFs.
Legacy (non-chunked) DIF upload methods have been already addressed in #120183.

Under this new flag, chunk-upload works as follows:

  1. On the first call to the assemble endpoint, we perform HEAD requests to Objectstore for all the chunks that belong to debug files to determine existence.
    To prevent race conditions where the file might be missing at the finalization step, files that expire in <= 1 hour are still reported as missing.
    For all the missing chunks, we record an intent in Redis chunk-upload-objectstore:{org-id}:{hash}, which allows us to tell which chunks need to go to Objectstore when they're uploaded via chunk-upload.
  2. When a chunk is uploaded to the chunk-upload endpoint, if the intent is present in Redis, we additionally write that chunk to Objectstore at /chunk-upload/org={org.id}/{hash} with a 25 hours TTL (-1 hour for the check above = 1 day).
    Note that the chunks are still being uploaded as FileBlobs, as they might be needed for other kinds of assembled files that are not DIFs, which currently don't support Objectstore as backend.
    The Redis intents serve to prevent the inverse from being true: only if a chunk is needed for DIFs then it will be uploaded to Objectstore.
  3. In the assemble task, we spool the chunks from Objectstore to a temporay file on disk, then use that file validation with Symbolic and subsequent upload to Objectstore as the final materialized DIF, creating a ProjectDebugFile row with just the storage_path.

A new constant MAX_OBJECTSTORE_DEBUG_FILE_SIZE is introduced, initially matching the existing MAX_DEBUG_FILE_SIZE but ready to be increased or made into a feature flag to support larger debug files.

Note: an additional Redis key chunk-upload-objectstore-mode:{organization_id} with values 0/1 is introduced in an effort to make this a graceful rollout without failures for customers.
The first DIF manifest writes this key with a 1 day TTL. If the feature flag is off, it pins that org to legacy FileBlob assembly.
If we don't do this, then it's possible that the flag gets enabled from an org between the first manifest request and the final assembly request. The assemble task would then fail to find chunks in Objectstore, and fail with an unrecoverable error.
After the flag is fully rolled out for 1 day, we can remove this mechanism and we need to make sure we clear the assemble status, so that the user can immediately retry the upload of a chunk that disappeared from Objectstore due to TTL (which should not happen normally, see point 1 above).

Close FS-365

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 21, 2026
@lcian
lcian force-pushed the feat/objectstore-debugfiles-assemble branch from 19fa98b to a23c245 Compare July 21, 2026 10:17
@lcian
lcian force-pushed the feat/objectstore-debugfiles-assemble branch from a23c245 to 410a5ad Compare July 21, 2026 10:31
@lcian lcian changed the title feat(debug-files): assemble chunks into objectstore feat(difs): Assemble directly and exclusively to Objectstore Jul 29, 2026
@linear-code

linear-code Bot commented Jul 29, 2026

Copy link
Copy Markdown

FS-365

@lcian lcian changed the title feat(difs): Assemble directly and exclusively to Objectstore feat(difs): Assemble exclusively to Objectstore Jul 30, 2026
@lcian
lcian requested a review from a team July 30, 2026 10:09
@lcian
lcian marked this pull request as ready for review July 30, 2026 10:09
@lcian
lcian requested review from a team as code owners July 30, 2026 10:09
Comment thread src/sentry/api/endpoints/debug_files.py Outdated
@lcian

This comment has been minimized.

Comment thread src/sentry/api/endpoints/chunk.py Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit e5d288f. Configure here.

@lcian lcian changed the title feat(difs): Stage DIF chunks in Objectstore feat(difs): Assemble directly to Objectstore Jul 31, 2026
@lcian lcian changed the title feat(difs): Assemble directly to Objectstore feat(chunk-upload): Assemble DIFs exclusively to Objectstore Jul 31, 2026
@lcian

This comment has been minimized.

Comment thread src/sentry/chunk_upload.py Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 0c470f5. Configure here.

@lcian

This comment has been minimized.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9dfd79f. Configure here.

Comment thread src/sentry/chunk_upload.py Outdated
@sentry

This comment has been minimized.

lcian added 4 commits July 31, 2026 16:12
Keep the Objectstore-specific clone handling and coverage separate from chunk assembly changes.
@lcian
lcian marked this pull request as ready for review August 3, 2026 11:07
@lcian

lcian commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

Superseded by #121065, where we continue writing FileBlobs and assemble to Objectstore.
Even though the approach in this PR makes it easier to extend to other types of files, it only increases cost for no other benefit.

@lcian lcian closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants