feat(chunk-upload): Assemble DIFs exclusively to Objectstore - #120184
feat(chunk-upload): Assemble DIFs exclusively to Objectstore#120184lcian wants to merge 30 commits into
Conversation
19fa98b to
a23c245
Compare
a23c245 to
410a5ad
Compare
…tstore-debugfiles-assemble
…tstore-debugfiles-assemble
…tstore-debugfiles-assemble
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit e5d288f. Configure here.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 0c470f5. Configure here.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9dfd79f. Configure here.
This comment has been minimized.
This comment has been minimized.
…tstore-debugfiles-assemble
|
Superseded by #121065, where we continue writing |

Follow-up PR, and stacked on #120183.
This updates the chunk-upload/assemble flow to stop relying on
FileBlob/Filewhenorganizations:objectstore-debugfiles-exclusive-write, paving the way for a complete migration of DIFs.Legacy (non-chunked) DIF upload methods have been already addressed in #120183.
Under this new flag, chunk-upload works as follows:
assembleendpoint, we perform HEAD requests to Objectstore for all the chunks that belong to debug files to determine existence.To prevent race conditions where the file might be missing at the finalization step, files that expire in <= 1 hour are still reported as missing.
For all the missing chunks, we record an intent in Redis
chunk-upload-objectstore:{org-id}:{hash}, which allows us to tell which chunks need to go to Objectstore when they're uploaded viachunk-upload.chunk-uploadendpoint, if the intent is present in Redis, we additionally write that chunk to Objectstore at/chunk-upload/org={org.id}/{hash}with a 25 hours TTL (-1 hour for the check above = 1 day).Note that the chunks are still being uploaded as
FileBlobs, as they might be needed for other kinds of assembled files that are not DIFs, which currently don't support Objectstore as backend.The Redis intents serve to prevent the inverse from being true: only if a chunk is needed for DIFs then it will be uploaded to Objectstore.
assembletask, we spool the chunks from Objectstore to a temporay file on disk, then use that file validation with Symbolic and subsequent upload to Objectstore as the final materialized DIF, creating aProjectDebugFilerow with just thestorage_path.A new constant
MAX_OBJECTSTORE_DEBUG_FILE_SIZEis introduced, initially matching the existingMAX_DEBUG_FILE_SIZEbut ready to be increased or made into a feature flag to support larger debug files.Note: an additional Redis key
chunk-upload-objectstore-mode:{organization_id}with values0/1is introduced in an effort to make this a graceful rollout without failures for customers.The first DIF manifest writes this key with a 1 day TTL. If the feature flag is off, it pins that org to legacy
FileBlobassembly.If we don't do this, then it's possible that the flag gets enabled from an org between the first manifest request and the final assembly request. The
assembletask would then fail to find chunks in Objectstore, and fail with an unrecoverable error.After the flag is fully rolled out for 1 day, we can remove this mechanism and we need to make sure we clear the assemble status, so that the user can immediately retry the upload of a chunk that disappeared from Objectstore due to TTL (which should not happen normally, see point 1 above).
Close FS-365