perf(files_external): opt-in same-endpoint S3 MultipartCopy - #62673
Open
TobyTheHutt wants to merge 2 commits into
Open
perf(files_external): opt-in same-endpoint S3 MultipartCopy#62673TobyTheHutt wants to merge 2 commits into
TobyTheHutt wants to merge 2 commits into
Conversation
Fixes nextcloud#62645. Assisted-by: ClaudeCode:claude-opus-4-7 Signed-off-by: Tobias Harnickell <tobias.harnickell@bedag.ch>
TobyTheHutt
requested review from
leftybournes,
nfebe,
salmart-dev and
sorbaugh
and removed request for
a team
July 29, 2026 14:53
Assisted-by: ClaudeCode:claude-opus-4-7 Signed-off-by: Tobias Harnickell <tobias.harnickell@bedag.ch>
Member
|
It's a lot of new code 😬 |
Author
|
I know, sorry for that. Most of the diff is the S3 multipart protocol (Initiate, UploadPartCopy × N, Complete) plus its invariants (SSE-C rewrap, orphan-MPU cleanup, wrapper transparency). I had a hard time to trim it further without dropping invariants and risking silent data or key leaks. The tests account for about half the diff. If it helps the review, I could split them off into a spearate PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #62645.
Assisted-by: ClaudeCode:claude-opus-4-7
Summary
Add opt-in server-side
MultipartCopydispatch on theAmazonS3external-storage backend. Cross-mount MOVE and COPY between two mounts on the same S3 endpoint stop streaming object bytes through the PHP host. Behaviour on upgrade remains identical to master until an admin enables the flag.Enable:
occ config:app:set files_external amazons3_server_side_copy --value=true --type=booleanBehavioural matrix on upgrade
false(default)s3:GetObjecton sourcetrues3:GetObjecttrueS3Exception(403)per top-level op. One WARNING per op. Extra S3 round-trips. Streamed fallback completes. Failure limit caps log flood at four per request per endpoint.UploadPartCopy(Storj, GCS XML)trueRequired IAM on the destination-mount credential
Source bucket:
s3:GetObjects3:GetObjectVersionfor versioned sourceskms:Decryptfor SSE-KMS sourcesDestination bucket:
s3:PutObjects3:AbortMultipartUploads3:ListBucketMultipartUploadskms:GenerateDataKeyfor SSE-KMS destinationsGuards that force the streamed parent path
Encryption.AmazonS3.$preserveMtime === true.Checklist
apps/files_external/tests/Storage/. Gated byrun_cross_mount => trueinconfig.amazons3.php, provisioned by the updatedenv/start-amazons3-ceph.sh.occ config:app:list files_external.AI (if applicable)