fix(s3): fix file deletion on versioned S3 buckets#673
Merged
Conversation
The S3 RemoveFile implementation had two bugs: 1. Dead legacy-fallback code: S3's DeleteObject API returns success even for non-existent objects, so the fallback from new to legacy object naming never triggered. 2. Versioned bucket deletion: On S3-compatible stores with versioning enabled (e.g. Backblaze B2), DeleteObject without a VersionID only creates a delete marker — the actual data persists indefinitely. Fix: introduce a removeObject helper that calls StatObject first to check existence (enabling the legacy fallback) and captures the VersionID, which is then passed to RemoveObject for permanent deletion. SSE options are forwarded to StatObject for SSE-C compatibility. Add per-backend storage-level integration tests for S3, GCS, and Swift that verify objects are truly removed after RemoveFile. These tests use t.Skip when PLIKD_CONFIG is not set, so they are transparent to make test but can be run on demand against real backends.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix S3
RemoveFileto properly delete objects from versioned S3-compatiblebuckets (e.g. Backblaze B2 with "Keep all versions" enabled).
Why
Two bugs in the previous implementation:
Dead legacy fallback:
DeleteObjectreturns success even for non-existentkeys, so the fallback from
{uploadID}.{fileID}to legacy{fileID}namingnever triggered.
Versioned buckets:
DeleteObjectwithout aVersionIDonly creates adelete marker — actual data persists indefinitely, silently accumulating
storage costs.
Changes
server/data/s3/s3.go— RewriteRemoveFilewith a newremoveObjecthelper that uses
StatObject→RemoveObject(VersionID)flow. Forwards SSEoptions for SSE-C compatibility.
server/data/s3/s3_integration_test.go— 5 new integration tests (newformat, legacy fallback, idempotent, not found, prefix)
server/data/gcs/gcs_integration_test.go— 3 new integration testsserver/data/swift/swift_integration_test.go— 3 new integration testsdocs/backends/data.md— Add versioning warning notetesting/ARCHITECTURE.md— Document per-backend integration testsTesting
make lint✅make test✅ (integration tests skip correctly withoutPLIKD_CONFIG)make docs✅make vuln✅TestCleanagainst B2 ✅plikd file delete --all→ bucket empty ✅