Skip to content

feat(storage): Add full object checksum validation for appendable uploads#16110

Open
v-pratap wants to merge 9 commits into
googleapis:mainfrom
v-pratap:fastbyte-full-object-checksum
Open

feat(storage): Add full object checksum validation for appendable uploads#16110
v-pratap wants to merge 9 commits into
googleapis:mainfrom
v-pratap:fastbyte-full-object-checksum

Conversation

@v-pratap
Copy link
Copy Markdown
Contributor

No description provided.

@v-pratap v-pratap requested review from a team as code owners May 21, 2026 09:00
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label May 21, 2026
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements dynamic CRC32C validation for appendable object uploads by updating the connection and writer logic to initialize hash functions from RPC response metadata. It also adds a stateful constructor to the CRC32C hash function. Feedback highlights a critical regression where a large suite of existing integration tests was deleted and replaced with experimental code. Additional suggestions include adding defensive checks for CRC32C field presence, utilizing existing library thread management instead of a custom thread pool in tests, and using environment variables instead of hardcoded strings for test configuration.

Comment thread google/cloud/storage/tests/async_client_integration_test.cc Outdated
Comment thread google/cloud/storage/internal/async/connection_impl.cc Outdated
Comment thread google/cloud/storage/tests/async_client_integration_test.cc Outdated
Comment thread google/cloud/storage/tests/async_client_integration_test.cc Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented May 21, 2026

Codecov Report

❌ Patch coverage is 91.59664% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.71%. Comparing base (a899f46) to head (8167423).

Files with missing lines Patch % Lines
...d/storage/internal/async/writer_connection_impl.cc 55.00% 9 Missing ⚠️
...le/cloud/storage/internal/async/connection_impl.cc 90.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16110      +/-   ##
==========================================
- Coverage   92.71%   92.71%   -0.01%     
==========================================
  Files        2353     2353              
  Lines      218986   219098     +112     
==========================================
+ Hits       203031   203129      +98     
- Misses      15955    15969      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


persisted_state_ = response->persisted_size();

if (response->has_persisted_data_checksums()) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible that: if a connection retry is triggered during the session, and the server's reported persisted_size causes a resend that overlaps with (but is not exactly aligned to) the client's previous chunk boundaries, the request fails?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the request will fail if we try to send overlapping data.
I think the server strictly requires the next write to start exactly at the persisted_size it has saved.

@v-pratap v-pratap force-pushed the fastbyte-full-object-checksum branch from 18389a0 to 58d4b1d Compare May 25, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants