Storage - Decoder Perf Improvements#49205
Merged
ibrandes merged 11 commits intoMay 20, 2026
Merged
Conversation
ibrandes
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors Storage structured-message decoding to emit validated payload as a sequence of ByteBuffers (rather than a single aggregated buffer), and updates the decoder policy and unit tests accordingly. This aligns the decoding path with streaming/backpressure considerations and reduces large intermediate allocations during CRC validation.
Changes:
- Refactors
StructuredMessageDecoder.decodeChunkto returnList<ByteBuffer>and emit per-chunk/per-segment validated payload buffers without consolidating into one array. - Updates
StorageContentValidationDecoderPolicyto stream decoder output viaFlux.fromIterable(...)and addslimitRate(1)to avoid upstream prefetch retaining multiple wire buffers. - Updates and extends decoder tests for the new return type and adds additional multi-segment CRC round-trip coverage.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| sdk/storage/azure-storage-common/src/main/java/com/azure/storage/common/implementation/contentvalidation/StructuredMessageDecoder.java | Changes decoder output contract to List<ByteBuffer> and switches segment buffering/emission strategy. |
| sdk/storage/azure-storage-common/src/main/java/com/azure/storage/common/policy/StorageContentValidationDecoderPolicy.java | Adapts policy to the new decoder output and adjusts reactive backpressure behavior (limitRate(1)). |
| sdk/storage/azure-storage-common/src/test/java/com/azure/storage/common/implementation/contentvalidation/StructuredMessageDecoderTests.java | Updates existing tests for list-based output, adds CRC multi-segment tests, and adds helper methods for collecting/writing decoded buffers. |
…om/Azure/azure-sdk-for-java into content-validation/decoderRefactoring
ibrandes
marked this pull request as ready for review
May 19, 2026 18:35
ibrandes
requested review from
a team,
alzimmermsft,
browndav-msft,
gunjansingh-msft,
kyleknap and
seanmcc-msft
as code owners
May 19, 2026 18:35
gunjansingh-msft
approved these changes
May 20, 2026
gunjansingh-msft
left a comment
gunjansingh-msft
left a comment
Member
There was a problem hiding this comment.
the perf numbers look good hence approving
ibrandes
merged commit May 20, 2026
09b8e15
into
Azure:feature/storage/content-validation
22 of 24 checks passed
ibrandes
added a commit
that referenced
this pull request
Jun 23, 2026
* Crc polynomial (#43752) * Message encoder (#43803) * wip * basic message encoder logic working * removing print statements and making slightly more readable * fixing several bugs * adding more tests * adding comments for building purposes and to branch off for service testing * wip * wip * working encode that takes in bytebuffer * removing redundant 'this' and making small readability changes * fixing flag bug, adding more comments, adjusting incorrect test case * adding structuredmessage package in implemenation and addressing other comments * refactoring * adding more tests, adjusting test encoder, adding more validation * re-ordering empty buffer return * adding correct comments * addressing comments and fixing ci issues * trying to resolve spotbugs issue * formatting * Extracting non-decoder changes from the decoder PR (#45546) * creating class * replacing encoder constants with new class constants * adding fromValue to StructuredMessageFlags * updating module info to contain more packages * style * undoing module info change * Storage Content Validation - Encoder Performance Improvements (#47531) * adjusting encoder logic * adjusting tests to work with encoder * addressing copilot comments * adding more documentation * Storage - Content Validation Public Interface (#48074) * public facing interface wip * adjusting output stream constructors * removing redundant apis * fixing pageblob api * bloboutputstream adjustment * wip * wip * importing things correctly * formatting * throwing error with client logger * fixing revapi visibility increased error * overload adjustments * adding final to options bags and hopefully resolving revapi error * more edits * more edits * more edits * spotless * spotless * more edits * removing javadoc again * Storage - Content Validation Message Encoder Perf Updates (#47158) * all perf adjustments * addressing copilot comments * Consolidate content validation types under contentvalidation package (#48272) * Storage - Content Validation Encoder Pipeline Implementation (#48354) * adding perf to message encoder tests * changing package name * changing package name * moving checksum algorithm to new package * wip * moving storageschecksumalgorithm out of impl so it can actually be used * removing unused file * bruh * small cleanups * implementation for other APIs * test updates * undoing silly agent changes * async tests * perf tests for all APIs * new path for crc64s without accessible array * md5 compatibility tests * progress reporter tests * changing upload file test to stream file * consolidating conflicting transactional checksum logic * addressing copilot comments * changing policy to compute crc64 header in a non-blocking manner * fixing crc64 policy * addressing copilot comments * deleting perf tests * more tests * addressing more copilot comments * adding recordings and marking multi part tests as live only due to the random block ids * removing unused tests * updating assets * addressing github comments * addressing comments * renaming missed setters * idk * mode resolver cleanup * adjusting mode resolver tests * un-deprecating compute md5 * addressing API view comments * addressing comments * swapping test behavior for new md5 compatibility util * wip * adding validation to prevent progress listener being used with content val * adding comment about flux.generate in encoder class * adding recordings * removing unused imports * Storage - Content Validation Decoder Implementation (#47016) * adding the StructuredMessageDecoder * adding the pipeline policy changes * smart retry changes * fixing smart retry impl * smart retry changes * smart retry changes * smart retry changes * smart retry changes * adding content validation tests * code refactoring * fixing errors i introduced :( * addressing review comments * code refactoring based on latest review comments * code refactoring based on latest review comments * simplifying retry mechanism * removing dead code * addressing Kyle's review comments * addressing latest review comments * refactoring based on latest review comments * refactoring based on latest review comments * refactoring based on latest review comments * refactoring based on latest review comments from kyle * expanding test coverage * removing unused imports * recordings * adding documentation to decoder classes * small fixes and failure path tests * addressing context comment * analyze error * removing close override from decodedresponse * line removal --------- Co-authored-by: Isabelle <ibrandes@microsoft.com> * Storage Content Validation - Defer MD5 + CRC64 Val Conflict to Service (#48999) * impl and tests * cleanup * updating assets * strengthening sync assertations * Storage Content Validation - adding failure tests for content validation decoder (#49073) * adding failure tests for content validation decoder * adding Decoder Random Byte Failure Case Test * rearrange code to make it a little more human readable * fix linting --------- Co-authored-by: browndav <browndav@microsoft.com> * Storage Content Validation - Audit DecodedResponse (#49147) * Audit DecodedResponse: tighten override surface, pin UTF-8, add unit tests * Reduce DecodedResponse to minimal override surface Strip overrides and tests that aren't required for transparent HttpResponse behavior, leaving exactly the compiler-required abstract methods plus one discretionary override that fixes a proven base-class bug. DecodedResponse.java: - Remove close() override. Current consumers fully drain the body Flux, so Reactor's onComplete signals release the underlying transport; the explicit close-forwarding was defensive against a try-with-resources / status-only pattern that no caller in the codebase actually uses. - Collapse no-arg getBodyAsString() to delegate to the charset overload, pinning UTF-8 in one place instead of duplicating the lambda. - Final override surface (8 methods): the 7 abstract methods on HttpResponse in azure-core 1.57.1 (compiler-required) plus getBodyAsBinaryData, which is concrete in the base but seeds BinaryData with the wire Content-Length header, making BinaryData.getLength() return the encoded size (frames + CRC trailers) instead of the decoded payload size. DecodedResponseTests.java: - Remove closeDelegatesToWrappedResponse and closeDoesNotSubscribeToDecodedBody (no override left to validate). - Drop the close-counting assertion from the writeBodyTo test and rename it to inheritedWriteBodyToWritesDecodedBytes; it now uses MockHttpResponse directly. - Strengthen getBodyAsBinaryDataReportsDecodedSizeNotContentLength: assert data.getLength() equals the decoded size (post-consumption). Without the override this returns 71 instead of 7, empirically reproduced. - Drop now-unused trackingResponse helper, AtomicInteger import. Tests: 10 of 10 passing. Each test maps 1:1 to one override or one inherited integration; no test is redundant. * some code cleanup * Address Copilot PR #49147 review comments DecodedResponse.java (comment r3220562536): - Restore HttpResponse#getBodyAsString() base contract by switching from unconditional UTF-8 to CoreUtils.bomAwareToString(bytes, contentType), matching BufferedHttpResponse's idiom. The previous UTF-8 pin silently dropped BOM detection and Content-Type charset honoring, which callers viewing this as a generic HttpResponse expect. DecodedResponseTests.java (comment r3220562553): - Add inheritedGetBodyAsBinaryDataReturnsDecodedBytes: exercises the inherited HttpResponse#getBodyAsBinaryData() and asserts the bytes match the decoded payload. Uses a divergent Content-Length header to make the wire vs decoded distinction explicit and guard against header-forwarding regressions. - Add getBodyAsStringHonorsCharsetFromContentTypeHeader: proves the bom-aware fix actually honors a charset declared in Content-Type (would fail if the previous UTF-8 pinning were still in place). - Add getBodyAsStringDetectsUtf8BomAndStripsIt: pins the BOM-detection arm of CoreUtils.bomAwareToString. - Update getBodyAsStringDefaultsToUtf8WhenNoCharsetSpecified docstring to describe the new bom-aware fallback semantics rather than the old unconditional UTF-8 behavior. Tests: 12 of 12 passing. * some code cleanup * analyze error --------- Co-authored-by: Isabelle <ibrandes@microsoft.com> * Storage Content Validation - Assert Request and Response Headers for Decoder Tests (#49075) * add checks for response headers * fix linting * refactor async, change messageDownloadRequestHeaders signatures * fix methods with wrong method names * refactor sync * remove unnecessary helper in BlobTestBase * reuse hasStructuredMessageDownloadRequestHeaders with List<HttpHeaders> * remove unused imports * apply copilot suggestions * fix linting * inline local variables * remove try-finally, replace with @tempdir * add checks for response headers * add recording for downloadStreamWithResponseContentValidationRange * readd recording * rerecord recordings * Storage Content Validation - Verify Download Progress Listener Support (#49157) * tests and blob client changes * addressing copilot comments * updating assets * Storage Content Validation - Fuzzy Upload Tests (#49031) * tests and assets * addressing copilot comments * making things actually fuzzy * cleanup * cleanup * addressing copilot comments * recordings * resolving conflicts * updating assets * Storage Content Validation - adding fuzzy tests for content validation decoder (#49058) * adding fuzzy tests for content validation decoder * adding recordings * adding few fuzzy tests * randomizing fuzzy test payloads * randomizing fuzzy test payloads * fixing code review comments * fix spotless formatting violation in BlobContentValidationAsyncDownloadTests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Storage - Content Validation Decoder Dynamic Segment Size Tests and Large GET Behavior Notes (#49172) * tests for dynamic segment size encoding and decoding * cleanup * cleanup * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * style * cant change pr name while ci is running i guess. no-op change * cleaning up test file --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Storage - Decoder Perf Improvements (#49205) * refactoring * removing unused code * reverting unecessary changes * reverting unecessary changes * reverting unecessary changes * not pre-allocating entire byte array of segment size * renaming variable * adding back unecessary removals * renaming variable * addressing copilot comments * Storage - Content validation content-length override (#49226) * content length override changes * content length override changes * removing stuff * addressing review comments * addressing review comments * addressing review comments * resolving merge conflicts * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fixing spotbugs * fixing spellchecks --------- Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Storage - Content Validation Decoder Stress Tests (#49255) * adding stress tests for content validation decoder * fix TelemetryHelper, add tru to registerOberservers() * Storage - Fix Flaky Stress Tests (#48359) * removed enableDeterministic * change .delete() to .deleteIfExists() * remove Sinks.EmitFailureHandler.FAIL_FAST from CrcInputStream - read functions had FAIL_FAST which would throw an error when the stream had reached then end and we wanted to read from the stream again. So we removed from both reads. - refactor code so that the exit criteria is a tthe beginning - refactor the emitContentInfo for dry * prevent crashes on reattempted close on stream - changed emitValue to tryEmitValue - remove Sinks.EmitFailureHandler.FAIL_FAST so that multiple closes does not cause an error to be thrown * fix telemetry so that it doesnt swallow errors * roll back two deps because they were causing failures in the containers - opentelemetry-runtime-telemetry-java8 from 2.24.0-alpha -> 2.15.0-alpha - opentelemetry-logback-appender-1.0 from 2.24.0-alpha -> 2.15.0-alpha * rollback azure-client-sdk-parent linting extensions from 1.0.0-beta.2 t0 beta.1 * revert linting extensions to beta2 * remove comments with old code * add logging for errors * remove catches for double close issue and okay status * recursively delete files then delete the directory * change to sync deletes, refactor for easier reading * restructing share clean up so super calls only once * incorporate copilot suggestions * incorporate copilot suggestions * incorporate copilot suggestions * incorporate copilot suggestions * fix all deletes to make sync and wrap in try-catch * fix tests so that super.globalCleanupAsync() is only called once * change telemetry to loggin only returns final state instead of failed retries when ultimately successful * undo versio downgrade for linting-extensions * Fixing spacing in error messages Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * refactor datalake delete all so that it is easier to read * refactor runAsync in ShareScenarioBase so retry failures dont show as failures upon success --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * changes to cvdownload content, lazyload versus eagerloading * update to TelemetryHelper based on previous cherry picked stress tests fixes * Fix storage stress fault injector certificate trust Export the fault injector certificate as PEM and wait for it before importing it into the Java truststore so storage stress tests fail fast instead of hitting PKIX errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * bump opentelemetry version * add cv tests to App.class * add script to delete resource groups * update scenarios matrix with tests for cv * fixing stress tests * stageblocksmall scenario adjustment * wip * avoid depending on raw short baseName being available globally * dep pinning * reversion * adding back * wip * storageseekablebytechannel change * bare minimum changes * removing perf core change * removing redundant test file and non-blob package changes * removing redundant test file and non-blob package changes * renaming files and hardcoding crc64 option * removing unused test * removing unused imports --------- Co-authored-by: browndav <browndav@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Isabelle <ibrandes@microsoft.com> * Storage - Content Validation Encoder Stress Tests (#48979) * Add encoder stress tests * enabling fault injector * restoring commitblocklist * renaming files and hardcoding crc64 option * adding scenarios back to app.java --------- Co-authored-by: Rabab Ibrahim <ibrahimr@microsoft.com> Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com> Co-authored-by: Isabelle <ibrandes@microsoft.com> Co-authored-by: browndav <browndav@microsoft.com> Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves structured-message download decode performance and memory behavior by decoding one wire buffer at a time and emitting validated payload without consolidating entire segments into a single growing buffer.
Memory and allocation
ByteArrayOutputStreamwith aList<byte[]>that holds one copy per inbound chunk. Retained memory grows with bytes received, not with the full declared segment size, which avoids large heap spikes on multi‑MiB segments.ByteBuffer.wrap(chunk)views instead ofwriteTointo anotherByteArrayOutputStreamand a finaltoByteArray()copy.singleSegmentEmissionIsExactSized), so downstream consumers do not see oversized arrays from a growing buffer.Pipeline backpressure (download path)
limitRate(1)on the encoded flux — Aligns download decoding with the upload policy: process one network buffer at a time so the decoder can copy only the current chunk, release the inbound buffer, and limit concurrent in-flight segment payload before the next chunk arrives.API shape (supports the above)
decodeChunkreturnsList<ByteBuffer>— Empty list means “no validated bytes yet”; multiple buffers when a segment was fed across several wire chunks. The decoder policy emits them viaFlux.fromIterablewithout an extra merge step.Correctness coverage
Tables
Before
downloadStreamWithResponsedownloadContentWithResponsedownloadToFileWithResponseAfter
downloadStreamWithResponsedownloadContentWithResponsedownloadToFileWithResponse