Skip to content

Storage - Content Validation Public Interface#48074

Merged
ibrandes merged 21 commits into
Azure:feature/storage/content-validationfrom
ibrandes:content-validation/publicInterface
Feb 25, 2026
Merged

Storage - Content Validation Public Interface#48074
ibrandes merged 21 commits into
Azure:feature/storage/content-validationfrom
ibrandes:content-validation/publicInterface

Conversation

@ibrandes

@ibrandes ibrandes commented Feb 23, 2026

Copy link
Copy Markdown
Member

Content Validation and Options-Based APIs for Azure Storage Blob

Summary

This PR adds content validation (checksum) support across blob upload and download operations and introduces options-based overloads for append blob, page blob, and blob download APIs. Content validation is opt-in and will default to NONE. A new shared enum StorageChecksumAlgorithm in azure-storage-common drives both request-side (upload) and response-side (download) validation.


Content Validation (Checksum Support)

New type: StorageChecksumAlgorithm (azure-storage-common)

  • NONE – No checksum (default).
  • AUTO – SDK chooses (currently CRC64 where supported).
  • MD5 – Standard MD5 (e.g. Content-MD5 header) where the API supports it.
  • CRC64 – Azure Storage 64-bit CRC.

When set to AUTO, CRC64, or MD5 (where applicable), the SDK can compute and send checksums on upload and/or validate response payload checksums on download.

Upload (request checksum)

Request content validation is plumbed through options for:

  • Block blob: BlobParallelUploadOptions, BlobUploadFromFileOptions, BlockBlobOutputStreamOptions, BlockBlobSeekableByteChannelWriteOptions, BlockBlobSimpleUploadOptions, BlockBlobStageBlockOptions
  • Append blob: AppendBlobAppendBlockOptions, AppendBlobOutputStreamOptions
  • Page blob: PageBlobOutputStreamOptions, PageBlobUploadPagesOptions

BlobAsyncClient.uploadWithResponse (parallel upload) now passes requestChecksumAlgorithm from BlobParallelUploadOptions into the upload pipeline.

Download (response checksum)

Response content validation is plumbed through options for:

  • Blob download: BlobDownloadContentOptions, BlobDownloadStreamOptions, BlobDownloadToFileOptions
  • Stream / channel read: BlobInputStreamOptions, BlobSeekableByteChannelReadOptions

New Options Classes and Overloads

New options classes

Class Purpose
AppendBlobAppendBlockOptions Block data (InputStream or Flux + length), contentMd5, request conditions, requestChecksumAlgorithm
AppendBlobOutputStreamOptions Request conditions, requestChecksumAlgorithm for append blob output stream
BlobDownloadContentOptions Range, download retry, request conditions, range MD5, responseChecksumAlgorithm
BlobDownloadStreamOptions Same as above for stream-based download
PageBlobOutputStreamOptions Page range, request conditions, requestChecksumAlgorithm for page blob output stream
PageBlobUploadPagesOptions Page range, body (InputStream or Flux), contentMd5, request conditions, requestChecksumAlgorithm

New/updated API overloads

  • Append blob
    • appendBlockWithResponse(AppendBlobAppendBlockOptions) (sync and async); existing overloads delegate to this.
    • getBlobOutputStream(AppendBlobOutputStreamOptions) (sync).
  • Page blob
    • uploadPagesWithResponse(PageBlobUploadPagesOptions) (sync and async); existing overloads delegate to this.
    • getBlobOutputStream(PageBlobOutputStreamOptions) (sync).
  • Blob download
    • downloadStreamWithResponse(BlobDownloadStreamOptions) (sync and async); existing overloads delegate to this.
    • downloadContentWithResponse(BlobDownloadContentOptions) (sync and async); existing overloads delegate to this.

Existing methods that take individual parameters are unchanged in signature and now delegate to the options-based overloads, preserving backward compatibility.


Implementation Notes

  • Revapi: New internal methods (e.g. appendBlockWithResponseInternal, downloadStreamWithResponseInternal, uploadPagesWithResponseInternal) carry the checksum parameter; existing package-visible methods call into these with null for the checksum to avoid “visibility increased” issues.
  • BlobOutputStream: Append, block, and page blob output streams now accept and use requestChecksumAlgorithm from their respective options.
  • Error handling: A few async paths were updated to use monoError(LOGGER, ...) instead of Mono.error(...) for consistent logging.
  • Tests: Unused import removed in MessageEncoderTests (azure-storage-common).

Affected Packages

  • azure-storage-common: New StorageChecksumAlgorithm enum; minor test cleanup.
  • azure-storage-blob: All options and client changes above (Blob, Block, Append, Page clients and options).

@github-actions github-actions Bot added the Storage Storage Service (Queues, Blobs, Files) label Feb 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

API Change Check

APIView identified API level changes in this PR and created the following API reviews

com.azure:azure-storage-blob
com.azure:azure-storage-common

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request introduces public interfaces for content validation in Azure Storage operations by adding the StorageChecksumAlgorithm enum and integrating it into upload and download options across the Blob storage SDK.

Changes:

  • Introduces StorageChecksumAlgorithm enum with NONE, AUTO, MD5, and CRC64 options for content validation
  • Adds requestChecksumAlgorithm and responseChecksumAlgorithm fields to existing and new options classes
  • Creates new options classes (PageBlobUploadPagesOptions, AppendBlobAppendBlockOptions, BlobDownloadStreamOptions, BlobDownloadContentOptions, PageBlobOutputStreamOptions, AppendBlobOutputStreamOptions) to support checksum configuration
  • Refactors client methods to use new options objects and introduces "Internal" method variants to maintain backward compatibility while adding checksum functionality

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
StorageChecksumAlgorithm.java New enum defining checksum algorithms (NONE, AUTO, MD5, CRC64) with comprehensive documentation
Constants.java Adds REQUEST_CHECKSUM_ALGORITHM context key constant
PageBlobUploadPagesOptions.java New options class for page blob upload with Flux/InputStream constructors and checksum algorithm support
PageBlobOutputStreamOptions.java New options class for page blob output stream with checksum algorithm support
AppendBlobAppendBlockOptions.java New options class for append block with Flux/InputStream constructors and checksum algorithm support
AppendBlobOutputStreamOptions.java New options class for append blob output stream with checksum algorithm support
BlobDownloadStreamOptions.java New options class for download to stream with response checksum validation
BlobDownloadContentOptions.java New options class for download content with response checksum validation
BlockBlobStageBlockOptions.java Adds requestChecksumAlgorithm field and accessors
BlockBlobSimpleUploadOptions.java Adds requestChecksumAlgorithm field and accessors
BlockBlobSeekableByteChannelWriteOptions.java Adds requestChecksumAlgorithm field and accessors
BlockBlobOutputStreamOptions.java Adds requestChecksumAlgorithm field and accessors
BlobUploadFromFileOptions.java Adds requestChecksumAlgorithm field and accessors
BlobSeekableByteChannelReadOptions.java Adds responseChecksumAlgorithm field and accessors
BlobParallelUploadOptions.java Adds requestChecksumAlgorithm field and accessors
BlobInputStreamOptions.java Adds responseChecksumAlgorithm field and accessors
BlobDownloadToFileOptions.java Adds responseChecksumAlgorithm field and accessors
PageBlobClient.java Adds uploadPagesWithResponse(options) and getBlobOutputStream(options) methods; refactors existing methods to delegate to new implementations
PageBlobAsyncClient.java Adds new public methods and uploadPagesWithResponseInternal to support checksum while maintaining backward compatibility
AppendBlobClient.java Adds appendBlockWithResponse(options) and getBlobOutputStream(options) methods; removes unused imports; adds explicit cast for method disambiguation
AppendBlobAsyncClient.java Adds new public methods and appendBlockWithResponseInternal to support checksum while maintaining backward compatibility
BlobOutputStream.java Updates internal stream implementations to accept and use checksum algorithms for append, block, and page blobs
BlobClientBase.java Adds downloadStreamWithResponse(options) and downloadContentWithResponse(options) methods; refactors download methods to support response checksum validation; reorganizes downloadContent method location
BlobAsyncClientBase.java Adds downloadStreamWithResponse(options) and downloadContentWithResponse(options) methods; updates error handling to use monoError helper; adds downloadStreamWithResponseInternal for checksum support
BlobAsyncClient.java Updates uploadInChunks and uploadFullBlob methods to pass requestChecksumAlgorithm parameter
MessageEncoderTests.java Removes unused assertNotNull import

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 8 comments.

@ibrandes
ibrandes merged commit 0d10088 into Azure:feature/storage/content-validation Feb 25, 2026
20 checks passed
@ibrandes
ibrandes deleted the content-validation/publicInterface branch February 25, 2026 23:16
ibrandes added a commit that referenced this pull request Jun 23, 2026
* Crc polynomial (#43752)

* Message encoder (#43803)

* wip

* basic message encoder logic working

* removing print statements and making slightly more readable

* fixing several bugs

* adding more tests

* adding comments for building purposes and to branch off for service testing

* wip

* wip

* working encode that takes in bytebuffer

* removing redundant 'this' and making small readability changes

* fixing flag bug, adding more comments, adjusting incorrect test case

* adding structuredmessage package in implemenation and addressing other comments

* refactoring

* adding more tests, adjusting test encoder, adding more validation

* re-ordering empty buffer return

* adding correct comments

* addressing comments and fixing ci issues

* trying to resolve spotbugs issue

* formatting

* Extracting non-decoder changes from the decoder PR (#45546)

* creating class

* replacing encoder constants with new class constants

* adding fromValue to StructuredMessageFlags

* updating module info to contain more packages

* style

* undoing module info change

* Storage Content Validation - Encoder Performance Improvements (#47531)

* adjusting encoder logic

* adjusting tests to work with encoder

* addressing copilot comments

* adding more documentation

* Storage - Content Validation Public Interface (#48074)

* public facing interface wip

* adjusting output stream constructors

* removing redundant apis

* fixing pageblob api

* bloboutputstream adjustment

* wip

* wip

* importing things correctly

* formatting

* throwing error with client logger

* fixing revapi visibility increased error

* overload adjustments

* adding final to options bags and hopefully resolving revapi error

* more edits

* more edits

* more edits

* spotless

* spotless

* more edits

* removing javadoc again

* Storage - Content Validation Message Encoder Perf Updates (#47158)

* all perf adjustments

* addressing copilot comments

* Consolidate content validation types under contentvalidation package (#48272)

* Storage - Content Validation Encoder Pipeline Implementation (#48354)

* adding perf to message encoder tests

* changing package name

* changing package name

* moving checksum algorithm to new package

* wip

* moving storageschecksumalgorithm out of impl so it can actually be used

* removing unused file

* bruh

* small cleanups

* implementation for other APIs

* test updates

* undoing silly agent changes

* async tests

* perf tests for all APIs

* new path for crc64s without accessible array

* md5 compatibility tests

* progress reporter tests

* changing upload file test to stream file

* consolidating conflicting transactional checksum logic

* addressing copilot comments

* changing policy to compute crc64 header in a non-blocking manner

* fixing crc64 policy

* addressing copilot comments

* deleting perf tests

* more tests

* addressing more copilot comments

* adding recordings and marking multi part tests as live only due to the random block ids

* removing unused tests

* updating assets

* addressing github comments

* addressing comments

* renaming missed setters

* idk

* mode resolver cleanup

* adjusting mode resolver tests

* un-deprecating compute md5

* addressing API view comments

* addressing comments

* swapping test behavior for new md5 compatibility util

* wip

* adding validation to prevent progress listener being used with content val

* adding comment about flux.generate in encoder class

* adding recordings

* removing unused imports

* Storage - Content Validation Decoder Implementation (#47016)

* adding the StructuredMessageDecoder

* adding the pipeline policy changes

* smart retry changes

* fixing smart retry impl

* smart retry changes

* smart retry changes

* smart retry changes

* smart retry changes

* adding content validation tests

* code refactoring

* fixing errors i introduced :(

* addressing review comments

* code refactoring based on latest review comments

* code refactoring based on latest review comments

* simplifying retry mechanism

* removing dead code

* addressing Kyle's review comments

* addressing latest review comments

* refactoring based on latest review comments

* refactoring based on latest review comments

* refactoring based on latest review comments

* refactoring based on latest review comments from kyle

* expanding test coverage

* removing unused imports

* recordings

* adding documentation to decoder classes

* small fixes and failure path tests

* addressing context comment

* analyze error

* removing close override from decodedresponse

* line removal

---------

Co-authored-by: Isabelle <ibrandes@microsoft.com>

* Storage Content Validation - Defer MD5 + CRC64 Val Conflict to Service (#48999)

* impl and tests

* cleanup

* updating assets

* strengthening sync assertations

* Storage Content Validation - adding failure tests for content validation decoder (#49073)

* adding failure tests for content validation decoder

* adding Decoder Random Byte Failure Case Test

* rearrange code to make it a little more human readable

* fix linting

---------

Co-authored-by: browndav <browndav@microsoft.com>

* Storage Content Validation - Audit DecodedResponse (#49147)

* Audit DecodedResponse: tighten override surface, pin UTF-8, add unit tests

* Reduce DecodedResponse to minimal override surface

Strip overrides and tests that aren't required for transparent HttpResponse behavior, leaving exactly the compiler-required abstract methods plus one discretionary override that fixes a proven base-class bug.

DecodedResponse.java:
- Remove close() override. Current consumers fully drain the body Flux, so Reactor's onComplete signals release the underlying transport; the explicit close-forwarding was defensive against a try-with-resources / status-only pattern that no caller in the codebase actually uses.
- Collapse no-arg getBodyAsString() to delegate to the charset overload, pinning UTF-8 in one place instead of duplicating the lambda.
- Final override surface (8 methods): the 7 abstract methods on HttpResponse in azure-core 1.57.1 (compiler-required) plus getBodyAsBinaryData, which is concrete in the base but seeds BinaryData with the wire Content-Length header, making BinaryData.getLength() return the encoded size (frames + CRC trailers) instead of the decoded payload size.

DecodedResponseTests.java:
- Remove closeDelegatesToWrappedResponse and closeDoesNotSubscribeToDecodedBody (no override left to validate).
- Drop the close-counting assertion from the writeBodyTo test and rename it to inheritedWriteBodyToWritesDecodedBytes; it now uses MockHttpResponse directly.
- Strengthen getBodyAsBinaryDataReportsDecodedSizeNotContentLength: assert data.getLength() equals the decoded size (post-consumption). Without the override this returns 71 instead of 7, empirically reproduced.
- Drop now-unused trackingResponse helper, AtomicInteger import.

Tests: 10 of 10 passing. Each test maps 1:1 to one override or one inherited integration; no test is redundant.

* some code cleanup

* Address Copilot PR #49147 review comments

DecodedResponse.java (comment r3220562536):
- Restore HttpResponse#getBodyAsString() base contract by switching from unconditional UTF-8 to CoreUtils.bomAwareToString(bytes, contentType), matching BufferedHttpResponse's idiom. The previous UTF-8 pin silently dropped BOM detection and Content-Type charset honoring, which callers viewing this as a generic HttpResponse expect.

DecodedResponseTests.java (comment r3220562553):
- Add inheritedGetBodyAsBinaryDataReturnsDecodedBytes: exercises the inherited HttpResponse#getBodyAsBinaryData() and asserts the bytes match the decoded payload. Uses a divergent Content-Length header to make the wire vs decoded distinction explicit and guard against header-forwarding regressions.
- Add getBodyAsStringHonorsCharsetFromContentTypeHeader: proves the bom-aware fix actually honors a charset declared in Content-Type (would fail if the previous UTF-8 pinning were still in place).
- Add getBodyAsStringDetectsUtf8BomAndStripsIt: pins the BOM-detection arm of CoreUtils.bomAwareToString.
- Update getBodyAsStringDefaultsToUtf8WhenNoCharsetSpecified docstring to describe the new bom-aware fallback semantics rather than the old unconditional UTF-8 behavior.

Tests: 12 of 12 passing.

* some code cleanup

* analyze error

---------

Co-authored-by: Isabelle <ibrandes@microsoft.com>

* Storage Content Validation - Assert Request and Response Headers for Decoder Tests (#49075)

* add checks for response headers

* fix linting

* refactor async, change messageDownloadRequestHeaders signatures

* fix methods with wrong method names

* refactor sync

* remove unnecessary helper in BlobTestBase

* reuse hasStructuredMessageDownloadRequestHeaders with List<HttpHeaders>

* remove unused imports

* apply copilot suggestions

* fix linting

* inline local variables

* remove try-finally, replace with @tempdir

* add checks for response headers

* add recording for downloadStreamWithResponseContentValidationRange

* readd recording

* rerecord recordings

* Storage Content Validation - Verify Download Progress Listener Support (#49157)

* tests and blob client changes

* addressing copilot comments

* updating assets

* Storage Content Validation - Fuzzy Upload Tests (#49031)

* tests and assets

* addressing copilot comments

* making things actually fuzzy

* cleanup

* cleanup

* addressing copilot comments

* recordings

* resolving conflicts

* updating assets

* Storage Content Validation - adding fuzzy tests for content validation decoder (#49058)

* adding fuzzy tests for content validation decoder

* adding recordings

* adding few fuzzy tests

* randomizing fuzzy test payloads

* randomizing fuzzy test payloads

* fixing code review comments

* fix spotless formatting violation in BlobContentValidationAsyncDownloadTests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Local Merge <merge@local.tmp>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Storage - Content Validation Decoder Dynamic Segment Size Tests and Large GET Behavior Notes (#49172)

* tests for dynamic segment size encoding and decoding

* cleanup

* cleanup

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* style

* cant change pr name while ci is running i guess. no-op change

* cleaning up test file

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Storage - Decoder Perf Improvements (#49205)

* refactoring

* removing unused code

* reverting unecessary changes

* reverting unecessary changes

* reverting unecessary changes

* not pre-allocating entire byte array of segment size

* renaming variable

* adding back unecessary removals

* renaming variable

* addressing copilot comments

* Storage - Content validation content-length override (#49226)

* content length override changes

* content length override changes

* removing stuff

* addressing review comments

* addressing review comments

* addressing review comments

* resolving merge conflicts

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fixing spotbugs

* fixing spellchecks

---------

Co-authored-by: Local Merge <merge@local.tmp>
Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Storage - Content Validation Decoder Stress Tests (#49255)

* adding stress tests for content validation decoder

* fix TelemetryHelper, add tru to registerOberservers()

* Storage - Fix Flaky Stress Tests (#48359)

* removed enableDeterministic

* change .delete() to .deleteIfExists()

* remove Sinks.EmitFailureHandler.FAIL_FAST from CrcInputStream

- read functions had FAIL_FAST which would throw an error when  the stream had reached then end and we wanted to read from the stream again. So we removed from  both reads.
- refactor code so that the exit criteria is a tthe beginning
- refactor the emitContentInfo for dry

* prevent crashes on reattempted close on stream

- changed emitValue to tryEmitValue
- remove Sinks.EmitFailureHandler.FAIL_FAST so that multiple closes does not cause an error to be thrown

* fix telemetry so that it doesnt swallow errors

* roll back two deps because they were causing failures in the containers

- opentelemetry-runtime-telemetry-java8 from 2.24.0-alpha -> 2.15.0-alpha
- opentelemetry-logback-appender-1.0 from 2.24.0-alpha -> 2.15.0-alpha

* rollback azure-client-sdk-parent linting extensions from 1.0.0-beta.2 t0 beta.1

* revert linting extensions to beta2

* remove comments with old code

* add logging for errors

* remove catches for double close issue and okay status

* recursively delete files then delete the directory

* change to sync deletes, refactor for easier reading

* restructing share clean up so super calls only once

* incorporate copilot suggestions

* incorporate copilot suggestions

* incorporate copilot suggestions

* incorporate copilot suggestions

* fix all deletes to make sync and wrap in try-catch

* fix tests so that super.globalCleanupAsync() is only called once

* change telemetry to loggin only returns final state instead of failed retries when ultimately successful

* undo versio downgrade for linting-extensions

* Fixing spacing in error messages

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* refactor datalake delete all so that it is easier to read

* refactor runAsync in ShareScenarioBase so retry failures dont show as failures upon success

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* changes to cvdownload content, lazyload versus eagerloading

* update to TelemetryHelper based on previous cherry picked stress tests fixes

* Fix storage stress fault injector certificate trust

Export the fault injector certificate as PEM and wait for it before importing it into the Java truststore so storage stress tests fail fast instead of hitting PKIX errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* bump opentelemetry version

* add cv tests to App.class

* add script to delete resource groups

* update scenarios matrix with tests for cv

* fixing stress tests

* stageblocksmall scenario adjustment

* wip

* avoid depending on raw short baseName being available globally

* dep pinning

* reversion

* adding back

* wip

* storageseekablebytechannel change

* bare minimum changes

* removing perf core change

* removing redundant test file and non-blob package changes

* removing redundant test file and non-blob package changes

* renaming files and hardcoding crc64 option

* removing unused test

* removing unused imports

---------

Co-authored-by: browndav <browndav@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Local Merge <merge@local.tmp>
Co-authored-by: Isabelle <ibrandes@microsoft.com>

* Storage - Content Validation Encoder Stress Tests (#48979)

* Add encoder stress tests

* enabling fault injector

* restoring commitblocklist

* renaming files and hardcoding crc64 option

* adding scenarios back to app.java

---------

Co-authored-by: Rabab Ibrahim <ibrahimr@microsoft.com>
Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com>
Co-authored-by: Isabelle <ibrandes@microsoft.com>
Co-authored-by: browndav <browndav@microsoft.com>
Co-authored-by: Local Merge <merge@local.tmp>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants