Fix ZstandardStream truncating multi-frame zstd responses to the first frame

[christosk92]

Fixes #129038.

Problem

A zstd stream can be several frames concatenated back to back (RFC 8878 §3). Many encoders and CDNs emit one frame per buffer, so a large Content-Encoding: zstd response often arrives as multiple frames. ZstandardStream only decoded the first frame and dropped the rest, which surfaced downstream as truncated data. In the linked issue, System.Text.Json fails at exactly BytePositionInLine: 65536.

The cause is in ZstandardDecoder.Decompress:

if (result == 0) { _finished = true; return OperationStatus.Done; }

A 0 return from ZSTD_decompressStream means the current frame is done, not that the whole stream is done (lib/zstd.h: "0 when a frame is completely decoded and fully flushed"). Because _finished is a one-way latch and Decompress returns early once it is set, finishing the first frame permanently stops decoding and every later frame is discarded.

Fix

ZstandardStream now continues into the next frame instead of stopping at the first. This follows the pattern GZipStream and DeflateStream already use for concatenated members, adapted to zstd.

• TryDecompress decodes a single frame and returns when the decoder reports Done. The frame-to-frame loop now lives in Read and ReadAsync, with a single AdvanceToNextFrame(bool async, ...) helper handling the boundary for both.
• At a frame boundary, AdvanceToNextFrame reads up to the next 4-byte frame magic and feeds exactly those bytes to the decoder through the public ZstandardDecoder.Decompress (after a session-only Reset()). A valid magic comes back as NeedMoreData, so the stream discards the magic from its buffer and keeps decoding; anything else is trailing data after the final frame. Feeding only the magic (not the whole buffer) means a frame whose magic is valid but whose body is corrupt is not mistaken for trailing data: the magic is accepted and the corrupt body is rejected by the following decode.
• When the boundary check decides the stream is finished it sets _endOfStream, so later reads return 0 without re-entering the decoder (which may have been left in an error state by feeding it non-magic bytes). On a seekable stream the position is rewound so trailing data is still readable by the caller.
• Reset() is ZSTD_DCtx_reset in session-only mode, so window size and any referenced dictionary carry over to the next frame. It also releases the single-use prefix, which is only valid for one frame anyway.
• An empty frame (one that produces no output) is no longer mistaken for end of stream, so an empty leading or middle frame doesn't cut decoding short.
• End of input counts as a clean end only when the last frame finished on a frame boundary, so the existing truncation check still fires for a genuinely truncated frame.

The one-shot helpers were already correct across frames (ZstandardDecoder.TryDecompress calls ZSTD_decompress and TryGetMaxDecompressedLength calls ZSTD_decompressBound, both multi-frame aware), so they are unchanged.

Tests

Added to CompressionStreamUnitTests.Zstandard.cs, each run sync and async:

• ZstandardStream_ConcatenatedFrames_DecompressesAllFrames: two concatenated frames, asserts the whole payload comes back (only the first frame decoded before the fix).
• ZstandardStream_ConcatenatedFrames_AcrossReads_DecompressesAllFrames: a non-seekable stream that returns one byte per read, so a frame magic can land split across reads. Covers the HttpClient style path with no rewind.
• ZstandardStream_EmptyFramesAmongFrames_DecompressesAllFrames: empty frames mixed in with real ones, to confirm an empty frame isn't read as end of stream.
• ZstandardStream_FrameFollowedByShortTrailingData_StopsAtEndOfFrame: a frame followed by 1 to 3 trailing bytes (shorter than the magic), checks the stream ends cleanly and the trailing bytes stay available on the base stream.
• ZstandardStream_FrameFollowedByCorruptFrame_Throws: a frame followed by a second frame with a valid magic but a corrupt body, asserts InvalidDataException so a corrupt continuation isn't swallowed as trailing data.
• ZstandardStream_SkippableFrameBetweenFrames_DecompressesAllFrames: a skippable frame between two real frames, asserts both real frames decode.

Existing tests, including StreamTruncation_IsDetected, still pass.

Notes

Next-frame detection feeds the 4 boundary bytes straight to the decoder rather than recognizing the magic in managed code: a valid frame magic (standard or skippable) comes back as OperationStatus.NeedMoreData, anything else as InvalidData. This keeps the managed side out of knowing the frame-format magics, and a corrupt frame body is still surfaced as InvalidDataException by the subsequent decode rather than being swallowed as trailing data.

Repro: https://github.com/christosk92/zstd-net11-repro

[dotnet-policy-service]

Tagging subscribers to this area: @karelz, @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates Zstandard decompression to correctly handle multiple concatenated zstd frames (common in HTTP Content-Encoding: zstd) without truncation, and adds regression coverage for multi-frame and trailing-data scenarios.

Changes:

• Extend ZstandardStream decompression to continue decoding across concatenated frames and distinguish frames vs. trailing non-zstd data.
• Add tests to verify full output for concatenated frames (including across fragmented reads) and correct handling of trailing data after the final frame.
• Add decoder support method to clear managed end-of-frame state between frames.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/libraries/System.IO.Compression/tests/Zstandard/CompressionStreamUnitTests.Zstandard.cs	Adds regression tests for concatenated frames, across-read boundaries, and trailing-data handling.
src/libraries/System.IO.Compression/src/System/IO/Compression/Zstandard/ZstandardStream.Decompress.cs	Implements multi-frame decoding logic and adjusts truncation detection at frame boundaries.
src/libraries/System.IO.Compression/src/System/IO/Compression/Zstandard/ZstandardDecoder.cs	Adds PrepareForNextFrame to clear managed end-of-frame state for concatenated streams.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[christosk92]

@dotnet-policy-service agree

[rzikm]

LGTM in general, left some comments.

Since multiple concatenated frames are actually common, I am not sure it makes sense to do hard stop in ZstandardDecoder at the end of the frame anymore, but since we have Reset() then users can just call that to get the decoder ready for the next frame, and it becomes just a matter of documentation.

[christosk92]

makes sense. plan from the comments:

• decoder: drop PrepareForNextFrame(), use Reset(). it's session_only so the dict + windowLogMax still carry over, and it unpins the single-use prefix the old path was leaking.
• stream: move the frame-to-frame loop up into Read/ReadAsync (one frame per TryDecompress, Reset() at the boundary), and handle the 0-byte-frame case so an empty frame doesn't read as EOF. adding a test for that.
• keep the per-frame stop in the public decoder and just document it (Done is per-frame, can be 0 bytes; Reset() readies it for the next one).
• tests: drop the trailing-data test (the inherited rewind test covers it now), keep the short-trailing one, keep the big-payload concat test but retarget its comment.
• plus the small stuff: = 4, and pulling the issue links / "previously" wording out of the test comments.

i'll push these as a few separate commits.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[MihaZupan]

Since multiple concatenated frames are actually common, I am not sure it makes sense to do hard stop in ZstandardDecoder at the end of the frame anymore, but since we have Reset() then users can just call that to get the decoder ready for the next frame, and it becomes just a matter of documentation.

Is there any risk with moving the logic into the decoder so that dealing with such streams "just works" there as well?
Seems like it might be a commonly encountered issue

[rzikm]

Is there any risk with moving the logic into the decoder so that dealing with such streams "just works" there as well?
Seems like it might be a commonly encountered issue.

I am not sure how that would work, what if the frame happens to end at the end of supplied buffer, do we report Done or NeedMoreData (since we don't know if another frame follows)? And then we can get into two weird situations:

• reporting NeedMoreData when user provided the entire input and nothing extra
• if there is garbage data supplied later, then would we report Done without consuming anything.

If we stick to returning Done at the end of the frame, then the only change we can do at the Decoder level is not requiring Reset before processing next frame, and I don't think that's that big of a deal and I would prefer consistency with other Decoder classes.

But this actually brings up another idea: instead of detecting the zstd headers in managed code, we could just call Decompress on the bytes that follow and watch out for OperationStatus.InvalidData. If we get InvalidData on the first call on the new frame, then it is not actually a zstd frame and we can stop/rewind there. That would minimize the amount of logic we have in Managed code while still addressing the bug. @christosk92 would you be willing to give this approach a try?

[christosk92]

Is there any risk with moving the logic into the decoder so that dealing with such streams "just works" there as well?
Seems like it might be a commonly encountered issue.

I am not sure how that would work, what if the frame happens to end at the end of supplied buffer, do we report Done or NeedMoreData (since we don't know if another frame follows)? And then we can get into two weird situations:

• reporting NeedMoreData when user provided the entire input and nothing extra
• if there is garbage data supplied later, then would we report Done without consuming anything.

If we stick to returning Done at the end of the frame, then the only change we can do at the Decoder level is not requiring Reset before processing next frame, and I don't think that's that big of a deal and I would prefer consistency with other Decoder classes.

But this actually brings up another idea: instead of detecting the zstd headers in managed code, we could just call Decompress on the bytes that follow and watch out for OperationStatus.InvalidData. If we get InvalidData on the first call on the new frame, then it is not actually a zstd frame and we can stop/rewind there. That would minimize the amount of logic we have in Managed code while still addressing the bug. @christosk92 would you be willing to give this approach a try?

Hi,

Thanks for the replies. I was busy the entire week so I haven't been able to take a look at the comments. I'll take a look at the suggested changes over the weekend.

Happy coding

[rzikm]

Hey, @christosk92, gentle ping here, did you have time to take a look on the review comments?

[christosk92]

Hey, @christosk92, gentle ping here, did you have time to take a look on the review comments?

unfortunately not, im sorry. I will take a look at them this evening so expect an update tmrw.

[christosk92]

fwiw this is exactly what the gzip path we're mirroring does: after a member ends, Inflater.ResetStreamForLeftoverInput peeks the next bytes for the gzip header id (0x1f 0x8b) and only continues if they match, otherwise it treats the leftover as trailing data and stops.

and because it keys off the magic, a trailing chunk that does start like a member but is corrupt still gets decoded and throws, rather than being silently dropped. that's the throw-vs-rewind split i want to keep for zstd, and dropping the magic check loses it.

[christosk92]

Since multiple concatenated frames are actually common, I am not sure it makes sense to do hard stop in ZstandardDecoder at the end of the frame anymore, but since we have Reset() then users can just call that to get the decoder ready for the next frame, and it becomes just a matter of documentation.

Is there any risk with moving the logic into the decoder so that dealing with such streams "just works" there as well? Seems like it might be a commonly encountered issue

@MihaZupan yeah it's a common one, that's basically why i opened this. for the streaming case (ZstandardStream) it does just work after this change, which is the scenario people actually hit (httpclient, Content-Encoding: zstd, etc). for the low level ZstandardDecoder i think @rzikm 's point is the real blocker: if a frame ends right at the end of the supplied buffer the decoder can't tell whether another frame follows, so it'd either report NeedMoreData on input that's actually complete, or report Done without consuming anything when there's just trailing garbage.

neither is a good default. so the plan is to keep the per-frame Done on the decoder (now documented, and Reset() readies it for the next frame) and let ZstandardStream own the continuation, which also keeps it consistent with the other Decoder classes.

[rzikm]

@christosk92 Another ping here, did you manage to take a look a the feedback? We would like to get this in soon-ish so that it gets released in one of the next previews (I think it is too late for preview 6 already)

I understand that you do this outside of your job, so it's fine to not have time to invest into it If you don't want to spend the time to drag this into completion than I am okay with taking it over.

[christosk92]

hi @rzikm. Sorry It was just a bad timing with some deadlines. it is now better so I will address your comments and come with a fix by end of the weekend so you can take a look at them by monday.

[rzikm]

Makes sense. We tend to add a bool isFinalBlock parameter to deal with such cases but I see we haven't done so for compression APIs.

we have those for compression, but not decompression, yeah

[rzikm]

LGTM, but I still want to try avoiding recognizing Zstd headers in managed code, if that does not pan out, I will merge this as-is

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[christosk92]

LGTM, but I still want to try avoiding recognizing Zstd headers in managed code, if that does not pan out, I will merge this as-is

Implementing this looks a bit above my paygrade however Im willing to give this a shot, so I ran your request in copilot and it came up with these potential risks:

1. The decoder currently collapses every bad result into InvalidData, so it'd need to surface prefix_unknown for the stream to tell trailing data from corruption.
2. The sub-4-byte trailing case still needs a bit of read-ahead, since the decoder eats those bytes before it can tell it's not a frame.

Please let me know if these risks are worth the change and I will go ahead and try to implement it, will also be a nice learning experience for me.

[rzikm]

Both are solved by feeding the decoder only 4 bytes (the magic prefix)

• valid header => NeedMoreData
• invalid header => InvalidData
• less than 4 bytes => Don't even pass them to decoder, trailing garbage.

I have a patch ready locally, but I can wait so that you can give it a try if you want.

[christosk92]

@rzikm
I gave the decoder-driven version a shot. there's a new internal ZstandardDecoder.IsFrameStart that feeds the decoder exactly the 4 boundary bytes instead of checking the magic in managed code: a valid magic comes back NeedMoreData (it's a frame, keep going), anything else InvalidData (trailing data, stop). the probe doesn't eat anything from the buffer (it resets the decoder before and after), so the next read decodes the frame normally. corrupt frames still throw: the probe only looks at the 4 magic bytes, so a bad header gets caught when the frame actually decodes. StartsWithZstdFrame and the magic constants are gone, and skippable frames should work now.

[rzikm]

@christosk92 You have replied to my comments but did not push any new changes, is that intentional?

stale

[christosk92]

@christosk92 You have replied to my comments but did not push any new changes, is that intentional?

hi,

yes i was awaiting some confirmation on some points before i make changes.

[rzikm]

I replied now

[christosk92]

thanks, will make these changes today and test it e2e with couple of zstd streams

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[christosk92]

hi,

I pushed the inline version.

I dropped IsFrameStart from the decoder. the check is now inlined in AdvanceToNextFrame and feeds the next 4 magic bytes straight into the public Decompress, with the scratch byte from the buffer's free space (can't stackalloc sadly in an async method). a valid magic comes back NeedMoreData so i Discard it and keep decoding, anything else is trailing data.
I also added _endOfStream and short-circuited Read/ReadAsync to return 0 once it's set, so after trailing data we never re-enter the loop with a decoder that's been left in an error state (which is fine, the only thing left to do with it at that point is dispose/reset it).
FInally I kept the if (_stream.CanSeek) TryRewindStream(...) block after the _endOfStream assignment, like you said.

on testing, i went a bit overboard haha. i put the full Read/ReadAsync/AdvanceToNextFrame path through a heavy stress test that exercises it e2e against zstd over a large matrix of multi-frame streams (3100+ distinct streams, 325 assertions), sync and async, over seekable, single-byte-per-read, and chunked non-seekable base streams. all passed.

• the original repro itself, two frames back to back where the second was getting dropped. it still repros with small frames too like you said, not just the big >64k one, and i checked the first frame landing right on the 64k boundary and a byte either side of it since that's where it actually broke.
• the continuation boundary, which was the main thing i wanted to be sure about. i truncated a two-frame stream at every byte offset and checked each one. the only clean endings are the real frame boundary and the spots where you've got a complete first frame plus fewer than 4 bytes of the next magic. once there's a full magic and the body is cut short, it throws.
• corrupt continuation: a valid magic with a bad body throws, but 4 bytes that aren't a real magic just get treated as trailing data and rewound. that's the bit feeding only the magic was meant to fix.
• checksums: turning on AppendChecksum puts the 4-byte trailer right where the probe looks. two checksummed frames concatenated still decode, a frame missing part of its trailer throws, and a flipped checksum byte in the second frame throws.
• skippable frames: all 16 magics, plus zero-length, bigger-than-the-buffer, a stream that's only skippables, and a truncated one whose declared size runs off the end (throws).
• empty frames in the awkward spots: leading, a long run of them between real frames, and a stream that's nothing but empty frames.
• trailing junk of a few lengths (0 through 5 bytes, and 64k) on both seekable and non-seekable. on the seekable ones i checked the base stream ends up exactly at the end of the frame and the trailing bytes read back untouched.
• reading again after the end: returns 0 after trailing data, still throws after a truncated frame.
• dictionary frames still decode after the per-frame Reset, and a missing dictionary throws.
• windowTooLarge throws whether it's the first frame or a later one.
• frames at different compression levels in the same stream.
• all the read entry points: Read, ReadByte, ReadAsync, CopyTo/CopyToAsync, zero-byte reads in the middle, and a 1-byte destination.
• and a fair bit of fuzzing for safety: a few thousand random/garbage inputs and a couple thousand single-bit flips, none of which hang or AV, they all either throw or come back clean. plus round-trip fuzzing a few hundred random multi-frame streams (random counts, sizes, levels, checksum on/off, skippables and empties mixed in) and checking they come back byte for byte.

[rzikm]

LGTM, thanks!

[christosk92]

thanks, and again i appreciate your patience on this.
have a great summer