GZipStream cannot decompress multipart gzip files.

[mburbea]

See the following stackoverflow thread for more information.

GZipStream currently fails if you try to decompress a concatenated gzip file. The RFC does state that two gzip files can be cated together and it is also a valid gzip file. However, the current implementation can only read the first gzip stream and stops there. Here is a simple repro::

using System;
using System.IO;
using System.IO.Compression;
class Program
{
    static void Main()
    {
        using (var fs = new FileStream(Path.Combine(Environment.CurrentDirectory,"test.txt.gz"), FileMode.Create))
        {
            using(var gz = new GZipStream(fs, CompressionLevel.NoCompression, true))
            using(var sw = new StreamWriter(gz))
                sw.WriteLine("Stream 1");
                
            using(var gz = new GZipStream(fs, CompressionLevel.NoCompression, true))
            using(var sw = new StreamWriter(gz))
                sw.WriteLine("Stream 2");
            
            fs.Seek(0, SeekOrigin.Begin);
            using (var gz = new GZipStream(fs, CompressionMode.Decompress))
            using (var sr = new StreamReader(gz))
            Console.WriteLine(sr.ReadToEnd());
        }
    }
}

The expected output should be

Stream 1
Stream 2

But the actual output is just Stream 1.

If you open up "test.txt.gz" in your favorite gzip util, like 7z, you will be able to extract the file and you will see the correct output.

[ianhays]

This is a pretty common occurrence with gzip implementations. Our guidance in the past has been that there is a 1:1 mapping between a gzipstream and an independent gzip member, so the workaround for multi-part gzip files is to use multiple gzip streams.

I think we can do better than that, though it will require a good amount of reworking the relationship between DeflateStream and GZipStream and may incur a compat cost. One way of doing this would be to modify the GZipStream Read* members that wrap around DeflateStream Read* to cache a read-block before passing it to be inflated. Then if the inflation returns a number of "read" bytes less than the full read-block, we might have another gzip data member. We could then pass the cached read-block minus the length of the successful previous inflate read to a new DeflateStream object, This would mean that we would end up throwing exceptions in cases where a single-block GZipStream was appended with garbage data because we would then be trying to parse that garbage data as an additional gzip member. So we might want to catch exceptions there or do some manual parsing for GZip Header ID bits before passing the next potential member to a new DeflateStream.

Another option would be to remove the GZipStream->DeflateStream relationship in favor of a custom GZip wrapper around zlib, but I think that would be a mistake as it would fork our zlib wrapper code.

I don't have the time to do this myself right now, but I think it would be a nice quality of life addition and bring us in line with the spec.

[mburbea]

@ianhays , well trying to open the second stream seems to not work at all with the current implementation. It seems that after reading content of the first stream it immediately jumps to the end of the file. E.g. the first stream ends at byte 33 the second stream starts at byte 34. However, the position jumps to position 66 (the end of file) when you are done reading the first stream. Thus there is no easy way for a consumer to consume a concatenated gzip file using the corefx GZipStream. SharpZipLib can handle this, but its actually slow for large files.

[Kentalot]

I've tested this before and it doesn't actually jump to the end of the stream (it does for you because it's a small stream). It actually jumps 1-2kb or so on larger files. Maybe it tries to fill the buffer or something? I would definitely love to not have to use SharpZipLib, so this issue would really help me out.

[ianhays]

@Kentalot you're correct, it is filling a buffer.

I finally got some spare time, so I'm going to take a stab at this.