fetch regressions

[andrewrk]

some example urls to fix:

• git+https://gitlab.com/fjc/zigby.git#57d01bac78e1a152172d1daee4868593eb1c5156
• https://github.com/squeek502/compressed_mingw_includes/releases/download/1.0.2/compressed_mingw_includes.tar
• https://www.ryanliptak.com/misc/notin.html
• building my music player project with empty cache results in "invalid HTTP response" errors sometimes, which disappear on retry.

[ianprime0509]

For what it's worth, here's another one that triggers an assertion failure (this output is from 0.15.0-dev.1455+46b34949c, that is, the current head of #24698):

ian@ian-laptop:~/src/zig/build$ stage4/bin/zig fetch git+https://github.com/ianprime0509/zig-xml
thread 327893 panic: reached unreachable code
/home/ian/src/zig/lib/std/posix.zig:1861:23: 0x1934ff7 in openatZ (std.zig)
            .NOENT => return error.FileNotFound,
                      ^
/home/ian/src/zig/lib/std/fs/Dir.zig:1590:21: 0x199474c in openDirFlagsZ (std.zig)
        else => |e| return e,
                    ^
/home/ian/src/zig/lib/std/fs/Dir.zig:1554:5: 0x1977707 in openDirZ (std.zig)
    return self.openDirFlagsZ(sub_path_c, symlink_flags);
    ^
/home/ian/src/zig/lib/std/fs/Dir.zig:1498:5: 0x193767e in openDir (std.zig)
    return self.openDirZ(&sub_path_c, args);
    ^
/home/ian/src/zig/lib/std/debug.zig:559:14: 0x181df59 in assert (std.zig)
    if (!ok) unreachable; // assertion failure
             ^
/home/ian/src/zig/lib/std/http.zig:828:23: 0x34fdf5f in endUnflushed (std.zig)
                assert(len == 0); // Trips when end() called before all bytes written.
                      ^
/home/ian/src/zig/lib/std/http.zig:809:25: 0x2d1f58f in end (std.zig)
        try endUnflushed(w);
                        ^
/home/ian/src/zig/lib/std/http/Client.zig:892:19: 0x26a7721 in sendBodyComplete (std.zig)
        try bw.end();
                  ^
/home/ian/src/zig/src/Package/Fetch/git.zig:919:37: 0x21a57f5 in listRefs (main.zig)
        try request.sendBodyComplete(body.buffered());
                                    ^
/home/ian/src/zig/src/Package/Fetch.zig:1047:29: 0x21aefd4 in initResource (main.zig)
            session.listRefs(&ref_iterator, .{
                            ^
/home/ian/src/zig/src/Package/Fetch.zig:404:35: 0x1ca19d2 in run (main.zig)
                try f.initResource(uri, &resource, &server_header_buffer);
                                  ^
/home/ian/src/zig/src/main.zig:7039:14: 0x1b68926 in cmdFetch (main.zig)
    fetch.run() catch |err| switch (err) {
             ^
/home/ian/src/zig/src/main.zig:328:24: 0x1b7cd6e in mainArgs (main.zig)
        return cmdFetch(gpa, arena, cmd_args);
                       ^
/home/ian/src/zig/src/main.zig:205:20: 0x19ed620 in main (main.zig)
    return mainArgs(gpa, arena, args);
                   ^
/home/ian/src/zig/lib/std/start.zig:627:37: 0x19ea709 in posixCallMainAndExit (std.zig)
            const result = root.main() catch |err| {
                                    ^
/home/ian/src/zig/lib/std/start.zig:232:5: 0x19e9f91 in _start (std.zig)
    asm volatile (switch (native_arch) {
    ^
???:?:?: 0x0 in ??? (???)
Aborted (core dumped)

[andrewrk]

Thanks. I have 4 local fixes that includes that one, and I'm working on the next problem after that before pushing.

The next problem is the interaction between:

   596              else => |remaining_chunk_len| {
-> 597                  const n = try in.stream(w, limit.min(.limited64(@intFromEnum(remaining_chunk_len) - 2)));
   598                  chunk_len_ptr.* = .init(@intFromEnum(remaining_chunk_len) - n);
   599                  return n;
   600              },

...in the HTTP chunk stream, and

        .application_data => {
            if (@intFromEnum(limit) < cleartext.len) return failRead(c, error.OutputBufferUndersize);

...in the TLS client. The TLS client needs to be made to respect limit.

[alosarjos]

Not sure if this helps.

I have a dependency on https://www.sqlite.org/2025/sqlite-amalgamation-3500200.zip

Looks like with version v0.15.0-dev.1429+04fe1bfe3 (Not sure when changes happened) fetch generates a new hash, running the zig build on a CICD with no cache says so. If I run fetch to update the dependency, I start getting some issues with Zig not being able to find some symbols on the sqlite3.h file.

Keeping my local version of the dependency with the old hash cached, it's all still working.

[alexrp]

With 1440519, https://github.com/neurocyte/flow fails to build with:

/home/alexrp/Source/flow/build.zig.zon:18:20: error: unable to unpack git files: ReadFailed
            .url = "git+https://github.com/neurocyte/thespian?ref=master#0a386496cda74ef827d5770f6f071a7d2d54b91a",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/alexrp/Source/flow/build.zig.zon:30:20: error: unable to unpack git files: ReadFailed
            .url = "git+https://github.com/neurocyte/libvaxis?ref=main#846ddb8bf483e8a7eb25628d6c34ba7e781155b6",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(I'm unsure if this is actually a regression on master or somehow related to it currently targeting Zig 0.14.1 in build.zig.zon.)

[alexrp]

With 1440519, https://github.com/kristoff-it/superhtml fails to build with:

thread 239283 panic: reached unreachable code
/home/alexrp/Source/ziglang/zig/lib/std/debug.zig:559:14: 0x1a1fa09 in assert (std.zig)
    if (!ok) unreachable; // assertion failure
             ^
/home/alexrp/Source/ziglang/zig/lib/std/Io/Reader.zig:531:11: 0x1b92d41 in toss (std.zig)
    assert(r.seek <= r.end);
          ^
/home/alexrp/Source/ziglang/zig/src/Package/Fetch/git.zig:1098:43: 0x22798e1 in stream (main.zig)
                                input.toss(1);
                                          ^
/home/alexrp/Source/ziglang/zig/lib/std/Io/Reader.zig:178:34: 0x1a278fe in stream (std.zig)
    const n = try r.vtable.stream(r, w, limit);
                                 ^
/home/alexrp/Source/ziglang/zig/lib/std/Io/Reader.zig:257:27: 0x2ba6125 in streamRemaining (std.zig)
        offset += r.stream(w, .unlimited) catch |err| switch (err) {
                          ^
/home/alexrp/Source/ziglang/zig/src/Package/Fetch.zig:1351:49: 0x28d4d5e in unpackGitPack (main.zig)
            _ = try fetch_reader.streamRemaining(&pack_file_writer.interface);
                                                ^
/home/alexrp/Source/ziglang/zig/src/Package/Fetch.zig:1222:42: 0x22633ff in unpackResource (main.zig)
        .git_pack => return unpackGitPack(f, tmp_directory.handle, &resource.git) catch |err| switch (err) {
                                         ^
/home/alexrp/Source/ziglang/zig/src/Package/Fetch.zig:516:47: 0x1fe336e in runResource (main.zig)
        var unpack_result = try unpackResource(f, resource, uri_path, tmp_directory);
                                              ^
/home/alexrp/Source/ziglang/zig/src/Package/Fetch.zig:472:25: 0x1e53401 in run (main.zig)
    return f.runResource(try uri.path.toRawMaybeAlloc(arena), &resource, remote.hash);
                        ^
/home/alexrp/Source/ziglang/zig/src/Package/Fetch.zig:833:8: 0x1fa276c in workerRun (main.zig)
    run(f) catch |err| switch (err) {
       ^
/home/alexrp/Source/ziglang/zig/lib/std/Thread/Pool.zig:118:39: 0x28b3d3c in runFn (std.zig)
            @call(.auto, func, closure.arguments);
                                      ^
/home/alexrp/Source/ziglang/zig/lib/std/Thread/Pool.zig:293:27: 0x22d7e4d in worker (std.zig)
            runnable.runFn(runnable, id);
                          ^
/home/alexrp/Source/ziglang/zig/lib/std/Thread.zig:510:13: 0x20777b0 in callFn__anon_182877 (std.zig)
            @call(.auto, f, args);
            ^
/home/alexrp/Source/ziglang/zig/lib/std/Thread.zig:782:30: 0x1ea3b29 in entryFn (std.zig)
                return callFn(f, args_ptr.*);
                             ^

Panicked during a panic: reached unreachable code
... snip ...

[andrewrk]

@alexrp thanks for helping work on this. that toss needs to become a discardAll:

/// Skips the next `n` bytes from the stream, advancing the seek position. This
/// is typically and safely used after `peek`.
///
/// Asserts that the number of bytes buffered is at least as many as `n`.
///
/// The "tossed" memory remains alive until a "peek" operation occurs.
///
/// See also:
/// * `peek`.
/// * `discard`.
pub fn toss(r: *Reader, n: usize) void {

/// Skips the next `n` bytes from the stream, advancing the seek position.
///
/// Unlike `toss` which is infallible, in this function `n` can be any amount.
///
/// Returns `error.EndOfStream` if fewer than `n` bytes could be discarded.
///
/// See also:
/// * `toss`
/// * `discardRemaining`
/// * `discardShort`
/// * `discard`
pub fn discardAll(r: *Reader, n: usize) Error!void {