Replace `readUntilDelimiter*` with `streamUntilDelimiter`

[silversquirl]

readUntilDelimiterArrayList is great if you want to read delimited data into a dynamically allocated buffer. However, sometimes you may want to read delimited data into a BoundedArray (for allocation-free code, or when you know the max size will be small), or send it directly to a writer (eg. for converting a newline-separated stream into a NUL-separated stream or similar).

These usecases are possible today, but they're a bit awkward and don't work as well as they should:

const delimiter = '\n';

// Read into BoundedArray
// Doesn't properly handle EndOfStream like readUntilDelimiterArrayList does
var buf = std.BoundedArray(u8, 4096);
buf.len = (try reader.readUntilDelimiter(buf.buffer, delimiter)).len;

// Read into writer
// Literally just an adaptation of what readUntilDelimiterArrayList does
const writer = std.io.getStdOut().writer();
while (true) {
    const byte = try reader.readByte();
    if (byte == delimiter) break;
    try writer.writeByte(byte);
}

I propose to replace readUntilDelimiter, readUntilDelimiterOrEof, and readUntilDelimiterArrayList with a new function, streamUntilDelimiter (name up for discussion, I don't hugely like this one but can't think of anything better):

// Read into ArrayList
var array = std.ArrayList(u8).init(allocator);
try reader.streamUntilDelimiter(array.writer(), delimiter, max_size);

// Read into BoundedArray
var buf = std.BoundedArray(u8, 4096);
try reader.streamUntilDelimiter(buf.writer(), delimiter, null); // null max_size = unbounded

// Read into writer
const writer = std.io.getStdOut().writer();
try reader.streamUntilDelimiter(writer, delimiter, null);

I would recommend removing readUntilDelimiterAlloc and readUntilDelimiterOrEofAlloc, since they're already pretty much superseded by readUntilDelimiterArrayList which is never slower, often faster, and rarely significantly harder to use.

[nektro]

why not set up var buffer: [4096]u8 = undefined; directly?

[silversquirl]

Because BoundedArray provides a nicer API to work with when doing other operations on the data later on.
Anyway, BoundedArray is only one example. The point of this proposal is to allow readers to work better with all datastructures that could expose a writer, not just BoundedArray

[nmichaels]

I think it's worth keeping readUntilDelimiter and the friends that use slices directly around. Slices are right there in the language, and it's simpler to write

var some_array : [128]u8 = undefined;
try stream.readUntilDelimiter(some_array[0..], '\n');

than

    var some_array : [128]u8 = undefined;
    var buf_stream = io.FixedBufferStream(@TypeOf(some_array[0..])){.buffer = &some_array, .pos=0};
    try stream.streamUntilDelimiter(buf_stream, '\n', some_array.len);
}

Even if there were a more ergonomic way to initialize the FixedBufferStream, it's just getting in the way. I agree that for data structures that aren't part of the language streamUntilDelimiter makes loads of sense, but slices are core and easy to work with.

[silversquirl]

@nmichaels The idiomatic new way to write that doesn't use FixedBufferStream at all:

var array: std.BoundedArray(128) = .{};
try stream.streamUntilDelimiter(array.writer(), '\n', array.len);

std.BoundedArray is, imo, strictly better than an array for this application, as it unifies the slice view and the backing buffer. The old way had a separate storage buffer and slice, which clutters the local namespace with an extra name and requires more mental overhead to interpret, especially for someone unfamiliar with C-style APIs. The newer API avoids manual low level buffer handling while providing the exact same behaviour, which leads to a more accessible API for newcomers, and more readable and flexible code.

---

It's also worth noting that, if you really do want to use FixedBufferStream, the idiomatic way to create it is:

var buf_stream = std.io.fixedBufferStream(&some_array)

[nmichaels]

@silversquirl That's much, much better than what I thought. I was probably misled by comments like

/// Deprecated: use `streamUntilDelimiter` with FixedBufferStream's writer instead.
/// Reads from the stream until specified byte is found. If the buffer is not
/// large enough to hold the entire contents, `error.StreamTooLong` is returned.
/// If end-of-stream is found, `error.EndOfStream` is returned.
/// Returns a slice of the stream data, with ptr equal to `buf.ptr`. The
/// delimiter byte is written to the output buffer but is not included
/// in the returned slice.

Maybe those should instead point to std.BoundedArray. I don't understand how just passing a pointer to an array can get me an initialized fixedBufferStream, but that's a me problem.

[silversquirl]

I don't understand how just passing a pointer to an array can get me an initialized fixedBufferStream, but that's a me problem.

fixedBufferStream is a function that constructs a FixedBufferStream for you :)

[nmichaels]

Ahh, I missed that the f was lower case. Thanks.

[T3kla]

@nmichaels The idiomatic new way to write that doesn't use FixedBufferStream at all:

var array: std.BoundedArray(128) = .{};
try stream.streamUntilDelimiter(array.writer(), '\n', array.len);

std.BoundedArray is, imo, strictly better than an array for this application, as it unifies the slice view and the backing buffer. The old way had a separate storage buffer and slice, which clutters the local namespace with an extra name and requires more mental overhead to interpret, especially for someone unfamiliar with C-style APIs. The newer API avoids manual low level buffer handling while providing the exact same behaviour, which leads to a more accessible API for newcomers, and more readable and flexible code.

It's also worth noting that, if you really do want to use FixedBufferStream, the idiomatic way to create it is:

var buf_stream = std.io.fixedBufferStream(&some_array)

This code doesn't work properly, it should use array.capacity() instead of array.len.

(sorry for the nitpick but I'm learning Zig and I was stuck on this more time than what I want to admit)