build runner: actually limit the peak RSS rather than enforcing it post-hoc

[andrewrk]

Extracted from #14647.

Currently, it is enforced post-hoc here:

zig/lib/std/Build/Step.zig

Lines 176 to 182 in b4d58e9

	if (s.max_rss != 0 and s.result_peak_rss > s.max_rss) {
	const msg = std.fmt.allocPrint(arena, "memory usage peaked at {d} bytes, exceeding the declared upper bound of {d}", .{
	s.result_peak_rss, s.max_rss,
	}) catch @panic("OOM");
	s.result_error_msgs.append(arena, msg) catch @panic("OOM");
	return error.MakeFailed;
	}

Instead, this issue proposes to enforce it using operating system APIs.

I looked briefly into this, and it looked like a giant pain in the ass to do this, at least on Linux.

[matu3ba]

For Linux (Unixoids):

Afaiu, one way to go is to read /proc/self/status https://stackoverflow.com/questions/1558402/memory-usage-of-current-process-in-c/47531152#47531152. Also, fopen them may fail and, afaiu, they are racy on reading.

The alternatives are to 1. invoke every time getrusage as syscall, 2. track all allocations synchronized in memory (meaning atomic access to write new memory usage on every "page allocation and free") , 3. use cgroups, 4. landlock offers an api to the memory allocator https://docs.kernel.org/userspace-api/landlock.html

In an ideal world, we could ask the Kernel upfront for memory up to a process limit in the syscall similar to how io_uring bundles multiple info together for better perf.
Option 1+2 offer no protection against mailicious use/bypassing the Zig allocator interface.
Option 3+4 dont work on Windows and will likely not work in future on Windows.

Please let me know, if I forgot anything.

APPENDUM: On Windows there is win32 with GetProcessMemoryInfo syscall, which probably uses NtQuerySystemInformation to get the used memory.

APPENDUM2: From quick glimpse, it looks like the wait statistics are used, but getrusage approach has not been implemented.

[matu3ba]

Must have not read properly and I need this for a thing (root requirement ok though).

Limiting works rather simple, but we would need a declarative table for it on Posix to talk to the child process after clone/fork before it does exec*.
If you think the complexity is worth it, then I'd lean towards an environment variable specifying tcp connection or pipe and closing it before execve. See https://stackoverflow.com/questions/7889594/setrlimit-on-another-process.
However, this would push us into "handle the leaking file descriptors territory" on Posix.

I'll soonish get to this:

• job api in windows limitation done. its kind of a clusterfuck, so I'm very much unsure if we want all of that additional api surface in libstd. We might want to limit it or use the ntdll calls instead. On the other hand there is no other way to limit memory consumption.
• still working on tests (processes really terminated, memory limits correct etc), since I'm concerned if pinning if necessary and windows behaves as advertised
• posix api boils down to soft limit and exception in descendant or hard limit and SIGKILL the descendant by kernel

https://devblogs.microsoft.com/oldnewthing/20230209-00/?p=107812 of course the error free way uses the extended api on Windows. Windows offers apparently no other way to restrict process memory, see https://superuser.com/questions/1263090/is-it-possible-to-limit-the-memory-usage-of-a-particular-process-on-windows.
However, I suspect by some evidence, that one needs to to some more involved setup as it was reported getting correct time and memory usage requires to pin things or they are wrong.