32Bit Processes - maximum usable memory

[Markus87]

How much memory should I be able to use on a 32Bit process with mimalloc?

Given the simple example:

struct SingleMByte{
	uint8_t bytes[ 1024 * 1024 ];
};
std::vector< std::unique_ptr< SingleMByte > > allTheMBytes;
while( 1 )
{
	std::this_thread::sleep_for( std::chrono::milliseconds{ 25 } );
	for( auto step = 0; step < 10; ++step )
		allTheMBytes.push_back( std::make_unique< SingleMByte >() );
	std::cout << allTheMBytes.size() << " MBytes allocated" << std::endl;
}

System	Allocator	Usable memory [MBytes]
Windows 32Bit	default	1890
Windows 32Bit	mimalloc 2.1.9	1490
wasm32 emsdk 4.0.0	default	4090
wasm32 emsdk 4.0.0	mimalloc 2.1.7	2920

Is this expected?
Is there a way to improve on that?

If I check the process with System Informer on Windows when it runs out of memory I see this:

Details

How can the difference between the commit and the work set size be explained?

This issue is a result of this discussion: emscripten-core/emscripten#20936 (comment)

[res2k]

How can the difference between the commit and the work set size be explained?

This has been explained around the internet already, eg here: https://stackoverflow.com/questions/7954781/whats-the-difference-between-working-set-and-commit-size

Generally, keep in mind that on 32-bit, there's only a maximum of 2GB user space for 32-bit processes - or 3GB, if "large address aware" is enabled for an executable; however, the "Windows 32Bit" numbers seem to be consistent with it being disabled.

Why does "Windows 32Bit + mimalloc 2.1.9" get 400MB less than "default"?
I can only guess, but perhaps mimalloc is trying to obtain a rather large chunk of memory that simply doesn't fit into the available address space, resulting in a failure.

"wasm32 emsdk 4.0.0" +"mimalloc 2.1.7" seems to be consistent with the 3GB limit I mentioned above.

"wasm32 emsdk 4.0.0" +"default" is actually quite astonishing; it's almost 4GB, so more than should actually be supported by the OS. Maybe there's some stuff like using NUMA allocations happening behind the scenes?

[Markus87]

The funny thing is with Windows 32Bit, if I reserve and commit the memory ahead I still run into the problem.

OS? The "OS" is the browser, this is 64Bit in an 64Bit OS. 🤔
wasm32 has a Limit of 4GByte.
Chrome 32Bit has a limit of 512MByte if remember it right.

[res2k]

The funny thing is with Windows 32Bit, if I reserve and commit the memory ahead I still run into the problem.

Can you elaborate how you did that?

OS? The "OS" is the browser, this is 64Bit in an 64Bit OS. 🤔 wasm32 has a Limit of 4GByte (in Chrome 64Bit). Chrome 32Bit has a limit of 512MByte if remember it right.

Ah, that's an important detail, and explains nicely why the ceiling is at about 4GB.

On why mimalloc then fails much earlier, I can only speculate as well - maybe it's, again, mimalloc trying to get a chunk of memory that's "too large" compared to the available space.

To eliminate as much guesswork as possible, it might be best to actually get some insight into what's mimalloc doing.
For example, I can see some _mi_warning_message in the code, among them when OS memory allocation fails.
So maybe make sure you obtain all the output you can get.

[Markus87]

Windows 32-Bit was just a test after seeing what happened in WebAssembly, to see if mimalloc behaves differently there.

Can you elaborate how you did that?

I did the following:

	mi_option_enable( mi_option_verbose );
	mi_option_enable( mi_option_disallow_os_alloc );
	mi_option_enable( mi_option_limit_os_alloc );
	mi_reserve_os_memory( ( 1024 * 1024 ) * 1790, true, true );
	mi_option_set( mi_option_retry_on_oom, 0 );

Output Windows 32Bit with reserved memory

mimalloc: reserved 1835008 KiB memory
Singlethreaded
mimalloc: using 1 numa regions
10 MBytes allocated
20 MBytes allocated
30 MBytes allocated
...
1320 MBytes allocated
1330 MBytes allocated
1340 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x8000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x8000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x8000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x8000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: error: unable to allocate memory (1048576 bytes)
mimalloc: error: out of memory in 'new'

In that case I dont understand why mimalloc needs to allocate something new, there should be reserved space left? 🤔

---

Output Windows 32Bit without special options

mimalloc: using 1 numa regions
mimalloc: reserved 131072 KiB memory
10 MBytes allocated
...
90 MBytes allocated
mimalloc: reserved 131072 KiB memory
100 MBytes allocated
...
190 MBytes allocated
mimalloc: reserved 131072 KiB memory
200 MBytes allocated
...
280 MBytes allocated
mimalloc: reserved 131072 KiB memory
290 MBytes allocated
...
380 MBytes allocated
mimalloc: reserved 131072 KiB memory
390 MBytes allocated
...
480 MBytes allocated
mimalloc: reserved 131072 KiB memory
490 MBytes allocated
...
570 MBytes allocated
mimalloc: reserved 131072 KiB memory
580 MBytes allocated
...
670 MBytes allocated
mimalloc: reserved 131072 KiB memory
680 MBytes allocated
...
760 MBytes allocated
mimalloc: reserved 262144 KiB memory
770 MBytes allocated
...
960 MBytes allocated
mimalloc: reserved 262144 KiB memory
970 MBytes allocated
...
1150 MBytes allocated
mimalloc: reserved 262144 KiB memory
1160 MBytes allocated
...
1340 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1350 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1360 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1370 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1380 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1390 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1400 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1410 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1420 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1430 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1440 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1450 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1460 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1470 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1480 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
1490 MBytes allocated
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x400000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x3000)
mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x400000 bytes, address: 0x00010000, alignment: 0x400000, commit: 1)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x800000 bytes, align: 0x01, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x10000000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x2000)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x10000000 bytes, align: 0x400000, commit: 0, allow large: 0)
mimalloc: warning: unable to allocate aligned OS memory (0x400000 bytes, error code: 0x08, address: 0x00000000, alignment: 0x400000, flags: 0x3000)
mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x400000 bytes, address: 0x00010000, alignment: 0x400000, commit: 1)
mimalloc: warning: unable to allocate OS memory (error: 8 (0x08), addr: 0x00000000, size: 0x800000 bytes, align: 0x01, commit: 0, allow large: 0)
mimalloc: error: unable to allocate memory (1048576 bytes)
mimalloc: error: out of memory in 'new'

In this case it can shoot over the process memory limit, but should mimalloc consider this case and make sure as much of the limited memory is available as possible?

---

WebAssembly - no special options

15:58:51.397 TestMemory.js:1 mimalloc: 
15:58:51.397 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.400 TestMemory.html:1 10 MBytes allocated
15:58:51.429 TestMemory.html:1 20 MBytes allocated
15:58:51.457 TestMemory.js:1 mimalloc: 
15:58:51.457 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.458 TestMemory.html:1 30 MBytes allocated
...
15:58:51.514 TestMemory.html:1 50 MBytes allocated
15:58:51.540 TestMemory.js:1 mimalloc:
15:58:51.540 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.543 TestMemory.html:1 60 MBytes allocated
15:58:51.571 TestMemory.html:1 70 MBytes allocated
15:58:51.598 TestMemory.js:1 mimalloc: 
15:58:51.598 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.599 TestMemory.html:1 80 MBytes allocated
15:58:51.627 TestMemory.html:1 90 MBytes allocated
15:58:51.656 TestMemory.js:1 mimalloc: 
15:58:51.656 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.656 TestMemory.html:1 100 MBytes allocated
...
15:58:51.712 TestMemory.html:1 120 MBytes allocated
15:58:51.739 TestMemory.js:1 mimalloc: 
15:58:51.739 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.741 TestMemory.html:1 130 MBytes allocated
15:58:51.769 TestMemory.html:1 140 MBytes allocated
15:58:51.797 TestMemory.js:1 mimalloc: 
15:58:51.797 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.798 TestMemory.html:1 150 MBytes allocated
...
15:58:51.855 TestMemory.html:1 170 MBytes allocated
15:58:51.881 TestMemory.js:1 mimalloc: 
15:58:51.881 TestMemory.js:1 reserved 32768 KiB memory
15:58:51.884 TestMemory.html:1 180 MBytes allocated
15:58:51.913 TestMemory.html:1 190 MBytes allocated
15:58:51.941 TestMemory.js:1 mimalloc: 
15:58:51.941 TestMemory.js:1 reserved 65536 KiB memory
15:58:51.942 TestMemory.html:1 200 MBytes allocated
...
15:58:52.054 TestMemory.html:1 240 MBytes allocated
15:58:52.082 TestMemory.js:1 mimalloc: 
15:58:52.082 TestMemory.js:1 reserved 65536 KiB memory
15:58:52.084 TestMemory.html:1 250 MBytes allocated
...
15:58:52.197 TestMemory.html:1 290 MBytes allocated
15:58:52.224 TestMemory.js:1 mimalloc: 
15:58:52.224 TestMemory.js:1 reserved 65536 KiB memory
15:58:52.227 TestMemory.html:1 300 MBytes allocated
...
15:58:52.312 TestMemory.html:1 330 MBytes allocated
15:58:52.342 TestMemory.js:1 mimalloc: 
15:58:52.342 TestMemory.js:1 reserved 65536 KiB memory
15:58:52.342 TestMemory.html:1 340 MBytes allocated
...
15:58:52.454 TestMemory.html:1 380 MBytes allocated
15:58:52.482 TestMemory.js:1 mimalloc: 
15:58:52.482 TestMemory.js:1 reserved 65536 KiB memory
15:58:52.483 TestMemory.html:1 390 MBytes allocated
...
15:58:52.595 TestMemory.html:1 430 MBytes allocated
15:58:52.624 TestMemory.js:1 mimalloc: 
15:58:52.625 TestMemory.js:1 reserved 65536 KiB memory
15:58:52.626 TestMemory.html:1 440 MBytes allocated
...
15:58:52.738 TestMemory.html:1 480 MBytes allocated
15:58:52.764 TestMemory.js:1 mimalloc: 
15:58:52.764 TestMemory.js:1 reserved 65536 KiB memory
15:58:52.766 TestMemory.html:1 490 MBytes allocated
...
15:58:52.879 TestMemory.html:1 530 MBytes allocated
15:58:52.908 TestMemory.js:1 mimalloc: 
15:58:52.908 TestMemory.js:1 reserved 65536 KiB memory
15:58:52.911 TestMemory.html:1 540 MBytes allocated
...
15:58:52.995 TestMemory.html:1 570 MBytes allocated
15:58:53.026 TestMemory.js:1 mimalloc: 
15:58:53.026 TestMemory.js:1 reserved 131072 KiB memory
15:58:53.027 TestMemory.html:1 580 MBytes allocated
...
15:58:53.280 TestMemory.html:1 670 MBytes allocated
15:58:53.311 TestMemory.js:1 mimalloc: 
15:58:53.311 TestMemory.js:1 reserved 131072 KiB memory
15:58:53.312 TestMemory.html:1 680 MBytes allocated
...
15:58:53.566 TestMemory.html:1 770 MBytes allocated
15:58:53.596 TestMemory.js:1 mimalloc:
15:58:53.596 TestMemory.js:1 reserved 131072 KiB memory
15:58:53.599 TestMemory.html:1 780 MBytes allocated
...
15:58:53.824 TestMemory.html:1 860 MBytes allocated
15:58:53.857 TestMemory.js:1 mimalloc: 
15:58:53.857 TestMemory.js:1 reserved 131072 KiB memory
15:58:53.858 TestMemory.html:1 870 MBytes allocated
...
15:58:54.112 TestMemory.html:1 960 MBytes allocated
15:58:54.143 TestMemory.js:1 mimalloc: 
15:58:54.143 TestMemory.js:1 reserved 131072 KiB memory
15:58:54.145 TestMemory.html:1 970 MBytes allocated
...
15:58:54.371 TestMemory.html:1 1050 MBytes allocated
15:58:54.405 TestMemory.js:1 mimalloc: 
15:58:54.405 TestMemory.js:1 reserved 131072 KiB memory
15:58:54.406 TestMemory.html:1 1060 MBytes allocated
...
15:58:54.660 TestMemory.html:1 1150 MBytes allocated
15:58:54.694 TestMemory.js:1 mimalloc: 
15:58:54.694 TestMemory.js:1 reserved 131072 KiB memory
15:58:54.696 TestMemory.html:1 1160 MBytes allocated
...
15:58:54.964 TestMemory.html:1 1250 MBytes allocated
15:58:54.998 TestMemory.js:1 mimalloc: 
15:58:54.998 TestMemory.js:1 reserved 131072 KiB memory
15:58:55.002 TestMemory.html:1 1260 MBytes allocated
...
15:58:55.243 TestMemory.html:1 1340 MBytes allocated
15:58:55.281 TestMemory.js:1 mimalloc: 
15:58:55.281 TestMemory.js:1 reserved 262144 KiB memory
15:58:55.282 TestMemory.html:1 1350 MBytes allocated
...
15:58:55.820 TestMemory.html:1 1530 MBytes allocated
15:58:55.861 TestMemory.js:1 mimalloc: 
15:58:55.861 TestMemory.js:1 reserved 262144 KiB memory
15:58:55.862 TestMemory.html:1 1540 MBytes allocated
...
15:58:56.434 TestMemory.html:1 1730 MBytes allocated
15:58:56.475 TestMemory.js:1 mimalloc: 
15:58:56.476 TestMemory.js:1 reserved 262144 KiB memory
15:58:56.480 TestMemory.html:1 1740 MBytes allocated
...
15:58:57.020 TestMemory.html:1 1920 MBytes allocated
15:58:57.063 TestMemory.js:1 mimalloc: 
15:58:57.063 TestMemory.js:1 reserved 262144 KiB memory
15:58:57.066 TestMemory.html:1 1930 MBytes allocated
...
15:58:57.609 TestMemory.html:1 2110 MBytes allocated
15:58:57.655 TestMemory.js:1 mimalloc: 
15:58:57.655 TestMemory.js:1 reserved 262144 KiB memory
15:58:57.658 TestMemory.html:1 2120 MBytes allocated
...
15:58:58.210 TestMemory.html:1 2300 MBytes allocated
15:58:58.260 TestMemory.js:1 mimalloc: 
15:58:58.260 TestMemory.js:1 reserved 262144 KiB memory
15:58:58.262 TestMemory.html:1 2310 MBytes allocated
...
15:58:58.828 TestMemory.html:1 2490 MBytes allocated
15:58:58.881 TestMemory.js:1 mimalloc: 
15:58:58.881 TestMemory.js:1 reserved 262144 KiB memory
15:58:58.882 TestMemory.html:1 2500 MBytes allocated
...
15:58:59.458 TestMemory.html:1 2690 MBytes allocated
15:58:59.516 TestMemory.js:1 mimalloc: 
15:58:59.516 TestMemory.js:1 reserved 262144 KiB memory
15:58:59.521 TestMemory.html:1 2700 MBytes allocated
...
15:59:00.067 TestMemory.html:1 2880 MBytes allocated
15:59:00.094 TestMemory.js:1 mimalloc: warning: 
15:59:00.094 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.095 TestMemory.js:1 mimalloc: warning: 
15:59:00.095 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.096 TestMemory.js:1 mimalloc: warning: 
15:59:00.096 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.097 TestMemory.html:1 2890 MBytes allocated
15:59:00.124 TestMemory.js:1 mimalloc: warning: 
15:59:00.124 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.125 TestMemory.js:1 mimalloc: warning: 
15:59:00.125 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.127 TestMemory.js:1 mimalloc: warning: 
15:59:00.127 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.128 TestMemory.html:1 2900 MBytes allocated
15:59:00.154 TestMemory.js:1 mimalloc: warning: 
15:59:00.154 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.155 TestMemory.js:1 mimalloc: warning: 
15:59:00.155 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.157 TestMemory.js:1 mimalloc: warning: 
15:59:00.157 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.158 TestMemory.html:1 2910 MBytes allocated
15:59:00.184 TestMemory.js:1 mimalloc: warning: 
15:59:00.184 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.186 TestMemory.js:1 mimalloc: warning: 
15:59:00.186 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.187 TestMemory.js:1 mimalloc: warning: 
15:59:00.187 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.189 TestMemory.js:1 mimalloc: warning: 
15:59:00.189 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.219 TestMemory.html:1 2920 MBytes allocated
15:59:00.246 TestMemory.js:1 mimalloc: warning: 
15:59:00.246 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.247 TestMemory.js:1 mimalloc: warning: 
15:59:00.247 TestMemory.js:1 unable to allocate OS memory (error: 48 (0x30), size: 0x20000000 bytes, align: 0x400000, commit: 0, allow large: 0)
15:59:00.248 TestMemory.js:1 Uncaught RuntimeError: memory access out of bounds

If I add up the reserved amount I should be 3GB so I do not know why the last allocation of 512MByte fails ... 🤔
Without mimalloc I can do ~4GByte of single 1MByte allocations.

---

mimalloc is much quicker in certain cases, if the tradeof is this overhead than this is explanation enough for me.
But since I do not know much about the inner workings of mimalloc I can simply not tell if thats acceptable.
In my mind the way mimalloc allocates memory for such limited processes leaves it to chance if as much memory as possible is usable. Because the next "current allocation size"-block to be allocated may just be too much, but some of it may be ok and enough to keep the program running.

Ah, that's an important detail, and explains nicely why the ceiling is at about 4GB.

Sorry I should have started with that. 😅

[res2k]

In that case I dont understand why mimalloc needs to allocate something new, there should be reserved space left? 🤔

Indeed. "Where does the memory go?" is what we should ask. (And since it applies in all the scenarios you tested, it's perhaps not unlikely the answer(s) are the same for both "native" Windows 32-bit and WebAssembly.)

Here are some theories I can think of.

• Measuring inaccuracies. You're summing up the amount of memory you get from the allocator. However, in practice, any allocator has some per-allocation overhead, so that sum is not an accurate measurement of "memory used". But, of course, every allocator probably strives to minimize that overhead, so I doubt it accounts for anything close to the seen difference. Still, it's there and should be considered.
• "Wastage". In the output above, you can see a bunch of "mimalloc: reserved N KiB memory" messages. The sizes are multiple of 1 MiB. You're allocating exactly 1MiB. Most likely, such a block reserved by mimalloc is not filled to the brim with "your" allocations, as allocation overhead/mimalloc's own structures need to go somewhere as well. Though it seems this couldn't account for the whole difference in allocatable memory, either.
• OS heap fragmention. So the OS returns blocks of memory to mimalloc. Those blocks may not be located exactly back-to-back, either; so it could happen that, at some point, the already allocated blocks are distributed in such a manner that the next allocation attempt simply doesn't fit anyway.
• Other memory use. There'll also be some memory allocated by OS components, and thus not usable by mimalloc.

...but as I said, these are theories/guesses. And TBH, none of them feel like they could properly explain the difference in usable memory right now. And, it's also possible none of these are actually what's happening. ¯\_(ツ)_/¯

What to do now? Well, depends on how much further you want to go down the rabbit hole.

• A reasonably easy-to-do analysis would be to sum up the values from all the "mimalloc: reserved N KiB memory" messages in your log(s). Comparing that to your own "MBytes allocated" messages could give an idea of "wastage" by mimalloc itself (ie memory requested from the OS but not used by mimalloc).
• There are probably also tools available to get detailed information of how much memory was allocated from the OS, and where it's placed. (In the wasm case, perhaps even built into the browser?) Might be an idea of OS-level fragmentation is an actual concern. But it may not be trivial to extract useful information from the raw memory details.

But, given there's quite a bit of information and ideas collected in this discussion, maybe it's time to ping @daanx and ask whether he might have some insights 😅

[Markus87]

The changes from @kleisauke (and/or mimalloc 3.0.1) seem to improve the situation a lot.
emscripten-core/emscripten#20936 (reply in thread)
I tested it again with -> emscripten-core/emscripten#23037
This branch => https://github.com/kleisauke/emscripten/tree/mimalloc-update-3.0.1

mimalloc single threaded: 3580 MBytes (1 MByte allocations)
mimalloc multi threaded: 3580 MBytes (1 MByte allocations) / (3670 MByte - 10 MByte allocations)

If the "wasted" amount is overhead of mimalloc I am happy with that if it stays as fast as it was. 👍

I see no reason to keep the issue open, Win32 was just an example for me - I have no use for it.
If somebody else has a usecase for Win32 I guess the issue can be reopened for further investigation. 🤷‍♂

@res2k Thank you for your help!