Avoid emitting library calls in standalone WASM

[nepx]

Hi all,

I'm looking into generating standalone WASM (#12757), and so far things are looking great, especially when these things are going on inside of a Web Worker.

I've been looking into implementing system calls, making a list of the ones that I'll need to implement in order to get some applications running, but I've noticed that sometimes Emscripten will generate imports to things other than system calls (__syscalNNN) and various WASI functions (i.e. fd_read). I've seen this happen for time-related calls, but I'm sure this will happen in other cases as well.

For reference, the imports to a module looked like this, produced by wasm-objdump:

 - func[0] sig=2 <wasi_snapshot_preview1.proc_exit> <- wasi_snapshot_preview1.proc_exit
 - func[1] sig=3 <env.__syscall4> <- env.__syscall4
 - func[2] sig=3 <wasi_snapshot_preview1.args_get> <- wasi_snapshot_preview1.args_get
 - func[3] sig=3 <wasi_snapshot_preview1.args_sizes_get> <- wasi_snapshot_preview1.args_sizes_get
 - func[4] sig=0 <env.localtime> <- env.localtime
 - func[5] sig=0 <env.time> <- env.time

IIRC, both localtime and time are C library functions, and I'd really like to avoid rewriting sections of libc in JavaScript. For the record, however, it's only these functions that are called as exports: other libc functions like malloc are statically linked in, which is great. I really have no idea which ones are inlined and which ones aren't. The others, like __syscall4 and args_sizes_get are fine, and I expect to implement them myself, in my own compatibility layer. I can also implement time-related system calls, but I'd be better if libc handled localtime and friends itself and called the raw system calls instead.

The command that I used to compile this file is emcc $1.c -O3 -s WASM=1 -s STANDALONE_WASM -o $1.wasm -static -g4. Are there any other flags that I need in order to avoid generating calls to localtime and time, or will I just need to bite the bullet and implement them myself?

Thanks!

[sbc100]

That background there is that, historically, it as sometimes made sense to implement some higher level APIs such as localtime in JavaScript, rather than trying to implement the lower level system calls and re-use the musl code at the higher level.

We are working on another option called PURE_WASI which should do less of this over time: #12704

I'm rather confused by __syscall4 import. Syscalls should have names in emscripten rather than numbers these day. What version of emscripten are you using?

BTW, you can skip some of the arguments you are using. WASM=1 is that default. -static is the default. STANDALONE_WASM is the default when the output filename ends in .wasm.

[nepx]

Thanks for the response! I have an older version of Emscripten, didn't realize how outdated it was until now, haha.

It doesn't matter whether I implement WASI or just raw system calls. Personally, I like system calls because they're a more familiar interface, but WASI would obviously be the more future-proof and standards-compliant option.

It seems like #10536 added some time functions to Emscripten's version of musl. So would the solution here just be upgrading to a newer version of Emscripten? Furthermore, are there any other JS-libc functions that I should be aware of?

Thanks for the information about the flags!

[sbc100]

You certainly want to be using the latest version of emscripten as this is one area we are currently working on and improving. That should be your first goal.

Sadly the JS<->wasm ABI/API used by emscripten is not fixed and so can change over time. As we move more to WASI this should happen less but it can still happen.

There is some background here but its a little out of day: https://github.com/emscripten-core/emscripten/wiki/WebAssembly-Standalone.

There are various other open bugs about trying to do more pure with pure WASM/WASI. See
#12341 and #12073.

Maybe this issue can be closed as a duplicate of one of those ? Or are you looking for something different?

[nepx]

I'd love to use PURE_WASI when it becomes available, but I'm okay with implementing system calls as well (I'm more familiar with those than WASI calls). All I wanted was to avoid writing libc functions in JS.

But it seems like updating Emscripten did the trick, so I'll close for now and reopen if something else happens.

[nepx]

It seems like the WASI interface as supported by Emscripten is quite limited. This is a major roadblock for me; is there a way to have Emscripten generate imports so that I can write my own system call layer, a la src/library_syscall.js?

To be precise, I'm not looking for PURE_WASI, although it would be helpful. I'm looking for a way to have Emscripten emit imports to low-level building blocks like system calls and WASI so that I can implement them myself (please don't ask me why). I'd actually prefer system calls since WASI has some safety restrictions (i.e. preopens) that aren't really useful in my case and actually hinder operations.

Right now, the WASI interface is quite limited, and I like the flexibility that implementing individual system calls gives me. Compiling to standalone WASM gives me pure WASI imports, which, again, led to some problems with limitations in the Emscripten system call layer.

Compiling with emcc -O3 ls.c -g3 -o ls.html, the resulting .wasm file has the import env.__sys_open, which is exactly what I need! The only problem is that these names become mangled once I drop -g3. It also has a few other extraneous functions, like emscripten_resize_heap, that I don't really need and aren't present in the standalone version.

Is there a way to get the best of both worlds?

[sbc100]

To avoid inport/export name minification you can build in standalone mode (-o somefile.wasm) or you can build with --profiling-funcs or you can build with --minify 0 I believe. Plenty of options there :)

[nepx]

Is there any way to get rid of the Emscripten-specific functions? Like emscripten_resize_heap, etc.

Basically, I have two options at this point:

• Use -o file.html and take the WASM from that. It's great because it uses raw syscalls, but it also produces Emscripten-specific functions.
• Use -o file.wasm. No Emscripten-specific functions, which is great, but it also doesn't generate the raw system calls, which appears to be more feature complete.

Is there a way to produce imports to system calls without generating Emscripten-specific imports? I'd be open to using WASI, if needed, but at the moment the Emscripten WASI layer isn't complete enough for my needs. (and there's a perfectly nice system call implementation in the source code too!)

[sbc100]

Note: Just to be clear the difference between -o file.html and -o file.wasm aside from the fact the one generates html is that -o file.wasm also sets STANDALONE_WASM by default. If you like you can do -o file.html -s STANDALONE_WASM if you want the standalone APIs and the html.

I'm not sure what you question is exactly. The WASI API is a set of syscalls, and we use it where it makes sense instead of linux/musl syscall (the ones that start with __sys) because its more compatible with other wasm runtimes.

Over time we hope to move away from the linux/musl syscalls and use more of WASI.. so you will to implement the WASI syscalls that we use. When you build in STANDALONE_WASM mode we try harder to avoid emscripten-specific calls and will use WASI syscall in more places. While we try to maintain stability here, things at the wasm boundary will change, so as you upgrade emscripten you will need to change you loader. Eventually we might get to place where this API is fixed but we are not there yet.

[nepx]

It doesn't really matter which set of "system calls" that I implement. I personally prefer the __sys functions since I'm more familiar with them as a C programmer, but WASI is definitely something to look into as it matures.

Here's the precise problem that I'm having:

When I generate files using -s STANDALONE_WASM, it links in system/lib/standalone/standalone.c, which at the moment lacks completeness (in particular, the open system call appears to be a stub, but I'd assume that others are as well). Interestingly enough, it's not pure WASI code: a few system calls are linked in as well, and that's totally fine.

Now, if I generate an ordinary HTML project, system/lib/standalone/standalone.c isn't linked in, and there's direct access to __sys_open, so I know some form of this is very much possible. However, it comes at the price of additional Emscripten-specific code and name mangling, the latter of which you can somewhat prevent by having -g3 or --profiling-funcs or --minify 0.

What I'm struggling with right now is not linking in system/lib/standalone/standalone.c and instead having direct access to something like __sys_open. However, I'd also like to prevent imports to Emscripten-specific code. I know that WASI has filesystem sandboxing that would make having a direct open call a security risk, but in my case, I'm not bound to "pure" WASI, and I'm okay with implementing some of the __sys functions myself. In any case, it's running in the browser, and the security risk is greatly reduced.

[nepx]

Basically, the question is: is there way to avoid linking in system/lib/standalone/standalone.c while keeping all the advantages of standalone WASM (i.e. no Emscripten-specific imports)? I'm OK with using complex/compiler linker flags if needed and implementing any number of functions, as long as the number is sustainable. The only requirement is that it needs to work.

[sbc100]

Ah I see... I hadn't realized we had stubs in system/lib/standalone/standalone.c. No there is no way to do what you want.

I think once I land #10536 we can remove the stubs from standalone.c and you should have solution that works for you.