Add openmp library from llvm

[duerrbaby]

Original PR here: #25937

Imports the openmp library from LLVM and enables it using the standard -fopenmp flag.

When updating openmp, it needs to be compiled. This is not strictly necessary, but we need to invoke cmake in order to generate the headers. When compiling openmp, we need to make use of cmake as well, as the build system is quite long and complicated. I am sure it can be done another way using ninja, and I encourage others to implement this, but I do not have the capacity right now to make this happen.

Fixes: #13892, #17637

[anutosh491]

Thanks a lot for looking into this. I'm looking forward to this for supporting openmp with clang-repl in the browser !

[duerrbaby]

@anutosh491 tests are resolved. But I haven't tried it on my app yet.
EDIT: Confirmed to work on my side.

[duerrbaby]

hi, @JAicewizard have you taken a look yet?

[duerrbaby]

@sbc100 can you help to review?

[sbc100]

We should probably add this the ChangeLog too.

[anutosh491]

Interested to know which release would/could this be a part of ?

[sbc100]

All changes always go into the next release. In this case that next release is going to be 6.0.3.

[JAicewizard]

Thanks for getting this finished! It was quite a bit of work to get it working initially, but getting merged was a longer process than anticipated. I'm happy someone got finishing it and getting this merged!

[JAicewizard]

Also noticed that the final squashed commit doesn't have me as co-autho, probably because of how github works. Would it be possible to add credits on the changelog?

[sbc100]

Oh sorry about that, yes we can add you to AUTHORS (And the release notes) for sure. Would you mind send a PR adding your name to AUTHORS?

[aheejin]

So is this from LLVM 21.1.8 or 22.1.8? (The README says 21.1.8, but I'm just making sure, because the current latest version is 22.1.8)

And do we expect to update OpenMP with other LLVM libraries (comiler-rt, libc, libcxx, libcxxabi, and libunwind) going forward?

[sbc100]

To answer the second question, yes we should update each time there is an LLVM stable release.

[sbc100]

I'm pretty sure this version of openmp was imported from the emscripten-libs-21 branch in our fork. I remember asks it to come from our fork and that emscripten-libs-22 did not yet exist I believe?

[duerrbaby]

Yeah I took it from the latest HEAD of the emscripten fork for llvm.

See the detailed message

[aheejin]

Based on emscripten-core/llvm-project: 1823581ecb2ff84e1adb159d30216bd2cff48cfd

This is neither emscripten-libs-21 nor emscripten-libs-22. It's main (which I synced with the upstream LLVM's main like 3 weeks ago). Are we planning to follow tot for this?

[duerrbaby]

No idea. Ask the maintainers on what procedure to follow.

[sbc100]

IIRC there are no local patches, so you should be able to just run the update_openmp.py script to update our local copy to emscripten-libs-22.

[aheejin]

@duerrbaby We don't have any local patches on top of the tot?
And what test suite should we run to make sure this works?

[sbc100]

IIRC we only have one test so far: test_openmp_max_threads (but becuase its in test_core.py can be run in many different modes I guess).

[sbc100]

I did land one change to openmp. Doing run of system/lib/update_openmp.py (after fix it up a little) from 1823581ecb2ff84e1adb159d30216bd2cff48cfd shows the only diff is the one I made:

diff --git a/system/lib/openmp/src/kmp_dispatch.h b/system/lib/openmp/src/kmp_dispatch.h
index 1f45bf02c4..f161a80170 100644
--- a/system/lib/openmp/src/kmp_dispatch.h
+++ b/system/lib/openmp/src/kmp_dispatch.h
@@ -291,8 +291,8 @@ template <typename T> kmp_uint32 __kmp_eq(T value, T checker) {
     TODO: make inline function (move to header file for icl)
 */
 template <typename UT>
-UT __kmp_wait(volatile UT *spinner, UT checker,
-              kmp_uint32 (*pred)(UT, UT) USE_ITT_BUILD_ARG(void *obj)) {
+static UT __kmp_wait(volatile UT *spinner, UT checker,
+                     kmp_uint32 (*pred)(UT, UT) USE_ITT_BUILD_ARG(void *obj)) {
   // note: we may not belong to a team at this point
   volatile UT *spin = spinner;
   UT check = checker;

[sbc100]

@aheejin I created this no-op PR while investigating this. Feel free to land it, or ignore it: #27383.

[arsnyder16]

@duerrbaby @sbc100 @kripken Thank you for working through this. I haven't had a chance to try this, but prior to this PR i was compiling openmp manually and linking it to my module. There was still one gap that still exists between what i need and what is upstream in llvm. I am curious if there is any insight from the emscripten/binaryen maintainers .

Its related to how openmp invoked microtasks. __kmp_invoke_microtask You will notice that specifically for WebAssembly, a change was made to make sure each function pointers has an explicit cast to avoid a runtime error (The code has since been shifted since it was originally added)
cpp

// Because WebAssembly will use `call_indirect` to invoke the microtask and
// WebAssembly indirect calls check that the called signature is a precise
// match, we need to cast each microtask function pointer back from `void *` to
// its original type.

For my version of openmp i have to apply a patch that expands this all the way to microtask_t25 . Otherwise i will hit fprintf(stderr, "Too many args to microtask: %d!\n", argc);. This doesn't seems like a scalable long term solution by any means.

1. Its unclear if this is related to some optimization being applied that collapsing code and adding more arguments (binaryen?). I use -Os.
2. Maybe this is rooted in how clang is compiling, based on comment above __kmp_invoke_microtask functiib

// we really only need the case with 1 argument, because CLANG always build
// a struct of pointers to shared variables referenced in the outlined function
int __kmp_invoke_microtask(microtask_t pkfn, int gtid, int tid, int argc,
                           void *p_argv[]

To me this smells like some underlying issue. It also appear that a lot of "__kmp_invoke_microtask" implementations are actually in assembly instead of c++. Maybe that would be a better approach for WebAssembly?

[sbc100]

It seems like the limit on the number of args in a microtask is hardcoded into llvm and not really specific to WebAssembly. I think you might be better of opening an issue upstream in LLVM where this code lives.

Sadly swtiching to assembly I don't think will help with the WebAssembly build because we still have the issue call_indirect must have the correct signature.

That comment on __kmp_invoke_microtask does seem very strange. If the 1 argument case is all that is needed, why do that allow up to 15 args? And why are you seeing up to 25? These are all really question for upstream I think and not specific to emscripten or related to binaryen I believe.

[arsnyder16]

Yea I can try to bring it up there.

I started here because it seems strange that when running under Webassembly/emscripten toolchain I hit the >15 path but on a native build I don't see that. Sort of why I had a suspicion on the optimizer. I suppose I can test this assumption by just not optimizing

[sbc100]

Did you try a native build an one of the architectures that doesn't implement __kmp_invoke_microtask in asm? i.e. that actually uses that switch statement?

[arsnyder16]

@sbc100 Thanks! Turns out that this is specific to how the non asm version of __kmp_invoke_microtask works. I was able to confirm that with my native binary the args do exceed 15 but the x86-64 assembly implementation handles it fine.

I am going to submit a PR in llvm side, to add a test or two to show constraint and increase the headroom. That will leave it to the maintainers if they want to remove the hard constraint at all with more in depth signature changes