link against jemalloc

[nickdesaulniers]

not sure if it would make a difference, but I noticed that in bolt's dockerfile they're linking llvm against jemalloc.
https://github.com/llvm/llvm-project/blob/main/bolt/utils/docker/Dockerfile#L23
That might be interesting to try, perhaps if any of the memory allocations are showing up in profiles of kernel builds. I guess we're using whatever malloc implementation musl has? I wonder if it has its own? "mallocng?" https://git.musl-libc.org/cgit/musl/tree/src/malloc/mallocng/malloc.c

[nickdesaulniers]

https://github.com/jemalloc/jemalloc/wiki/Getting-Started

[nickdesaulniers]

Here's a profile of the kernel build with stage5 pgo:

Samples: 13M of event 'cycles:ppu', Event count (approx.): 9699067643159                                            
  Overhead  Shared Object              Symbol                                                                       
+    2.91%  clang                      [.] clang::SourceManager::getFileIDLocal
+    2.15%  clang                      [.] llvm::StringMapImpl::LookupBucketFor
+    1.72%  clang                      [.] __libc_malloc_impl
+    1.61%  clang                      [.] clang::TokenLexer::Lex
+    1.49%  clang                      [.] clang::Lexer::LexTokenInternal
+    1.40%  clang                      [.] memcpy
+    1.00%  clang                      [.] clang::Preprocessor::Lex
+    0.99%  clang                      [.] clang::ASTContext::getDeclAttrs
+    0.95%  clang                      [.] clang::Lexer::LexIdentifierContinue
+    0.95%  clang                      [.] GetFullTypeForDeclarator
+    0.90%  clang                      [.] (anonymous namespace)::IntExprEvaluator::VisitBinaryOperator
+    0.88%  clang                      [.] CheckICE
     0.80%  clang                      [.] __libc_free
+    0.75%  clang                      [.] clang::SourceManager::getFileID
+    0.74%  clang                      [.] clang::TokenLexer::ExpandFunctionArguments
+    0.67%  clang                      [.] alloc_slot
+    0.65%  clang                      [.] get_meta
+    0.65%  clang                      [.] memset
     0.62%  [unknown]                  [k] 0xffffffffaba01087 (kptr_restrict not disabled)
+    0.55%  clang                      [.] clang::Preprocessor::getMacroDefinition  (end top 20)
     0.50%  clang                      [.] clang::Parser::ParseDeclarationSpecifiers
     0.50%  clang                      [.] memcmp
     0.49%  clang                      [.] llvm::FoldingSetBase::FindNodeOrInsertPos
     0.47%  clang                      [.] GetDiagInfo
     0.46%  clang                      [.] clang::Sema::LookupName
     0.46%  clang                      [.] strlen

so __libc_malloc_impl, __libc_free, alloc_slot and get_meta all seem to be coming from musl's memory allocator. Seeing memcpy and memset in the top 20 is also with memcmp and strlen not too far behind :(.

[nickdesaulniers]

@petrhosek mentioned that https://github.com/mjansson/rpmalloc might be interesting to play with; it's used for host tools for fuchsia.

[petrhosek]

I'd also experiment with https://github.com/microsoft/mimalloc and https://github.com/microsoft/snmalloc, all of these are already supported by LLVM on Windows and we could consider extending that support to other platforms.

[nickdesaulniers]

heh, all of those are listed: https://github.com/llvm/llvm-project/blob/be6af89f85ebd04646b5704301470f02b70a0447/llvm/lib/Support/CMakeLists.txt#L89-L95 so it seems that we can just set LLVM_INTEGRATED_CRT_ALLOC=path/to/rpmalloc.

[nickdesaulniers]

Adding the following -D LLVM_INTEGRATED_CRT_ALLOC=/rpmalloc-1.4.4 to my cmake invocation produces:

CMake Error at CMakeLists.txt:575 (message):
  LLVM_INTEGRATED_CRT_ALLOC is only supported on Windows.

which is lame. I guess that's the purpose of https://reviews.llvm.org/D101427.