Implement Symbol Demanglers for Linux Binaries#2383
Conversation
Add ItaniumDemangler for demangling C++ symbols that follow the Itanium C++ ABI mangling scheme (used by GCC, Clang, and other compilers on Linux, macOS, and other non-Windows platforms). Key features: - Recursive descent parser handling the full Itanium ABI grammar including nested names, template args, expressions, function types, array types, pointer-to-member, decltype, and pack expansions. - Substitution table with indexed lookup and dedup. - GCC/LLVM linker suffix stripping (.cold, .isra, .constprop, etc.) and ELF symbol versioning (@glibc). - ELF build-ID hash stripping (40-char SHA-1 or 32-char MD5) for symbols carrying metadata suffixes. - ABI tag support (B<source-name>). - C4/C5 constructor variants and Dp pack expansion types. - Reusable instance-based Parser to minimize per-call allocations. - Pre-sized StringBuilders and optimized string building throughout. - Depth guard (256) to prevent StackOverflow on crafted input. Includes 528 xUnit test cases covering operators, templates, nested names, function types, special names, expressions, substitutions, lambda expressions, ABI tags, pack expansions, and linker annotations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add RustDemangler for demangling Rust symbols using the v0 mangling scheme (RFC 2603), identified by the _R prefix. Key features: - Recursive descent parser for the Rust v0 grammar including paths, types, const generics, closures, shims, and trait implementations. - Backref resolution with cached Func delegates to avoid per-call delegate allocation. - Base-62 number parsing for backrefs and disambiguators. - Punycode support for Unicode identifiers. - Basic type mapping, function signatures, dyn trait bounds, and higher-ranked lifetime binders. - Reusable instance-based Parser with scratch StringBuilder for leaf-level formatting and pre-sized loop StringBuilders. - Depth guard (200) to prevent StackOverflow on crafted input. Includes 33 xUnit test cases covering basic types, generics, closures, trait impls, const generics, backrefs, and edge cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add TraceEventBenchmarks project with MemoryDiagnoser and EtwProfiler for measuring CPU time and memory allocations of the Itanium C++ and Rust v0 demanglers. Benchmark classes: - ItaniumDemanglerBenchmarks (Short/Medium/Long/All with 45,694 symbols) - ItaniumDemanglerMetadataBenchmarks (49,349 symbols with ELF hashes) - RustDemanglerBenchmarks (Short/Medium/Long/All with 11,239 symbols) Infrastructure: - BenchmarkInput.cs loads real-world symbols from inputs/ directory and provides curated representative symbols for micro-benchmarks. - PublicSign with MSFT.snk (same pattern as TraceEvent.Tests) to satisfy InternalsVisibleTo without MicroBuild signing. - BenchmarkDotNet 0.14.0 and BenchmarkDotNet.Diagnostics.Windows 0.14.0 added to central package management. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
cc: @zachcmadsen |
leculver
left a comment
There was a problem hiding this comment.
I have a few questions/suggestions before marking approve:
Allocations: I do wonder how much we are allocating when demangling. There are several places where we could be using ReadOnlySpan back and forth instead of creating new strings. There seems to be quite a few substring calls, and similar. Ultimately though, you have to build a new string for the demangled name, and I'm not sure how much time/GC we'd save by trying to do less allocations.
You are using Benchmark.Net, can you check how much the DemangleAll tests allocate, and subtract the result of _allSymbols.Sum(r => demangler.Demangle(symbol).Length)? If it's a really high number, maybe consider having copilot rewrite some of these demangle methods to not allocate? (Mostly be using read only spans in helpers.)
If the number seems reasonable, then don't bother, of course.
AllSymbols Demangle Test: Would we catch regressions better by demangling all symbols in cpp_symbols.txt/rust_symbols.txt and comparing it against a pre-demangled version? IE run everything through the demangler now, store that result and assert it in one final "check-all" test. This does nothing for us now, but if you have to modify the demangler in the future, it will ensure we don't actually drift old functions. (Or that any drift is good and actually fixing real bugs.)
Or does the tests as the exist now fully cover the entirety of demangling, and this would just be overhead?
…arsing - Replace string + operator concatenation with string.Concat() throughout both ItaniumDemangler and RustDemangler to eliminate intermediate string allocations. - Change ItaniumDemangler to pass an end offset to the parser instead of allocating a Substring in StripLinkerAnnotations (renamed to ComputeLinkerAnnotationEnd). - Add _endOffset field to Itanium Parser, matching the pattern already used by the Rust parser. - Multi-target benchmark project for net462 and net8.0. - Remove [EtwProfiler] attribute from benchmarks (requires admin elevation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace 15 new StringBuilder(...) allocations in instance methods with a reusable pool (AcquireSb/ReleaseSb). The pool holds 8 pre-allocated StringBuilders and falls back to new allocations for deep recursion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make WrapModifier non-static so it can use the Parser's StringBuilder pool instead of allocating new StringBuilders on every pointer/reference type wrapping operation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add StringBuilder pool (AcquireSb/ReleaseSb) to the Rust demangler parser, replacing new StringBuilder allocations in ParsePath (generic args), ParseType (tuples), ParseFnSig, ParseDynBounds, and ParseDynTrait. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 42 new test cases covering real-world symbol patterns found in the benchmark input files that were not covered by existing tests. Itanium demangler (22 new tests): - Operator new/delete with nothrow_t - Sized operator delete (C++14) - Function pointer parameters from real libraries - ELF versioned symbols (@glibcxx, @CXXABI) - GCC linker suffixes on complex symbols - Variadic pack expansion (Dp) - Address-of in template arguments (Xad) - Real-world function signatures - Lambda with discriminators in std::_Function_handler Rust demangler (20 new tests): - Y/X trait impls with fn pointer and primitive types - Nested closures (2-6 levels deep) - Unsafe extern "C" fn pointer types in generics - Shim vtable patterns (NS) - Array types in generics - Dyn trait with higher-ranked lifetime binder (DG0_) - Multiple backreferences in complex contexts - Complex trait impls with nested closures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thank you for the review @leculver. Regarding allocations - I did a bunch of optimization in the allocation space before originally creating the PR. I had originally looked at potentially using DemangleAll — Full CorpusThe
Per-Symbol Benchmarks — .NET 8.0These benchmarks demangle a single symbol and show the allocation impact at different
Regarding the tests - The txt files are just input to the benchmarks and not to any correctness tests. My goal was to have enough variation of mangling to get a reasonable read on performance, rather than just hitting the simple or complex cases. From a correctness perspective, the existing test cases are reasonably good, but I asked copilot to do a comparison between what we have in the functional test space and what exists in the allsymbols benchmark, and so it identified another 42 test cases to add. All of these changes should now be pushed in separate commits so that you can see them without looking at the entire change. |
Implements C++ and Rust v0 demangling APIs. These were largely generated by Copilot, though I have lightly reviewed them. Extensive testing:
Performance validation:
Contributes to #2382.