Skip to content

Implement Symbol Demanglers for Linux Binaries#2383

Merged
brianrob merged 9 commits into
microsoft:mainfrom
brianrob:brianrob/demanglers
Mar 18, 2026
Merged

Implement Symbol Demanglers for Linux Binaries#2383
brianrob merged 9 commits into
microsoft:mainfrom
brianrob:brianrob/demanglers

Conversation

@brianrob
Copy link
Copy Markdown
Member

@brianrob brianrob commented Mar 17, 2026

Implements C++ and Rust v0 demangling APIs. These were largely generated by Copilot, though I have lightly reviewed them. Extensive testing:

  • Basic testing against the specs.
  • 100% successful demangling of all symbols in libcoreclr.so (from .NET 10).
  • 100% successful demangling of all symbols from 500 randomly selected Ubuntu dbg packages.
  • 100% successful demangling of all symbols from 500 randomly selected Rust crates from crates.io.

Performance validation:

  • Created and ran benchmarks using BenchmarkDotNet.
  • Selected optimizations based on findings to improve CPU and memory efficiency.

Contributes to #2382.

brianrob and others added 3 commits March 17, 2026 15:48
Add ItaniumDemangler for demangling C++ symbols that follow the Itanium
C++ ABI mangling scheme (used by GCC, Clang, and other compilers on
Linux, macOS, and other non-Windows platforms).

Key features:
- Recursive descent parser handling the full Itanium ABI grammar
  including nested names, template args, expressions, function types,
  array types, pointer-to-member, decltype, and pack expansions.
- Substitution table with indexed lookup and dedup.
- GCC/LLVM linker suffix stripping (.cold, .isra, .constprop, etc.)
  and ELF symbol versioning (@glibc).
- ELF build-ID hash stripping (40-char SHA-1 or 32-char MD5) for
  symbols carrying metadata suffixes.
- ABI tag support (B<source-name>).
- C4/C5 constructor variants and Dp pack expansion types.
- Reusable instance-based Parser to minimize per-call allocations.
- Pre-sized StringBuilders and optimized string building throughout.
- Depth guard (256) to prevent StackOverflow on crafted input.

Includes 528 xUnit test cases covering operators, templates, nested
names, function types, special names, expressions, substitutions,
lambda expressions, ABI tags, pack expansions, and linker annotations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add RustDemangler for demangling Rust symbols using the v0 mangling
scheme (RFC 2603), identified by the _R prefix.

Key features:
- Recursive descent parser for the Rust v0 grammar including paths,
  types, const generics, closures, shims, and trait implementations.
- Backref resolution with cached Func delegates to avoid per-call
  delegate allocation.
- Base-62 number parsing for backrefs and disambiguators.
- Punycode support for Unicode identifiers.
- Basic type mapping, function signatures, dyn trait bounds, and
  higher-ranked lifetime binders.
- Reusable instance-based Parser with scratch StringBuilder for
  leaf-level formatting and pre-sized loop StringBuilders.
- Depth guard (200) to prevent StackOverflow on crafted input.

Includes 33 xUnit test cases covering basic types, generics, closures,
trait impls, const generics, backrefs, and edge cases.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add TraceEventBenchmarks project with MemoryDiagnoser and EtwProfiler
for measuring CPU time and memory allocations of the Itanium C++ and
Rust v0 demanglers.

Benchmark classes:
- ItaniumDemanglerBenchmarks (Short/Medium/Long/All with 45,694 symbols)
- ItaniumDemanglerMetadataBenchmarks (49,349 symbols with ELF hashes)
- RustDemanglerBenchmarks (Short/Medium/Long/All with 11,239 symbols)

Infrastructure:
- BenchmarkInput.cs loads real-world symbols from inputs/ directory and
  provides curated representative symbols for micro-benchmarks.
- PublicSign with MSFT.snk (same pattern as TraceEvent.Tests) to
  satisfy InternalsVisibleTo without MicroBuild signing.
- BenchmarkDotNet 0.14.0 and BenchmarkDotNet.Diagnostics.Windows 0.14.0
  added to central package management.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@brianrob
Copy link
Copy Markdown
Member Author

cc: @zachcmadsen

@brianrob brianrob marked this pull request as ready for review March 18, 2026 03:02
@brianrob brianrob requested a review from leculver March 18, 2026 03:02
Copy link
Copy Markdown
Collaborator

@leculver leculver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few questions/suggestions before marking approve:

Allocations: I do wonder how much we are allocating when demangling. There are several places where we could be using ReadOnlySpan back and forth instead of creating new strings. There seems to be quite a few substring calls, and similar. Ultimately though, you have to build a new string for the demangled name, and I'm not sure how much time/GC we'd save by trying to do less allocations.

You are using Benchmark.Net, can you check how much the DemangleAll tests allocate, and subtract the result of _allSymbols.Sum(r => demangler.Demangle(symbol).Length)? If it's a really high number, maybe consider having copilot rewrite some of these demangle methods to not allocate? (Mostly be using read only spans in helpers.)

If the number seems reasonable, then don't bother, of course.

AllSymbols Demangle Test: Would we catch regressions better by demangling all symbols in cpp_symbols.txt/rust_symbols.txt and comparing it against a pre-demangled version? IE run everything through the demangler now, store that result and assert it in one final "check-all" test. This does nothing for us now, but if you have to modify the demangler in the future, it will ensure we don't actually drift old functions. (Or that any drift is good and actually fixing real bugs.)

Or does the tests as the exist now fully cover the entirety of demangling, and this would just be overhead?

brianrob and others added 6 commits March 18, 2026 07:57
…arsing

- Replace string + operator concatenation with string.Concat() throughout
  both ItaniumDemangler and RustDemangler to eliminate intermediate string
  allocations.
- Change ItaniumDemangler to pass an end offset to the parser instead of
  allocating a Substring in StripLinkerAnnotations (renamed to
  ComputeLinkerAnnotationEnd).
- Add _endOffset field to Itanium Parser, matching the pattern already used
  by the Rust parser.
- Multi-target benchmark project for net462 and net8.0.
- Remove [EtwProfiler] attribute from benchmarks (requires admin elevation).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace 15 new StringBuilder(...) allocations in instance methods with
a reusable pool (AcquireSb/ReleaseSb). The pool holds 8 pre-allocated
StringBuilders and falls back to new allocations for deep recursion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make WrapModifier non-static so it can use the Parser's StringBuilder
pool instead of allocating new StringBuilders on every pointer/reference
type wrapping operation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add StringBuilder pool (AcquireSb/ReleaseSb) to the Rust demangler
parser, replacing new StringBuilder allocations in ParsePath (generic
args), ParseType (tuples), ParseFnSig, ParseDynBounds, and
ParseDynTrait.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 42 new test cases covering real-world symbol patterns found in
the benchmark input files that were not covered by existing tests.

Itanium demangler (22 new tests):
- Operator new/delete with nothrow_t
- Sized operator delete (C++14)
- Function pointer parameters from real libraries
- ELF versioned symbols (@glibcxx, @CXXABI)
- GCC linker suffixes on complex symbols
- Variadic pack expansion (Dp)
- Address-of in template arguments (Xad)
- Real-world function signatures
- Lambda with discriminators in std::_Function_handler

Rust demangler (20 new tests):
- Y/X trait impls with fn pointer and primitive types
- Nested closures (2-6 levels deep)
- Unsafe extern "C" fn pointer types in generics
- Shim vtable patterns (NS)
- Array types in generics
- Dyn trait with higher-ranked lifetime binder (DG0_)
- Multiple backreferences in complex contexts
- Complex trait impls with nested closures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@brianrob brianrob requested a review from a team as a code owner March 18, 2026 19:15
@brianrob
Copy link
Copy Markdown
Member Author

Thank you for the review @leculver.

Regarding allocations - I did a bunch of optimization in the allocation space before originally creating the PR. I had originally looked at potentially using ReadOnlySpan, but it turns out that the bulk of the allocation is in materializing the strings for the demangled symbol and not in reading/parsing the mangled name. That said, I did send copilot off to push further on this using the benchmarks, and it made some improvements using pooled StringBuilder instances, since that's where the bulk of the allocation happens. Here's the before/after findings:

DemangleAll — Full Corpus

The DemangleAll benchmarks demangle every symbol in the test corpus and are the most
representative measure of real-world impact.

Demangler Runtime Alloc Before Alloc After Alloc Reduction Mean Before Mean After Speedup
Itanium .NET 8.0 93.8 MB 54.9 MB 41% 43.0 ms 35.0 ms 19%
Itanium .NET Framework 4.6.2 100.7 MB 58.2 MB 42% 73.7 ms 64.6 ms 12%
Itanium + Metadata .NET 8.0 110.5 MB 61.1 MB 45% 57.6 ms 47.2 ms 18%
Itanium + Metadata .NET Framework 4.6.2 118.4 MB 64.7 MB 45% 91.5 ms 80.4 ms 12%
Rust .NET 8.0 34.7 MB 24.0 MB 31% 17.5 ms 15.6 ms 11%
Rust .NET Framework 4.6.2 34.1 MB 25.5 MB 25% 27.5 ms 25.8 ms 6%

Per-Symbol Benchmarks — .NET 8.0

These benchmarks demangle a single symbol and show the allocation impact at different
symbol complexity levels.

Demangler Symbol Alloc Before Alloc After Alloc Reduction Mean Before Mean After
Itanium Short 288 B 72 B 75% 83 ns 72 ns
Itanium Medium 1,048 B 472 B 55% 471 ns 342 ns
Itanium Long 1,799 KB 926 KB 49% 303 μs 238 μs
Itanium + Metadata Short 352 B 88 B 75% 137 ns 114 ns
Itanium + Metadata Medium 904 B 488 B 46% 357 ns 269 ns
Itanium + Metadata Long 1,799 KB 926 KB 49% 302 μs 239 μs
Rust Short 120 B 120 B 0% 86 ns 86 ns
Rust Medium 1,400 B 1,088 B 22% 642 ns 592 ns
Rust Long 227 KB 121 KB 47% 54 μs 43 μs

Regarding the tests - The txt files are just input to the benchmarks and not to any correctness tests. My goal was to have enough variation of mangling to get a reasonable read on performance, rather than just hitting the simple or complex cases. From a correctness perspective, the existing test cases are reasonably good, but I asked copilot to do a comparison between what we have in the functional test space and what exists in the allsymbols benchmark, and so it identified another 42 test cases to add.

All of these changes should now be pushed in separate commits so that you can see them without looking at the entire change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants