Support Apple .dSYM bundles in symbol packages/indexing

[agocke]

We currently use .dwarf files, which are unsupported.

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We currently use .dwarf files, which are unsupported.

Author:	agocke
Assignees:	-
Labels:	area-Infrastructure
Milestone:	8.0.0

[BruceForstall]

Some notes on this issue:

Apple's supported symbol file format is the .dSYM bundle. A .dSYM bundle is a directory tree in a specific format that contains various files, including the DWARF format symbols for the executable, as well as various meta-data. In the macOS Finder it appears to be a single file (but can be explored using the "Show Package Contents" command). Confusingly, much documentation uses the term ".dSYM file", which is used essentially interchangeably with ".dSYM bundle". Note, however, that the .dSYM is always a bundle (directory tree), never a single file.

The .NET build process currently creates a single file for each executable file's symbols, with the file extension .dwarf. This is not the standard way that symbol files are created and used on macOS. There is some concern that this might not be supported long-term. There may also be developer experience improvements to using .dSYM format symbols, such as automatic Spotlight integration for symbol finding.

Creating a .dwarf file is done by running the dsymutil --flat command on an executable. Simply removing the --flat argument during the build process allows creating .dSYM bundles. However, there are lots of parts of the .NET build and packaging process that need to be updated to deal with the fact that symbols are now directory trees and not just single files.

It is best to consider a .dSYM bundle as opaque: do not make assumptions about the structure and contents of the bundle. There may be reasons to break this rule. For instance, one file inside the .dSYM bundle is the DWARF symbol file. We might want to read that file directly in order to index the symbols. However, even that might be better done by addressing the .dSYM bundle as a whole, e.g., using the following commands to get the build UUID for the contained symbols:

% dwarfdump --uuid createdump.dSYM 
UUID: D5BFA2CC-C73B-3E6F-8B2C-E96246EE2EAD (arm64) createdump.dSYM/Contents/Resources/DWARF/createdump

A description of some of the challenges in converting the .NET build to using .dSYM bundles follows.

MSBuild files versus directories

The .NET build uses msbuild for most activities. MSBuild has a strong bias towards files and not directories, at least when it comes to item glob expressions. For example, using:

<Files Include="*.pdb;*.dSYM"/>

Will include all the *.pdb files in the directory, but will ignore any *.dSYM bundles.

Using:

<Files Include="*.dSYM/**"/>

Will include all the files in the .dSYM bundle in the Files item, individually. This can be useful sometimes, but often it is useful to treat the .dSYM bundle as a "single", "whole" thing, and not as a set of pieces.

MSBuild item transform globbing

A lot of MSBuild code does transforms on items to form new items. One example is to transform a list of generated executables to a list of the related symbol files (or bundles):

<NonWindowsSymbolFile Include="@(NonWindowsNativeFile -> '%(Identity).dSYM')</NonWindowsSymbolFile>

This is a 1-to-1 mapping. However, if we need to expand this to the full list of files in the bundle (instead of just the bundle itself), we might do:

<NonWindowsSymbolFile Include="@(NonWindowsNativeFile -> '%(Identity).dSYM/**')</NonWindowsSymbolFile>

(Note the /** wildcard.)

This doesn't work, however, as the "globbing" does not get expanded by MSBuild during the transform. A workaround is to "go through" a property:

        <PropertyGroup>
            <SymFilesReplace>@(Files -> '%(Identity).dSYM/**')</SymFilesReplace>
        </PropertyGroup>
        <ItemGroup>
            <SymFiles Include="$(SymFilesReplace)" />
        </ItemGroup>

Note: if using the "expand all files" option in MSBuild, it seems to be very difficult to then associate a per-.dSYM attribute to each of the expanded files, such as the target directory each should be copied to in the NuGet package.

MSBuild copy directories

MSBuild has a built-in Copy task to copy files. However, to copy directories, preserving directory structure, requires a bit of a hack. A December 2023 change to the Copy task did add folder tree copying that might be helpful, but it isn't available in our build tools yet, and it requires distinguishing files and directories to be copied.

If we want to treat .dSYM bundles as an atomic unit, we might need to implement a CopyFileOrFolder custom MSBuild task that can handle either files (e.g., *.pdb) or folders (e.g. *.dSYM) and copy them appropriately.

NuGet packing

.NET packs native builds and symbols into NuGet "legacy" symbols packages. For example, the Microsoft.NETCore.App.Runtime.osx-arm64.9.0.0-dev.symbols.nupkg package contains (this is a small subset):

./runtimes/osx-arm64/native/createdump
./runtimes/osx-arm64/native/createdump.dwarf
./runtimes/osx-arm64/native/libclrjit.dylib
./runtimes/osx-arm64/native/libclrjit.dylib.dwarf

To support .dSYM, these .dwarf files would need to be replaced with their full .dSYM bundle replacements.

It appears that NuGet itself has assumptions about symbol files being single files with a distinguished file extension. Neither of these things are true for .dSYM bundles: they are directories, and the files they contain both have common file extensions not specific to debug information (e.g., ".plist", ".yml") and have no file extension at all (for the DWARF info file).

The MSBuild Shared Framework SDK file in the arcade repo, in src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets, defines an MSBuild property that determines how NuGet packages symbols. Specifically:

<AllowedOutputExtensionsInSymbolsPackageBuildOutputFolder>$(AllowedOutputExtensionsInSymbolsPackageBuildOutputFolder);.map;.r2rmap;.dbg;.debug;.dwarf</AllowedOutputExtensionsInSymbolsPackageBuildOutputFolder>

This is passed through many layers before ending up being used in the NuGet client packing code to determine how to pack symbols packages, here.

It needs more investigation to determine if NuGet needs to learn about .dSYM bundles as directories with multiple files, or whether all the packing changs that are needed can be accomplished without touching NuGet.

Symbol server uploading

One use for the generated symbols packages is apparently to give to the symbol server indexer. It cracks the NuGet package, looks for symbol files, and index and uploads the files. It is presumed that if the symbol package contains a .dSYM bundle, including the DWARF debug info file, that the symbol indexer will be able to understand that the debug info file actually is a debug info file (despite having no file extension), and handle it properly. This might need to be verified.

Question: where is the symbol server uploading code?

It is a separate issue about whether the symbol indexer should be augmented such that (1) the entire .dSYM bundle is indexed, (2) Mac native debuggers (lldb) can use the symbol server to automatically find and download the archived .dSYM bundle.

dSYM-in-a-file

It's easy to get the build to produce .dSYM bundles for native code symbols. The biggest problems seem to be in how to package the bundles in NuGet. It has been suggested that perhaps a halfway point would be to (1) produce the .dSYM bundle, then (2) immediately compress the bundle into a single file using zip or tar, e.g., create a .dSYMzip file. This could be packaged like we currently package .dwarf files. Since it's a single file, it might be easier to deal with than teaching all layers of the build system about folders. It would almost certainly require changes to the symbol server indexer.

MSBuild complexity

General comment on working in MSBuild in the .NET build system: the rules that affect the build are spread out widely, between the eng tree, the src/installer tree, the arcade repo (both MSBuild files and custom MSBuild targets), and the NuGet source tree. It is extremely difficult to find documentation on many MSBuild properties and items, determine where they are defined and used, and generally understand the order of operations. There is no IDE support nor any debugger.

[BruceForstall]

A partial implementation, or, a set of attempts: #96456