Make CoreCompileInputs.cache hash location-independent under DeterministicSourcePaths - #14559
Make CoreCompileInputs.cache hash location-independent under DeterministicSourcePaths#14559chsienki wants to merge 2 commits into
Conversation
…isticSourcePaths The Hash task feeding CoreCompileInputs.cache hashes absolute compile inputs (@(Compile), @(ReferencePath)) whose paths embed the checkout location, so an otherwise-identical build under a different repo root produces a different hash. That invalidates the incremental compile even though the sources are equivalent, which defeats seeding a build from another machine or agent. Add an optional PathMap input to the Hash task that applies the compiler's own path mapping (prefix replacement) to each hashed item, and pass $(PathMap) from _GenerateCompileDependencyCache when DeterministicSourcePaths is on. Because the map is applied per item and rewrites only a leading prefix, the raw $(PathMap) property - which embeds every root mid-string - is dropped from the hash under DeterministicSourcePaths; its effect is already captured by the mapped compile inputs. The parameter is empty by default, so the hash is unchanged for normal builds and the normalization is opt-in rather than a breaking change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e672bdd6-ac71-4e00-9085-71c0bf0a3059
|
Sibling PR: dotnet/sdk#55514 (the package-assets cache half of the same cross-root cache work). The two are validated together. |
| </ItemGroup> | ||
|
|
||
| <!-- | ||
| When the compiler is already producing location-independent output (DeterministicSourcePaths), apply the |
There was a problem hiding this comment.
Is DeterministicSourcePaths the right name for this? It seems like absolute paths are perfectly deterministic already (on the same machine and checkout location). Perhaps it should be called something like RelativeSourcePaths.
| normal builds. | ||
| --> | ||
| <PropertyGroup> | ||
| <_CoreCompileDependencyCachePathMap Condition="'$(DeterministicSourcePaths)' == 'true'">$(PathMap)</_CoreCompileDependencyCachePathMap> |
There was a problem hiding this comment.
So users need to specify both DeterministicSourcePaths and PathMap to opt into this? Perhaps we could default the path map to something like $(SolutionRoot)=/_/.
| /// Roslyn.Utilities.PathUtilities.NormalizePathPrefix. Comparison is ordinal because the compiler | ||
| /// expects consistent capitalization for path-map keys. | ||
| /// </summary> | ||
| private static string NormalizePathPrefix(string filePath, List<KeyValuePair<string, string>> pathMap) |
There was a problem hiding this comment.
Could this just be a custom msbuild task that takes the pathmap, a list of file paths to normalize; and returns a list of normalized file paths? That task could be defined in roslyn alongside Csc and the other tasks we have there.
Alternatively, I think we could just publish a source package from roslyn and consume it from both sdk and msbuild.
For example, there is already one being consumed from sdk: https://github.com/dotnet/roslyn/tree/c67ab9a38782b72900e4c758a6fcea476a600b44/src/NuGet/Microsoft.CodeAnalysis.BuildClient.Package
In dotnet/dotnet#1654 (comment) I have also a draft of one that would be consumed from msbuild.
There was a problem hiding this comment.
A task would work here but can't easily be used on the SDK (which also needs this) because there the real paths are also needed, so we'd have to duplicate its inputs to take both a real and mapped path.
Seems like a shared source package is the right way to go then.
TaskItem normalizes item specs to the host's native separator (backslashes become forward slashes on Unix via FixFilePath), while a path-map string passed to the Hash task is not normalized. The test hard-coded Windows drive paths and a backslash path map, so on Unix the slash-fixed item specs no longer shared the backslash prefix, the mapping was skipped, and the two roots failed to converge -- passing on Windows but failing on Linux and macOS. Build the paths and the path map from the host separator so the item specs and prefixes match, as they do in a real build where both derive from the same OS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e672bdd6-ac71-4e00-9085-71c0bf0a3059
2 participants
Note
Prototype / RFC — not requesting merge yet. This is part of the 1ES build-speedup effort and is opened as a draft to gather expert feedback on the approach. See the "Open questions" below.
Motivation
We are prototyping a content-addressed, cross-machine build cache (part of the 1ES build-speedup effort). For a cache to hit across different agents/checkout locations, a project built from identical sources under a different repository root must produce identical incremental state — otherwise every agent re-does work the cache should let it skip.
Today
_GenerateCompileDependencyCachewritesCoreCompileInputs.cacheasHash(@(Compile) + @(ReferencePath) + $(DefineConstants) + $(LangVersion) + $(Deterministic) + $(PathMap) + @(_CoreCompileResourceInputs)). Three of those inputs embed the absolute checkout path (the compile item paths, the reference paths, and$(PathMap)itself), so an otherwise-identical build under a different root produces a different hash and needlessly invalidates the compile — even when the compiler is already producing location-independent output via/pathmap.Change
Add an optional
PathMapinput to theHashtask that applies the compiler's own path mapping (prefix replacement) to each hashed item, and pass$(PathMap)from_GenerateCompileDependencyCacheonly when$(DeterministicSourcePaths)is true.$(PathMap)property (which embeds every root mid-string) is dropped from the hash underDeterministicSourcePaths; its effect is already captured by the mapped compile inputs.CommandLineParser.ParsePathMap/PathUtilities.NormalizePathPrefix).Validation
CoreCompile.Open questions for reviewers
_GenerateCompileDependencyCachethe right layer for this, or would you prefer it elsewhere?Hashtask now carries a copy of the compiler's path-map parsing helpers. There are sibling copies in dotnet/sdk and dotnet/roslyn. Suggestions for a shared, non-duplicated home welcome — given MSBuild sits below Roslyn in the dependency graph and can't referenceMicrosoft.CodeAnalysis.$(DeterministicSourcePaths)as the opt-in signal?