[WIP] Add Configuration references by rosebyte · Pull Request #129734 · dotnet/runtime

rosebyte · 2026-06-23T10:19:42Z

No description provided.

dotnet-policy-service · 2026-06-23T10:21:31Z

Tagging subscribers to this area: @dotnet/area-extensions-configuration
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

This PR introduces a new “configuration references” feature for Microsoft.Extensions.Configuration, enabling configuration values to be expanded via reference syntax (e.g., ref(Target:Key)) and composed values (e.g., format(template, key1, key2, ...)). It does so by adding a contextual configuration source/provider that materializes expanded values on top of upstream providers, plus a fairly comprehensive new test suite.

Changes:

Adds a new reference rule model + pattern matcher (ReferenceRule, KeyPattern) and a contextual configuration provider (ReferenceConfigurationProvider) to materialize expanded values and mirror subtrees.
Extends ConfigurationBuilder and ConfigurationManager to support building “contextual” configuration sources that require an upstream provider snapshot.
Introduces new public API surface (AllowReferences, ConfigurationReferenceBuilder, ConfigurationExpansion) and adds end-to-end tests.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/libraries/Microsoft.Extensions.Configuration/tests/ConfigurationReferenceTests.cs	New test coverage for reference resolution, templates, composition, reload behavior, and parsing.
src/libraries/Microsoft.Extensions.Configuration/src/Resources/Strings.resx	Adds new user-facing error strings for reference expansion and validation.
src/libraries/Microsoft.Extensions.Configuration/src/ReferenceRule.cs	Defines rule semantics (subject + allowed/denied targets) and validates parser output.
src/libraries/Microsoft.Extensions.Configuration/src/ReferenceConfigurationSource.cs	Contextual source that requires upstream provider snapshot to build its provider.
src/libraries/Microsoft.Extensions.Configuration/src/ReferenceConfigurationProvider.cs	Core implementation: resolves references, mirrors subtrees, composes format expansions, handles reload.
src/libraries/Microsoft.Extensions.Configuration/src/ReferenceConfigurationBuilderExtensions.cs	Public builder extension method `AllowReferences(...)`.
src/libraries/Microsoft.Extensions.Configuration/src/KeyPattern.cs	Wildcard/doublestar key-pattern matching for subjects/targets.
src/libraries/Microsoft.Extensions.Configuration/src/IContextualConfigurationSource.cs	Internal contract to allow sources to build with upstream provider list.
src/libraries/Microsoft.Extensions.Configuration/src/ConfigurationReferenceBuilder.cs	Public builder object for declaring rules and customizing the parser.
src/libraries/Microsoft.Extensions.Configuration/src/ConfigurationManager.cs	Adds contextual-source dispatch when adding/reloading sources.
src/libraries/Microsoft.Extensions.Configuration/src/ConfigurationExpansion.cs	Public struct representing expansion kinds (reference/literal/format) returned by parsers.
src/libraries/Microsoft.Extensions.Configuration/src/ConfigurationBuilder.cs	Adds contextual-source dispatch during build.
src/libraries/Microsoft.Extensions.Configuration/ref/Microsoft.Extensions.Configuration.cs	Updates public API contract to include the new APIs/types.

+        public static ConfigurationExpansion Format(string template, params string[] referencedKeys)
+        {
+            ArgumentNullException.ThrowIfNull(template);
+            ArgumentNullException.ThrowIfNull(referencedKeys);
+            return new ConfigurationExpansion(template, referencedKeys.Length switch
+            {
+                0 => StringValues.Empty,
+                1 => new StringValues(referencedKeys[0]),
+                _ => new StringValues(referencedKeys),
+            });
+        }


+        [Fact]
+        public void Configurable_ValidSelection_ResolvesThrough()
+        {
+            IConfigurationRoot root = BuilderWith(Dict(
+                    ("Shared:Prod:Credential", "prod"),


+    public sealed partial class ConfigurationReferenceBuilder
+    {
+        internal ConfigurationReferenceBuilder() { }
+        public Microsoft.Extensions.Configuration.ConfigurationReferenceBuilder Allow(string subject, string target, params string[] additionalTargets) { throw null; }
+        public Microsoft.Extensions.Configuration.ConfigurationReferenceBuilder Deny(string subject, string target, params string[] additionalTargets) { throw null; }


+            ReferenceRule rule = FindOrCreateRule(NormaliseSubject(subject));
+            rule.AddTarget(NormaliseTarget(target));
+            foreach (string additionalTarget in additionalTargets)
+            {
+                rule.AddTarget(NormaliseTarget(additionalTarget));
+            }
+            return this;


+            ReferenceRule rule = FindOrCreateRule(NormaliseSubject(subject));
+            rule.AddDisallowedTarget(NormaliseTarget(target));
+            foreach (string additionalTarget in additionalTargets)
+            {
+                rule.AddDisallowedTarget(NormaliseTarget(additionalTarget));
+            }
+            return this;


+        // Reads the value already materialised for <paramref name="key"/> if any; otherwise
+        // falls back to the first non-empty upstream literal at the same key.
+        private string? OverlayAwareRead(string key, Dictionary<string, string?> values)
+        {
+            if (values.TryGetValue(key, out string? ov))
+            {
+                return ov;
+            }
+            for (int i = _upstream.Count - 1; i >= 0; i--)
+            {
+                if (_upstream[i].TryGet(key, out string? pv) && !string.IsNullOrEmpty(pv))
+                {
+                    return pv;
+                }
+            }
+            return null;
+        }


+                    // A non-empty upstream literal at the overlay key should win on read;
+                    // skip the overlay entry so it does.
+                    string? sourceAtOverlay = ReadUpstream(overlayKey);
+                    if (string.IsNullOrEmpty(sourceAtOverlay))
+                    {
+                        string? targetValue = OverlayAwareRead(childKey, values);
+                        if (targetValue is not null)
+                        {
+                            values[overlayKey] = targetValue;
+                        }
+                    }


+  <data name="Error_ReferenceTargetsRequired" xml:space="preserve">
+    <value>At least one non-empty reference target must be provided.</value>
+  </data>
+  <data name="Error_ReferenceTargetInvalid" xml:space="preserve">
+    <value>'{0}' is not a valid reference target pattern. Each ':'-separated segment must be non-empty; '*' is permitted only as a whole segment.</value>
+  </data>


Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

+        private void OnUpstreamChanged()
+        {
+            if (!_disposed)
+            {
+                // Upstream changed, so the materialised state may be stale; drop it, then propagate so
+                // consumers re-read and re-materialise.
+                _valid = false;
+                OnReload();
+            }
+        }


+            // Configuration expansion is opt-out: unless disabled, append a reference provider at the
+            // last position so it sees every source and its resolved values win on read. Skipped when
+            // there are no sources so an empty builder still reports "no sources" on write.
+            if (providers.Count > 0 && ConfigurationExpansionSwitch.Enabled)
+            {
+                providers.Add(new ReferenceConfigurationProvider(providers.ToArray()));
+            }


+        private static InvalidOperationException ParseError(string whole) =>
+            new InvalidOperationException(SR.Format(SR.Error_ReferenceMarkerInvalid, whole));
+    }


Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

+    // A leading backslash escapes a marker (\ref(...) / \format(...)) and yields the literal text with the
+    // backslash removed. Any other value is taken verbatim, and a direct non-empty value always wins over a
+    // value inherited through an ancestor reference.


Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

+    // A leading backslash escapes a marker (\ref(...) / \format(...)) and yields the literal text with the
+    // backslash removed. Any other value is taken verbatim, and a direct non-empty value always wins over a
+    // value inherited through an ancestor reference.


+        // Upper bound on reference hops in a single resolution. Real configurations chain references a handful of
+        // levels deep at most; exceeding this indicates a self-feeding reference that would otherwise never settle.
+        private const int MaxDepth = 100;


+  <data name="Error_ReferenceComposedValueMissing" xml:space="preserve">
+    <value>The reference '{0}' used in the composed value at '{1}' resolved to no value.</value>
+  </data>
+  <data name="Error_ReferenceTemplateInvalid" xml:space="preserve">
+    <value>The composed value template at '{0}' is not a valid format string.</value>
+  </data>


tarekgh · 2026-07-16T16:03:20Z

@rosebyte could you please get some benchmarks reading configuration especially without references to ensure we don't have any perf regression with configuration reading. Thanks!

rosebyte · 2026-07-17T05:47:28Z

@tarekgh, I'm doing it continuously when iterating the code, with the current approach we are about the same level in general with the only exception of complete miss (where we really have to check all parents down the road). I'm still simplifying the code and getting better results though.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

+    // A value is a reference when it carries the marker ref(target): the subject takes the target's value and mirrors
+    // its subtree. A leading backslash escapes the marker (\ref(...)) and yields the literal text with the backslash
+    // removed. Any other value is taken verbatim, and a direct non-empty value always wins over a value inherited
+    // through an ancestor reference.


+        private void Promote(string key, CacheEntry entry)
+        {
+            // First writer wins: a key resolves to the same result for the life of this instance, so an existing entry
+            // is left untouched. Only a genuine insert counts toward the rotation threshold.
+            if (_hot.TryAdd(key, entry) && Interlocked.Increment(ref _hotCount) >= GenerationCapacity)
+            {
+                Rotate();
+            }
+        }


Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

+            private int _hops;
+            private HashSet<string>? _fired;
+
+            public void Advance(string key, int prefixLength)
+            {
+                if (++_hops <= GraceHops)
+                {
+                    return;
+                }
+
+                string governing = key.Substring(0, prefixLength);
+                if (!(_fired ??= new HashSet<string>(StringComparer.OrdinalIgnoreCase)).Add(governing))
+                {
+                    throw new InvalidOperationException(
+                        SR.Format(SR.Error_ReferenceCycle, string.Join(" -> ", _fired) + " -> " + governing));
+                }
+            }


+            bool fired = false;
+            ChangeToken.OnChange(root.GetReloadToken, () => fired = true);
+


Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

+            if (targets is null)
+            {
+                return Empty;
+            }
+
+            return new ReferenceIndex(Flatten(targets));
+        }


Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/libraries/Microsoft.Extensions.Configuration/src/ReferenceEngine.cs:210

CycleGuard builds the exception message from a HashSet (string.Join over _fired), which is not ordered and can produce nondeterministic, hard-to-debug cycle strings. Since this is a user-facing error, it should be stable and ideally reflect the actual walk order.

                string governing = key.Substring(0, prefixLength);
                if (!(_fired ??= new HashSet<string>(StringComparer.OrdinalIgnoreCase)).Add(governing))
                {
                    throw new InvalidOperationException(
                        SR.Format(SR.Error_ReferenceCycle, string.Join(" -> ", _fired) + " -> " + governing));
                }

+                foreach (KeyValuePair<string, string?> entry in ScanProvider(provider))
+                {
+                    // The highest provider that declares a path's $ref decides it; claim the path here so a lower
+                    // provider's $ref for the same path is ignored. An empty target claims the path but records no
+                    // reference, so a higher provider can drop a lower reference by setting its $ref to empty.
+                    if (IsRefKey(entry.Key, out string? referencePath)
+                        && (claimed ??= new HashSet<string>(StringComparer.OrdinalIgnoreCase)).Add(referencePath)
+                        && !string.IsNullOrEmpty(entry.Value))
+                    {
+                        (targets ??= new Dictionary<string, (string Target, int Level)>(StringComparer.OrdinalIgnoreCase))[referencePath] = (entry.Value!, i);
+                    }
+                }


+        // Exposes this provider's key/value pairs so the reference engine can build its reference index by scanning the
+        // already-loaded backing dictionary directly, instead of going through the O(n^2) GetChildKeys reconstruction.
+        internal IEnumerable<KeyValuePair<string, string?>> GetEntriesForReferenceScan() => Data;
+


Copilot AI review requested due to automatic review settings June 23, 2026 10:19

dotnet-policy-service Bot assigned rosebyte Jun 23, 2026

github-actions Bot added the area-Extensions-Configuration label Jun 23, 2026

Copilot started reviewing on behalf of rosebyte June 23, 2026 10:20 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

build-analysis Bot mentioned this pull request Jun 29, 2026

Flaky timeout in CreateFileChangeToken_RootDeletedAndRecreated_TokenFiresWhenFileCreated #129691

Closed

Copilot AI review requested due to automatic review settings July 14, 2026 13:41

rosebyte force-pushed the configuration-references branch from 9325454 to 494bd94 Compare July 14, 2026 13:41

Copilot started reviewing on behalf of rosebyte July 14, 2026 13:41 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

rosebyte force-pushed the configuration-references branch from 494bd94 to d34a11b Compare July 14, 2026 14:02

Copilot AI review requested due to automatic review settings July 16, 2026 07:49

rosebyte force-pushed the configuration-references branch from d34a11b to 8082ba9 Compare July 16, 2026 07:49

Copilot started reviewing on behalf of rosebyte July 16, 2026 07:50 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 16, 2026 10:59

Copilot started reviewing on behalf of rosebyte July 16, 2026 10:59 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

This was referenced Jul 16, 2026

Unable to pull image from mcr.microsoft.com #117164

Open

AF: *(_UNCHECKED_OBJECTREF *)handle == NULL (HndCreateHandle called by getJitHandleForObject) #117138

Open

Copilot AI review requested due to automatic review settings July 17, 2026 07:21

rosebyte force-pushed the configuration-references branch from 89f57b0 to f0f5076 Compare July 17, 2026 07:21

Copilot started reviewing on behalf of rosebyte July 17, 2026 07:22 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

This was referenced Jul 17, 2026

iOS tests failing with OSError: [Errno 28] No space left on device dotnet/dnceng#6473

Open

Helix: System.IO.IOException: No space left on device #128758

Open

Copilot AI review requested due to automatic review settings July 21, 2026 21:48

rosebyte force-pushed the configuration-references branch from f0f5076 to b7e0b86 Compare July 21, 2026 21:48

Copilot started reviewing on behalf of rosebyte July 21, 2026 21:48 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 22, 2026 05:05

rosebyte force-pushed the configuration-references branch from b7e0b86 to 452f9f0 Compare July 22, 2026 05:05

Copilot started reviewing on behalf of rosebyte July 22, 2026 05:05 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread src/libraries/Microsoft.Extensions.Configuration/src/ReferenceIndex.cs Outdated

Comment on lines +119 to +125

if (targets is null)

{

return Empty;

}

return new ReferenceIndex(Flatten(targets));

}

rosebyte added 2 commits July 23, 2026 08:51

draft

b087fc8

use keys

129b0bf

Copilot AI review requested due to automatic review settings July 23, 2026 06:51

rosebyte force-pushed the configuration-references branch from 452f9f0 to 129b0bf Compare July 23, 2026 06:51

Copilot started reviewing on behalf of rosebyte July 23, 2026 06:52 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

This was referenced Jul 23, 2026

XHarness package install failure on iOS due to devicectl NSPOSIXErrorDomain error 49 #123796

Open

AF: op.Next is null == op.Previous is null in ChannelUtilities.Remove (RendezvousChannel cancellation) #129796

Open

		bool fired = false;
		ChangeToken.OnChange(root.GetReloadToken, () => fired = true);

Uh oh!

Conversation

rosebyte commented Jun 23, 2026

Uh oh!

dotnet-policy-service Bot commented Jun 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

tarekgh commented Jul 16, 2026

Uh oh!

rosebyte commented Jul 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants