fix(governance): define adagents.json discovery redirect policy (#5450)

.changeset/5450-adagents-redirect-policy.md

@@ -0,0 +1,7 @@
+---
+"adcontextprotocol": minor
+---
+
+Define adagents.json discovery redirect policy and reconcile the reference implementation with it.
+
+The initial `/.well-known/adagents.json` fetch now follows **same-registrable-domain** redirects (apex↔www, HTTPS-preserving, ≤3 hops, SSRF re-validated per hop, anchored on the originally-requested domain) so that standard apex→www managed hosting resolves instead of being silently reported unauthorized. **Cross-registrable-domain** redirects are refused — declare delegation with `authoritative_location` instead — and the `authoritative_location` dereference continues to refuse all redirects. Docs: managed-networks "Why not HTTP redirects?" and L1 security SSRF/TLS-hardening sections; new conformance vectors in `static/test-vectors/adagents-discovery-redirects.json`.

docs/building/by-layer/L1/security.mdx

@@ -108,7 +108,7 @@ Before any outbound fetch to a counterparty-controlled URL, fetchers MUST:
    - IPv4: RFC 1918 (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), RFC 6598 CGNAT (`100.64.0.0/10`), loopback (`127.0.0.0/8`), link-local (`169.254.0.0/16` — explicitly includes `169.254.169.254` used by AWS/GCP/Azure/Alibaba instance metadata), broadcast (`255.255.255.255`), `0.0.0.0/8`, multicast (`224.0.0.0/4`).
    - IPv6: loopback (`::1`), unique-local (`fc00::/7`), link-local (`fe80::/10`), IPv4-mapped (`::ffff:0:0/96` — the most common bypass, mapping reserved IPv4 into IPv6), multicast (`ff00::/8`), and the AWS IMDSv2 fd00:ec2::254 address.
 3. **Pin the connection to the validated IP.** DNS-based filtering alone is vulnerable to DNS rebinding: an attacker serves a public IP at validation time and a private IP at connect time. Fetchers MUST pin the connection. **Preferred**: (a) pass the validated IP directly to the TCP connect call and set the `Host:` header from the URL. **Fallback** (only when the HTTP client cannot accept a pre-resolved IP): (b) validate the socket's post-handshake peer address against the reserved-range list before sending any request body. Note: (b) depends on the client library exposing a peer-address hook that fires before the first body byte ships; many common libraries do not, so implementations choosing (b) MUST verify the hook in testing. Re-resolving DNS without pinning is not sufficient.
-4. **Refuse to follow redirects** when fetching counterparty-controlled URLs (a 30x response lets the origin redirect to a reserved address that bypassed the initial check).
+4. **Refuse to follow redirects** when fetching counterparty-controlled URLs (a 30x response lets the origin redirect to a reserved address that bypassed the initial check). Two bounded exceptions, each re-validating steps 1–3 on every hop and capping the chain: the [brand.json resolution](#buyer-identity-resolution) (one redirect, no chains), and the initial `/.well-known/adagents.json` fetch, which follows **same-registrable-domain** redirects only (`apex ↔ www`, HTTPS-preserving, ≤ 3 hops, anchored on the originally-requested domain) so standard apex→www hosting resolves — see [managed networks](/docs/governance/property/managed-networks#why-not-http-redirects). The `adagents.json` `authoritative_location` dereference takes no exception: redirects on that second hop MUST be refused.
 5. **Cap response size and timeouts.** Recommended: 5 MB body cap, 10 s connect, 10 s read. The only exception is the dereferenced authoritative file in the managed-network indirection pattern — second-hop only, after a pointer file's `authoritative_location` redirects to the network origin — which uses a recommended 20 MB cap because it fans out across a publisher network. Pointer files themselves stay at 5 MB. See [managed networks security](/docs/governance/property/managed-networks#security-considerations).
 6. **Do not echo fetch errors to the agent that supplied the URL.** Detailed error messages (connection refused vs. timed out vs. TLS failure) are a side-channel for probing internal network topology.
 
@@ -1796,7 +1796,7 @@ Outbound-fetch code paths (governance JWKS, brand.json, revocation list, webhook
 - Use a connection pool with a fixed per-host cap and a fixed overall cap. Unbounded pools are a resource-exhaustion surface.
 - Cap TLS handshake time at 10 s and total request time at 30 s by default — counterparty-supplied URLs are a tarpit DoS vector otherwise.
 - Pin the connection to the IP address that passed the [SSRF controls](#webhook-url-validation-ssrf) — DNS re-resolution between the SSRF check and the actual connect is how TOCTOU bypasses land.
-- Refuse redirects on security-sensitive fetches. JWKS, brand.json, revocation list, and webhook-callback fetches MUST NOT follow redirects; the [brand.json resolution rule](#buyer-identity-resolution) already says "one redirect (`authoritative_location` or `house` variant), no chains" — everywhere else, zero.
+- Refuse redirects on security-sensitive fetches. JWKS, brand.json, revocation list, and webhook-callback fetches MUST NOT follow redirects; the [brand.json resolution rule](#buyer-identity-resolution) already says "one redirect (`authoritative_location` or `house` variant), no chains," and the initial `/.well-known/adagents.json` fetch follows same-registrable-domain redirects only (apex↔www, HTTPS-preserving, ≤ 3 hops, anchored on the originally-requested domain) — everywhere else, zero. The `adagents.json` `authoritative_location` dereference is "everywhere else": zero redirects.
 - Disable session resumption across trust boundaries. Resuming a TLS session with an attacker-controlled counterparty onto a later verified counterparty (same IP via DNS rebind) is a well-known class of confusion; library defaults are usually fine, but the operator MUST audit.
 
 ### TLS renegotiation and downgrade

docs/governance/property/managed-networks.mdx

@@ -141,7 +141,10 @@ For `adagents.json`, HTTP redirects cause problems:
 - **No scoping.** An HTTP redirect is all-or-nothing. A pointer file sits alongside a structured authorization model where the network can declare exactly what it is authorized to sell.
 - **Caching penalty.** With HTTP redirects, a validator has no way to know that 10,000 domains all redirect to the same file. It must treat each response as independent — 10,000 cache entries for identical content. With `authoritative_location`, the validator sees the same URL across all pointer files and caches the authoritative file once. For a network with thousands of domains, this is the difference between one cache entry and thousands.
 
-HTTP redirects to `/.well-known/adagents.json` are not prohibited, but they are not the recommended pattern. Use `authoritative_location` instead.
+Those problems are about using a redirect to express **delegation** — a cross-domain hop that says "my authorization lives somewhere else." That is what `authoritative_location` replaces. They are not about **hosting normalization**, which is the ordinary case where a publisher's apex domain `301`-redirects to `www` (the default on most managed hosting and CDNs). The two are distinguished by where the redirect points, so the fetch rule is split accordingly:
+
+- **Same-registrable-domain redirects MUST be followed** on the `/.well-known/adagents.json` fetch — `apex ↔ www`, and any redirect that stays on the same registrable domain (eTLD+1) — re-validating the [SSRF controls](/docs/building/by-layer/L1/security#webhook-url-validation-ssrf) on every hop, refusing any redirect to a non-HTTPS scheme, and capping the chain at 3 hops. Refusing these silently reports a correctly-configured publisher as unauthorized. The same-registrable-domain comparison MUST be anchored on the **originally-requested** domain at every hop, not the previous hop, so a `same → cross` two-hop chain cannot escape it.
+- **Cross-registrable-domain redirects MUST be refused** on this fetch. A cross-domain redirect is the ambiguous, unscoped delegation signal this pattern exists to replace — declare delegation explicitly with `authoritative_location` instead. The validator dereferences `authoritative_location` as a second hop with redirects **refused** (see [Security considerations](#security-considerations)): the named URL is the declared authoritative location, and a redirect away from it changes that declaration.
 
 ## Choosing the right delegation type
 

package.json

@@ -172,6 +172,7 @@
     "sharp": "^0.34.5",
     "stripe": "^22.2.0",
     "svix": "^1.95.2",
+    "tldts": "^7.0.29",
     "undici": "^8.4.1",
     "whoiser": "^2.0.0-beta.10",
     "ws": "^8.21.0",

server/src/adagents-manager.ts

@@ -5,6 +5,11 @@ import { canonicalizePublisherDomain } from './services/publisher-domain.js';
 
 const MCP_ACCEPT_HEADER = 'application/json, text/event-stream';
 const ADS_TXT_MAX_REDIRECTS = 5;
+// The initial /.well-known/adagents.json fetch follows same-registrable-domain
+// redirects only (apex↔www, HTTPS-preserving), capped at 3 hops, so standard
+// apex→www hosting resolves. Cross-domain hops are refused — see
+// docs/governance/property/managed-networks#why-not-http-redirects.
+const ADAGENTS_WELL_KNOWN_MAX_REDIRECTS = 3;
 const MCP_PREFLIGHT_INITIALIZE_BODY = {
   jsonrpc: '2.0',
   method: 'initialize',
@@ -243,6 +248,8 @@ export class AdAgentsManager {
       // unauthenticated and reachable on view (see PR #4128 / issue #4129).
       const response = await safeFetchAxiosLike(url, {
         timeoutMs: 10000,
+        maxRedirects: ADAGENTS_WELL_KNOWN_MAX_REDIRECTS,
+        sameSiteRedirectsOnly: true,
         headers: {
           'Accept': 'application/json',
           'User-Agent': AAO_UA_VALIDATOR,
@@ -540,20 +547,23 @@ export class AdAgentsManager {
         return null;
       }
 
-      // Fetch the authoritative file. safeFetch follows redirects with
-      // per-hop SSRF validation, which matters more here than on the
-      // /.well-known fetch — the authoritative_location URL was named
-      // by the publisher and could redirect anywhere.
+      // Fetch the authoritative file with redirects REFUSED. The named URL is
+      // the publisher's declared authoritative location; a redirect away from
+      // it changes that declaration, so it is treated as an error rather than
+      // followed (unlike the initial /.well-known fetch, which follows
+      // same-registrable-domain hosting redirects). See
+      // docs/governance/property/managed-networks#security-considerations.
       const response = await safeFetchAxiosLike(url, {
         timeoutMs: 10000,
+        maxRedirects: 0,
         headers: {
           'Accept': 'application/json',
           'User-Agent': AAO_UA_VALIDATOR,
         },
       });
 
-      // Overwrite the resolved URL — when authoritative_location is
-      // followed, the canonical document body came from THIS fetch,
+      // Overwrite the resolved URL — when the authoritative_location pointer
+      // is dereferenced, the canonical document body came from THIS fetch,
       // not the original /.well-known. Verifiers should pin trust to
       // the authoritative location's TLS chain, not the publisher's.
       // Bytes likewise reflect the canonical body, not the stub.
@@ -563,7 +573,9 @@ export class AdAgentsManager {
       if (response.status !== 200) {
         const statusMessage = response.status === 404
           ? `File not found at ${url}`
-          : `HTTP ${response.status} error fetching ${url}`;
+          : [301, 302, 303, 307, 308].includes(response.status)
+            ? `Redirect (HTTP ${response.status}) from authoritative_location ${url}; redirects on the authoritative file are refused because they change the declared location`
+            : `HTTP ${response.status} error fetching ${url}`;
         result.errors.push({
           field: 'authoritative_location',
           message: statusMessage,

server/src/utils/url-security.ts

@@ -2,6 +2,7 @@ import dns from 'dns/promises';
 import { lookup as dnsLookup, type LookupAddress, type LookupOptions } from 'dns';
 import { isIP, type LookupFunction } from 'net';
 import { Agent, fetch as undiciFetch, type Dispatcher } from 'undici';
+import { getDomain } from 'tldts';
 import { createLogger } from '../logger.js';
 
 const logger = createLogger('url-security');
@@ -353,6 +354,17 @@ export async function safeFetch(
     body?: string | Uint8Array;
     maxRequestBytes?: number;
     signal?: AbortSignal;
+    /**
+     * Restrict redirect-following to the originally-requested registrable
+     * domain (eTLD+1), HTTPS only. Cross-registrable-domain or scheme-downgrade
+     * hops throw instead of being followed. Used for the `/.well-known/adagents.json`
+     * discovery fetch, where standard apex→www hosting redirects MUST resolve but a
+     * cross-domain hop is an unscoped delegation signal that MUST be refused — see
+     * docs/governance/property/managed-networks#why-not-http-redirects. The
+     * comparison is anchored on the original request URL at every hop, not the
+     * previous hop, so a same→cross two-hop chain cannot escape it.
+     */
+    sameSiteRedirectsOnly?: boolean;
   },
 ): Promise<Response> {
   const parsedUrl = new URL(url);
@@ -364,6 +376,23 @@ export async function safeFetch(
   const body = options?.body;
   const maxRequestBytes = options?.maxRequestBytes ?? DEFAULT_MAX_REQUEST_BYTES;
   const signal = options?.signal;
+  const sameSiteRedirectsOnly = options?.sameSiteRedirectsOnly ?? false;
+  // Same-site mode preserves HTTPS across the whole chain, including hop 0 —
+  // assert it up front so the invariant holds regardless of caller discipline.
+  if (sameSiteRedirectsOnly && parsedUrl.protocol !== 'https:') {
+    throw new Error(
+      `sameSiteRedirectsOnly requires an HTTPS origin, got ${parsedUrl.protocol}//${parsedUrl.hostname}`,
+    );
+  }
+  // Anchor the same-site check on the ORIGINAL request domain, never the
+  // previous hop — otherwise a same→cross two-hop chain bypasses the rule.
+  // allowPrivateDomains:true so the PSL PRIVATE section (github.io, pages.dev,
+  // herokuapp.com, …) is the registrant boundary; without it two unrelated
+  // tenants on shared hosting collapse to one registrable domain and a
+  // cross-tenant redirect would be wrongly trusted.
+  const originRegistrableDomain = sameSiteRedirectsOnly
+    ? getDomain(parsedUrl.hostname, { allowPrivateDomains: true })
+    : null;
   const dispatcher = buildSsrfSafeDispatcher();
 
   // POST without a body would surprise downstream agent endpoints; reject
@@ -407,6 +436,21 @@ export async function safeFetch(
     if (!location) throw new Error('Redirect with no Location header');
     // Pre-flight check on the redirect hop, then dial through the same SSRF-safe dispatcher.
     const redirectUrl = await validateRedirectTarget(location, currentUrl);
+    if (sameSiteRedirectsOnly) {
+      // HTTPS-preserving: refuse any downgrade away from HTTPS.
+      if (redirectUrl.protocol !== 'https:') {
+        throw new Error(
+          `Refused non-HTTPS redirect on same-site fetch: ${currentUrl.hostname} -> ${redirectUrl.protocol}//${redirectUrl.hostname}`,
+        );
+      }
+      // Same registrable domain (eTLD+1), anchored on the ORIGINAL request.
+      const targetRegistrableDomain = getDomain(redirectUrl.hostname, { allowPrivateDomains: true });
+      if (!originRegistrableDomain || targetRegistrableDomain !== originRegistrableDomain) {
+        throw new Error(
+          `Refused cross-registrable-domain redirect: ${currentUrl.hostname} -> ${redirectUrl.hostname}`,
+        );
+      }
+    }
     // Per RFC 7231 §6.4.4 a 303 ALWAYS rewrites to GET; for 301/302 most
     // clients also rewrite for non-idempotent verbs even though the spec
     // only mandates user confirmation. We rewrite POST→GET on 301/302/303
@@ -483,6 +527,7 @@ export async function safeFetchAxiosLike(
     timeoutMs?: number;
     maxResponseBytes?: number;
     maxRedirects?: number;
+    sameSiteRedirectsOnly?: boolean;
   },
 ): Promise<SafeFetchAxiosLike> {
   const controller = new AbortController();
@@ -495,6 +540,7 @@ export async function safeFetchAxiosLike(
       headers: options?.headers,
       body: options?.body,
       maxRedirects: options?.maxRedirects,
+      sameSiteRedirectsOnly: options?.sameSiteRedirectsOnly,
       signal: controller.signal,
     });