dogfood: client router intermittently falls back to a full page load (tab spinner + whole-document flash)

[vivek7405]

Problem

Clicking links on webjs.dev intermittently shows a loading spinner in the browser tab and flashes the ENTIRE document, navbar included. Reproduction (live site, real browser, roughly every other lap):

1. Load https://webjs.dev
2. Go to /docs
3. Click the top-left logo to return to the landing page
4. Go to /docs again
5. Sometimes the whole document flashes with a tab spinner, sometimes it does not

The tab spinner and the flash are ONE event: a main-frame document load. With the client router enabled, clicking a same-origin link should never do that.

Root cause (found, captured, fixed on PR #1115)

The client router consumes a POISONED PREFETCH ENTRY, finds the fragment shares no boundary with the live DOM, and correctly degrades to location.href (the #1015 integrity fallback). Captured cause string, via the new webjs:navigation-fallback event, on every failure: no-shared-boundary.

How the poison is planted, from the captured request log:

/docs/getting-started   have=/:/                     prefetch=1   <- hover on home: correct
/docs/getting-started   have=/docs/...,/docs,/:/     prefetch=1   <- fired FROM the docs page itself
NAV-REQUESTED: scriptInitiated -> /docs/getting-started            <- the reload

1. Hover the /docs link on the landing page. The intent prefetcher starts its dwell timer.
2. Click. The soft nav lands; the DOM is now the docs page.
3. The hover's timer fires ANYWAY and prefetches /docs/getting-started, computing X-Webjs-Have from the NOW-SWAPPED DOM. The server correctly answers "you already have everything" with a near-empty fragment. (dogfood: the client router prefetches the page you are already on #1106 documented this same wasted request from the duplicate-fetch angle.)
4. The fragment is cached under the URL, TTL 30 s. Hovering again does NOT refresh it: the fresh-entry check skips the refetch.
5. Return to the landing page, click /docs again. The click consumes the poisoned entry, applySwap finds no shared boundary, and the router does the only safe thing: a full page load. Spinner, flash.
6. The reload wipes the in-page cache, so the next lap is clean, and the lap after re-poisons: the repro alternates, which is the "sometimes".

Two composing framework defects:

• prefetch() could prefetch the page the user is already on (dogfood: the client router prefetches the page you are already on #1106).
• prefetchTake() ignored the X-Webjs-Have context an entry was fetched against. The server marks the reduced response Vary: X-Webjs-Have; the prefetch cache is a client-side cache of that response and violated its own vary dimension.

Reproduced 6-of-24 clicks before the fix, 0-of-24 after, same randomized hover dwells, via a fine-pointer + hover-dwell puppeteer rig (headless defaults report hover: none, which is why earlier rigs missed it: intent prefetch never ran).

A third defect made all of this undiagnosable in production and is fixed on the same PR: the router's full-page-load degradations were logged by a DEV-ONLY console warning (devWarnFallback), and the deploy-mismatch reload had no diagnostic at all. reportFallback now dispatches webjs:navigation-fallback on document in every environment with { cause, href, willReload }.

Ruled out along the way (measured, do not re-investigate)

• CSSOM churn from a swapped page/layout <style> (dogfood: a page/layout <style> churns the CSSOM on every navigation, flashing preserved layouts #1109's attribution): about 4.6 ms of extra style recalc per navigation. Real waste, cannot flash a document.
• Cloudflare serving a reduced fragment to a full-document request (Vary: X-Webjs-Have unhonoured): every response on the repro path is cf-cache-status: DYNAMIC, nothing cached, cannot fire. Remains a latent hazard if HTML caching is ever enabled.
• The startup interception window (module boot is deferred, links clickable before the router listens): real, but 25 to 35 ms warm-cache, not human-clickable. Network-sized only with DevTools "Disable cache" or a genuine first visit; worth closing separately someday, not this bug.
• Deploy-detection reload (dogfood: SSR-only deploy not reflected on client nav (build id misses it) #899): webjs:importmap-reload flag was never set on any captured failure.
• Cloudflare Speed Brain (/cdn-cgi/speculation is 200 on webjs.dev): explains the second hover fetch a user sees in the Network tab; a speculation prefetch cannot hijack an intercepted click.

Acceptance criteria

• prefetch() never fetches the current page (also closes dogfood: the client router prefetches the page you are already on #1106)
• prefetchTake() rejects an entry whose stored have differs from the live one
• Every router full-page-load degradation dispatches webjs:navigation-fallback with its cause, in production
• A browser-level regression test drives hover, swap, then click and asserts the nav stays soft; counterfactual red without the fix
• Unit tests cover both guards
• The reported cycle, repeated in a real browser with hover, produces no document load
• Docs updated: the prefetch section and the new event alongside webjs:navigation-error

[vivek7405]

Decisive: it IS a full page load, and there is a concrete live-site mechanism

Two new observations that together close the diagnosis question and make this issue the only one worth working.

1. The browser tab shows a loading spinner during the repro. Reported directly:

https://webjs.dev -> go to /docs -> click a menu item -> click the top-left logo to go back to the landing page -> go to /docs -> loading in browser tab appears

A soft navigation is a fetch plus a DOM swap. It does not spin the tab throbber. The throbber is a MAIN-FRAME DOCUMENT LOAD. So the router is falling back to location.href = href on this path, intermittently, and the "whole document flashes, navbar included" symptom is the same event seen a different way. Both are what a full document load looks like. Neither is what a 4.6 ms style recalc looks like.

That retires the CSSOM-churn attribution for good.

2. Cloudflare almost certainly cannot honour the Vary that the reduced-response optimization depends on. Live headers today:

$ curl -sI https://webjs.dev/docs/getting-started
cache-control: public, max-age=0, s-maxage=600, stale-while-revalidate=86400
vary: Accept-Encoding
server: cloudflare
cf-cache-status: DYNAMIC

$ curl -sI https://webjs.dev/docs/getting-started -H 'X-Webjs-Have: /:/'
cache-control: public, max-age=0, s-maxage=600, stale-while-revalidate=86400
vary: X-Webjs-Have, Accept-Encoding
cf-cache-status: DYNAMIC

The pages are served public, s-maxage=600, stale-while-revalidate=86400, so they are explicitly CDN-cacheable for ten minutes with a day of stale-while-revalidate. The reduced fragment correctly marks itself Vary: X-Webjs-Have, which is exactly the protection AGENTS.md describes: "marked Vary: X-Webjs-Have so a shared cache can never serve the reduced body to a full-page navigation".

Cloudflare does not vary its cache on arbitrary request headers. It honours Vary: Accept-Encoding and ignores the rest. So the guarantee that invariant rests on does not hold behind this CDN. A cached REDUCED fragment served to a request expecting a full document (or the reverse) produces exactly the failure this issue is chasing: the incoming document has boundaries that do not match what the router expects, the scan reports incoming-boundaries-malformed or no-shared-boundary, and router-client.js:2902 degrades to a full page load. Cache hit versus miss is precisely the source of "sometimes".

Today every response shows cf-cache-status: DYNAMIC, meaning Cloudflare is not currently caching them, so the hazard may be latent rather than firing. That needs confirming rather than assuming: DYNAMIC on an uncached sample says nothing about what happens on a path that does get cached, and the repro visits /docs twice with different client state each time.

Revised investigation order:

1. Add the production-observable fallback signal described in the body. Without it every conclusion here stays inferential, which is the mistake that produced dogfood: a page/layout <style> churns the CSSOM on every navigation, flashing preserved layouts #1109.
2. Reproduce on webjs.dev with the signal listening, and read off the cause.
3. Check whether any response on the repro path is ever served from the Cloudflare cache (cf-cache-status: HIT), and whether a reduced fragment can be returned for a full-document request. If so, that is the bug, and the fix is at the caching layer (do not make the reduced response publicly cacheable, or move the discriminator into the URL rather than a header a CDN will ignore), NOT in the router.
4. Only if the cache is exonerated, look at the other producers of malformed boundaries.

Note the shape of the likely fix: if this is the cause, it is a caching-correctness bug, not a styling bug and not a router bug. The router's degradation is working exactly as designed (#1015 chose a bounded full page load over a heuristic recovery that could corrupt silently). It is the input that is wrong.

[vivek7405]

Diagnosis: it is a startup race, not the CDN and not styles. Measured.

I was wrong about Cloudflare too. Downgrading that; the real cause is simpler and reproduces locally.

The router is not listening yet when the links are already clickable.

Links arrive in the SSR'd HTML, so they are clickable from first paint. The router's document click listener is installed by enableClientRouter(), which runs when @webjsdev/core executes, and the boot script is <script type="module"> (packages/server/src/ssr.js:1270). Module scripts are deferred by spec, so they run AFTER HTML parsing, after the module is fetched, parsed, and its dependency graph resolved. Until then nothing intercepts a click, and the browser does what it always does with an <a href>: a full document navigation.

Measured against the real website, timing when the click listener is actually installed:

Network	Router starts listening at
Fast	32 ms
Slow (300 kbps, 200 ms RTT)	313 ms

That is the unprotected window, and it starts at first paint. Click inside it and you get a full page load: tab spinner, whole-document flash, preserved chrome destroyed because the whole document is destroyed. Click after it and you get a clean soft nav. Nothing else needs to vary for the bug to be intermittent; the user's reaction time versus the module load is the whole story.

Corroborating evidence from the same run. Driving 6 quick clicks with no wait produced 5 to 7 full page loads out of 6, and zero webjs:navigation-fallback events. If any degradation path in the router had fired, the new signal would have caught it. Silence plus a full load means the router never saw the click at all. That also rules out readyState-loading, no-shared-boundary, and the deploy-mismatch reload for this symptom: all of those run INSIDE the router and would have reported.

Why it fits the reported repro exactly. The sequence is four navigations in quick succession. Every full load restarts the window, so one mistimed click makes the next click more likely to be mistimed too. And it is worse on the live site than locally because the window scales with network latency, which is the "only on webjs.dev" part.

Why the CDN theory is out. Every response on the repro path returns cf-cache-status: DYNAMIC, so Cloudflare is not caching them, so the Vary: X-Webjs-Have gap cannot be firing. It remains a latent hazard worth fixing, but it is not this. Filing it separately rather than leaving it buried here.

This reframes the issue. The expectation is right: with the client router enabled, clicking a link should never do a full document load. Today there is a guaranteed window at the start of every page load where it always will. The fix is to close the window, not to make the fallback cheaper. Options to weigh:

1. Install the click interception from a tiny SYNCHRONOUS inline script in <head>, before the module boot. It only needs to capture the click, prevent default, and queue the URL; the real router replays the queue once it loads. Turbo takes essentially this approach. Costs a few hundred bytes inline on every page.
2. Keep the module boot but make the interception listener the first thing it does, and get the module earlier (it is already modulepreloaded, so the remaining cost is execution order, not fetch).
3. Accept the window but make it correct: capture the click, show nothing, and let the browser navigation proceed only if the router genuinely never arrives.

Option 1 is the only one that actually closes it, because the others still depend on module execution finishing.

[vivek7405]

Correction again: the startup race is real but too small to be this bug. I have now been wrong three times, so I am stopping the guessing.

The objection that killed it: the core bundle is served cache-control: public, max-age=31536000, immutable, so on any repeat visit there is no fetch at all. My 313 ms figure came from a throttled COLD load, which is not the situation in the reported repro.

Re-measured with a warm cache:

	core bundle	router listening at
Cold	36 KB, 9.5 ms	39 ms
Warm	0 bytes, 0 ms	34 ms
Warm	0 bytes, 0 ms	25 ms

So the unprotected window is about 25 to 35 ms from navigation start. A person cannot click inside that in normal use. My 6-click test hit it only because it clicked the instant the link existed, which is an artifact of the test being inhumanly fast, not a model of anyone's behaviour.

The window is a genuine latent gap and worth closing on its own merits, since a module script is deferred by spec and can never intercept during parse. But it is not the reported symptom, and I should not have presented it as the diagnosis.

Scorecard, so the next person does not retrace this. Three mechanisms proposed, three measured and eliminated:

1. CSSOM churn from a swapped page style block (dogfood: a page/layout <style> churns the CSSOM on every navigation, flashing preserved layouts #1109's attribution): about 4.6 ms of extra style recalc per navigation. Too small to flash a document.
2. Cloudflare ignoring Vary: X-Webjs-Have and serving a reduced fragment to a full-document request: every response on the repro path returns cf-cache-status: DYNAMIC, so nothing is being cached and the gap cannot be firing.
3. The startup interception window: about 25 to 35 ms warm, not human-clickable.

The pattern in all three is the same and it is mine: a plausible mechanism, reasoned to a conclusion, not measured against the symptom until afterwards.

So: instrument first, theorise second. #1115 already dispatches webjs:navigation-fallback for every degradation INSIDE the router. That is necessary but not sufficient, because the loudest thing the last experiment showed is that a full page load can happen with the router emitting nothing at all, which is exactly the case a router-internal signal cannot see.

What is needed alongside it is a signal that survives the reload and answers "was a click responsible":

• On every same-origin link click, record { href, t, routerWasListening } into sessionStorage.
• On pagehide, stamp whether a document unload followed that click and how long after.
• After a flash, read the key. It says whether the document unloaded within a few hundred ms of a click, and whether the router was live at the time.

sessionStorage survives a same-origin document load, so the record outlives the very event it is describing. That is the property every instrument so far has lacked.

Then reproduce on webjs.dev and read the answer off the page instead of arguing about it. Adding that to #1115.

[vivek7405]

State of evidence: the tab spinner IS the bug (one event, a document load), it reproduces only in a real persisted browser, and there is a console one-liner that discriminates the leading remaining cause

First, an instrument error of mine to retract before it misleads anyone: an earlier 15-lap run against the live site appeared to show 45 of 45 clicks doing full page loads. That was my rig, not the site. Puppeteer's framenavigated also fires for SAME-DOCUMENT navigations (pushState), so it counted every healthy soft nav as a reload. Re-run with in-page logging that survives reloads: on the live site every click is intercepted (defaultPrevented === true) and a whole lap produces ZERO document loads. The live site soft-navigates correctly in a clean browser.

Current facts, all measured:

• The reported spinner in the tab means a main-frame document load. The flash and the spinner are one event.
• Live site, clean headless browser, human-paced clicks: 0 document loads in 49 clicks, all intercepted.
• Local prod boot of the instrumented branch, 120 clicks at randomized 60 to 1200 ms timing: 0 document loads, 0 webjs:navigation-fallback events, 0 unintercepted clicks.
• Build/src signatures on the live site are stable across requests, so cross-replica drift is out.
• cf-cache-status: DYNAMIC on every probe from here, so CDN staleness is not observable from this vantage point (but my colo is not the reporter's colo, and stale-while-revalidate=86400 means a colo that ever cached a pre-deploy page may serve it stale for a day).

So the reproducing environment is specifically a REAL persisted browser profile against the live site. That is the same shape as #968, where a dev reload loop never reproduced on a fresh Chromium profile and only in a persisted browser.

The leading remaining suspect is the #899 deploy-detection hard reload, because it is the one mechanism whose intermittency is BUILT IN: it reloads on a build-id mismatch, sets the one-shot webjs:importmap-reload sessionStorage flag, suppresses the next reload, and a clean nav clears the flag. If something in a persisted browser keeps serving pre-deploy HTML for some navigations (an HTTP cache entry, a colo holding a stale-while-revalidate copy, an old service worker), the result is exactly "sometimes flashes, sometimes does not".

And it is discriminable TODAY, with no deploy, because the flag is already in production code. Immediately after a flash happens, open the console on the flashed page and run:

sessionStorage.getItem('webjs:importmap-reload')          // '1' => the deploy-detection reload did it
performance.getEntriesByType('navigation')[0]?.type       // 'navigate' confirms a fresh document load

If the first line prints '1', the #899 path caused the reload and the question becomes what served the stale build id (browser cache vs CDN vs SW). If it prints null while the second confirms a document load, the reload came from a path that does not set the flag (the boundary-integrity degradation, or an unintercepted click), and the #1115 event plus click log needs to be deployed to tell those apart.

Next steps in order: finish #1115 (tests for the event), get it deployed, then one real reproduction with the console read answers this conclusively.

[vivek7405]

ROOT CAUSE FOUND, CAPTURED, AND FIXED. Reproduced 6-of-24 clicks before the fix, 0-of-24 after.

The flash and the tab spinner are one event: the client router consuming a POISONED PREFETCH ENTRY, finding the fragment shares no boundary with the live DOM, and correctly degrading to location.href (the #1015 integrity fallback). Cause string, captured off the new webjs:navigation-fallback event on every single failure: no-shared-boundary.

The poison, step by step, from the captured request log:

/docs/getting-started   have=/:/                      prefetch=1   <- hover on home: correct
/docs/getting-started   have=/docs/…,/docs,/:/        prefetch=1   <- fired FROM the docs page itself
NAV-REQUESTED: scriptInitiated -> /docs/getting-started             <- the reload

1. You hover the /docs link on the landing page. The intent prefetcher starts its dwell timer.
2. You click. The soft nav lands. The DOM is now the docs page.
3. The hover's timer fires ANYWAY and prefetches /docs/getting-started, computing X-Webjs-Have from the NOW-SWAPPED DOM. The server correctly answers "you already have everything" with a near-empty fragment.
4. That fragment is cached under the URL, TTL 30 seconds. Hovering again does NOT refresh it (the fresh-entry check skips the refetch).
5. You go back to the landing page and click /docs again. The click consumes the poisoned entry, applySwap finds no shared boundary, and the router does the only safe thing: a full page load. Spinner, flash.
6. The reload wipes the in-page cache, so the next lap is clean, and the one after re-poisons. The reported cycle alternated clean and flashing laps: exactly "sometimes".

Two composing framework defects, both fixed on PR #1115:

• prefetch() could prefetch the page the user is already on. Now a no-op.
• prefetchTake() ignored the X-Webjs-Have context an entry was fetched against. The server marks the reduced response Vary: X-Webjs-Have; the prefetch cache is a client-side cache of that response and violated its own vary dimension. An entry whose stored have differs from the live one is now discarded (one extra round-trip on a stale hit, instead of a full document load).

Why every earlier theory missed it: the degradation was invisible in production (devWarnFallback was dev-only), and the poison is planted by a timer that outlives the very navigation the user is making, so it only reproduces with real hover-then-click sequences. The headless rigs clicked without hovering, or with a touch-profile pointer where intent prefetch never runs. The fine-pointer shim plus varied hover dwell reproduced it within one run, and the new event named the path immediately. The observability was not overhead; it was the fix's prerequisite.

Remaining work on the PR before it is ready: regression tests (unit for both guards, a browser test for the hover-swap-click sequence, the counterfactual), doc sync, and the standard review loop.