Skip to content

chore: bound getExternalLinkPaths file reads to a frontmatter-sized prefix#32252

Closed
mvvmm wants to merge 1 commit into
productionfrom
chore/reduce-external-link-scan-io
Closed

chore: bound getExternalLinkPaths file reads to a frontmatter-sized prefix#32252
mvvmm wants to merge 1 commit into
productionfrom
chore/reduce-external-link-scan-io

Conversation

@mvvmm

@mvvmm mvvmm commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

getExternalLinkPaths() (used at config-load time to build the sitemap exclusion set for pages with external_link: in frontmatter) was reading the entire body of every one of ~6,800 docs files just to regex-match one frontmatter key — 27.2MB total across all content, with one file at 606KB. This runs on every astro check, every dev-server start, and every build.

Checked the actual frontmatter block size across all current content: max is 736 bytes. Bounded reads to a 1KB prefix per file instead of the whole body, via a fs.open() + read() helper rather than readFile.

Verified byte-for-byte identical output (235 matched paths, same set) against the prior full-read implementation at the 1KB bound, then confirmed end-to-end in a real build: the sitemap correctly excludes the exact pages with external_link: in frontmatter (spot-checked security-center/cloudforce-one/cloudforce-one.mdx — excluded — while its sibling open-port-scanning.mdx, which lacks external_link:, remains).

Measured the actual effect on local dev/build workflows, since the isolated function-level win doesn't necessarily translate to a visible end-to-end difference:

Measurement Full read (old) Bounded 1KB read (new)
Standalone re-implementation benchmark (warm OS cache, 10 runs, median) 451ms 383ms (~15% faster)
In-context timing (console.time around the real call, actual astro sync, 3 runs, median) ~472ms ~450ms (~5%, overlaps with noise)
Whole-command wall time (astro sync, 3 runs) 3.4–4.8s 3.4–4.8s (no distinguishable difference)

Net: this is a correctness/I/O-hygiene improvement (avoids needlessly reading 27MB across 6,800 files, one of which is 606KB, just to check one frontmatter key) rather than a meaningful wall-clock speedup for typical dev workflows — worth doing, but not oversold as a build-time fix. Vite startup and type generation dominate total command time.

Considered gray-matter (the standard Node frontmatter-extraction library) as a more "standardized" alternative to the hand-rolled regex. It's already present in the dependency tree, but only transitively via astro-skills — not importable directly without adding it as an explicit dependency. It wouldn't reduce any further I/O either (still needs to read file content first), just swaps the regex for a library call; not worth a new dependency for a check this simple. No Astro-native or Nimbus-native utility exists for reading just frontmatter outside the content-collection lifecycle — the "standard" way to read frontmatter (getCollection) isn't available at this point in the build, since astro-config.ts runs during config-loading, before astro:content resolves.

Verification

  • pnpm run check / lint / format:check / test all clean
  • Full pnpm run build succeeds (8709 pages)
  • pnpm run test:postbuild passes

Documentation checklist

@github-actions

Copy link
Copy Markdown
Contributor

This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:

Pattern Owners
*.ts @cloudflare/content-engineering, @kodster28

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

…refix

getExternalLinkPaths() (used at config-load time to build the sitemap
exclusion set for pages with external_link: in frontmatter) was
reading the entire body of every one of ~6,800 docs files just to
regex-match one frontmatter key — 27.2MB total across all content,
with one file at 606KB. This runs on every astro check, every
dev-server start, and every build.

Checked the actual frontmatter block size across all current content:
max is 736 bytes. Bounded reads to a 1KB prefix per file instead of
the whole body, via a fs.open()+read() helper rather than Node's
readFile.

Verified byte-for-byte identical output (235 matched paths, same set)
against the prior full-read implementation at the 1KB bound via a
standalone comparison script, then confirmed end-to-end in a real
build: the sitemap correctly excludes the exact pages with
external_link: in frontmatter (spot-checked
security-center/cloudforce-one/cloudforce-one.mdx, excluded, while its
sibling open-port-scanning.mdx, which lacks external_link:, remains).

Measured the actual effect on local dev/build workflows before and
after, since the isolated function-level win doesn't necessarily
translate to a visible end-to-end difference:

- Standalone re-implementation benchmark (warm OS file cache, 10 runs
  each): full-read median 451ms -> bounded-read median 383ms (~15%).
- In-context timing (console.time around the real call inside
  astro-config.ts, actual `astro sync` process, 3 runs each):
  full-read ~472ms median -> bounded-read ~450ms median (~5%,
  overlapping with run-to-run noise).
- Whole-command wall time (`astro sync`, 3 runs each): both versions
  land in the same 3.4-4.8s band: no end-to-end difference
  distinguishable from noise, since Vite startup and type generation
  dominate total time.

Net: this is a correctness/I-O-hygiene improvement (avoids needlessly
reading 27MB across 6,800 files, one of which is 606KB, just to check
one frontmatter key) rather than a meaningful wall-clock speedup for
typical dev workflows — worth doing, but not oversold as a build-time
fix.

Also considered gray-matter (the standard Node frontmatter-extraction
library) as a more "standardized" alternative to the hand-rolled
regex. It's already present in the dependency tree, but only
transitively via astro-skills — not importable directly without adding
it as an explicit dependency. It wouldn't reduce any further I/O
either (still needs to read file content first), just swaps the regex
for a library call; not worth a new dependency for a check this
simple. No astro-native or nimbus-native utility exists for reading
just frontmatter outside the content-collection lifecycle (the
"standard" way to read frontmatter, getCollection, isn't available at
this point in the build — astro-config.ts runs during config-loading,
before astro:content resolves).

Verified: pnpm run check/lint/format:check/test all clean, full build
succeeds (8709 pages), test:postbuild passes.
@mvvmm
mvvmm force-pushed the chore/reduce-external-link-scan-io branch from 934a903 to 4a3d161 Compare July 22, 2026 20:57
@mvvmm

mvvmm commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

half a second savings isn't worth any complexity here, closing.

@mvvmm mvvmm closed this Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants