chore: bound getExternalLinkPaths file reads to a frontmatter-sized prefix#32252
Closed
mvvmm wants to merge 1 commit into
Closed
chore: bound getExternalLinkPaths file reads to a frontmatter-sized prefix#32252mvvmm wants to merge 1 commit into
mvvmm wants to merge 1 commit into
Conversation
Contributor
|
This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:
|
1 task
Contributor
|
Preview URL: https://4a3d161e.preview.developers.cloudflare.com |
kodster28
approved these changes
Jul 22, 2026
…refix getExternalLinkPaths() (used at config-load time to build the sitemap exclusion set for pages with external_link: in frontmatter) was reading the entire body of every one of ~6,800 docs files just to regex-match one frontmatter key — 27.2MB total across all content, with one file at 606KB. This runs on every astro check, every dev-server start, and every build. Checked the actual frontmatter block size across all current content: max is 736 bytes. Bounded reads to a 1KB prefix per file instead of the whole body, via a fs.open()+read() helper rather than Node's readFile. Verified byte-for-byte identical output (235 matched paths, same set) against the prior full-read implementation at the 1KB bound via a standalone comparison script, then confirmed end-to-end in a real build: the sitemap correctly excludes the exact pages with external_link: in frontmatter (spot-checked security-center/cloudforce-one/cloudforce-one.mdx, excluded, while its sibling open-port-scanning.mdx, which lacks external_link:, remains). Measured the actual effect on local dev/build workflows before and after, since the isolated function-level win doesn't necessarily translate to a visible end-to-end difference: - Standalone re-implementation benchmark (warm OS file cache, 10 runs each): full-read median 451ms -> bounded-read median 383ms (~15%). - In-context timing (console.time around the real call inside astro-config.ts, actual `astro sync` process, 3 runs each): full-read ~472ms median -> bounded-read ~450ms median (~5%, overlapping with run-to-run noise). - Whole-command wall time (`astro sync`, 3 runs each): both versions land in the same 3.4-4.8s band: no end-to-end difference distinguishable from noise, since Vite startup and type generation dominate total time. Net: this is a correctness/I-O-hygiene improvement (avoids needlessly reading 27MB across 6,800 files, one of which is 606KB, just to check one frontmatter key) rather than a meaningful wall-clock speedup for typical dev workflows — worth doing, but not oversold as a build-time fix. Also considered gray-matter (the standard Node frontmatter-extraction library) as a more "standardized" alternative to the hand-rolled regex. It's already present in the dependency tree, but only transitively via astro-skills — not importable directly without adding it as an explicit dependency. It wouldn't reduce any further I/O either (still needs to read file content first), just swaps the regex for a library call; not worth a new dependency for a check this simple. No astro-native or nimbus-native utility exists for reading just frontmatter outside the content-collection lifecycle (the "standard" way to read frontmatter, getCollection, isn't available at this point in the build — astro-config.ts runs during config-loading, before astro:content resolves). Verified: pnpm run check/lint/format:check/test all clean, full build succeeds (8709 pages), test:postbuild passes.
mvvmm
force-pushed
the
chore/reduce-external-link-scan-io
branch
from
July 22, 2026 20:57
934a903 to
4a3d161
Compare
Contributor
Author
|
half a second savings isn't worth any complexity here, closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
getExternalLinkPaths()(used at config-load time to build the sitemap exclusion set for pages withexternal_link:in frontmatter) was reading the entire body of every one of ~6,800 docs files just to regex-match one frontmatter key — 27.2MB total across all content, with one file at 606KB. This runs on everyastro check, every dev-server start, and every build.Checked the actual frontmatter block size across all current content: max is 736 bytes. Bounded reads to a 1KB prefix per file instead of the whole body, via a
fs.open()+read()helper rather thanreadFile.Verified byte-for-byte identical output (235 matched paths, same set) against the prior full-read implementation at the 1KB bound, then confirmed end-to-end in a real build: the sitemap correctly excludes the exact pages with
external_link:in frontmatter (spot-checkedsecurity-center/cloudforce-one/cloudforce-one.mdx— excluded — while its siblingopen-port-scanning.mdx, which lacksexternal_link:, remains).Measured the actual effect on local dev/build workflows, since the isolated function-level win doesn't necessarily translate to a visible end-to-end difference:
console.timearound the real call, actualastro sync, 3 runs, median)astro sync, 3 runs)Net: this is a correctness/I/O-hygiene improvement (avoids needlessly reading 27MB across 6,800 files, one of which is 606KB, just to check one frontmatter key) rather than a meaningful wall-clock speedup for typical dev workflows — worth doing, but not oversold as a build-time fix. Vite startup and type generation dominate total command time.
Considered
gray-matter(the standard Node frontmatter-extraction library) as a more "standardized" alternative to the hand-rolled regex. It's already present in the dependency tree, but only transitively viaastro-skills— not importable directly without adding it as an explicit dependency. It wouldn't reduce any further I/O either (still needs to read file content first), just swaps the regex for a library call; not worth a new dependency for a check this simple. No Astro-native or Nimbus-native utility exists for reading just frontmatter outside the content-collection lifecycle — the "standard" way to read frontmatter (getCollection) isn't available at this point in the build, sinceastro-config.tsruns during config-loading, beforeastro:contentresolves.Verification
pnpm run check/lint/format:check/testall cleanpnpm run buildsucceeds (8709 pages)pnpm run test:postbuildpassesDocumentation checklist