fix(server): prevent bot user-agent regex backtracking

[NathanDrake2406]

Overview

Goal	Keep attacker-controlled User-Agent matching linear for the default bot detector
Core change	Anchor the broad *-Google branch at the start of each word/hyphen token
Key boundary	Shared html-limited-bots utility used by Pages rendering and metadata bot gates
Expected impact	Long non-matching User-Agent values no longer cause quadratic request-time CPU work

Why

The default bot pattern included an unanchored greedy [\w-]+-Google branch. On a long word-only value without the suffix, the JavaScript regex engine retried that scan at every character. The detector runs on request User-Agent headers, so matching work must scale with input length.

Requiring either the start of the string or a non-token delimiter before the greedy token preserves the broad Google crawler contract while preventing overlapping retries.

What changed

Scenario	Before	After
Long non-matching User-Agent	Greedy suffix branch performed quadratic backtracking	Each token is considered from one boundary
Mediapartners-Google, AdsBot-Google, Storebot-Google	Detected	Still detected
Google-* and fixed crawler tokens	Detected	Unchanged
Custom htmlLimitedBots configuration	Evaluated as the configured regular expression	Unchanged

Maintainer review path

1. packages/vinext/src/utils/html-limited-bots.ts for the token-boundary decision.
2. tests/pages-page-response.test.ts for suffix compatibility and the long-input regression.

Validation

• The Pages response, Pages data, and streaming metadata suites pass: 134 assertions.
• Targeted format, lint, and type checks pass.
• The vinext package build passes.
• The repository pre-commit gate passes full check, staged unit/integration tests, and knip.
• Built-output timing scales linearly through a 128,000-character non-match.

Commands

vp test run tests/pages-page-response.test.ts
vp test run tests/streaming-metadata.test.ts tests/pages-page-data.test.ts
vp check packages/vinext/src/utils/html-limited-bots.ts tests/pages-page-response.test.ts
vp run vinext#build

Risk / compatibility

The production change is limited to the default suffix branch and preserves boolean matches for word/hyphen crawler tokens ending in -Google. The configured-regex path and public configuration shape are unchanged. The new timeout-backed regression has substantial headroom: the fixed 64,000-character case completes in well under the 500 ms limit locally.

References

Reference	Why it matters
Next.js default HTML-limited bot pattern	Defines the broad Google crawler compatibility contract mirrored by vinext

[NathanDrake2406]

@codex review

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@vinext/cloudflare@2765

npm i https://pkg.pr.new/create-vinext-app@2765

npm i https://pkg.pr.new/@vinext/types@2765

npm i https://pkg.pr.new/vinext@2765

commit: 97cfd70

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7e8f5c7793

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Performance benchmarks

Compared 97cfd70 against base dc05459 using alternating same-runner rounds. Next.js was unchanged and skipped.

0 improved · 0 regressed · 6 within ±1.5%

Scenario	Framework	Baseline	Current	Change
Client bundle size (gzip)	vinext	134.5 KB	134.5 KB	⚫ +0.0%
Client entry size (gzip)	vinext	122.1 KB	122.1 KB	⚫ +0.0%
Dev server cold start	vinext	2.85 s	2.82 s	⚫ -1.0%
Production build time	vinext	3.03 s	3.00 s	⚫ -1.0%
RSC entry closure size (gzip)	vinext	112.3 KB	112.3 KB	⚫ +0.0%
Server bundle size (gzip)	vinext	189.7 KB	189.7 KB	⚫ +0.0%

View detailed results and traces

_{🟢 improvement · 🔴 regression · ⚫ change below 1.5% · paired base/head}

[NathanDrake2406]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 4ff942acd9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[james-elicx]

/bigbonk review for issues

[ask-bonk]

Review: ReDoS fix looks correct and effective

I verified the change against the actual regex engine, the request-time call sites, and the Vitest runner source. The fix is sound and the regression test is valid. Approving in substance.

The vulnerability is real and request-triggerable

The pattern runs against the attacker-controlled User-Agent header at request time in multiple server paths: server/dev-server.ts:1090, server/pages-page-data.ts:1194, server/streaming-metadata.ts:8, and server/pages-page-response.ts:742. So quadratic matching on a long UA is a genuine request-time CPU DoS, not just a theoretical concern.

The fix is effective and correct

Measured old vs. new on "a".repeat(n) (Node regex engine):

| n | old [\w-]+-Google | new (?:^|[^\w-])[\w-]+-Google |
|---|---|---|
| 16,000 | 284 ms | 0.2 ms |
| 32,000 | 1,129 ms | 0.4 ms |
| 64,000 | 4,488 ms | 0.5 ms |

The old branch was quadratic; the new one is linear. I also confirmed all remaining branches (Google-[\w-]+, etc.) stay linear through 128k chars.

Correctness is preserved. Any -Google occurrence is preceded by a [\w-]+ run that is itself preceded by either the string start or a [^\w-] delimiter, so the anchor never causes a miss. Verified Mediapartners-Google, AdsBot-Google, Storebot-Google (leading, mid-string, and after !/space/tab), plus Google-Extended and negative cases — all match old behavior.

The Codex reviewer's concern about the timeout test is incorrect

Codex claimed the synchronous isPagesStreamingBot() call blocks the event loop so Vitest's timeout timer can't fire, making the test merely slow rather than failing. This is disproven by the Vitest 4.1.10 runner source. In @vitest/runner/dist/chunk-artifact.js, withTimeout records startTime = now() (line 2268) and, after a sync test returns, resolve() checks if (now() - startTime >= timeout) rejectTimeoutError() (lines 2288-2294 — the fix for vitest-dev/vitest#2920). So a synchronous body that overruns the budget does fail the test even though the setTimeout never fired. The author's response is accurate; the 500 ms bound genuinely enforces the regression. No change needed.

Minor notes (non-blocking)

• This anchoring is a deliberate divergence from Next.js's upstream [\w-]+-Google (confirmed still unanchored on canary). The added comment documents the intent, which satisfies the AGENTS.md "deliberate divergence is OK when documented" rule. Consider adding a one-line note that this is an intentional divergence from the upstream pattern (not just a perf tweak of vinext's own list) so a future sync with Next.js doesn't silently revert it.
• The 64,000-char test has ample headroom (fix runs in ~3 ms vs. the 500 ms limit), so it won't be flaky on slow CI runners.

Overall: correct fix, real security benefit, valid test. LGTM.

[ask-bonk]

Review posted successfully. Here's my summary:

Review Summary — PR #2765 (fix: prevent bot user-agent regex backtracking)

Verdict: LGTM. The fix is correct, effective, and addresses a real vulnerability.

Key findings

1. The vulnerability is real and request-triggerable. The regex runs against the attacker-controlled User-Agent header at request time in 4 server paths (dev-server.ts:1090, pages-page-data.ts:1194, streaming-metadata.ts:8, pages-page-response.ts:742). Confirmed old regex is quadratic — 4.5s at 64k chars — a genuine CPU DoS.

2. The fix works and is correct. The (?:^|[^\w-]) anchor makes matching linear (~0.5ms at 64k) while preserving all match behavior. I verified logically (every -Google has a [\w-]+ run preceded by start-or-delimiter) and empirically across leading/mid-string/delimiter cases.

3. The Codex reviewer's concern is wrong. Codex claimed the sync test can't fail because the blocked event loop prevents the timeout timer firing. I traced @vitest/runner@4.1.10's withTimeout (lines 2288-2294) — it records startTime and rejects on now() - startTime >= timeout after a sync test returns (the vitest#2920 fix). The author's rebuttal is accurate; the 500ms bound genuinely enforces the regression.

Minor non-blocking notes

• This is a deliberate divergence from Next.js's still-unanchored upstream pattern; suggested adding a note so a future upstream-sync doesn't silently revert it.
• Test has ample headroom (~3ms vs 500ms), so no flakiness risk.

I did not edit any files or run any git write operations, consistent with fork-review mode.

github run