i18n: heading-section chunked translation for long MDX pages#1159
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughIntroduces a generalized chunked translation engine ( ChangesChunked Translation Engine
Comfy Cloud Glossary and Preserve Term
Sequence Diagram(s)sequenceDiagram
participant translateFile
participant resolveFileChunkStrategy
participant getSectionSyncStatus
participant translateChunkedFile
participant serializeChunkedDocument
translateFile->>resolveFileChunkStrategy: relPath, enContent
resolveFileChunkStrategy-->>translateFile: ChunkStrategy | null
alt strategy resolved
translateFile->>translateChunkedFile: relPath, lang, strategy
translateChunkedFile->>getSectionSyncStatus: enContent, existingContent, strategy, force
getSectionSyncStatus-->>translateChunkedFile: SectionSyncStatus (upToDate, pendingBlocks, needsFrontmatter)
loop each pending block
translateChunkedFile->>translateChunkedFile: translate + validate block
translateChunkedFile->>translateChunkedFile: writeChunkedCheckpoint (per-block)
end
translateChunkedFile->>serializeChunkedDocument: frontmatter, slots, fileHash, enRel, blockHashes
serializeChunkedDocument-->>translateChunkedFile: final MDX with updated translationBlockHashes
end
🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Split oversized docs at ## boundaries with per-section hash sync and checkpointing, extend truncation detection, and pin Comfy Cloud as a preserved brand term in preserve_terms and glossary overrides.
comfyui-wiki
force-pushed
the
feat/i18n-heading-section-chunking
branch
from
61aff54 to
3741841
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/scripts/i18n/chunked-translate.test.ts (1)
14-147: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winAdd regression tests for label-key safety and fenced-heading parsing.
Please add focused tests for: (1) labels containing
:or quotes, (2) duplicate H2 titles, and (3)##inside fenced code blocks. These are high-risk sync boundaries and currently unguarded.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/scripts/i18n/chunked-translate.test.ts around lines 14 - 147, Add three focused regression tests to the test file to cover high-risk edge cases in heading parsing: First, add a test for the parseHeadingSections function to verify it handles labels containing special characters like colons and quotes correctly without breaking the label-key mapping. Second, add a test for parseHeadingSections to ensure it properly handles duplicate H2 titles and creates appropriate labels (possibly with numeric suffixes) to maintain uniqueness. Third, add a test for parseHeadingSections to verify that double hash symbols inside fenced code blocks are not treated as section boundaries and remain part of the content. These tests should be positioned within the existing describe blocks for the functions they test (parseHeadingSections, etc.) to ensure they catch potential sync boundary issues.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/scripts/i18n/chunked-translate.ts:
- Around line 160-170: The parseHeadingSections and countH2Sections functions
don't account for code fences, causing them to incorrectly treat heading-like
text within code blocks as real section boundaries. Add fence detection by
creating a FENCE_RE regex pattern to match code block delimiters, then in both
functions add an inFence boolean flag that toggles when encountering fence
markers. Modify the H2_HEADING_RE check to only apply when inFence is false,
ensuring that heading detection is skipped inside code blocks. Apply this logic
consistently in parseHeadingSections (checking the condition before processing
the line) and countH2Sections (looping through lines to track fence state and
only incrementing count when not in a fence).
- Around line 90-103: The regex pattern in parseBlockHashesFromFrontmatter uses
([^:]+) to capture hash keys, which breaks when keys contain colons (like
"Setup: Windows") since it stops at the first colon and only captures the text
before it. Update the regex to properly handle YAML-safe keys by quoting them
and adjusting the pattern to match quoted keys like "([^"]+)" instead of
([^:]+), ensuring that keys with colons round-trip correctly and prevent
indefinite retranslation of sections.
- Around line 347-355: The issue is that when English sections are deleted, the
function does not detect this as a pending sync because it only compares hashes
for sections that currently exist in enHashes. To fix this, in addition to the
existing filter that finds changed blocks in the pendingBlocks calculation, also
identify and include any sections that exist in storedHashes but are no longer
present in enHashes (these represent deleted sections). Combine the changed
blocks with the deleted blocks so that upToDate correctly returns false when
sections have been removed from the English document, ensuring that stale
translated sections for deleted English sections are properly flagged for
resync.
- Around line 63-67: The documentBlockHashes function silently overwrites block
hash entries when duplicate labels are encountered, causing data loss and
breaking pending-block detection. Before assigning the block hash to
out[b.label] in the loop, add a guard check to detect if the label already
exists in the output object. If a duplicate is found, either throw an error with
details about the conflicting label or log a warning to prevent silent
corruption of the block hash mapping.
---
Outside diff comments:
In @.github/scripts/i18n/chunked-translate.test.ts:
- Around line 14-147: Add three focused regression tests to the test file to
cover high-risk edge cases in heading parsing: First, add a test for the
parseHeadingSections function to verify it handles labels containing special
characters like colons and quotes correctly without breaking the label-key
mapping. Second, add a test for parseHeadingSections to ensure it properly
handles duplicate H2 titles and creates appropriate labels (possibly with
numeric suffixes) to maintain uniqueness. Third, add a test for
parseHeadingSections to verify that double hash symbols inside fenced code
blocks are not treated as section boundaries and remain part of the content.
These tests should be positioned within the existing describe blocks for the
functions they test (parseHeadingSections, etc.) to ensure they catch potential
sync boundary issues.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 1265bb33-5c2f-4c5e-9fde-ca96f547a903
📒 Files selected for processing (9)
.github/scripts/i18n/README.md.github/scripts/i18n/check-translation-truncation.ts.github/scripts/i18n/chunked-translate.test.ts.github/scripts/i18n/chunked-translate.ts.github/scripts/i18n/glossary/overrides/ja.json.github/scripts/i18n/glossary/overrides/ko.json.github/scripts/i18n/glossary/overrides/zh.json.github/scripts/i18n/translate-i18n.ts.github/scripts/i18n/translation-config.json
Skip code-fence lines when splitting on ## headings, quote YAML block-hash keys for labels with colons, detect removed EN sections for re-serialize, and throw on duplicate section labels instead of silently overwriting.
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
cat .github/scripts/i18n/chunked-translate.tsLength of output: 13848 🏁 Script executed: #!/bin/bash
# Check translate-i18n.ts for the needsReserialize handling
rg -n "needsReserialize\|translateChunkedFile\|getSectionSyncStatus" .github/scripts/i18n/translate-i18n.ts -A3 -B1Length of output: 430 🏁 Script executed: #!/bin/bash
# Search for needsReserialize in translate-i18n.ts with PCRE2
rg -n "needsReserialize" .github/scripts/i18n/translate-i18n.ts || echo "NOT FOUND"
# Also check for the function and broader chunked handling
rg -n "translateChunked\|getSectionSync\|needsReseria" .github/scripts/i18n/translate-i18n.ts || echo "NOT FOUND"Length of output: 459 🏁 Script executed: #!/bin/bash
# Get key sections of translate-i18n.ts around chunked translation
grep -n "needsReserialize\|translateChunked\|getSectionSync\|serializeChunked\|resolveFileChunk" .github/scripts/i18n/translate-i18n.ts | head -60Length of output: 1246
1. Fence-aware heading parsing ✓ 2. YAML-safe block-hash keys ✓ 3. Deleted EN sections not detected ✓ 4. Duplicate H2 titles throw instead of silently overwriting ✓ One minor pre-existing observation (not introduced by these fixes): Overall the implementation is solid. The chunked translation pipeline should now handle fenced code blocks, colon-containing section titles, removed English sections, and duplicate headings robustly. Great work, and thanks for the thorough follow-up! 🐇✨ |
…x passes. Extract shared frontmatter-stripping helpers and reuse them from translate-i18n.ts instead of maintaining a second copy.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
1 participant
Summary
heading_sectionschunking strategy: split long MDX at level-2##headings, translate per section, and checkpoint after each block to avoid output truncation (max_tokens: 8192).translationBlockHashesfor incremental sync; unify changelogupdate_blocksunder the same chunked pipeline.auto_chunk(body ≥ 10k chars, ≥ 4 sections) and explicitly configuretutorials/partner-nodes/pricing.mdx.missing_sectionsdetection.preserve_terms+ glossary overrides for zh/ja/ko).Test plan
bun .github/scripts/i18n/translate-i18n.ts --dry-run -- tutorials/partner-nodes/pricing.mdx --lang koshows[SECTIONS]with pending sectionspnpm translate -- tutorials/partner-nodes/pricing.mdx --lang koand verify full output (no truncation)pnpm translate:check-truncation -- --lang koon pricing after translation