Add toPdf(): searchable PDF export with invisible RTR text overlay - #27
Merged
Conversation
Renders each page's raster image into a PDF page and draws recognized handwriting (RTR) words as invisible (render mode 3) text at their source location, so PDF viewers can search/select/copy handwritten text. The word bounding boxes in the recognition JSON are stored in a coordinate space divided by a fixed 11.9 scale factor relative to the raster pixel space; this was verified empirically (cropping the raster image at candidate scales and visually confirming word alignment) and cross-checked against an independent reverse-engineering of the same Supernote file format.
philips
changed the title
philips
marked this pull request as ready for review
July 25, 2026 08:29
philips
added a commit
to philips/supernote-obsidian-plugin
that referenced
this pull request
Jul 25, 2026
… build Points the submodule at philips/supernote-typescript#27 (worktree-rtr-searchable-pdf-plan, commit 2ad5251), which adds toPdf(note, options?) to the library itself: real invisible PDF text (render mode 3, via pdf-lib) instead of alpha=0 fill, embedded fonts, and a properly verified recognition-box coordinate scale of 11.9 — found by cropping the raster image at candidate scales and visually confirming word alignment, cross-checked against an independent reverse-engineering of the same format. Our own RECOGNITION_BOX_SCALE = 13.5 here was an empirical guess against the same fixture without that cross-check; 11.9 supersedes it. Deletes buildNotePdf/drawPositionedRecognizedText/decodeRecognizedLabel/ RECOGNITION_BOX_SCALE from main.ts and the jspdf dependency (and its canvg/dompurify/html2canvas/core-js transitive tree) entirely — that whole PDF-assembly path now lives in, and is tested by, the library. VaultWriter.exportToPDF no longer needs its own ImageConverter pass either, since toPdf() rasterizes internally. SupernoteView.onLoadFile still needs `images` from ImageConverter separately for the thumbnail sidebar/save-to-vault/drag-out, so that page duplicates a render pass toPdf() also does internally — a small tradeoff for not having to thread pre-rendered images through the library's PDF builder.
- compositeImages() requires 8-bit RGBA on both layers, but user-uploaded background templates are arbitrary PNGs and can decode to other formats (e.g. 8-bit RGB with no alpha). Normalize via convertBitDepth(8).convertColor(RGBA) after decodePng(). - toPdf() aborted the entire PDF if any recognized word contained a character the active font couldn't encode (e.g. "→", which isn't in WinAnsi/Helvetica). Skip just that word instead of failing the whole page. Adds a 275KB fixture (tests/input/moonchild-user-bg-and-bad-glyph.note) trimmed from a real 23MB note that hit both bugs on the same page, by directly repacking the minimal set of address-referenced blocks the Supernote container format needs (see the file's structure notes in src/parsing.ts) rather than committing the full original file.
Previously the font was only shrunk when text overflowed the box width, so narrower words (e.g. "I", short words, or compressed cursive) left the invisible text run's advance width smaller than the actual handwriting, throwing off the search-hit highlight rectangle PDF viewers draw. Now font size is set to the box height and the `Tz` horizontal-scaling operator stretches or squeezes the text run to exactly match the box width in both directions. Verified with pdfjs-dist: the generated text item width now matches the expected box width (in points) to five decimal places.
Owner
Author
|
Follow-up: recognized words are now sized with the PDF horizontal-scaling operator (`Tz`) so the invisible text run's advance width matches the recognition bounding box exactly, in both directions — not just capped when it overflowed. This means a PDF viewer's search-hit highlight rectangle now lines up with the actual handwriting instead of sometimes being narrower (e.g. for short/compressed words). Verified with `pdfjs-dist` that the generated text item width matches the expected box width to 5 decimal places. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
plans/rtr-searchable-pdf.mdandplans/rtr-searchable-pdf-workers.md, the implementation plans for this feature and a follow-up worker-parallelism design.toPdf(note, options?)insrc/pdf.ts, exported from the package root: renders each page's raster image into a PDF page and overlays each recognized handwriting (RTR) word as invisible (PDF text-rendering mode 3) text at its source location, so the resulting PDF is searchable/selectable over the handwriting image — the standard "OCR text layer" technique.toPdfdefaults to the standard Helvetica font (Latin-only); callers can passfontBytesfor a Unicode TTF/OTF (embedded via@pdf-lib/fontkit) to support other scripts.options.dpi) since Supernote files don't record device DPI. This only affects print/physical sizing, not searchability.README.mdusage docs andtests/pdf.test.ts.Two follow-up fixes from real-world testing
While testing against a real 23MB multi-page note, found and fixed two crashes:
compositeImages()(src/conversion.ts) requires 8-bit RGBA on both layers, but user-uploaded background templates are arbitrary PNGs and can decode to other formats (this note had an 8-bit RGB template with no alpha channel). Now normalized viaconvertBitDepth(8).convertColor(RGBA)afterdecodePng().toPdf()previously aborted the entire PDF if any single recognized word contained a character the active font couldn't encode (e.g. "→", not in WinAnsi/Helvetica). Now skips just that word instead of failing the whole page.tests/input/moonchild-user-bg-and-bad-glyph.note, a 275KB fixture trimmed from the original 23MB note (which hit both bugs on the same page) by directly repacking the minimal set of address-referenced blocks the Supernote container format needs, rather than committing the full original file.Test plan
npm run build(tsc typecheck)npx eslint .npx vitest run— all 18 tests passpdf-parse'sgetScreenshotand visually confirmed the image renders correctly with no visible artifacts from the invisible text layer