feat(extraction): add progressive LiteParse PDF processing#627
Conversation
|
Warning Review limit reached
Next review available in: 37 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughAdds PDF readability validation, a LiteParse container and extraction provider, provisional markdown projections, workflow fallback handling, and telemetry for LiteParse and enhancement outcomes. ChangesLiteParse PDF flow
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant WorkspaceFileUpload
participant WorkspaceFileExtractionWorkflow
participant LiteParsePdfExtractor
participant WorkspaceKernel
WorkspaceFileUpload->>WorkspaceFileUpload: validate PDF readability
WorkspaceFileExtractionWorkflow->>LiteParsePdfExtractor: submit PDF bytes
LiteParsePdfExtractor-->>WorkspaceFileExtractionWorkflow: return markdown pages
WorkspaceFileExtractionWorkflow->>WorkspaceKernel: upsert provisional projection
WorkspaceFileExtractionWorkflow-->>WorkspaceFileExtractionWorkflow: record extraction outcome
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThis PR adds a faster staged PDF extraction path before the existing enhancement flow. The main changes are:
Confidence Score: 4/5The converted PDF upload path needs a buffer-copy fix before merging. Converted PDFs validate the same buffer that is later written to storage. Direct PDF uploads and the LiteParse extraction flow otherwise follow the changed contracts shown in the diff. Container and environment bindings are added consistently across the changed configuration sections. src/features/workspaces/upload/workspace-file-upload-normalization.ts
 T-Rex LogsWhat T-Rex did
Important Files Changed
Reviews (1): Last reviewed commit: "fix(upload): reject unreadable PDFs befo..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (input.descriptor.assetKind !== "pdf") { | ||
| return; | ||
| } | ||
|
|
There was a problem hiding this comment.
When a converter returns a PDF as an ArrayBuffer, this line validates the same buffer that the caller later persists as prepared.body. If the PDF parser takes ownership of or transfers that buffer while loading the document, validation can pass but the later R2 write receives a detached or consumed buffer, so converted PDF uploads can fail or be stored incorrectly.
Artifacts
Repro: focused Vitest harness for converted PDF ArrayBuffer consumption
- Contains supporting evidence from the run (text/typescript; charset=utf-8).
Repro: Vitest config used to run the artifact harness
- Contains supporting evidence from the run (text/typescript; charset=utf-8).
Repro: verbose failing test output showing detached prepared.body and failed persistence simulation
- Keeps the command output available without making the summary code-heavy.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b0f03b973c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| inputBytes = bytes.byteLength; | ||
| const result = await parser.parse(bytes); | ||
| const pages = result.pages.map((page) => ({ | ||
| markdown: page.markdown ?? page.text ?? "", |
There was a problem hiding this comment.
For PDFs where the LiteParse pass succeeds but enhancement later fails, this retains the provisional projection. The LiteParse Node API documents Markdown output as being returned on result.text, while result.pages[] exposes page text/items rather than a markdown field (https://developers.llamaindex.ai/liteparse/guides/library-usage/#markdown-output); this fallback therefore stores layout-preserved/plain page text in the projection, dropping the headings/tables/link/image Markdown that the fast projection is supposed to provide.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
containers/liteparse/server.mjs (1)
43-50: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAdd a timeout to
parser.parse()to prevent indefinite hangs.If LiteParse hangs on a malformed PDF, the parse promise remains pending even after the upstream workflow step times out and closes the fetch connection. With
max_instances: 2, accumulated hung operations could degrade container availability. Wrapping withPromise.raceagainst a timeout ensures the handler completes, though the underlying parse may still run in the background.♻️ Suggested timeout wrapper
const bytes = new Uint8Array(await file.arrayBuffer()); inputBytes = bytes.byteLength; - const result = await parser.parse(bytes); + const parseTimeout = new Promise<never>((_, reject) => { + setTimeout(() => reject(new Error("Parse timeout")), 90_000); + }); + const result = await Promise.race([parser.parse(bytes), parseTimeout]); const pages = result.pages.map((page) => ({🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@containers/liteparse/server.mjs` around lines 43 - 50, Wrap the parser.parse(bytes) promise in the request handler with a Promise.race timeout so malformed PDFs cannot leave the handler pending indefinitely. Preserve the existing pages mapping and successful response flow, and make the timeout reject or otherwise follow the handler’s existing error path after the configured duration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@containers/liteparse/Dockerfile`:
- Around line 1-12: Update the Dockerfile to run the container as the
image-provided non-root node user by adding USER node after copying the
application files and before EXPOSE/CMD runtime instructions. Do not change the
existing dependency installation or startup commands.
---
Nitpick comments:
In `@containers/liteparse/server.mjs`:
- Around line 43-50: Wrap the parser.parse(bytes) promise in the request handler
with a Promise.race timeout so malformed PDFs cannot leave the handler pending
indefinitely. Preserve the existing pages mapping and successful response flow,
and make the timeout reject or otherwise follow the handler’s existing error
path after the configured duration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5c713201-75bb-4a4b-a009-63d114267e17
⛔ Files ignored due to path filters (2)
containers/liteparse/package-lock.jsonis excluded by!**/package-lock.jsonpnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (19)
containers/liteparse/Dockerfilecontainers/liteparse/package.jsoncontainers/liteparse/server.mjspackage.jsonsrc/features/workspaces/extraction/liteparse-projection.tssrc/features/workspaces/extraction/providers/liteparse-response.tssrc/features/workspaces/extraction/providers/liteparse.test.tssrc/features/workspaces/extraction/providers/liteparse.tssrc/features/workspaces/extraction/types.tssrc/features/workspaces/extraction/workspace-file-extraction-observability.tssrc/features/workspaces/extraction/workspace-file-extraction-workflow.tssrc/features/workspaces/model/workspace-file/policy.tssrc/features/workspaces/upload/pdf-upload-validation.test.tssrc/features/workspaces/upload/pdf-upload-validation.tssrc/features/workspaces/upload/workspace-file-upload-normalization.tssrc/integrations/posthog/events.tssrc/server.tsworker-configuration.d.tswrangler.jsonc
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 21 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Use rendered LiteParse markdown directly. Preserve converted PDF upload buffers. Keep successful extraction writes from falling back. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews. |
There was a problem hiding this comment.
1 issue found across 10 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/features/workspaces/extraction/workspace-file-extraction-workflow.ts">
<violation number="1" location="src/features/workspaces/extraction/workspace-file-extraction-workflow.ts:222">
P2: A failed artifact deletion is now converted into a successful terminal step, so transient R2 failures permanently accumulate per-workflow artifacts and are invisible to telemetry. Preserve the successful extraction result, but schedule/retry cleanup or at least emit an operational failure for this branch.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| await this.env.WORKSPACE_KERNEL_FILES.delete(extraction.artifactKey); | ||
| return { deleted: extraction.artifactKey }; | ||
| } catch (error) { | ||
| return { |
There was a problem hiding this comment.
P2: A failed artifact deletion is now converted into a successful terminal step, so transient R2 failures permanently accumulate per-workflow artifacts and are invisible to telemetry. Preserve the successful extraction result, but schedule/retry cleanup or at least emit an operational failure for this branch.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/features/workspaces/extraction/workspace-file-extraction-workflow.ts, line 222:
<comment>A failed artifact deletion is now converted into a successful terminal step, so transient R2 failures permanently accumulate per-workflow artifacts and are invisible to telemetry. Preserve the successful extraction result, but schedule/retry cleanup or at least emit an operational failure for this branch.</comment>
<file context>
@@ -235,6 +213,41 @@ export class WorkspaceFileExtractionWorkflow extends WorkflowEntrypoint<
+ await this.env.WORKSPACE_KERNEL_FILES.delete(extraction.artifactKey);
+ return { deleted: extraction.artifactKey };
+ } catch (error) {
+ return {
+ deleted: null,
+ errorType: error instanceof Error ? error.name : "UnknownError",
</file context>
Use LiteParse's per-page markdown contract. Keep document text out of the projection protocol. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
0 issues found across 4 files (changes from recent commits).
Requires human review: Auto-approval blocked by 1 unresolved issue from previous reviews.
Re-trigger cubic
1 participant
Summary
Why
PDF uploads previously remained unavailable until the full LlamaParse job completed, and password-protected PDFs were accepted even though neither the viewer nor extraction pipeline could use them. This provides a fast usable projection while preserving the higher-quality enhancement path, and moves authoritative PDF readability validation ahead of persistence.
Changes
LiteParsePdfExtractorcontainer using pinned@llamaindex/liteparse@2.5.1standard-2instances with a maximum of two and no internet accessunpdf@1.6.2validation at the prepared-upload boundary so both direct and converted PDFs are checked before R2 writesPASSWORD_PROTECTED_PDFandINVALID_PDF422 errorsTesting
pnpm checkpnpm test— 23 tests passedpnpm buildunpdfverification against plain and AES-256 password-protected PDFsReview Notes
Start with
workspace-file-extraction-workflow.tsfor orchestration,liteparse-projection.tsfor the provisional publish boundary, andpdf-upload-validation.tsfor upload rejection behavior.The local Node runtime was 25.8.0 while the repository requests Node 24.11.x. Checks passed despite the engine warning.
pnpm peers checkalso reports the repository's existing Babel, Vite, and Tailwind peer mismatches.Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.Summary by CodeRabbit