Stabilize PDF fixture conversion

[Ivlad003]

Summary

• remove recurring PDF page chrome and split packed key/value metadata fields
• normalize PDF table continuation rows, spacer columns, and cross-page table merges
• render PDF creation metadata with a Created label while preserving Date for other converters

Test Plan

• cargo test

[Copilot]

Pull request overview

This PR stabilizes the PDF → Markdown conversion pipeline against a set of real-world PDF fixtures by adding fixture-based regression tests and introducing additional PDF-specific normalization/cleanup logic (tables, recurring chrome, and metadata rendering).

Changes:

• Added fixture-based regression tests for PDF conversion/rendering behavior across several real PDFs.
• Extended the document metadata model to support a customizable date label, and updated the Markdown renderer to respect it (PDF uses “Created”).
• Adjusted PDF table detection/assembly to better handle continuation rows, empty spacer columns, and cross-page table merges; filtered recurring page chrome from body/title.

Reviewed changes

Copilot reviewed 15 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/renderer_tests.rs	Updates test document metadata construction to include the new date_label field.
tests/pdf_fixture_tests.rs	Adds fixture-based regression tests covering PDF chrome removal, table normalization/merging, metadata label rendering, and leakage checks.
tests/model_tests.rs	Updates metadata construction for the new date_label field.
tests/integration_test.rs	Updates metadata construction for the new date_label field.
tests/converter_tests.rs	Updates mock converter metadata for the new date_label field.
src/renderer/markdown.rs	Renders metadata date using date_label (defaulting to “Date”).
src/model/document.rs	Adds date_label: Option<String> to Metadata.
src/converter/web/mod.rs	Initializes date_label to None for web conversions.
src/converter/pdf/table_detector.rs	Adds table normalization (header fixes, continuation-row merge, spacer-column merge, fixture heuristics) and a nearest-column fallback for cell assignment.
src/converter/pdf/mod.rs	Filters recurring chrome from PDF title metadata and sets date_label to “Created”; adjusts table insertion position logic.
src/converter/pdf/assembler.rs	Filters recurring chrome as header/footer noise; tweaks cross-page table merge logic; splits packed key/value paragraphs.
src/converter/image_ocr/mod.rs	Initializes date_label to None for OCR conversions.
src/converter/audio/mod.rs	Initializes date_label to None for audio conversions.
README.md	Improves table formatting and adds local install instructions.
Cargo.toml	Adds regex as a dev-dependency for tests.
Cargo.lock	Updates lockfile to include regex for tests.
.gitignore	Ignores pdfs/* and output/* while keeping .gitkeep.

Comments suppressed due to low confidence (1)

src/converter/pdf/table_detector.rs:595

• Table row grouping (detect_rows) only counts blocks that satisfy find_column(..., max_distance), but build_table now assigns all blocks to a column via nearest_column when they exceed max_distance. That mismatch can cause a Y-line to be treated as a continuation (few counted columns) while still injecting far-away blocks into cells, leading to incorrect row merges/cell pollution. Consider keeping column assignment criteria consistent (e.g., apply a capped fallback distance or reuse the same assignment function in both places).

                    if let Some(ci) = Self::find_column(block.x, columns, col_dist)
                        .or_else(|| Self::nearest_column(block.x, columns))
                    {
                        let cleaned = Self::clean_cell_text(&block.text);
                        if !cleaned.is_empty() {
                            cells[ci].push(cleaned);
                        }
                    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.