diff --git a/claude.md b/claude.md index 788e2df..4f6ca88 100644 --- a/claude.md +++ b/claude.md @@ -35,21 +35,29 @@ Entry point is `VerifyPDFium.Initialize(dpi = 96)` which registers a stream conv 3. One `png` target per page, named `page_0001` style To keep snapshots stable for PDFs freshly generated at test time, the non-deterministic fields are neutralized two ways: -- **In the `pdf` bytes** (`PdfNormalizer.Normalize`): the trailer `/ID`, the info-dictionary `/CreationDate`/`/ModDate`, and the XMP metadata dates plus `xmpMM:DocumentID`/`InstanceID` are overwritten by a length-preserving, in-place byte scan (no string round-trip, no regex) — only the volatile characters change, so cross-reference offsets stay valid. `Normalize` mutates in place and runs *after* the `PdfiumDocument` (which reads lazily from the same buffer) is disposed, so no defensive copy is needed. Values inside a compressed object/metadata stream (`/ObjStm`, flate-compressed XMP) are not reachable by the plaintext scan. -- **In the info file** (`PdfNormalizer.NormalizeProperties`): `Properties` is a `Dictionary` whose `CreationDate`/`ModDate` values are parsed (`PdfDate`) into `DateTimeOffset`, so Verify's built-in date scrubbing renders them deterministically (`DateTimeOffset_1` etc.). Properties are read from the original bytes, before `Normalize` zeroes them. +- **In the `pdf` bytes** (`DeterministicPdf.PdfNormalizer.Normalize`): the trailer `/ID`, the info-dictionary `/CreationDate`/`/ModDate`, and the XMP dates plus `xmpMM:DocumentID`/`InstanceID`/`OriginalDocumentID` and `dc:date` are overwritten by an in-place byte scan (no string round-trip, no regex). That value-zeroing is length-preserving, so cross-reference offsets survive it. A final `CanonicalizeXmp` pass then collapses the XMP packet's whitespace — Apache FOP serializes the packet through the platform's XML writer, so indentation varies by JRE — and that pass is **not** length-preserving: it rebuilds the buffer and repairs both the metadata stream length and the classic cross-reference table. A document it cannot safely rewrite (cross-reference stream, incremental update, unlocatable stream length) comes back unchanged. `Normalize` copies its input, so a caller keeps ownership of the array passed in; the call is still made only after the `PdfiumDocument` (which reads lazily from the same buffer) is disposed. Values compressed away inside an `/ObjStm` or a flate-compressed XMP packet are not reachable by the plaintext scan and are left alone. +- **In the info file** (`PdfProperties.Normalize`): `Properties` is a `Dictionary` whose `CreationDate`/`ModDate` values are parsed (`PdfDate`) into `DateTimeOffset`, so Verify's built-in date scrubbing renders them deterministically (`DateTimeOffset_1` etc.). Properties are read from the original bytes, before the byte-level pass zeroes them. + +Both can be opted out of per verification, via a context key set by a `SettingsTask` extension and read in `Convert`: +- `ExcludePdfDocument` — drops the `.verified.pdf` entirely, for producers whose bytes can never be made deterministic (Aspose.Cells embeds the machine's system fonts). +- `SkipPdfNormalization` — keeps the `.verified.pdf` but snapshots the producer's own bytes, for producers that are already byte-deterministic. Worth knowing when changing this: `CanonicalizeXmp` is the pass that alters bytes even for such a producer, so toggling the setting on an existing suite shifts every stored `.verified.pdf` once. Key files: -- **VerifyPDFium.cs** — initialization and the converter -- **PdfNormalizer.cs** — neutralizes the `pdf` bytes and projects the info-file properties to the scrubbable object map +- **VerifyPDFium.cs** — initialization, the converter, and the two opt-out extensions +- **PdfProperties.cs** — projects the pdfium-reported property map to the scrubbable object map - **PdfDate.cs** — parses PDF date strings (`D:YYYYMMDD…`) to `DateTimeOffset` - **PdfInfo.cs** / **PageInfo.cs** — info shape for the snapshot (per-page width/height in points and text via `PdfPage.GetText()`) +The byte-level normalizer is **not** a type in this repo — it is `PdfNormalizer` from the [DeterministicPdf](https://github.com/SimonCropp/DeterministicPdf) package, which owns and tests the algorithm. + Style note: only public types get a namespace declaration (`VerifyTests`); internal types live in the global namespace. ### Tests (`src/Tests/`) - **ModuleInitializer.cs** — calls `VerifyPDFium.Initialize()` via `[ModuleInitializer]` (also the readme `enable` snippet); PNGs compare via SSIM (`VerifierSettings.UseSsimForPng()`) - **Samples.cs** — snapshot tests, also the readme usage snippets +- **PdfNormalizerTests.cs** — asserts the wiring rather than the algorithm: that a normalized document still loads in pdfium, that volatile values are neutralized, and that the pass is idempotent +- **SkipPdfNormalizationTests.cs** — asserts the opposite pairing, that the skipped snapshot holds the producer's own bytes while the default one holds the neutralized bytes. `SampleIsNotAlreadyNormalized` pins the premise, so the pair cannot both pass vacuously if `sample.pdf` ever becomes normalization-invariant - Test assets `sample.pdf` (1 page) and `multi-page.pdf` (4 pages), both Letter size, were produced by [Morph](https://github.com/Papyrine/Morph)'s PDF exporter with embedded font subsets, so rendering is machine-independent. Engine-level unit tests (load/render/page sizes/metadata/threading) live in the Morph.PDFium repo, not here. @@ -57,6 +65,7 @@ Engine-level unit tests (load/render/page sizes/metadata/threading) live in the ## Dependency notes - Morph.PDFium pins the PDFium native version; bumping it may shift rendered pixels — expect to regenerate `*.verified.png` files in the same commit. +- DeterministicPdf owns the byte-level normalization, so bumping it may shift `.verified.pdf` files across every consuming suite without any document having changed. The 1.3.0 release did exactly that by adding `CanonicalizeXmp`: downstream snapshots moved by the size of the collapsed XMP whitespace while the rendered pages and info files stayed identical. When a bump changes the `.verified.pdf` but leaves the pages and info alone, that is the expected shape of it — not a rendering regression. - Morph.PDFium is a Papyrine package: `Papyrine_SponsorshipLicenseIgnored` is set in `src/Directory.Build.props` and `SC023` is in `NoWarn` (same arrangement as Verify.OpenXml's Morph dependency). ## Build Configuration diff --git a/readme.md b/readme.md index 3a9d191..60ad0b7 100644 --- a/readme.md +++ b/readme.md @@ -12,7 +12,7 @@ Verifying a `pdf` produces: * The pdf itself as `.verified.pdf`. This can be omitted with [`ExcludePdfDocument`](#exclude-the-pdf-document). * A PNG render of every page as `#page_0001.verified.png`, `#page_0002.verified.png`, etc. -The non-deterministic fields of the pdf (the trailer `/ID`, the `/CreationDate` and `/ModDate`, and the equivalent XMP metadata) are neutralized so the same source document produces a byte-identical `.verified.pdf` across runs. +The non-deterministic fields of the pdf (the trailer `/ID`, the `/CreationDate` and `/ModDate`, and the equivalent XMP metadata) are neutralized so the same source document produces a byte-identical `.verified.pdf` across runs. A producer that already emits deterministic bytes can skip that work with [`SkipPdfNormalization`](#skip-pdf-normalization). Rendering is provided by [Morph.PDFium](https://github.com/Papyrine/Morph.PDFium), which wraps the prebuilt PDFium binaries from [pdfium-binaries](https://github.com/bblanchon/pdfium-binaries) (Windows, Linux, and macOS). Rendering is deterministic for a given Morph.PDFium version: the same input produces byte-identical PNGs on every machine and OS, and no image library dependency is added. @@ -100,6 +100,26 @@ public Task ExcludePdfDocument() => +### Skip pdf normalization + +By default the pdf bytes are normalized before being snapshotted: the trailer `/ID`, the `/CreationDate` and `/ModDate`, and the XMP dates and identifiers are neutralized, and the XMP packet is canonicalized. A producer that already emits byte-deterministic documents gains nothing from that, and pays for a full buffer copy, a rescan, and — when the XMP is canonicalized — a rebuild plus a cross-reference repair. `SkipPdfNormalization` snapshots the bytes exactly as produced: + + + +```cs +[Test] +public Task SkipPdfNormalization() => + VerifyFile("sample.pdf") + .SkipPdfNormalization(); +``` +snippet source | anchor + + +Only skip it when the producer is genuinely deterministic. Without normalization a freshly generated pdf carries a wall-clock `/CreationDate` and a fresh `/ID`, so the snapshot differs on every run. + +The XMP canonicalization is worth calling out, because it is the pass that changes bytes even for an already-deterministic producer: it collapses the packet's whitespace. Turning this setting on for an existing suite therefore shifts every stored `.verified.pdf` once, without anything about the documents having changed. + + ## Icon [PDF](https://thenounproject.com/icon/pdf-7564953//) designed by [Meilia](https://thenounproject.com/creator/meilia1/) from [The Noun Project](https://thenounproject.com). diff --git a/src/Directory.Build.props b/src/Directory.Build.props index 4c6b67c..ffe92d1 100644 --- a/src/Directory.Build.props +++ b/src/Directory.Build.props @@ -2,7 +2,7 @@ CS1591;CS0649;CA1416;NU1608;NU1109;SC023 - 1.3.0 + 1.4.0 14.0 1.0.0 PDF, PDFium, Verify diff --git a/src/Tests/Samples.SkipPdfNormalization#page_0001.verified.png b/src/Tests/Samples.SkipPdfNormalization#page_0001.verified.png new file mode 100644 index 0000000..0e0cadf Binary files /dev/null and b/src/Tests/Samples.SkipPdfNormalization#page_0001.verified.png differ diff --git a/src/Tests/Samples.SkipPdfNormalization.verified.pdf b/src/Tests/Samples.SkipPdfNormalization.verified.pdf new file mode 100644 index 0000000..acb06ce Binary files /dev/null and b/src/Tests/Samples.SkipPdfNormalization.verified.pdf differ diff --git a/src/Tests/Samples.SkipPdfNormalization.verified.txt b/src/Tests/Samples.SkipPdfNormalization.verified.txt new file mode 100644 index 0000000..9540e7d --- /dev/null +++ b/src/Tests/Samples.SkipPdfNormalization.verified.txt @@ -0,0 +1,16 @@ +{ + PageCount: 1, + Properties: { + CreationDate: DateTimeOffset_1, + Creator: Morph, + ModDate: DateTimeOffset_1, + Producer: PDFsharp 6.2.4 + }, + Pages: [ + { + Width: 612.0, + Height: 792.0, + Text: Hello, World! This is a simple paragraph. + } + ] +} \ No newline at end of file diff --git a/src/Tests/Samples.cs b/src/Tests/Samples.cs index 23ea3e4..0e69a63 100644 --- a/src/Tests/Samples.cs +++ b/src/Tests/Samples.cs @@ -32,4 +32,13 @@ public Task ExcludePdfDocument() => .ExcludePdfDocument(); #endregion + + #region SkipPdfNormalization + + [Test] + public Task SkipPdfNormalization() => + VerifyFile("sample.pdf") + .SkipPdfNormalization(); + + #endregion } diff --git a/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes#page_0001.verified.png b/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes#page_0001.verified.png new file mode 100644 index 0000000..0e0cadf Binary files /dev/null and b/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes#page_0001.verified.png differ diff --git a/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes.verified.pdf b/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes.verified.pdf new file mode 100644 index 0000000..e8f6d78 Binary files /dev/null and b/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes.verified.pdf differ diff --git a/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes.verified.txt b/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes.verified.txt new file mode 100644 index 0000000..9540e7d --- /dev/null +++ b/src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes.verified.txt @@ -0,0 +1,16 @@ +{ + PageCount: 1, + Properties: { + CreationDate: DateTimeOffset_1, + Creator: Morph, + ModDate: DateTimeOffset_1, + Producer: PDFsharp 6.2.4 + }, + Pages: [ + { + Width: 612.0, + Height: 792.0, + Text: Hello, World! This is a simple paragraph. + } + ] +} \ No newline at end of file diff --git a/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes#page_0001.verified.png b/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes#page_0001.verified.png new file mode 100644 index 0000000..0e0cadf Binary files /dev/null and b/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes#page_0001.verified.png differ diff --git a/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes.verified.pdf b/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes.verified.pdf new file mode 100644 index 0000000..acb06ce Binary files /dev/null and b/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes.verified.pdf differ diff --git a/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes.verified.txt b/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes.verified.txt new file mode 100644 index 0000000..9540e7d --- /dev/null +++ b/src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes.verified.txt @@ -0,0 +1,16 @@ +{ + PageCount: 1, + Properties: { + CreationDate: DateTimeOffset_1, + Creator: Morph, + ModDate: DateTimeOffset_1, + Producer: PDFsharp 6.2.4 + }, + Pages: [ + { + Width: 612.0, + Height: 792.0, + Text: Hello, World! This is a simple paragraph. + } + ] +} \ No newline at end of file diff --git a/src/Tests/SkipPdfNormalizationTests.cs b/src/Tests/SkipPdfNormalizationTests.cs new file mode 100644 index 0000000..423f49a --- /dev/null +++ b/src/Tests/SkipPdfNormalizationTests.cs @@ -0,0 +1,25 @@ +// The Samples entry for SkipPdfNormalization only proves the setting round-trips through a +// verification. These assert the wiring it is actually there for: with the setting the snapshotted +// bytes are the producer's own, and without it they are the normalized ones. +[TestFixture] +public class SkipPdfNormalizationTests +{ + [Test] + public void SampleIsNotAlreadyNormalized() + { + // The premise the two tests below rest on. If sample.pdf were already byte-identical to its + // normalized form they would both pass while asserting nothing. + var raw = File.ReadAllBytes("sample.pdf"); + + Assert.That(PdfNormalizer.Normalize(raw), Is.Not.EqualTo(raw)); + } + + [Test] + public Task SkippedSnapshotHoldsTheProducerBytes() => + Verify(new MemoryStream(File.ReadAllBytes("sample.pdf")), "pdf") + .SkipPdfNormalization(); + + [Test] + public Task NormalizedSnapshotHoldsTheNeutralizedBytes() => + Verify(new MemoryStream(File.ReadAllBytes("sample.pdf")), "pdf"); +} diff --git a/src/Verify.PDFium/VerifyPDFium.cs b/src/Verify.PDFium/VerifyPDFium.cs index 42c9392..90fd4de 100644 --- a/src/Verify.PDFium/VerifyPDFium.cs +++ b/src/Verify.PDFium/VerifyPDFium.cs @@ -9,6 +9,10 @@ public static class VerifyPDFium // pdf can never be made byte-deterministic. The rendered pages and info file still verify. const string excludeDocumentKey = "VerifyPDFium.ExcludeDocument"; + // Context key set by SkipPdfNormalization. When present the pdf bytes are snapshotted as + // produced, for producers that already emit byte-deterministic documents. + const string skipNormalizationKey = "VerifyPDFium.SkipNormalization"; + public static bool Initialized { get; private set; } /// @@ -43,6 +47,30 @@ public static SettingsTask ExcludePdfDocument(this SettingsTask settings) return settings; } + /// + /// Snapshots the pdf bytes exactly as produced, skipping the normalization that neutralizes the + /// trailer /ID, the /CreationDate and /ModDate, and the XMP dates and + /// identifiers. Use it when the producer already emits byte-deterministic documents, since + /// normalizing them again copies the whole buffer, rescans it, and — when the XMP packet is + /// canonicalized — rebuilds it and repairs the cross-reference table, all to change nothing. + /// + /// + /// Only skip this when the producer is genuinely deterministic. Without it a freshly generated + /// pdf carries a wall-clock /CreationDate and a fresh /ID, so the snapshot differs + /// on every run. + /// + /// The XMP canonicalization is worth calling out because it is the pass that changes bytes for + /// an already-deterministic producer: it collapses the packet's whitespace, so enabling or + /// disabling this setting on an existing suite shifts the stored .verified.pdf even + /// though nothing about the document changed. Expect to re-accept those snapshots once. + /// + /// + public static SettingsTask SkipPdfNormalization(this SettingsTask settings) + { + settings.CurrentSettings.Context[skipNormalizationKey] = true; + return settings; + } + static ConversionResult Convert(Stream stream, IReadOnlyDictionary context) { using var buffer = new MemoryStream(); @@ -81,9 +109,13 @@ static ConversionResult Convert(Stream stream, IReadOnlyDictionary context) => !context.TryGetValue(excludeDocumentKey, out var value) || value is not true; + + static bool Normalize(IReadOnlyDictionary context) => + !context.TryGetValue(skipNormalizationKey, out var value) || + value is not true; }