diff --git a/openspec/README.md b/openspec/README.md new file mode 100644 index 0000000..1ca5aa3 --- /dev/null +++ b/openspec/README.md @@ -0,0 +1,64 @@ +# SAPP fork — OpenSpec + +This folder contains the implementation contracts for the +text-replacement feature being staged on the `work/text-replacement` +integration branch of `ConductionNL/sapp`. Each change corresponds to +one PR that we eventually intend to contribute back upstream to +`dealfonso/sapp` once the feature stabilises in our primary consumer +(OpenRegister's PDF anonymisation pipeline). + +## Why an OpenSpec scaffold inside a PHP library? + +SAPP is not a Nextcloud app, but the text-replacement work is being +done under time pressure for a Nextcloud-app consumer (OpenRegister) +that lives in the wider Conduction OpenSpec workflow. Putting the +specs next to the code that implements them keeps the contract close +to the source, makes upstream-PR submission easier later (each spec +becomes the PR's design doc), and gives us a place to record the +implementation notes worth surfacing in the upstream review. + +The `docs/upstream-prs/01-asciihex-decode/` ... `08-text-replacement-api/` +folders are the actual artefacts we'll paste into the upstream PRs. +These directories are created lazily by the individual implementation +PRs; this scaffold change ships only the `openspec/` tree. The +`openspec/changes/` folder is where the work is planned and tracked +**before** those upstream-PR-draft folders get the polished final +write-up. + +## Folder structure + +| File / Folder | Purpose | +|---|---| +| `config.yaml` | OpenSpec project config — rules tuned for a PHP library, not a Nextcloud app | +| `architecture/` | ADRs specific to the fork (e.g. fork-and-give-back ordering) | +| `specs/` | Accepted capability specs — created when a `change/` is archived | +| `changes/` | One folder per planned upstream PR — proposal, design, spec, tasks | + +## Mapping to the upstream-PR series + +Rows are listed in dependency order — `feat-filter-chain-dispatch` is +the foundation that PRs `#01`–`#04` attach to. The "Upstream-PR draft" +column references the `docs/upstream-prs/NN-/` folder index, +which intentionally differs from the dependency-order rows. To +disambiguate "upstream-PR draft" indices from ConductionNL/sapp PR +numbers throughout the rest of this scaffold, references are written +as `upstream-PR #NN` (or by change name). + +| OpenSpec change (dependency order) | Upstream-PR draft | PDF 1.7 ref | +|---|---|---| +| `feat-filter-chain-dispatch` (foundation) | `docs/upstream-prs/05-filter-chaining/` | §7.4 (Filters, array form) | +| `feat-asciihex-decode` | `docs/upstream-prs/01-asciihex-decode/` | §7.4.2 | +| `feat-runlength-decode` | `docs/upstream-prs/02-runlength-decode/` | §7.4.5 | +| `feat-ascii85-decode` | `docs/upstream-prs/03-ascii85-decode/` | §7.4.3 | +| `feat-lzw-decode` | `docs/upstream-prs/04-lzw-decode/` | §7.4.4 | +| `feat-tounicode-cmap` | `docs/upstream-prs/06-tounicode-cmap/` | §9.10 (ToUnicode CMaps) | +| `feat-tj-flattening` | `docs/upstream-prs/07-tj-flattening/` | §9.4 (text-showing operators) | +| `feat-text-replacement-api` | `docs/upstream-prs/08-text-replacement-api/` | n/a (new public API) | + +## Workflow + +- `/opsx-ff` — Create + scaffold a new change end-to-end +- `/opsx-continue` — Resume work on an in-progress change +- `/opsx-apply` — Drive the implementation from `tasks.md` +- `/opsx-verify` — Verify implementation matches the spec +- `/opsx-archive` — Move a completed change into `specs/` diff --git a/openspec/architecture/.gitkeep b/openspec/architecture/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/openspec/changes/.gitkeep b/openspec/changes/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/openspec/changes/feat-ascii85-decode/.openspec.yaml b/openspec/changes/feat-ascii85-decode/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-ascii85-decode/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-ascii85-decode/design.md b/openspec/changes/feat-ascii85-decode/design.md new file mode 100644 index 0000000..8768c27 --- /dev/null +++ b/openspec/changes/feat-ascii85-decode/design.md @@ -0,0 +1,60 @@ +## Context + +ASCII85 base-85 encoding maps 4 binary bytes (interpreted as a 32-bit unsigned big-endian integer `n`) to 5 ASCII characters via `n = c0 * 85^4 + c1 * 85^3 + c2 * 85^2 + c3 * 85 + c4` where each `ci` is in `[0, 84]` and emitted as `ci + 33` (codepoint range `!..u`). + +Special cases: + +- Single character `z` decodes to 4 zero bytes (compact form for runs of zeros — common in image data). +- EOD marker `~>` terminates decoding. +- Whitespace anywhere is ignored. +- Trailing partial groups: a group of `k` ASCII chars (where `2 ≤ k ≤ 4`) decodes to `k - 1` binary bytes, padded with `u` (codepoint 117 = value 84) to a full 5-char group before computing. + +## Goals / Non-Goals + +**Goals:** + +- Lossless round-trip on arbitrary binary input. +- Honour the `z` shortcut on both encode and decode. +- Tolerate whitespace anywhere in the encoded stream. + +**Non-Goals:** + +- The alternative "btoa version 4.2" syntax with `<~` start marker — PDF 1.7 §7.4.3 doesn't use it and Adobe Acrobat doesn't emit it. +- ASCII85 in other contexts (PostScript level-2 has a slightly different EOD convention). + +## Decisions + +### D1 — Mirror upstream's filter helper shape + +`protected static function ASCII85Decode($_stream, $params)` and `protected static function ASCII85Encode($_stream, $params)` as static helpers on `PDFObject`. + +### D2 — Use PHP's native arithmetic, not `gmp` or `bcmath` + +The 32-bit big-endian integer never exceeds `2^32 - 1`. PHP's `int` type is at least 32-bit (64-bit on 64-bit platforms, which sapp's `composer.json` requires via the PHP 7.4 floor). No external math extension needed. + +### D3 — Encode emits `z` for runs of 4 zero bytes (greedy, aligned) + +The encoder emits `z` only when 4 consecutive zero bytes are aligned on a group boundary. Unaligned zero runs go through the standard 5-char encoding. This matches Adobe's de-facto encoding behaviour and keeps the implementation simple. + +### D4 — Decode rejects illegal characters via `p_error` + raw input return + +Characters outside `!..u` plus `z` plus whitespace plus the EOD bytes `~>` cause `p_error()` and a return of the raw input. Same convention as the other filters. + +### D5 — Decode tolerates missing EOD (Adobe-compatible) + +If the stream ends without a `~>` marker, the decoder finishes the last partial group (treating trailing `u`s as padding) and returns the result. Adobe Reader does this. Spec is mildly ambiguous on whether EOD is mandatory; the lenient interpretation is reader-compatible. + +## Risks / Trade-offs + +- **Risk**: A 5-char group might overflow `2^32 - 1` if all 5 chars are at their maximum value (`uuuuu` = `85^4 * 84 + ... + 84 = 4294967295`). → **Mitigation**: that's exactly `2^32 - 1`, the maximum representable value; PHP's int handles it. Groups beyond this value (`uuuu\x76` would compute to > `2^32`) are spec-illegal — the decoder fails them via the `p_error` path. + +- **Trade-off**: Greedy z-encoding leaves a few bytes on the floor when zero runs straddle group boundaries. → **Mitigation**: documented; in practice this matters only for image data which we don't anonymise. + +## Open Questions + +- **OQ1**: Some readers tolerate the `<~` start marker (btoa convention). PDF 1.7 §7.4.3 doesn't mention it. Reject or tolerate? **Provisional**: tolerate on decode (strip the leading `<~` if present); never emit on encode. Maximum reader-compatibility on input; spec-faithful on output. + +## Migration Plan + +Strictly additive change. No migration required for consumers — string-form callers (and unaware-of-this-change callers) observe byte-for-byte identical behaviour after merge. Rollback path is a clean revert of the commit. + diff --git a/openspec/changes/feat-ascii85-decode/proposal.md b/openspec/changes/feat-ascii85-decode/proposal.md new file mode 100644 index 0000000..c5c6691 --- /dev/null +++ b/openspec/changes/feat-ascii85-decode/proposal.md @@ -0,0 +1,29 @@ +## Why + +`ASCII85Decode` (PDF 1.7 §7.4.3) encodes 4 binary bytes per 5 printable ASCII characters in the range `!..u`, with a single-character shortcut `z` for four zero bytes. It's used as an outer envelope in PDFs that need to pass through 7-bit-clean transports while still being more compact than ASCIIHex. Real-world appearances include certain Adobe Distiller outputs and PDFs produced by older Mac toolchains. Upstream sapp doesn't implement it; without it, any chain-encoded content stream that uses an ASCII85 outer wrapper falls through to the no-op path and the text-replacement pipeline produces corrupted output on those PDFs. + +## What Changes + +- Add `ASCII85Decode` encode + decode primitives in `PDFObject`. +- Wire `/ASCII85Decode` into the chain dispatcher introduced by `feat-filter-chain-dispatch`. +- Spec-faithful per §7.4.3: + - Decode: 5-char groups in `!..u` (codepoints 33..117) decode to 4 bytes via base-85. The shortcut `z` decodes to four zero bytes. EOD marker is `~>`. Trailing groups of fewer than 5 chars decode to fewer than 4 bytes (1 → impossible; 2 → 1 byte; 3 → 2 bytes; 4 → 3 bytes), padded with `u` (codepoint 117) for the missing chars. + - Whitespace anywhere is ignored. +- No `/DecodeParms` (Table 5). + +## Capabilities + +### New Capabilities + +- `ascii85-decode-filter`: PDF 1.7 §7.4.3 ASCII85Decode encode + decode plugged into the filter chain dispatcher. + +### Modified Capabilities + +- `filter-chain-dispatch`: extend the `case` table with `/ASCII85Decode`. + +## Impact + +- **Touched files**: `src/PDFObject.php` (+ASCII85 helpers, +dispatch arms; ~80 LOC). +- **Public API**: none. +- **Depends on**: `feat-filter-chain-dispatch`. +- **Upstream-PR draft**: `docs/upstream-prs/03-ascii85-decode/`. diff --git a/openspec/changes/feat-ascii85-decode/specs/ascii85-decode-filter/spec.md b/openspec/changes/feat-ascii85-decode/specs/ascii85-decode-filter/spec.md new file mode 100644 index 0000000..d0b4438 --- /dev/null +++ b/openspec/changes/feat-ascii85-decode/specs/ascii85-decode-filter/spec.md @@ -0,0 +1,127 @@ +**Status**: planned +**Scope**: change `feat-ascii85-decode` (delta spec) +**OpenSpec changes**: +- `feat-ascii85-decode` (in-progress) + +## Purpose + +Capability contract for `ascii85-decode-filter` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/ascii85-decode-filter/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: ASCII85Decode SHALL decode per PDF 1.7 §7.4.3 + +The decoder MUST interpret 5-character groups in the range `!..u` (codepoints 33..117) as base-85 integers, emit 4 bytes per group in big-endian order, recognise the single-character shortcut `z` as 4 zero bytes, terminate at the `~>` EOD marker, and ignore whitespace anywhere. + +#### Scenario: Standard 5-char group decode + +- GIVEN the input is `87cURD]i,"Ebo80~>` (encoding `"Hello world!"`) +- WHEN `ASCII85Decode` is invoked +- THEN the decoder MUST return `Hello world!` + +#### Scenario: `z` shortcut + +- GIVEN the input is `z~>` +- WHEN `ASCII85Decode` is invoked +- THEN the decoder MUST return `\x00\x00\x00\x00` (4 zero bytes) + +#### Scenario: Whitespace ignored + +- GIVEN the input is `87cURD\n]i,"E\tbo80~>` +- WHEN `ASCII85Decode` is invoked +- THEN the decoder MUST return `Hello world!` (whitespace stripped) + +#### Scenario: Trailing partial group + +- GIVEN the input is `87cURDZ~>` (5-char group `87cUR` decoding to 4 bytes `0x48 0x65 0x6C 0x6C` = `"Hell"`, followed by 2-char partial group `DZ` decoding to the single byte `0x6F` = `"o"` per §7.4.3 padding rules) +- WHEN `ASCII85Decode` is invoked +- THEN the decoder MUST return the 5-byte string `"Hello"` (concatenation `0x48 0x65 0x6C 0x6C 0x6F`) + +### REQ-002: ASCII85Encode SHALL produce round-trip-compatible output + +The encoder MUST emit 5 ASCII chars per 4 input bytes, use the `z` shortcut for aligned 4-zero-byte groups, terminate with `~>`, and handle trailing partial groups by padding with `u` and emitting only the relevant prefix. + +#### Scenario: Basic encode + +- GIVEN the input is the string `Hello world!` +- WHEN `ASCII85Encode` is invoked +- THEN the encoder MUST return a stream ending in `~>` that round-trips to the input + +#### Scenario: Aligned zero run uses `z` + +- GIVEN the input is `\x00\x00\x00\x00` (4 zero bytes) +- WHEN `ASCII85Encode` is invoked +- THEN the encoder MUST return `z~>` + +#### Scenario: Empty input + +- GIVEN the input is the empty string +- WHEN `ASCII85Encode` is invoked +- THEN the encoder MUST return `~>` (EOD only) + +### REQ-003: ASCII85Decode round-trip MUST be lossless + +For any input byte string `$P`, `ASCII85Decode(ASCII85Encode($P, null), null)` MUST equal `$P` byte-for-byte. + +#### Scenario: Binary round-trip + +- GIVEN the input is `random_bytes(1024)` +- WHEN `ASCII85Decode` is invoked +- THEN the round-trip MUST equal the input byte-for-byte + +### REQ-004: ASCII85Decode SHALL fail safely on illegal characters + +Characters outside `!..u` plus `z` plus whitespace plus the EOD bytes `~>` MUST cause the decoder to call `p_error()` and return the raw input unchanged. + +#### Scenario: Illegal character + +- GIVEN the input is `87c{RD~>` (contains `{`, codepoint 123, outside the range) +- WHEN `ASCII85Decode` is invoked +- THEN the decoder MUST call `p_error()` and return the original input unchanged + +## MODIFIED Requirements + +### REQ-001: Array-form `/Filter` SHALL decode in forward chain order + +The dispatcher MUST recognise `/ASCII85Decode` and route to the ASCII85Decode helper. + +#### Scenario: ASCII85 outer + Flate inner + +- WHEN an object has `/Filter [/ASCII85Decode /FlateDecode]` and `_stream` is `ASCII85Encode(gzcompress("BT...ET"))` +- THEN `get_stream(false)` MUST return `"BT...ET"` byte-for-byte + +### REQ-002: Array-form `/Filter` SHALL encode in reverse chain order + +The dispatcher MUST recognise `/ASCII85Decode` on the encode path and route to the ASCII85Encode helper. + +#### Scenario: Two-filter encode round-trip with ASCII85 + +- WHEN `set_stream($P, false)` is called on `/Filter [/ASCII85Decode /FlateDecode]`, then `get_stream(false)` is called +- THEN the value returned by `get_stream(false)` MUST equal `$P` diff --git a/openspec/changes/feat-ascii85-decode/tasks.md b/openspec/changes/feat-ascii85-decode/tasks.md new file mode 100644 index 0000000..35fe225 --- /dev/null +++ b/openspec/changes/feat-ascii85-decode/tasks.md @@ -0,0 +1,50 @@ +## 1. Pre-flight + +- [ ] 1.1 Confirm `feat-filter-chain-dispatch` is merged into `work/text-replacement` +- [ ] 1.2 Branch off as `feat/ascii85-decode` +- [ ] 1.3 Re-read PDF 1.7 §7.4.3 — confirm `z` shortcut + EOD + partial-group padding semantics + +## 2. Decode helper + +- [ ] 2.1 Add `protected static function ASCII85Decode($_stream, $params)` in `src/PDFObject.php` +- [ ] 2.2 Strip whitespace, optionally strip leading `<~` (OQ1 — Adobe-tolerant) +- [ ] 2.3 Process `z` shortcut as 4 zero bytes +- [ ] 2.4 Process 5-char groups as base-85 integers, big-endian to 4 bytes +- [ ] 2.5 Handle trailing partial group with `u` padding +- [ ] 2.6 Fail-safe on illegal characters via `p_error()` + +## 3. Encode helper + +- [ ] 3.1 Add `protected static function ASCII85Encode($_stream, $params)` +- [ ] 3.2 Walk input 4 bytes at a time; emit `z` for aligned 4-zero-byte groups (D3) +- [ ] 3.3 Emit standard 5-char groups otherwise +- [ ] 3.4 Handle trailing partial group (pad with `\x00` bytes, emit only `k+1` chars where `k` is the partial length) +- [ ] 3.5 Emit trailing `~>` EOD marker + +## 4. Chain-dispatch integration + +- [ ] 4.1 Add `case '/ASCII85Decode'` arms in both `apply_filter_chain_decode` and `apply_filter_chain_encode` + +## 5. Tests / verification + +- [ ] 5.1 Round-trip fixture `examples/poc-filter-roundtrip-ascii85.php`: 1024-byte buffer including 4-zero-byte runs (exercises `z` shortcut) +- [ ] 5.2 Verify `examples/poc-replace-text.php` still exits 0 +- [ ] 5.3 Negative test: illegal character → `p_error` + unchanged +- [ ] 5.4 Chain test `[/ASCII85Decode /FlateDecode]` round-trip +- [ ] 5.5 Adobe-tolerance test: input with leading `<~` decodes correctly + +## 6. Upstream-PR draft + +- [ ] 6.1 Update `docs/upstream-prs/03-ascii85-decode/{proposal,design,tasks}.md` +- [ ] 6.2 Leave `Posted at: ` placeholder + +## 7. Quality gate + +- [ ] 7.1 PHP 7.4 compatibility; verify int is 32-bit-or-wider on target platform +- [ ] 7.2 No new composer dependencies +- [ ] 7.3 snake_case discipline + +## 8. Commit + PR + +- [ ] 8.1 Commit on `feat/ascii85-decode` +- [ ] 8.2 Open PR `feat/ascii85-decode → work/text-replacement` diff --git a/openspec/changes/feat-asciihex-decode/.openspec.yaml b/openspec/changes/feat-asciihex-decode/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-asciihex-decode/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-asciihex-decode/design.md b/openspec/changes/feat-asciihex-decode/design.md new file mode 100644 index 0000000..2b88364 --- /dev/null +++ b/openspec/changes/feat-asciihex-decode/design.md @@ -0,0 +1,67 @@ +## Context + +ASCIIHexDecode is the simplest filter in PDF 1.7 §7.4. Decode rules: + +- Input alphabet: `0..9 A..F a..f` (case-insensitive). +- Whitespace (`\x00 \t \n \f \r \x20`) is permitted anywhere and ignored. +- Termination: `>` (the EOD marker). Any input after `>` is ignored. +- Odd length: if EOD is reached after an odd number of hex digits, the missing trailing nibble is treated as `0`. +- Illegal characters: raise an error. + +Encode rules: + +- Output uppercase hex pairs, one pair per byte. +- Terminate with `>`. +- Line-wrap at 80 columns (not normative, but the de-facto convention in Adobe Distiller output; readers must tolerate any width). + +Upstream sapp's `FlateDecode` static helper is the structural reference — same calling convention, same `p_error` failure mode. + +## Goals / Non-Goals + +**Goals:** + +- Round-trip lossless: `decode(encode($x))` MUST equal `$x` for arbitrary binary input. +- Permissive decode: tolerate the full set of whitespace placements that real-world ASCIIHex streams use (Adobe Distiller emits 80-col line wraps; some sanitisers use single-line; some use 64-col). +- Spec-faithful encode: PDF 1.7 §7.4.2 doesn't mandate uppercase but explicitly allows it; we pick uppercase for determinism / diff-stability of test fixtures. + +**Non-Goals:** + +- ASCII85 (separate change `feat-ascii85-decode`). +- `/DecodeParms` handling — ASCIIHexDecode has no parameters per PDF 1.7 Table 5. +- Encryption (`/Crypt`). + +## Decisions + +### D1 — Static helpers symmetric to `FlateDecode` + +Mirror upstream's existing `protected static function FlateDecode(...)` shape: add `protected static function ASCIIHexDecode($_stream, $params)` and `protected static function ASCIIHexEncode($_stream, $params)` inside `PDFObject`. Keeps the dispatch table's call sites uniform. + +### D2 — Decode tolerates malformed input via `p_error` + return raw input + +When the decoder encounters an illegal character (anything outside `0..9A..Fa..f \s >`), it calls `p_error()` and returns the input bytes unchanged. Consistent with the rest of sapp's filter machinery. Strictness is the caller's responsibility. + +### D3 — Encode line-wraps at 80 columns + +Hard-coded `80` constant in the encoder. Spec-permissive on either side — readers MUST tolerate any width — but 80 matches the most common in-the-wild emission. No knob exposed; if a future caller needs a different width, it's a one-line patch. + +### D4 — Odd-length input pads with `0` + +PDF 1.7 §7.4.2 ¶3 is explicit: "If the filter encounters the EOD marker after reading an odd number of hexadecimal digits, it shall behave as if a 0 (zero) followed the last digit." Implementation: track parity in a local variable, finish the partial byte on EOD. + +### D5 — `>` past column boundaries is recognised + +The EOD marker `>` MUST be recognised regardless of line-wrap state. The state machine treats whitespace as no-op and only acts on hex digits or `>`. + +## Risks / Trade-offs + +- **Risk**: A PDF in the wild uses an unusual whitespace character (vertical tab, BOM). → **Mitigation**: PDF 1.7 §7.5.1 enumerates the whitespace set; we match that set exactly. Other characters are illegal per spec. + +- **Trade-off**: Hard-coding 80-column wrap means a forensic round-trip against a 64-column-wrapped source produces a byte-different (but semantically identical) re-encoded stream. → **Mitigation**: spec confirms readers MUST tolerate any width; document the choice in PHPDoc. + +## Migration Plan + +Strictly additive. No migration needed. Rollback = revert the commit (no public API surface to clean up). + +## Open Questions + +- **OQ1**: Lowercase hex on encode for compactness in the (rare) case that a downstream consumer is case-sensitive? **Provisional**: uppercase. Spec-permissive; reader behaviour identical; uppercase is more common in fixtures. diff --git a/openspec/changes/feat-asciihex-decode/proposal.md b/openspec/changes/feat-asciihex-decode/proposal.md new file mode 100644 index 0000000..498743c --- /dev/null +++ b/openspec/changes/feat-asciihex-decode/proposal.md @@ -0,0 +1,31 @@ +## Why + +`ASCIIHexDecode` is the PDF 1.7 §7.4.2 filter that wraps binary stream bytes in `0..9A..Fa..f` characters terminated by `>`. It's used by sanitisers, older toolchains, and a small but real fraction of in-the-wild PDFs as the outer envelope of a filter chain (e.g. `[/ASCIIHexDecode /FlateDecode]`). Upstream sapp does not implement this filter — the dispatcher in `PDFObject` rejects it via the `p_error('unknown compression method ...')` path. Without it, any chain-encoded content stream that uses an ASCIIHex outer wrapper falls through to the no-op path and the text-replacement pipeline produces corrupted output on those PDFs. + +## What Changes + +- Add `ASCIIHexDecode` encode + decode primitives in `PDFObject`, mirroring the shape of the existing `FlateDecode` static helper. +- Wire the filter name `/ASCIIHexDecode` into the chain dispatcher introduced by `feat-filter-chain-dispatch` (upstream-PR #05). +- Spec-faithful behaviour per §7.4.2: + - Decode: accept `0..9A..Fa..f`, ignore whitespace + EOL anywhere, terminate at `>` (EOD marker), tolerate odd-length input by treating the trailing hex digit as if followed by `0`. + - Encode: emit uppercase hex pairs with a trailing `>` EOD marker; line-wrap at 80 columns (Adobe's de-facto convention; not normative but matches reader expectations). +- No `/DecodeParms` support — ASCIIHexDecode has no parameters (Table 5). + +## Capabilities + +### New Capabilities + +- `asciihex-decode-filter`: PDF 1.7 §7.4.2 ASCIIHexDecode encode + decode, plugged into the filter chain dispatcher. + +### Modified Capabilities + +- `filter-chain-dispatch`: extend the dispatcher's `case` table to recognise `/ASCIIHexDecode` and route encode/decode calls to the new helper. + +## Impact + +- **Touched files**: `src/PDFObject.php` (+ASCIIHexDecode static helpers, +chain-dispatch `case` arm; ~60 LOC). +- **Public API**: none. Pure dispatch-table extension. +- **Depends on**: `feat-filter-chain-dispatch` (upstream-PR #05). +- **Unblocks**: real-world PDFs whose content streams use `[/ASCIIHexDecode /FlateDecode]`. +- **Upstream-PR draft**: `docs/upstream-prs/01-asciihex-decode/`. +- **No new composer dependencies.** PHP ≥ 7.4. snake_case method names preserved. diff --git a/openspec/changes/feat-asciihex-decode/specs/asciihex-decode-filter/spec.md b/openspec/changes/feat-asciihex-decode/specs/asciihex-decode-filter/spec.md new file mode 100644 index 0000000..d26afc8 --- /dev/null +++ b/openspec/changes/feat-asciihex-decode/specs/asciihex-decode-filter/spec.md @@ -0,0 +1,155 @@ +**Status**: planned +**Scope**: change `feat-asciihex-decode` (delta spec) +**OpenSpec changes**: +- `feat-asciihex-decode` (in-progress) + +## Purpose + +Capability contract for `asciihex-decode-filter` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/asciihex-decode-filter/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: ASCIIHexDecode SHALL decode per PDF 1.7 §7.4.2 + +The decoder MUST accept input over the alphabet `0..9 A..F a..f` plus whitespace plus the EOD marker `>`. It MUST ignore whitespace, terminate at `>`, treat an odd trailing digit as if followed by `0`, and emit raw binary bytes corresponding to the hex pairs. + +#### Scenario: Even-length decode + +- GIVEN the encoded stream is `48656C6C6F>` (encoding `"Hello"`) +- WHEN `ASCIIHexDecode` is invoked +- THEN the decoder MUST return the 5-byte string `Hello` + +#### Scenario: Odd-length decode pads with zero + +- GIVEN the encoded stream is `41B>` (3 hex digits, then EOD) +- WHEN `ASCIIHexDecode` is invoked +- THEN the decoder MUST return the 2-byte string `\x41\xB0` (the trailing `B` paired with implicit `0`) + +#### Scenario: Whitespace is ignored + +- GIVEN the encoded stream is `48 65\n6C\t6C\r\n6F>` +- WHEN `ASCIIHexDecode` is invoked +- THEN the decoder MUST return `Hello` byte-for-byte + +#### Scenario: Lowercase hex digits are accepted + +- GIVEN the encoded stream is `48656c6c6f>` +- WHEN `ASCIIHexDecode` is invoked +- THEN the decoder MUST return `Hello` (case-insensitive) + +#### Scenario: Trailing bytes after EOD are ignored + +- GIVEN the encoded stream is `48656C6C6F>garbage` +- WHEN `ASCIIHexDecode` is invoked +- THEN the decoder MUST return `Hello` and MUST NOT raise an error + +### REQ-002: ASCIIHexEncode SHALL emit uppercase hex pairs with EOD + +The encoder MUST emit one uppercase hex pair per input byte, terminated by `>`, with `\n` line wraps inserted at 80 columns (deterministic; readers tolerate any width per §7.4.2). + +#### Scenario: Basic encode + +- GIVEN the input is the 5-byte string `Hello` +- WHEN `ASCIIHexEncode` is invoked +- THEN the encoder MUST return `48656C6C6F>` + +#### Scenario: Line wrap at 80 columns + +- GIVEN the input is 50 bytes (which encode to 100 hex characters) +- WHEN `ASCIIHexEncode` is invoked +- THEN the encoder output MUST contain at least one `\n` and no line MUST exceed 80 characters + +#### Scenario: Empty input + +- GIVEN the input is the empty string +- WHEN `ASCIIHexEncode` is invoked +- THEN the encoder MUST return just `>` (the EOD marker) + +### REQ-003: ASCIIHexDecode round-trip MUST be lossless + +For any input byte string `$P`, `ASCIIHexDecode(ASCIIHexEncode($P, null), null)` MUST equal `$P` byte-for-byte. + +#### Scenario: Binary round-trip + +- GIVEN the input is a 1024-byte stream of pseudo-random bytes (PHP `random_bytes(1024)`) +- WHEN `ASCIIHexDecode` is invoked +- THEN the round-trip MUST equal the input byte-for-byte + +### REQ-004: ASCIIHexDecode SHALL fail safely on illegal characters + +When the decoder encounters a character outside the alphabet plus whitespace plus EOD, it MUST call `p_error()` with a message identifying the failure and MUST return the input bytes unchanged. + +#### Scenario: Illegal character fails safely + +- GIVEN the encoded stream is `48656C!6C6F>` (contains `!`) +- WHEN `ASCIIHexDecode` is invoked +- THEN the decoder MUST call `p_error()` with a message naming the illegal character +- AND the return value MUST equal the original encoded stream (unchanged) +- AND no exception MUST be thrown + +## MODIFIED Requirements + +### REQ-001: Array-form `/Filter` SHALL decode in forward chain order + +When `PDFObject::get_stream($raw = false)` is called on an object whose `/Filter` is an array, the dispatcher MUST apply the named filter decoders in FORWARD order (outermost first, innermost last), per PDF 1.7 §7.4.1 ¶3. The dispatcher MUST recognise `/ASCIIHexDecode` as a valid filter name and route to the ASCIIHexDecode helper. + +#### Scenario: Two-filter chain decode with ASCIIHex outer + Flate inner + +- WHEN an object has `/Filter [/ASCIIHexDecode /FlateDecode]` and `_stream` equal to `ASCIIHexEncode(gzcompress("BT...ET"))` +- THEN `get_stream(false)` MUST return `"BT...ET"` byte-for-byte + +#### Scenario: Single-element array form is semantically identical to string form + +- WHEN an object has `/Filter [/FlateDecode]` (1-element array) +- THEN `get_stream(false)` MUST return the same plaintext as it would for `/Filter /FlateDecode` (string form) on identical stream bytes + +#### Scenario: Empty array `/Filter` MUST be treated as no filtering + +- WHEN an object has `/Filter []` (empty array) +- THEN `get_stream(false)` MUST return `$this->_stream` unchanged + +#### Scenario: ASCIIHex-only single-filter chain + +- WHEN an object has `/Filter /ASCIIHexDecode` (string form) and `_stream` equal to `ASCIIHexEncode("plaintext")` +- THEN `get_stream(false)` MUST return `"plaintext"` byte-for-byte + +### REQ-002: Array-form `/Filter` SHALL encode in reverse chain order + +When `PDFObject::set_stream($plaintext, $raw = false)` is called on an object whose `/Filter` is an array, the dispatcher MUST apply the named filter encoders in REVERSE order (innermost first, outermost last), per PDF 1.7 §7.4.1 ¶3, then update `/Length` to the final encoded byte count. The dispatcher MUST recognise `/ASCIIHexDecode` as a valid filter name on the encode path and route to the ASCIIHexEncode helper. + +#### Scenario: Two-filter chain encode with ASCIIHex outer + +- WHEN `set_stream("plaintext", false)` is called on an object with `/Filter [/ASCIIHexDecode /FlateDecode]` +- THEN the resulting `_stream` MUST equal `ASCIIHexEncode(gzcompress("plaintext"))` and `_value['Length']` MUST equal `strlen($_stream)` after encoding + +#### Scenario: Encode-decode round-trip is lossless + +- WHEN `set_stream($P, false)` is called on an array-form-`/Filter` object that includes `/ASCIIHexDecode`, then `get_stream(false)` is called +- THEN the value returned by `get_stream(false)` MUST equal `$P` byte-for-byte diff --git a/openspec/changes/feat-asciihex-decode/tasks.md b/openspec/changes/feat-asciihex-decode/tasks.md new file mode 100644 index 0000000..70582f7 --- /dev/null +++ b/openspec/changes/feat-asciihex-decode/tasks.md @@ -0,0 +1,45 @@ +## 1. Pre-flight + +- [ ] 1.1 Confirm `feat-filter-chain-dispatch` (upstream-PR #05) is merged into `work/text-replacement` before starting — it's a hard dependency +- [ ] 1.2 Branch off `work/text-replacement` as `feat/asciihex-decode` +- [ ] 1.3 Re-read PDF 1.7 §7.4.2 and confirm decoder rules (whitespace + odd-length + EOD + alphabet) + +## 2. Decode helper + +- [ ] 2.1 Add `protected static function ASCIIHexDecode($_stream, $params)` in `src/PDFObject.php`, modelled on the existing `FlateDecode` static helper +- [ ] 2.2 Implement the decode state machine: accept hex digits + whitespace, terminate at `>`, pad odd trailing nibble with `0`, fail-safe on illegal characters via `p_error()` + return raw input + +## 3. Encode helper + +- [ ] 3.1 Add `protected static function ASCIIHexEncode($_stream, $params)` — emit uppercase hex pairs, line-wrap at 80 columns, terminate with `>` +- [ ] 3.2 Handle the empty-input edge case (emit just `>`) + +## 4. Chain-dispatch integration + +- [ ] 4.1 Add a `case '/ASCIIHexDecode'` arm to `apply_filter_chain_decode` (introduced in upstream-PR #05) routing to `self::ASCIIHexDecode(...)` +- [ ] 4.2 Add the symmetric arm in `apply_filter_chain_encode` routing to `self::ASCIIHexEncode(...)` + +## 5. Tests / verification + +- [ ] 5.1 Add a round-trip fixture in `examples/poc-filter-roundtrip-asciihex.php`: random 1024-byte buffer → encode → decode → assert byte-for-byte equality +- [ ] 5.2 Verify the existing `examples/poc-replace-text.php` still exits 0 (FlateDecode-only path unchanged) +- [ ] 5.3 Add a chain test exercising `[/ASCIIHexDecode /FlateDecode]` end-to-end (encode plaintext → assert decoded matches) +- [ ] 5.4 Add a negative test asserting `p_error` on an illegal-character input and stream-unchanged behaviour + +## 6. Upstream-PR draft + +- [ ] 6.1 Update `docs/upstream-prs/01-asciihex-decode/proposal.md` with the spec REQ references +- [ ] 6.2 Update `docs/upstream-prs/01-asciihex-decode/design.md` with D1–D5 decision log +- [ ] 6.3 Update `docs/upstream-prs/01-asciihex-decode/tasks.md` with the implementation summary +- [ ] 6.4 Leave `Posted at: ` placeholder + +## 7. Quality gate + +- [ ] 7.1 PHP 7.4 compatibility check (no typed properties, no enums, no readonly) +- [ ] 7.2 No new composer dependencies +- [ ] 7.3 snake_case method discipline (`ASCIIHexDecode` / `ASCIIHexEncode` follow upstream's PascalCase-on-filter-names + snake_case-on-utility-method convention; verify against `FlateDecode`) + +## 8. Commit + PR + +- [ ] 8.1 Commit on `feat/asciihex-decode` +- [ ] 8.2 Open PR `feat/asciihex-decode → work/text-replacement` diff --git a/openspec/changes/feat-filter-chain-dispatch/.openspec.yaml b/openspec/changes/feat-filter-chain-dispatch/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-filter-chain-dispatch/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-filter-chain-dispatch/design.md b/openspec/changes/feat-filter-chain-dispatch/design.md new file mode 100644 index 0000000..87879dc --- /dev/null +++ b/openspec/changes/feat-filter-chain-dispatch/design.md @@ -0,0 +1,101 @@ +## Context + +`PDFObject::set_stream` and `PDFObject::get_stream` are the two methods every other piece of stream-handling code in sapp routes through. Today they branch on `$this->_value['Filter']` with a single `switch` statement that has one arm — `'/FlateDecode'` — and a `p_error("unknown compression method ...")` default. The branch assumes the `/Filter` entry is a string. PDF 1.7 §7.4.1 defines the entry as **either** a name **or** an array of names, with the array form representing a chain of filters to apply in sequence. + +Real-world content streams that arrive with array-form filters: + +- Forensic and sanitiser pipelines that emit `[/ASCIIHexDecode /FlateDecode]` so the compressed payload survives transports that mangle binary data. +- Older toolchains (PDFKit 1.x, some report generators) that wrap a Flate stream in ASCII85 for the same reason. +- xref streams encoded as `[/FlateDecode]` (single-entry array form — semantically equivalent to the string form but lexically different). + +Constraints from upstream `dealfonso/sapp`: + +- PHP ≥ 7.4 minimum (composer constraint `^7.4 || ^8.0`). Do not raise it. +- Zero external composer dependencies. The library is intentionally lean. +- snake_case method names (`get_stream`, `set_stream`, `to_pdf_entry`, etc.). No camelCase. +- `PDFObject` / `PDFDoc` / `PDFValue*` class boundaries are load-bearing. Refactors across them are rejected on principle in upstream review. +- Each change in our fork should map 1:1 to an eventual upstream PR. This change is the foundation that PRs `#01`–`#04` (per-filter implementations) attach to. + +The PoC verify (`examples/poc-replace-text.php`) currently passes on a single-FlateDecode fixture; the dispatch refactor must keep it green. + +## Goals / Non-Goals + +**Goals:** + +- Accept both the string and array forms of `/Filter` on encode (`set_stream($_, false)`) and decode (`get_stream(false)`) without changing method signatures. +- Apply chain ordering normatively per PDF 1.7 §7.4.1 ¶3: encode reverses the chain (innermost-first), decode follows the chain (outermost-first). +- Handle the parallel `/DecodeParms` array shape — one entry per filter, `null` permitted for filters that take no parameters (Table 5 of §7.4.1). +- Fail safely on unknown filter names in the chain: emit `p_error()` and leave the stream unchanged so the caller can detect the failure. Do not throw. +- Keep all string-form callers passing through the existing code paths exactly (no behavioural drift on the 95% case). +- Provide the plug-in surface (`case` arms in the dispatcher) that PRs `#01`–`#04` attach to without further structural changes. + +**Non-Goals:** + +- Implementing new filters (`ASCIIHexDecode`, `ASCII85Decode`, `RunLengthDecode`, `LZWDecode`) — those are separate upstream PRs `#01`–`#04`. +- Changing the public `PDFObject` API surface beyond the internal dispatch. +- Refactoring `PDFDoc::get_object_iterator()` / `to_pdf_file_b()` — they don't see filter chains directly. +- Compression-level / quality knobs on FlateDecode. Out of scope. +- Stream encryption (`/Crypt` filter) — separate concern, not part of the contribution series. +- xref-stream-specific compression handling — separate concern. + +## Decisions + +### D1 — Inline private dispatch helpers, not new public methods + +We add two `private` helpers — `apply_filter_chain_decode(string $bytes, array $filters, array $decodeParms): string|false` and `apply_filter_chain_encode(string $bytes, array $filters, array $decodeParms): string|false` — both inside `PDFObject`. The public `get_stream` / `set_stream` shape stays identical; they normalise the `/Filter` value (string → 1-element array) and the parallel `/DecodeParms` (single value → 1-element array), then delegate. + +**Alternative considered:** add a public `apply_filter()` method to `PDFObject`. **Rejected** because it would expose internal-only mechanics, widen the API surface, and create a separate code path that callers might bypass to inject malformed input. The two existing public methods already encapsulate the right level of abstraction. + +### D2 — Normalise string → 1-element array on the way in, keep the original shape on the way out + +Inside `set_stream` / `get_stream`, immediately coerce the `/Filter` entry to an array regardless of source shape. The chain helpers always operate on arrays. When `set_stream` updates the object's `Filter` value after a chain encode, it preserves the original shape: string in → string out, array in → array out. This avoids gratuitous diff churn when round-tripping a single-filter stream (matters for `to_pdf_file_b`'s rebuild path — a single-filter stream that comes in as a string should not flip to an array on write-back). + +**Alternative considered:** always normalise to array form on write-back. **Rejected** because it produces unnecessary diffs against upstream-format-compatible PDFs and complicates byte-for-byte comparison testing. + +### D3 — `/DecodeParms` shape parity, with `null` as the per-filter no-params marker + +PDF 1.7 §7.4.1 Table 5 says `DecodeParms` "shall be either a dictionary or an array of dictionaries", parallel to `Filter`. When a filter in the chain has no parameters, the corresponding slot in the array is the PDF `null` value. We handle this by indexing the parameters array by chain position and treating `null` / missing as "no params" (existing FlateDecode path already tolerates this). + +**Alternative considered:** require callers to always supply a parallel `/DecodeParms` array. **Rejected** — PDF 1.7 makes `/DecodeParms` optional even when `/Filter` is present, and our test fixtures don't include it. + +### D4 — Unknown filter ⇒ `p_error()` + return `false` from the chain helper + +If any filter in the chain is unknown, the chain helper logs via `p_error()` (mirroring how the existing FlateDecode `switch` handles unsupported predictors / colours / bit-counts) and returns `false`. `get_stream` and `set_stream` propagate the failure: `get_stream` returns `$this->_stream` unchanged (matching existing behaviour); `set_stream` leaves the object's `_stream` field and `Length` unchanged. Result: the caller observes the original bytes still in place, and the `p_error` is logged for debugging. + +**Alternative considered:** throw an exception. **Rejected** — upstream sapp's existing convention is `p_error()` + soft-fail, not exceptions. Diverging would block the upstream PR. + +### D5 — Implementation lives entirely inside `src/PDFObject.php` + +No new files, no signature changes on dependent classes (`PDFDoc`, `PDFUtilFnc`). The two helpers are added inside `PDFObject` next to the existing `FlateDecode()` static method, keeping the change set tight and the diff narrowly reviewable. + +**Alternative considered:** factor filter implementations out into a `Filter\` namespace (one class per filter, registry pattern). **Rejected** for this PR — it's the right long-term refactor but would touch every existing callsite and inflate the upstream PR beyond what a single round-of-review can absorb. We'll revisit after PRs `#01`–`#04` land and the per-filter implementations have settled. + +### D6 — Tests live as round-trip fixture assertions in `examples/` + +The existing `examples/poc-replace-text.php` validation gate proves the single-FlateDecode path. We add a sibling `examples/poc-filter-chain-roundtrip.php` (introduced in this change) that builds a synthetic two-filter PDF (`[/ASCIIHexDecode /FlateDecode]`) using only FlateDecode-implemented machinery plus hand-crafted ASCIIHex envelope bytes (since ASCIIHexDecode itself doesn't ship until upstream-PR #01). The fixture asserts that the dispatcher correctly applies the chain even when only one filter is implemented — the ASCII-hex envelope gets handed off to a stub case that returns `false` and the test asserts that path is taken. + +**Alternative considered:** defer chain-aware tests until upstream-PR #01 (when there's a real second filter to chain). **Rejected** — the dispatch behaviour is the entire contract of this PR; testing it via the stub path locks the contract in before any real filter depends on it. + +## Risks / Trade-offs + +- **Risk**: A PDF in the wild has an unusual `/Filter` representation (e.g. `null`, an empty array, or a single-element nested array). → **Mitigation**: defensive normalisation in `set_stream` / `get_stream` — treat `null` and empty array as "no filter" (existing pass-through path), single-element array identically to the string form. Unit-test these shapes explicitly. + +- **Risk**: A caller mutates `$this->_value['Filter']` directly after `set_stream` and the object ends up in an inconsistent state. → **Mitigation**: out of scope — same risk exists in the current code. Document the contract: callers must use `set_stream` (which manages `Filter` + `Length` together) and not poke at `_value` directly. + +- **Trade-off**: We commit to applying filters in a specific order that's normative for the PDF 1.7 spec but contrary to the way some PDF inspectors visualise the chain (some inspectors show outermost-first regardless of operation). → **Mitigation**: the rule is in the dispatcher's PHPDoc with the §7.4.1 ¶3 reference; tests anchor the ordering with explicit `encode(decode($x)) === $x` round-trips. + +- **Trade-off**: Adding a chain dispatcher without any second filter to chain means the new helper is technically dead code until upstream-PR #01 lands. → **Mitigation**: the stub-based test in D6 exercises the dispatcher today; upstream-PR #01 only needs to swap the stub `case` for a real `case`. + +- **Trade-off**: Choosing `p_error()` over an exception means callers won't be jolted into handling failures explicitly. → **Mitigation**: this is upstream convention. We document the failure mode clearly in PHPDoc; the `p_error()` log gives operators the signal they need. + +## Migration Plan + +No migration is needed for consumers. The change is strictly additive on the input shape: callers handing in a string-form `/Filter` continue to work; callers (mostly OR-side once upstream-PR #01–`#04` land) handing in array-form `/Filter` start working where they were silently broken before. No data migrations, no schema changes, no version bumps required on `dealfonso/sapp` consumers. + +For the rollback story: revert the single commit on `feat-filter-chain-dispatch`. Because all behaviour for string-form `/Filter` is preserved exactly, the revert is safe even if downstream code has already started using the array form (it just resumes failing in the same way it failed before this change). + +## Open Questions + +- **OQ1**: Should we proactively log a `p_debug()` line on every successful array-form decode for observability during the upstream review window? Leaning **no** — it'd pollute the log on every page-content read of a chain-encoded PDF, and the failure path already logs via `p_error()`. **Provisional**: skip `p_debug`. Confirm during PR review. + +- **OQ2**: When `/DecodeParms` is an array shorter than `/Filter`, do we treat missing trailing entries as `null` or as a parse error? PDF 1.7 doesn't speak to under-length arrays explicitly. **Provisional**: treat trailing missing entries as `null` (no params) — most lenient, matches reader behaviour in Adobe / pdf.js / poppler. Lock this with a unit test. diff --git a/openspec/changes/feat-filter-chain-dispatch/proposal.md b/openspec/changes/feat-filter-chain-dispatch/proposal.md new file mode 100644 index 0000000..d338e52 --- /dev/null +++ b/openspec/changes/feat-filter-chain-dispatch/proposal.md @@ -0,0 +1,33 @@ +## Why + +Upstream sapp's `PDFObject::set_stream` / `PDFObject::get_stream` hard-code a single string-typed `/Filter` value (only the literal `'/FlateDecode'` branch is wired up). PDF 1.7 §7.4 explicitly defines an array form for `/Filter` so a stream can be decoded by a chain of filters (e.g. `[/ASCIIHexDecode /FlateDecode]` — outer ASCII hex envelope wrapping a Flate-compressed payload). Real-world content streams produced by sanitisers, older toolchains, and some forensic pipelines use this array form regularly, and every downstream filter PR in the upstream-PR series (`#01` ASCIIHex / `#02` ASCII85 / `#03` RunLength / `#04` LZW) needs a place to plug in. Without chain dispatch first, the four follow-up filter PRs have nowhere to attach, and our text-replacement feature (`replaceTextInDocument` on `work/text-replacement`) can't operate on chain-encoded content streams at all. + +## What Changes + +- Refactor `PDFObject::set_stream($stream, $raw)` to walk `/Filter` when `$raw === false`: + - **String form** (e.g. `/FlateDecode`) — behaviour unchanged (one-shot encode through that filter). + - **Array form** (e.g. `[/ASCIIHexDecode /FlateDecode]`) — encode by applying filters in REVERSE order (innermost-first, PDF 1.7 §7.4.1 ¶3); update `/Length` to the final encoded byte count. +- Refactor `PDFObject::get_stream($raw)` symmetrically: array form decodes in FORWARD order (outermost-first). +- Handle the parallel `/DecodeParms` array shape (one entry per filter; `null` permitted as a placeholder for filters without parameters per PDF 1.7 §7.4.1 Table 5). +- Unknown filter names in the chain emit a `p_error()` warning (matches upstream's existing convention for unsupported predictors / colours / bit-counts) and leave the stream unchanged so the caller can detect the failure mode. +- No new filter implementations land in this change — that's upstream-PR #01–`#04` territory. This change is purely the dispatch surface. + +## Capabilities + +### New Capabilities + +- `filter-chain-dispatch`: PDF 1.7 §7.4 array-form `/Filter` chain processing on encode + decode, with parallel `/DecodeParms` array handling and defensive logging for unknown filter names. Provides the plug-in surface that the per-filter PRs (`#01`–`#04`) attach to. + +### Modified Capabilities + + + +## Impact + +- **Touched files**: `src/PDFObject.php` (the `set_stream` and `get_stream` switches; ~40 LOC delta). +- **Public API**: No changes. Existing string-form `/Filter` callers continue to work unmodified. The array-form input path is additive. +- **Downstream PRs**: Unblocks the upstream-PR series `#01` (ASCIIHex), `#02` (ASCII85), `#03` (RunLength), `#04` (LZW) — each of those just appends a `case` to the new chain-aware dispatcher. +- **Consumer (OpenRegister `pdf-anonymisation`)**: The PoC's `replaceTextInDocument()` on `work/text-replacement` already detects FlateDecode-only streams; once chain dispatch lands, the heuristic can extend to "any chain that contains FlateDecode" without further public-API churn. +- **Validation gate**: `examples/poc-replace-text.php` remains green (single FlateDecode round-trip). +- **Upstream-PR draft**: Final implementation notes land in `docs/upstream-prs/05-filter-chaining/` for eventual submission to `dealfonso/sapp`. +- **No external dependencies added.** Stays PHP ≥ 7.4 compatible. snake_case method names preserved (upstream convention). diff --git a/openspec/changes/feat-filter-chain-dispatch/specs/filter-chain-dispatch/spec.md b/openspec/changes/feat-filter-chain-dispatch/specs/filter-chain-dispatch/spec.md new file mode 100644 index 0000000..26447a3 --- /dev/null +++ b/openspec/changes/feat-filter-chain-dispatch/specs/filter-chain-dispatch/spec.md @@ -0,0 +1,121 @@ +**Status**: planned +**Scope**: change `feat-filter-chain-dispatch` (delta spec) +**OpenSpec changes**: +- `feat-filter-chain-dispatch` (in-progress) + +## Purpose + +Capability contract for `filter-chain-dispatch` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/filter-chain-dispatch/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: Array-form `/Filter` SHALL decode in forward chain order + +When `PDFObject::get_stream($raw = false)` is called on an object whose `/Filter` is an array, the dispatcher MUST apply the named filter decoders in FORWARD order (outermost first, innermost last), per PDF 1.7 §7.4.1 ¶3. The output MUST be the same plaintext that would be produced by an external PDF reader (Adobe / pdf.js / poppler) consuming the stream. + +#### Scenario: Two-filter chain decode + +- WHEN an object has `/Filter [/ASCIIHexDecode /FlateDecode]`, `/Length` matching the encoded byte count, and an encoded `_stream` produced by the sequence `gzcompress("BT...ET") |> ascii_hex_envelope` +- THEN `get_stream(false)` MUST return the original `"BT...ET"` plaintext (ASCIIHex un-envelope, then gzuncompress) + +#### Scenario: Single-element array form is semantically identical to string form + +- WHEN an object has `/Filter [/FlateDecode]` (1-element array) +- THEN `get_stream(false)` MUST return the same plaintext as it would for `/Filter /FlateDecode` (string form) on identical stream bytes + +#### Scenario: Empty array `/Filter` MUST be treated as no filtering + +- WHEN an object has `/Filter []` (empty array) +- THEN `get_stream(false)` MUST return `$this->_stream` unchanged + +### REQ-002: Array-form `/Filter` SHALL encode in reverse chain order + +When `PDFObject::set_stream($plaintext, $raw = false)` is called on an object whose `/Filter` is an array, the dispatcher MUST apply the named filter encoders in REVERSE order (innermost first, outermost last), per PDF 1.7 §7.4.1 ¶3, then update `/Length` to the final encoded byte count. + +#### Scenario: Two-filter chain encode + +- WHEN `set_stream("BT...ET", false)` is called on an object with `/Filter [/ASCIIHexDecode /FlateDecode]` +- THEN the resulting `_stream` MUST equal `ascii_hex_envelope(gzcompress("BT...ET"))` and `_value['Length']` MUST equal `strlen($_stream)` after encoding + +#### Scenario: Encode-decode round-trip is lossless + +- WHEN `set_stream($P, false)` is called on an array-form-`/Filter` object, then `get_stream(false)` is called on the same object +- THEN the value returned by `get_stream(false)` MUST equal `$P` byte-for-byte + +### REQ-003: `/DecodeParms` array SHALL be applied positionally per filter + +When `/Filter` is an array of length N, the dispatcher MUST accept a parallel `/DecodeParms` value of either: an array of length ≤ N (where each entry is either a parameter dictionary or `null` for filters with no parameters), or absent entirely (equivalent to `null` for every filter), or a single dictionary when `/Filter` has length 1 (string-form-equivalent shape). + +#### Scenario: Per-filter parameters dispatched to the correct decoder + +- WHEN an object has `/Filter [/FlateDecode]` and `/DecodeParms [<>]` +- THEN the `FlateDecode` decoder MUST receive `Predictor=12` and `Columns=4` (per PDF 1.7 §7.4.4 PNG predictor semantics) + +#### Scenario: Trailing `null` entries in `/DecodeParms` MUST be tolerated + +- WHEN an object has `/Filter [/ASCIIHexDecode /FlateDecode]` and `/DecodeParms [null null]` +- THEN both decoders MUST be invoked with no parameters (equivalent to the `/DecodeParms` entry being absent) + +#### Scenario: Under-length `/DecodeParms` MUST be tolerated + +- WHEN an object has `/Filter [/ASCIIHexDecode /FlateDecode]` and `/DecodeParms [null]` (length 1, less than `/Filter`'s length 2) +- THEN the missing trailing entry MUST be treated as `null` (no params for the unspecified filter), matching reader behaviour in Adobe / pdf.js / poppler + +### REQ-004: String-form `/Filter` callers MUST observe unchanged behaviour + +Callers that pass a single-name `/Filter` (the upstream-default shape, e.g. `/FlateDecode`) MUST observe byte-for-byte identical results from `get_stream` and `set_stream` after this change as before it. The dispatcher coerces string-form to a 1-element array internally but MUST NOT change the persisted `/Filter` shape on write-back. + +#### Scenario: String-form `/Filter` is preserved on write-back + +- WHEN an object loaded with `/Filter /FlateDecode` (string form) has `set_stream($P, false)` called on it +- THEN after the call, `$this->_value['Filter']` MUST still serialise as `/FlateDecode` (string), NOT as `[/FlateDecode]` (array) + +#### Scenario: Existing PoC verify gate stays green + +- WHEN `php examples/poc-replace-text.php` is executed on this change +- THEN it MUST exit with status 0 and `residual_needles=0, placeholder_hits=1, streams_modified=1` (same as before the change) + +### REQ-005: Unknown filter names in a chain MUST fail safely + +If any filter name in `/Filter` is not implemented, the dispatcher MUST log via `p_error()` (consistent with sapp's existing unsupported-feature convention) and MUST leave `$this->_stream` and `$this->_value['Length']` unchanged. It MUST NOT throw an exception. + +#### Scenario: Unknown filter in a chain leaves the stream untouched + +- WHEN `set_stream($P, false)` is called on an object with `/Filter [/UnknownFilter /FlateDecode]` +- THEN `p_error()` MUST be called with a message identifying the unknown filter name +- AND `$this->_stream` MUST remain unchanged from its pre-call value +- AND `$this->_value['Length']` MUST remain unchanged +- AND the method MUST NOT throw + +#### Scenario: Unknown filter on get_stream returns the raw stream + +- WHEN `get_stream(false)` is called on an object with `/Filter [/UnknownFilter /FlateDecode]` +- THEN `p_error()` MUST be called identifying the unknown filter +- AND the return value MUST equal `$this->_stream` (the raw, undecoded bytes) diff --git a/openspec/changes/feat-filter-chain-dispatch/tasks.md b/openspec/changes/feat-filter-chain-dispatch/tasks.md new file mode 100644 index 0000000..78dd775 --- /dev/null +++ b/openspec/changes/feat-filter-chain-dispatch/tasks.md @@ -0,0 +1,45 @@ +## 1. Pre-flight + +- [ ] 1.1 Branch off `work/text-replacement` as `feat/filter-chain-dispatch` +- [ ] 1.2 Re-read PDF 1.7 §7.4.1 ¶3 (chain ordering) + Table 5 (`/DecodeParms`) and confirm the dispatcher's contract matches the spec word-for-word +- [ ] 1.3 Capture the baseline: `php examples/poc-replace-text.php` exits 0 with `residual_needles=0, placeholder_hits=1, streams_modified=1` against today's `src/PDFObject.php` + +## 2. Dispatcher helpers in `src/PDFObject.php` + +- [ ] 2.1 Add `private static function normalise_filter_chain($filterValue): array` — coerces a `PDFValue` for `/Filter` (string-form, array-form, missing, or empty) into a plain PHP array of filter-name strings +- [ ] 2.2 Add `private static function normalise_decode_parms_chain($decodeParmsValue, int $chainLength): array` — coerces `/DecodeParms` (single dict, array of dicts/null, missing) into a fixed-length array indexed by chain position (D3) +- [ ] 2.3 Add `private static function apply_filter_chain_decode(string $bytes, array $filters, array $params)` — applies filters in FORWARD order; on unknown filter emits `p_error()` and returns `false` (D4) +- [ ] 2.4 Add `private static function apply_filter_chain_encode(string $bytes, array $filters, array $params)` — applies filters in REVERSE order; same failure semantics + +## 3. Refactor `get_stream` / `set_stream` + +- [ ] 3.1 Refactor `PDFObject::get_stream($raw)` (`src/PDFObject.php`) to call `apply_filter_chain_decode()` when `$raw === false` and the object has a `/Filter` entry; preserve the existing pass-through behaviour when `/Filter` is missing or coerces to an empty chain +- [ ] 3.2 Refactor `PDFObject::set_stream($stream, $raw)` to call `apply_filter_chain_encode()` when `$raw === false`; preserve the existing `_value['Length']` update and the unchanged-on-failure semantics (D4) +- [ ] 3.3 Implement the string-shape preservation rule from D2 — do not flip `/Filter` from string to array on write-back when input was string-form + +## 4. Tests / verification fixtures + +- [ ] 4.1 Add `examples/poc-filter-chain-roundtrip.php` exercising the two-filter scenario from spec REQ-1 / REQ-2 (`[/ASCIIHexDecode /FlateDecode]`), using a hand-crafted ASCII-hex envelope around a `gzcompress`'d payload — asserts the dispatcher hits the chain path even though `ASCIIHexDecode` itself is a stub `case` in this change +- [ ] 4.2 Add a defensive case in the dispatcher that returns the input unchanged when the chain is empty (REQ-1 scenario "Empty array `/Filter`") +- [ ] 4.3 Add an unknown-filter assertion in `examples/poc-filter-chain-roundtrip.php` per REQ-5 (`p_error` logged, `_stream` + `Length` unchanged) +- [ ] 4.4 Verify `php examples/poc-replace-text.php` still exits 0 (string-form FlateDecode path unchanged, REQ-4 "Existing PoC verify gate stays green") + +## 5. Upstream-PR draft + +- [ ] 5.1 Update `docs/upstream-prs/05-filter-chaining/proposal.md` with the dispatcher contract that landed (mirror what's in this change's `proposal.md`) +- [ ] 5.2 Update `docs/upstream-prs/05-filter-chaining/design.md` with the D1–D6 decision log +- [ ] 5.3 Update `docs/upstream-prs/05-filter-chaining/tasks.md` with the implementation note about upstream-PR #01–`#04` plug-in points (one `case` arm per filter) +- [ ] 5.4 Carry forward the `Posted at: ` placeholder in `issue.md` — do NOT post upstream yet (per the fork-and-give-back ordering) + +## 6. Quality gate + +- [ ] 6.1 Confirm no new composer dependencies were added (`composer.json` diff empty other than possible doc-only edits) +- [ ] 6.2 Confirm PHP 7.4 compatibility — no typed properties, no `readonly`, no `enum`, no first-class callable syntax +- [ ] 6.3 Confirm snake_case discipline on all new method names +- [ ] 6.4 Confirm no `PDFObject` / `PDFDoc` / `PDFValue*` class-boundary refactors crept in + +## 7. Commit + PR + +- [ ] 7.1 Commit on `feat/filter-chain-dispatch` with a single message summarising the dispatcher contract +- [ ] 7.2 Open PR `feat/filter-chain-dispatch → work/text-replacement` on `ConductionNL/sapp`, citing spec REQ-1 through REQ-5 +- [ ] 7.3 Add a note in the PR description that the four follow-up filter PRs (`#01`–`#04`) will attach via new `case` arms in `apply_filter_chain_decode` / `apply_filter_chain_encode` diff --git a/openspec/changes/feat-lzw-decode/.openspec.yaml b/openspec/changes/feat-lzw-decode/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-lzw-decode/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-lzw-decode/design.md b/openspec/changes/feat-lzw-decode/design.md new file mode 100644 index 0000000..107576e --- /dev/null +++ b/openspec/changes/feat-lzw-decode/design.md @@ -0,0 +1,76 @@ +## Context + +LZW decode (§7.4.4.2): + +- Initial dictionary has 256 entries (one per byte value) plus the two reserved codes (256 = clear, 257 = EOD). Code width starts at 9 bits. +- Read variable-width codes from the input bit stream MSB-first. +- On clear code (256): reset the dictionary to its initial 258-entry state, set code width back to 9 bits, continue. +- On EOD code (257): stop. +- On a user code (≥ 258): look up the string in the dictionary, emit it, append `prev_string + first_byte_of_current_string` to the dictionary, advance code width when the dictionary fills. +- On the special "code is exactly one past the last added code" case: the entry doesn't exist yet — emit `prev_string + first_byte_of_prev_string`. + +`EarlyChange` parameter: + +- `1` (default per §7.4.4.3 Table 8): bit width increases when the dictionary index reaches `2^width - 1` (the LAST entry of the current width is unused before advancing). +- `0`: bit width increases when the dictionary index reaches `2^width` (the full current width is used). + +PNG predictor support is identical to `FlateDecode` (§7.4.4.4) — same predictor algorithm, applied after decompression. + +## Goals / Non-Goals + +**Goals:** + +- Lossless round-trip on arbitrary binary input. +- Correct handling of clear codes and dictionary overflow at 12 bits. +- `EarlyChange` parity with both values (`0` and `1`). +- PNG predictor support matching `FlateDecode`. + +**Non-Goals:** + +- TIFF-style differencing predictor (PDF 1.7 §7.4.4.4 lists it but it's rare in PDFs — implement only if a real-world fixture demands it). +- Streaming / incremental decode. + +## Decisions + +### D1 — Mirror upstream's filter helper shape + +`protected static function LZWDecode($_stream, $params)` and `protected static function LZWEncode($_stream, $params)` as static helpers on `PDFObject`. + +### D2 — Bit-stream helpers as private static methods + +Add `protected static function lzw_read_code(string $stream, int $bitPos, int $codeWidth): int` and a corresponding write helper. These are LZW-specific because the MSB-first packing order matters for compatibility with Acrobat-emitted streams. + +### D3 — Dictionary as a flat PHP array indexed by code + +Code `c` maps to string `$dict[$c]`. Init with `chr(0)` through `chr(255)` for the first 256 entries; reserve indices 256–257 for clear/EOD; user entries start at 258. Cap dictionary growth at 4096 entries (the 12-bit code-width ceiling). + +### D4 — Reuse `FlateDecode`'s PNG predictor implementation + +Refactor the predictor logic in the existing `FlateDecode` method into a private `protected static function applyPngPredictor(string $bytes, array $params): string|false` helper, then call it from both `FlateDecode` and `LZWDecode`. Pure refactor, no behaviour change for FlateDecode callers (validated by the existing PoC verify). + +### D5 — `EarlyChange = 1` default, honour `EarlyChange = 0` if specified + +The default per spec is `1`; we honour the value passed via `/DecodeParms` and fall back to `1` when unset. Encode emits with the same `EarlyChange` value to make the round-trip lossless. + +### D6 — Fail-safe on dictionary overflow + +If the bit stream demands codes after the dictionary is full at 4096 entries without a clear code, `p_error()` is called and the raw input is returned. This is the recovery path for malformed streams — Acrobat handles this gracefully but corrupted streams hit it. + +## Risks / Trade-offs + +- **Risk**: LZW patent expired in 2003 but lingering FUD might make upstream sapp reviewers cautious. → **Mitigation**: link to the patent-expiry references in the upstream PR description; PDF 1.7 §7.4.4 is the normative reference. + +- **Risk**: `EarlyChange = 1` is subtle — bit-width advances happen one code earlier than the naive "fill the width" rule. Easy to get wrong; affects compatibility with every other PDF reader. → **Mitigation**: lock the behaviour with a fixture decoded by Adobe Reader; the round-trip test ensures we encode and decode using the same rule. + +- **Trade-off**: ~180 LOC is large for a single change. → **Mitigation**: the LZW state machine doesn't decompose into smaller PRs without breaking — the encoder, decoder, and predictor reuse all need to land together for a usable round-trip. + +## Open Questions + +- **OQ1**: TIFF predictor (`Predictor` values 2 and 3) — implement now or defer? **Provisional**: defer until a real-world fixture demonstrates the need. PNG predictor (values 10-15) covers the modern case. + +- **OQ2**: Should the encoder emit a clear code at the start of every stream (Acrobat-compatible) or only on overflow (minimal-output)? **Provisional**: emit on the start (Acrobat-style). Adds ~2 bytes but maximises reader compatibility. + +## Migration Plan + +Strictly additive change. No migration required for consumers — string-form callers (and unaware-of-this-change callers) observe byte-for-byte identical behaviour after merge. Rollback path is a clean revert of the commit. + diff --git a/openspec/changes/feat-lzw-decode/proposal.md b/openspec/changes/feat-lzw-decode/proposal.md new file mode 100644 index 0000000..6b814cb --- /dev/null +++ b/openspec/changes/feat-lzw-decode/proposal.md @@ -0,0 +1,31 @@ +## Why + +`LZWDecode` (PDF 1.7 §7.4.4) is the Lempel-Ziv-Welch variable-width-code compression filter inherited from TIFF/PostScript. It predates `FlateDecode` and shows up primarily on older PDFs (Adobe Acrobat 3-4 era) and PDFs produced by toolchains targeting wide reader compatibility. Like the other filters, it's a hard requirement for downstream PRs to plug into; without it, any chain containing `/LZWDecode` falls through to the no-op path and the text-replacement pipeline produces corrupted output on those PDFs. + +## What Changes + +- Add `LZWDecode` encode + decode primitives in `PDFObject`. +- Wire `/LZWDecode` into the chain dispatcher. +- Spec-faithful per §7.4.4: + - Variable-width codes start at 9 bits and grow to a maximum of 12 bits as the dictionary fills. + - Reserved codes: `256` (clear table — reset to 9-bit codes), `257` (EOD). User codes start at `258`. + - `EarlyChange` parameter (default `1`) controls whether the bit-width advance happens before or after emitting the code that fills the current width. +- Support the PNG predictor scheme per §7.4.4.4 (parity with `FlateDecode`'s predictor handling). +- `/DecodeParms` accepts `Predictor`, `Colors`, `BitsPerComponent`, `Columns`, `EarlyChange`. + +## Capabilities + +### New Capabilities + +- `lzw-decode-filter`: PDF 1.7 §7.4.4 LZWDecode encode + decode with `EarlyChange` + PNG predictor support, plugged into the filter chain dispatcher. + +### Modified Capabilities + +- `filter-chain-dispatch`: extend the `case` table with `/LZWDecode`. + +## Impact + +- **Touched files**: `src/PDFObject.php` (+LZW helpers + predictor reuse from `FlateDecode`; ~180 LOC — this is the largest of the four filter PRs). +- **Public API**: none. +- **Depends on**: `feat-filter-chain-dispatch`. +- **Upstream-PR draft**: `docs/upstream-prs/04-lzw-decode/`. diff --git a/openspec/changes/feat-lzw-decode/specs/lzw-decode-filter/spec.md b/openspec/changes/feat-lzw-decode/specs/lzw-decode-filter/spec.md new file mode 100644 index 0000000..62fff60 --- /dev/null +++ b/openspec/changes/feat-lzw-decode/specs/lzw-decode-filter/spec.md @@ -0,0 +1,140 @@ +**Status**: planned +**Scope**: change `feat-lzw-decode` (delta spec) +**OpenSpec changes**: +- `feat-lzw-decode` (in-progress) + +## Purpose + +Capability contract for `lzw-decode-filter` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/lzw-decode-filter/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: LZWDecode SHALL decode per PDF 1.7 §7.4.4 + +The decoder MUST read variable-width codes (9–12 bits) MSB-first from the input bit stream, maintain a dictionary starting with the 256 single-byte entries plus reserved codes 256 (clear) and 257 (EOD), and emit the dictionary string for each code per the standard LZW state machine. + +#### Scenario: Standard LZW round-trip + +- GIVEN the input is a 256-byte buffer encoded by `LZWEncode(...)` with `EarlyChange = 1` (default) +- WHEN `LZWDecode` is invoked +- THEN `LZWDecode(...)` MUST return the original 256-byte buffer byte-for-byte + +#### Scenario: Clear code resets the dictionary + +- GIVEN the encoded stream contains a clear code (256) mid-stream +- WHEN `LZWDecode` is invoked +- THEN the decoder MUST reset the dictionary to its initial 258-entry state and set the code width back to 9 bits before reading the next code + +#### Scenario: EOD code halts decoding + +- GIVEN the encoded stream contains the EOD code (257) +- WHEN `LZWDecode` is invoked +- THEN the decoder MUST stop and return the output accumulated so far +- AND any trailing bits MUST be ignored + +#### Scenario: KwKwK special case + +- WHEN the next code in the bit stream equals the dictionary's next-to-be-assigned index (the case where the dictionary entry doesn't exist yet at lookup time) +- THEN the decoder MUST emit `prev_string + first_byte_of_prev_string` per the standard LZW rule + +### REQ-002: LZWDecode SHALL honour the `EarlyChange` parameter + +The decoder MUST honour the `EarlyChange` entry from `/DecodeParms`. Default value is `1` per PDF 1.7 §7.4.4.3 Table 8. + +#### Scenario: EarlyChange = 1 (default) + +- WHEN the dictionary index reaches `2^width - 1` (one less than the full width capacity) +- THEN the decoder MUST increase the code width by 1 bit before reading the next code + +#### Scenario: EarlyChange = 0 + +- WHEN `/DecodeParms` specifies `EarlyChange = 0` and the dictionary index reaches `2^width` (full width capacity) +- THEN the decoder MUST increase the code width by 1 bit before reading the next code + +### REQ-003: LZWDecode round-trip MUST be lossless + +For any input byte string `$P` and any valid `$params`, `LZWDecode(LZWEncode($P, $params), $params)` MUST equal `$P` byte-for-byte. + +#### Scenario: Binary round-trip + +- GIVEN the input is `random_bytes(2048)` and `$params` is `null` +- WHEN `LZWDecode` is invoked +- THEN the round-trip MUST equal the input byte-for-byte + +#### Scenario: Round-trip with EarlyChange = 0 + +- GIVEN the input is `random_bytes(2048)` and `$params['EarlyChange']` is `0` +- WHEN `LZWDecode` is invoked +- THEN the round-trip MUST equal the input byte-for-byte + +### REQ-004: LZWDecode SHALL support the PNG predictor scheme + +When `/DecodeParms` specifies `Predictor >= 10` (PNG predictors), the decoder MUST apply the same PNG row-filter algorithm used by `FlateDecode`, parameterised by `Colors`, `BitsPerComponent`, and `Columns`. + +#### Scenario: PNG predictor unchanged from FlateDecode + +- WHEN `LZWDecode` is invoked with `Predictor = 12` and the same `Colors` / `BitsPerComponent` / `Columns` parameters as a `FlateDecode` call +- THEN the predictor output MUST match `FlateDecode`'s output byte-for-byte for an equivalent decompressed stream + +### REQ-005: LZWDecode SHALL fail safely on dictionary overflow + +If the bit stream demands codes beyond the 4096-entry dictionary limit without an intervening clear code, the decoder MUST call `p_error()` and MUST return the raw input bytes unchanged. + +#### Scenario: Dictionary overflow + +- GIVEN the input is a synthetic LZW stream crafted to produce more than 4096 distinct codes without a clear code +- WHEN `LZWDecode` is invoked +- THEN the decoder MUST call `p_error()` +- AND MUST return the original input bytes unchanged + +## MODIFIED Requirements + +### REQ-001: Array-form `/Filter` SHALL decode in forward chain order + +The dispatcher MUST recognise `/LZWDecode` as a valid filter name and route to the LZWDecode helper, propagating `/DecodeParms` per the positional rule from `feat-filter-chain-dispatch`. + +#### Scenario: LZW with PNG predictor + +- WHEN an object has `/Filter /LZWDecode` and `/DecodeParms <>` +- THEN `get_stream(false)` MUST apply LZW decode followed by PNG predictor inversion + +#### Scenario: LZW outer + Flate inner chain + +- WHEN an object has `/Filter [/LZWDecode /FlateDecode]` +- THEN the chain decode MUST apply LZW first, then Flate + +### REQ-002: Array-form `/Filter` SHALL encode in reverse chain order + +The dispatcher MUST recognise `/LZWDecode` on the encode path and route to the LZWEncode helper. + +#### Scenario: LZW-only encode round-trip + +- WHEN `set_stream($P, false)` is called on `/Filter /LZWDecode`, then `get_stream(false)` is called +- THEN the value returned by `get_stream(false)` MUST equal `$P` diff --git a/openspec/changes/feat-lzw-decode/tasks.md b/openspec/changes/feat-lzw-decode/tasks.md new file mode 100644 index 0000000..a6f4141 --- /dev/null +++ b/openspec/changes/feat-lzw-decode/tasks.md @@ -0,0 +1,64 @@ +## 1. Pre-flight + +- [ ] 1.1 Confirm `feat-filter-chain-dispatch` is merged into `work/text-replacement` +- [ ] 1.2 Branch off as `feat/lzw-decode` +- [ ] 1.3 Re-read PDF 1.7 §7.4.4 + §7.4.4.3 (`EarlyChange`) + §7.4.4.4 (predictor) +- [ ] 1.4 Find or create a reference LZW-encoded fixture decodable by Adobe Reader to lock the EarlyChange behaviour + +## 2. Refactor: lift PNG predictor out of FlateDecode + +- [ ] 2.1 Add `protected static function applyPngPredictor(string $bytes, array $params)` in `src/PDFObject.php` containing the existing predictor logic +- [ ] 2.2 Update `FlateDecode` to call `applyPngPredictor` instead of inlining the row-filter loop +- [ ] 2.3 Verify `examples/poc-replace-text.php` still exits 0 (predictor path unchanged for FlateDecode callers) + +## 3. Bit-stream helpers + +- [ ] 3.1 Add `protected static function lzw_read_code(string $stream, int &$bitPos, int $codeWidth): int` — MSB-first variable-width code reader +- [ ] 3.2 Add a symmetric write helper for the encoder + +## 4. Decode helper + +- [ ] 4.1 Add `protected static function LZWDecode($_stream, $params)` +- [ ] 4.2 Initialise dictionary (256 single-byte entries + reserved 256/257) +- [ ] 4.3 Implement the state machine: clear code, EOD, user codes, KwKwK special case +- [ ] 4.4 Honour `EarlyChange` (default 1; honour 0 if specified) — D5 +- [ ] 4.5 Cap dictionary growth at 4096; `p_error` + return raw on overflow — D6 +- [ ] 4.6 After LZW decode, call `applyPngPredictor` if `Predictor >= 10` + +## 5. Encode helper + +- [ ] 5.1 Add `protected static function LZWEncode($_stream, $params)` +- [ ] 5.2 Emit clear code at stream start (OQ2 — Acrobat-compatible) +- [ ] 5.3 Standard LZW encode state machine; emit clear code on dictionary overflow +- [ ] 5.4 Emit EOD code at the end +- [ ] 5.5 Honour `EarlyChange` symmetrically with the decoder + +## 6. Chain-dispatch integration + +- [ ] 6.1 Add `case '/LZWDecode'` arms in both `apply_filter_chain_decode` and `apply_filter_chain_encode` + +## 7. Tests / verification + +- [ ] 7.1 Round-trip fixture `examples/poc-filter-roundtrip-lzw.php`: 2048-byte buffer, default params +- [ ] 7.2 Round-trip fixture with `EarlyChange = 0` +- [ ] 7.3 Round-trip fixture with PNG predictor (`Predictor = 12, Columns = 4`) +- [ ] 7.4 Adobe-compat test: decode a reference fixture produced by Adobe Distiller (or Ghostscript with `-sFilter=LZW`) and assert the expected plaintext +- [ ] 7.5 Verify `examples/poc-replace-text.php` still exits 0 +- [ ] 7.6 Negative test: dictionary-overflow stream → `p_error` + unchanged input +- [ ] 7.7 Chain test `[/LZWDecode /FlateDecode]` round-trip + +## 8. Upstream-PR draft + +- [ ] 8.1 Update `docs/upstream-prs/04-lzw-decode/{proposal,design,tasks}.md` — include the predictor-refactor note for reviewers +- [ ] 8.2 Leave `Posted at: ` placeholder + +## 9. Quality gate + +- [ ] 9.1 PHP 7.4 compatibility — no bitwise operations that depend on 64-bit ints (LZW max code width is 12 bits → safe on 32-bit) +- [ ] 9.2 No new composer dependencies +- [ ] 9.3 snake_case discipline on the bit-stream helpers; PascalCase on filter helper names (matches `FlateDecode`) + +## 10. Commit + PR + +- [ ] 10.1 Commit on `feat/lzw-decode` +- [ ] 10.2 Open PR `feat/lzw-decode → work/text-replacement` — flag the predictor refactor as a noteworthy review point diff --git a/openspec/changes/feat-runlength-decode/.openspec.yaml b/openspec/changes/feat-runlength-decode/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-runlength-decode/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-runlength-decode/design.md b/openspec/changes/feat-runlength-decode/design.md new file mode 100644 index 0000000..bf0c8c6 --- /dev/null +++ b/openspec/changes/feat-runlength-decode/design.md @@ -0,0 +1,57 @@ +## Context + +RunLengthDecode operates on bytes with a simple length-prefix scheme: + +| Length byte `L` | Interpretation | +|---|---| +| `0 ≤ L ≤ 127` | Copy the next `L + 1` literal bytes | +| `L == 128` | EOD — stop decoding | +| `129 ≤ L ≤ 255` | Copy the next byte `257 - L` times | + +Encode is greedy: scan the input for runs of 2+ identical bytes (max 128 per repeat) and emit the repeat form; otherwise accumulate literals (max 128 per literal block). + +## Goals / Non-Goals + +**Goals:** + +- Lossless round-trip on arbitrary binary input. +- Recognise EOD at any position; bytes after EOD are ignored. +- Encode produces compact output for runs (≥3 bytes per emitted repeat group). + +**Non-Goals:** + +- Optimal encoding (we use greedy; some pathological inputs may compress slightly larger than an optimal RLE). +- Streaming / incremental decode (operates on full buffers, matching upstream's filter convention). + +## Decisions + +### D1 — Mirror `FlateDecode` shape + +`protected static function RunLengthDecode($_stream, $params)` and `protected static function RunLengthEncode($_stream, $params)` as static helpers on `PDFObject`. Same calling convention as the existing FlateDecode helper. + +### D2 — Encode literal-then-runs greedy heuristic + +Walk the input; when we see ≥ 2 identical adjacent bytes, flush any pending literal run and emit a repeat block (capped at 128 bytes per block). Otherwise accumulate literals (capped at 128 per block). Greedy — not optimal but ~5 LOC vs. dozens for an optimal encoder, and the difference is irrelevant for the small content streams we deal with. + +### D3 — Decode rejects truncated input via `p_error` + return raw + +If the decoder reads a length byte `L ∈ [0, 127]` but the stream runs out before delivering `L + 1` literal bytes, OR reads `L ∈ [129, 255]` but the stream runs out before delivering the repeat byte, OR reaches end-of-stream before encountering EOD (`L == 128`) — `p_error()` is called and the raw input is returned unchanged. Matches sapp's existing convention. + +### D4 — EOD is mandatory on encode, tolerated on decode + +The encoder MUST emit the EOD marker (`\x80`) at the end of every output stream. The decoder treats end-of-input without EOD as truncation (D3), but `\x80` followed by trailing garbage is fine — garbage is ignored. + +## Risks / Trade-offs + +- **Trade-off**: Greedy encoding leaves a few bytes on the floor vs. optimal RLE. → **Mitigation**: documented; in-the-wild RLE streams are typically short and the size difference is negligible. Future optimisation is a one-method swap. + +- **Risk**: An attacker-crafted input could request a 128-byte literal run with only 1 byte present, causing a buffer over-read in the decoder. → **Mitigation**: defensive bounds check before each `substr` call; raise `p_error` and return raw input on underflow. + +## Open Questions + +- **OQ1**: Should the encoder emit an empty stream for empty input, or a single EOD byte `\x80`? **Provisional**: single EOD byte. Matches the convention used by other RLE codecs and makes the encoder's output non-empty even for trivial input (easier round-trip testing). + +## Migration Plan + +Strictly additive change. No migration required for consumers — string-form callers (and unaware-of-this-change callers) observe byte-for-byte identical behaviour after merge. Rollback path is a clean revert of the commit. + diff --git a/openspec/changes/feat-runlength-decode/proposal.md b/openspec/changes/feat-runlength-decode/proposal.md new file mode 100644 index 0000000..826835f --- /dev/null +++ b/openspec/changes/feat-runlength-decode/proposal.md @@ -0,0 +1,29 @@ +## Why + +`RunLengthDecode` is the PDF 1.7 §7.4.5 filter used to compress streams with simple run-length encoding — primarily monochrome image data, but also occasionally seen on small content streams emitted by older toolchains. Upstream sapp doesn't implement it; without it, any chain containing `/RunLengthDecode` falls through to the no-op path and the text-replacement pipeline produces corrupted output. + +## What Changes + +- Add `RunLengthDecode` encode + decode primitives in `PDFObject`. +- Wire `/RunLengthDecode` into the chain dispatcher introduced by `feat-filter-chain-dispatch`. +- Spec-faithful semantics per §7.4.5: + - Length byte `L`: if `0 ≤ L ≤ 127`, copy `L + 1` literal bytes that follow. If `129 ≤ L ≤ 255`, repeat the next single byte `257 - L` times. If `L == 128`, EOD marker — stop decoding. + - No `/DecodeParms` (Table 5). +- Encode: emit greedy run-length encoding. Single-byte literals are wasteful; runs of 2+ identical bytes use the repeat form. + +## Capabilities + +### New Capabilities + +- `runlength-decode-filter`: PDF 1.7 §7.4.5 RunLengthDecode encode + decode plugged into the filter chain dispatcher. + +### Modified Capabilities + +- `filter-chain-dispatch`: extend the `case` table with `/RunLengthDecode`. + +## Impact + +- **Touched files**: `src/PDFObject.php` (+RunLength helpers, +dispatch arms; ~70 LOC). +- **Public API**: none. +- **Depends on**: `feat-filter-chain-dispatch`. +- **Upstream-PR draft**: `docs/upstream-prs/02-runlength-decode/`. diff --git a/openspec/changes/feat-runlength-decode/specs/runlength-decode-filter/spec.md b/openspec/changes/feat-runlength-decode/specs/runlength-decode-filter/spec.md new file mode 100644 index 0000000..5d14dfb --- /dev/null +++ b/openspec/changes/feat-runlength-decode/specs/runlength-decode-filter/spec.md @@ -0,0 +1,140 @@ +**Status**: planned +**Scope**: change `feat-runlength-decode` (delta spec) +**OpenSpec changes**: +- `feat-runlength-decode` (in-progress) + +## Purpose + +Capability contract for `runlength-decode-filter` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/runlength-decode-filter/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: RunLengthDecode SHALL decode per PDF 1.7 §7.4.5 + +The decoder MUST interpret each length byte `L` as: copy `L + 1` literal bytes if `0 ≤ L ≤ 127`, repeat the next byte `257 - L` times if `129 ≤ L ≤ 255`, halt at EOD if `L == 128`. Bytes after EOD MUST be ignored. + +#### Scenario: Literal run + +- GIVEN the input is `\x04Hello\x80` (length byte 4 = "copy 5 literals", then `Hello`, then EOD) +- WHEN `RunLengthDecode` is invoked +- THEN the decoder MUST return the 5-byte string `Hello` + +#### Scenario: Repeat run + +- GIVEN the input is `\xFAX\x80` (length byte 250 = "repeat next byte `257 - 250 = 7` times", then `X`, then EOD) +- WHEN `RunLengthDecode` is invoked +- THEN the decoder MUST return `XXXXXXX` (7 X's) + +#### Scenario: Mixed literal + repeat + +- GIVEN the input is `\x02ABC\xFEY\x80` (3 literal bytes `ABC`, then 3 `Y`s, then EOD) +- WHEN `RunLengthDecode` is invoked +- THEN the decoder MUST return `ABCYYY` + +#### Scenario: EOD halts decoding + +- GIVEN the input is `\x02ABC\x80garbage` +- WHEN `RunLengthDecode` is invoked +- THEN the decoder MUST return `ABC` and MUST NOT raise an error on the trailing `garbage` + +### REQ-002: RunLengthEncode SHALL produce a valid round-trip-compatible stream + +The encoder MUST emit a sequence of length-prefixed blocks (literal or repeat) terminated by the EOD byte `\x80`. The output MUST decode back to the exact input via the §7.4.5 decoder. + +#### Scenario: Literal-only input + +- GIVEN the input is the 5-byte string `Hello` (all distinct adjacent bytes) +- WHEN `RunLengthEncode` is invoked +- THEN the encoder output MUST begin with `\x04Hello` (literal-5-bytes block) and MUST end with `\x80` (EOD) + +#### Scenario: Repeat-only input + +- GIVEN the input is the 7-byte string `XXXXXXX` +- WHEN `RunLengthEncode` is invoked +- THEN the encoder output MUST contain a single repeat block `\xFAX` followed by `\x80` (EOD) + +#### Scenario: Empty input + +- GIVEN the input is the empty string +- WHEN `RunLengthEncode` is invoked +- THEN the encoder MUST return the single byte `\x80` (EOD only) + +### REQ-003: RunLengthDecode round-trip MUST be lossless + +For any input byte string `$P`, `RunLengthDecode(RunLengthEncode($P, null), null)` MUST equal `$P` byte-for-byte. + +#### Scenario: Binary round-trip + +- GIVEN the input is a 1024-byte buffer with a mix of runs and literals (e.g. 200 zero bytes, followed by random bytes, followed by 100 `\xFF` bytes) +- WHEN `RunLengthDecode` is invoked +- THEN the round-trip MUST equal the input byte-for-byte + +### REQ-004: RunLengthDecode SHALL fail safely on truncation + +If the decoder encounters end-of-input before delivering all bytes for a literal block, before delivering the repeat byte for a repeat block, or before encountering EOD — it MUST call `p_error()` and MUST return the raw input bytes unchanged. + +#### Scenario: Truncated literal block + +- GIVEN the input is `\x05ABC` (length byte says "copy 6 literals" but only 3 follow) +- WHEN `RunLengthDecode` is invoked +- THEN the decoder MUST call `p_error()` +- AND MUST return the original input bytes unchanged + +#### Scenario: Truncated repeat block + +- GIVEN the input is `\xFE` (length byte says "repeat next byte 3 times" but no byte follows) +- WHEN `RunLengthDecode` is invoked +- THEN the decoder MUST call `p_error()` +- AND MUST return the original input bytes unchanged + +## MODIFIED Requirements + +### REQ-001: Array-form `/Filter` SHALL decode in forward chain order + +The dispatcher MUST recognise `/RunLengthDecode` as a valid filter name and route to the RunLengthDecode helper. + +#### Scenario: RunLength single-filter chain + +- WHEN an object has `/Filter /RunLengthDecode` and `_stream` equal to `RunLengthEncode("plaintext")` +- THEN `get_stream(false)` MUST return `"plaintext"` byte-for-byte + +#### Scenario: RunLength outer + Flate inner chain + +- WHEN an object has `/Filter [/RunLengthDecode /FlateDecode]` +- THEN the chain decode MUST be applied in forward order (RunLength first, then Flate) + +### REQ-002: Array-form `/Filter` SHALL encode in reverse chain order + +The dispatcher MUST recognise `/RunLengthDecode` on the encode path and route to the RunLengthEncode helper. + +#### Scenario: RunLength encode + Flate inner chain + +- WHEN `set_stream("plaintext", false)` is called on an object with `/Filter [/RunLengthDecode /FlateDecode]` +- THEN the resulting `_stream` MUST equal `RunLengthEncode(gzcompress("plaintext"))` and `_value['Length']` MUST equal `strlen($_stream)` diff --git a/openspec/changes/feat-runlength-decode/tasks.md b/openspec/changes/feat-runlength-decode/tasks.md new file mode 100644 index 0000000..0ae9f15 --- /dev/null +++ b/openspec/changes/feat-runlength-decode/tasks.md @@ -0,0 +1,45 @@ +## 1. Pre-flight + +- [ ] 1.1 Confirm `feat-filter-chain-dispatch` is merged into `work/text-replacement` +- [ ] 1.2 Branch off as `feat/runlength-decode` +- [ ] 1.3 Re-read PDF 1.7 §7.4.5 — confirm length-byte semantics + +## 2. Decode helper + +- [ ] 2.1 Add `protected static function RunLengthDecode($_stream, $params)` in `src/PDFObject.php` +- [ ] 2.2 Implement the state machine: literal blocks (`0..127`), repeat blocks (`129..255`), EOD (`128`) +- [ ] 2.3 Defensive bounds checks before `substr()` calls — `p_error()` + return raw on underflow (D3) + +## 3. Encode helper + +- [ ] 3.1 Add `protected static function RunLengthEncode($_stream, $params)` — greedy run detection (D2) +- [ ] 3.2 Cap literal blocks at 128 bytes; cap repeat blocks at 128 bytes +- [ ] 3.3 Emit trailing `\x80` EOD marker (always, including empty-input case) + +## 4. Chain-dispatch integration + +- [ ] 4.1 Add `case '/RunLengthDecode'` to `apply_filter_chain_decode` +- [ ] 4.2 Add symmetric arm to `apply_filter_chain_encode` + +## 5. Tests / verification + +- [ ] 5.1 Round-trip fixture `examples/poc-filter-roundtrip-runlength.php`: 1024-byte mixed-run input +- [ ] 5.2 Verify `examples/poc-replace-text.php` still exits 0 +- [ ] 5.3 Negative test: truncated literal block → `p_error` + stream unchanged +- [ ] 5.4 Chain test `[/RunLengthDecode /FlateDecode]` round-trip + +## 6. Upstream-PR draft + +- [ ] 6.1 Update `docs/upstream-prs/02-runlength-decode/{proposal,design,tasks}.md` +- [ ] 6.2 Leave `Posted at: ` placeholder + +## 7. Quality gate + +- [ ] 7.1 PHP 7.4 compatibility +- [ ] 7.2 No new composer dependencies +- [ ] 7.3 snake_case discipline on new helpers + +## 8. Commit + PR + +- [ ] 8.1 Commit on `feat/runlength-decode` +- [ ] 8.2 Open PR `feat/runlength-decode → work/text-replacement` diff --git a/openspec/changes/feat-text-replacement-api/.openspec.yaml b/openspec/changes/feat-text-replacement-api/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-text-replacement-api/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-text-replacement-api/design.md b/openspec/changes/feat-text-replacement-api/design.md new file mode 100644 index 0000000..9bd1080 --- /dev/null +++ b/openspec/changes/feat-text-replacement-api/design.md @@ -0,0 +1,106 @@ +## Context + +PDF 1.7 §9.6.2.2 defines a "standard 14" set of fonts — Helvetica, Helvetica-Bold, Helvetica-Oblique, Helvetica-BoldOblique, Times-Roman (and its variants), Courier (and its variants), Symbol, ZapfDingbats — that every conformant reader MUST support without an embedded font program. They take a minimal Type1 font dictionary: + +``` +<< + /Type /Font + /Subtype /Type1 + /BaseFont /Helvetica + /Encoding /WinAnsiEncoding +>> +``` + +This is the same shape our PoC `poc-make-fixture.php` already uses. To use it as a placeholder font: + +1. Add the dictionary as an object in the PDF (if not already present). +2. Add a reference to the new font under `/Resources/Font/` on the page being modified (or its inherited Resources dictionary). +3. Wrap the placeholder emission in `q ... Q` so the original font, font size, and any other graphics-state knobs are restored. + +The `q/Q` pair (PDF 1.7 §8.4.4 ¶3) saves/restores the entire graphics state stack including the current font (`Tf` state), font size, text matrix, and rendering mode. So `q\n/F-fallback 12 Tf\n(placeholder) Tj\nQ` is locally scoped — the next operator after `Q` sees the original font as if nothing happened. + +## Goals / Non-Goals + +**Goals:** + +- Recover the substitution that upstream-PRs #06 / #07 skip due to subset-font encoding misses. +- Produce a content stream that renders correctly in every PDF reader (Adobe, Foxit, pdf.js, poppler, Preview). +- Inject the fallback font resource at most once per page (idempotent). +- The polished `replaceTextInDocument` API is stable: same signature, same diagnostic-key naming, full PHPDoc — ready for the upstream submission. + +**Non-Goals:** + +- Visual matching with the original font. The placeholder is intentionally a synthetic marker; Helvetica is fine. +- Font-size matching. We hard-code 12pt; if the surrounding text was 10pt the placeholder will be slightly taller. Acceptable for an anonymisation marker. +- Helvetica-Bold / Helvetica-Italic variants. The single regular Helvetica covers the placeholder set we use. +- Vertical text fallback. Out of scope for the Woo use case. + +## Decisions + +### D1 — Synthesise the Helvetica font once per page, lazily + +The resource injector tracks a per-page `$fallbackFontInjected` boolean. On the first match in a page that requires the fallback, it: + +1. Creates a new `PDFObject` for the Helvetica dictionary via `$pdfDoc->create_object($value)` (existing API). +2. Adds `/F-anonymisation-fallback` as a key under the page's `/Resources/Font` dictionary, pointing at the new object via an indirect reference. +3. Sets `$fallbackFontInjected = true` for the page so subsequent matches reuse the same resource. + +The resource name `/F-anonymisation-fallback` is namespaced enough to avoid collisions with the document's existing font resource names (which are typically `/F1`, `/F2`, etc.). If a document happens to have a real `/F-anonymisation-fallback` (vanishingly unlikely), we collide-detect and pick `/F-anonymisation-fallback-2`, `-3`, ... + +### D2 — Fallback wraps the placeholder in `q/Q` + +The splicer emits `q\n/F-anonymisation-fallback 12 Tf\n(placeholder) Tj\nQ`. The `q` pushes the graphics state; the `Q` pops it. Anything that comes after the `Q` operates under the original font/size, so the rest of the line (or the rest of the TJ array's surviving fragments) renders correctly. + +### D3 — Inherited resources require placeholder-page-level injection + +If the page being modified doesn't have its own `/Resources` (it inherits from a parent), we have to promote the resources dictionary to the page level so we can add our fallback font without affecting siblings. The promotion is: copy the inherited resources dict to the page's `/Resources`, then add our fallback font there. Sibling pages continue to use the original inherited resources unchanged. + +### D4 — Fallback ONLY fires when the active font can't encode + +We do NOT use the fallback as the default. The active-font-first rule from upstream-PR #06 is still primary; the fallback is only the recovery path. This minimises the visual disruption (a placeholder in the document's actual font is preferable to a placeholder in Helvetica when both are options). + +### D5 — `subset_font_fallbacks_used` is the diagnostic counter + +The returned diagnostic surface gains `subset_font_fallbacks_used: int` — counts the number of placeholder emissions that used the fallback. Combined with `font_encoding_misses` (which now ONLY contains matches that BOTH the active font AND the fallback couldn't encode — very rare, requires the placeholder to use characters outside Helvetica's WinAnsi encoding), this gives operators a clear picture of how the substitution played out. + +### D6 — `replaceTextInDocument` parameter validation + +The polished API rejects: + +- Empty-string needles (would match everywhere, almost certainly a caller bug) +- Placeholders containing the bytes `\x00` through `\x1F` (control characters — PDF 1.7 string-literal rules make these awkward) +- Placeholders containing `(`, `)`, or `\` (require PDF-string-escape handling — out of scope for this PR; can re-enable later if a use case demands) + +Rejection emits `p_error` with the offending entry and skips that single substitution; other substitutions in the same call proceed. + +### D7 — Final API documentation + +Full PHPDoc on `replaceTextInDocument` including: + +- Parameter types and constraints +- All diagnostic keys with their semantics +- Worked example +- Reference to the OpenSpec changes that built up the contract (`feat-filter-chain-dispatch` through `feat-text-replacement-api`) +- PDF 1.7 spec section references + +## Risks / Trade-offs + +- **Risk**: Adding a font resource to an existing PDF could trigger validation failures in strict readers. → **Mitigation**: the standard 14 fonts don't require font programs and have been a PDF spec promise since PDF 1.0. Every reader handles this case. Add a real-world-reader spot-check task. + +- **Risk**: Multiple `replaceTextInDocument` calls on the same `PDFDoc` instance must not inject the resource twice. → **Mitigation**: D1's per-page idempotency boolean handles this. Add a regression test for "call replaceTextInDocument twice". + +- **Trade-off**: Promoting inherited resources to the page level (D3) duplicates resource dict bytes for affected pages. → **Mitigation**: only fires when a fallback is actually used; for pages without subset-font encoding misses (the common case once OpenRegister documents start being modern), the promotion never happens. + +## Migration Plan + +For the upstream contribution: the public API contract this PR sets is what gets submitted to `dealfonso/sapp`. The upstream PR description references each of the 8 OpenSpec changes as the contract chain. + +For OpenRegister: the existing `pdf-anonymisation` change starts producing fully-redacted output across all Woo fixtures once this PR lands. The diagnostic surface stays backwards-compatible — consumers that read only the PoC's original keys continue to work. + +Rollback: revert the commit. Helvetica fallback feature unavailable; matcher still works but skips subset-font-encoding-miss cases. + +## Open Questions + +- **OQ1**: Should we expose font choice (`Helvetica` vs `Courier`) as an option in the public API? **Provisional**: no. One sensible default keeps the API tight. Internal call sites that want a different font can edit the synthesiser directly. + +- **OQ2**: Placeholder size — hard-coded 12pt or auto-detected from surrounding text? **Provisional**: hard-coded 12pt. Auto-detection requires parsing the current Tf state from the operator history, which is doable but adds complexity for marginal visual benefit on an explicitly synthetic marker. diff --git a/openspec/changes/feat-text-replacement-api/proposal.md b/openspec/changes/feat-text-replacement-api/proposal.md new file mode 100644 index 0000000..efc3dc1 --- /dev/null +++ b/openspec/changes/feat-text-replacement-api/proposal.md @@ -0,0 +1,36 @@ +## Why + +After PRs `#01`–`#07` land, the text-replacement machinery handles all four PDF filters (FlateDecode + ASCIIHex + ASCII85 + RunLength + LZW), filter chains, Identity-H + ToUnicode CMap resolution, and TJ kerning-array flattening. What's still missing for the production use case is **the Helvetica fallback when a subset font can't encode the placeholder**. This is the final blocker for the >95% Woo case. + +Word emits subset fonts containing only the glyphs the document actually uses. A document about "Jan Jansen" produced by a typical Dutch government template won't have `[`, `:`, or digits in the subset's font program. Our placeholder `[PERSON: 7]` then triggers `font_encoding_misses` and the substitution is skipped — usable output but with the wrong-but-not-corrupt behaviour of leaving the name in place. + +The fix is the PDF `q/Q` save/restore pattern: temporarily switch to a built-in (non-subset) font like Helvetica before emitting the placeholder, then restore. Built-in fonts (also called "base 14") don't require font programs in the PDF — they're guaranteed available in every reader. + +This PR also polishes the public API surface: documentation, idempotency tests, and final cleanup before we start drafting the upstream submission. + +## What Changes + +- Add the Helvetica fallback path in the placeholder-emit logic introduced by `feat-tounicode-cmap`. When the active font's forward map can't encode every character of the placeholder, the splicer: + 1. Adds `/F-anonymisation-fallback` to the page's `/Resources/Font` if not already present, pointing at a synthesised standard `/Helvetica` Type1 font with `/WinAnsiEncoding`. + 2. Wraps the placeholder in `q\n/F-anonymisation-fallback 12 Tf\n() Tj\nQ` (the `q/Q` pair saves and restores the graphics state, isolating the font switch). + 3. The trailing `Q` restores the original font; subsequent operators in the content stream continue under the original `Tf`. +- Polish `PDFDoc::replaceTextInDocument` API: stable diagnostic-key naming, full PHPDoc, parameter validation (e.g. reject empty-string keys, reject placeholders containing reserved PDF string characters that would require escape handling). +- Add `examples/upstream-poc.php` — the upstream-PR's demo script that exercises the full pipeline on a representative real-world fixture. + +## Capabilities + +### New Capabilities + +- `subset-font-fallback`: graphics-state-isolated Helvetica fallback for placeholders that the active subset font can't encode. + +### Modified Capabilities + +- `text-replacement`: the public API contract gains parameter-validation guarantees and a documented stable diagnostic shape suitable for upstream submission. + +## Impact + +- **Touched files**: `src/PDFDoc.php` (fallback path in the splicer + resource-injection logic, ~80 LOC), `src/PDFObject.php` (helper for synthesising the Helvetica resource object, ~30 LOC), `examples/upstream-poc.php` (new — ~60 LOC). +- **Public API**: `replaceTextInDocument`'s contract becomes stable and documented. Diagnostic shape: all keys from PRs `#06` and `#07` plus `subset_font_fallbacks_used: int` (counter). +- **Depends on**: `feat-tounicode-cmap`, `feat-tj-flattening` (uses both — the fallback fires inside the splicer for either Tj or TJ). +- **Unblocks**: production Woo PDFs with subset fonts that don't include placeholder characters. +- **Upstream-PR draft**: `docs/upstream-prs/08-text-replacement-api/` — this is the upstream-submission-ready package. diff --git a/openspec/changes/feat-text-replacement-api/specs/subset-font-fallback/spec.md b/openspec/changes/feat-text-replacement-api/specs/subset-font-fallback/spec.md new file mode 100644 index 0000000..93f823e --- /dev/null +++ b/openspec/changes/feat-text-replacement-api/specs/subset-font-fallback/spec.md @@ -0,0 +1,110 @@ +**Status**: planned +**Scope**: change `feat-text-replacement-api` (delta spec) +**OpenSpec changes**: +- `feat-text-replacement-api` (in-progress) + +## Purpose + +Capability contract for `subset-font-fallback` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/subset-font-fallback/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: Fallback Helvetica font SHALL be injected when the active font fails + +When the splicer's encoding step fails (active font's forward map can't encode every character of the placeholder), it MUST inject a `/Helvetica` Type1 resource into the affected page's `/Resources/Font` dictionary (idempotent — at most once per page) and emit the placeholder under that font. + +#### Scenario: Subset font missing `[` triggers fallback + +- WHEN the active font's forward map can encode `PERSON: 7` but not `[` or `]`, and the placeholder is `[PERSON: 7]` +- THEN a `/Helvetica` Type1 font object MUST be created via `PDFDoc::create_object` +- AND `/F-anonymisation-fallback` MUST be added to the page's `/Resources/Font` +- AND the spliced content stream MUST emit `q\n/F-anonymisation-fallback 12 Tf\n() Tj\nQ` +- AND `subset_font_fallbacks_used` MUST equal 1 in the returned diagnostic surface + +#### Scenario: Fallback font injected once per page across multiple matches + +- WHEN the same page contains three matches that all require the fallback +- THEN the Helvetica object MUST be created exactly once +- AND the page's `/Resources/Font/F-anonymisation-fallback` MUST be set exactly once +- AND `subset_font_fallbacks_used` MUST equal 3 (one per spliced placeholder) + +#### Scenario: Inherited Resources are promoted to page-level on fallback + +- WHEN a page inherits its `/Resources` from a parent node and has no own `/Resources` entry +- AND a match on that page requires the fallback +- THEN the inherited Resources dictionary MUST be copied to the page's own `/Resources` BEFORE the fallback font is added +- AND the parent's Resources dictionary MUST remain unchanged (sibling pages are unaffected) + +#### Scenario: Resource name collision SHALL be detected + +- WHEN a page already has a font resource named `/F-anonymisation-fallback` (vanishingly unlikely but spec-legal) +- THEN the injector MUST use the next available variant (`/F-anonymisation-fallback-2`, `/F-anonymisation-fallback-3`, ...) +- AND the injected resource name MUST be the one used in the spliced content stream's `Tf` operator + +### REQ-002: Fallback emission SHALL preserve surrounding graphics state + +The spliced content stream's `q ... Q` pair MUST be syntactically correct (matched push/pop) and MUST be the ONLY graphics-state mutation in the placeholder splice. Operators AFTER the `Q` MUST observe the same graphics state as operators BEFORE the `q`. + +#### Scenario: Operators after the placeholder see the original font + +- WHEN the active font before the splice was `/F1` and the splice emits the fallback placeholder +- THEN the next operator after `Q` MUST operate under `/F1` (verifiable by tokenising the spliced stream and checking the active Tf state at that operator's index) + +### REQ-003: Fallback SHALL ONLY fire when the active font fails + +The fallback path MUST NOT be the default. The splicer MUST first attempt to encode the placeholder through the active font's forward map. Only when that returns null/unencodable for at least one character does the fallback engage. + +#### Scenario: Active font that can encode placeholder MUST NOT trigger fallback + +- WHEN the active font is `/WinAnsiEncoding` Helvetica (built-in, can encode the entire placeholder character set) +- AND the placeholder is `[PERSON: 7]` +- THEN the placeholder MUST be emitted via the active font (not the fallback) +- AND the fallback Helvetica resource MUST NOT be injected into the page's `/Resources` +- AND `subset_font_fallbacks_used` MUST equal 0 + +## MODIFIED Requirements + +### REQ-001: Unencodable placeholders SHALL be diagnosed, not corrupted + +Replaces the `feat-tounicode-cmap` rule. The active font's forward map is tried first; if it can't encode every character, the fallback (`feat-text-replacement-api`) is tried. If BOTH fail (active subset font missing characters AND placeholder contains characters outside Helvetica WinAnsiEncoding), the substitution at this position MUST be skipped and `font_encoding_misses` MUST be populated as before. + +#### Scenario: Placeholder uses a character outside both fonts + +- WHEN the active font is a subset Helvetica missing `€` (U+20AC) and the placeholder contains `€` +- AND the fallback Helvetica `/WinAnsiEncoding` (which DOES include `€` at byte 0x80) is also available +- THEN the fallback MUST be used (WinAnsi has `€` even when the subset doesn't) +- AND the substitution MUST succeed + +#### Scenario: Placeholder uses a character outside both fonts (e.g. Hebrew) + +- WHEN the placeholder contains a character that's neither in the subset font NOR in Helvetica's `/WinAnsiEncoding` (e.g. U+05D0 Hebrew Alef) +- THEN the substitution MUST be skipped +- AND `font_encoding_misses[$oid][$needle] = $font_base_name` MUST be recorded +- AND `subset_font_fallbacks_used` MUST NOT be incremented for this match diff --git a/openspec/changes/feat-text-replacement-api/specs/text-replacement/spec.md b/openspec/changes/feat-text-replacement-api/specs/text-replacement/spec.md new file mode 100644 index 0000000..85d44b6 --- /dev/null +++ b/openspec/changes/feat-text-replacement-api/specs/text-replacement/spec.md @@ -0,0 +1,126 @@ +**Status**: planned +**Scope**: change `feat-text-replacement-api` (delta spec) +**OpenSpec changes**: +- `feat-text-replacement-api` (in-progress) + +## Purpose + +Capability contract for `text-replacement` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/text-replacement/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: replaceTextInDocument SHALL validate input parameters + +The polished API MUST validate input parameters and reject the following with `p_error` + skipping the offending substitution (other substitutions in the same call proceed): + +- Empty-string needles (would match everywhere) +- Placeholders containing control characters (`\x00`–`\x1F`) +- Placeholders containing PDF-string-escape-significant characters (`(`, `)`, `\`) — out of scope for this version; can be re-enabled when a use case demands + +#### Scenario: Empty-string needle is rejected + +- WHEN `replaceTextInDocument(['' => 'placeholder'])` is called +- THEN `p_error` MUST be called identifying the empty-string needle +- AND the substitution MUST NOT fire +- AND the method MUST return its normal diagnostic shape with `replacements_per_needle` excluding the rejected entry + +#### Scenario: Placeholder containing `\` is rejected + +- WHEN `replaceTextInDocument(['Jan Jansen' => 'foo\\bar'])` is called +- THEN `p_error` MUST be called identifying the offending placeholder +- AND the `Jan Jansen` substitution MUST NOT fire +- AND other substitutions in the same call MUST proceed normally + +#### Scenario: Valid call processes all substitutions + +- WHEN `replaceTextInDocument(['Jan Jansen' => '[PERSON: 7]', 'Acme B.V.' => '[ORG: 3]'])` is called with both valid +- THEN both substitutions MUST be attempted +- AND the returned `replacements_per_needle` MUST include both keys + +### REQ-002: Diagnostic surface SHALL be stable and documented + +The returned array MUST contain exactly these keys with the documented semantics. The API contract is frozen at this point (this is the upstream-submission shape). + +| Key | Type | Semantics | +|---|---|---| +| `streams_scanned` | int | Total content streams visited (regardless of match) | +| `streams_modified` | int | Streams with ≥ 1 spliced match | +| `replacements_per_needle` | `array` | Per-needle match count summed across all streams | +| `unmatched_needles` | `string[]` | Needles with zero matches across the entire document | +| `tj_arrays_modified` | int | TJ operators that had at least one match-driven splice | +| `subset_font_fallbacks_used` | int | Placeholder emissions that used the fallback font | +| `font_encoding_misses` | `array>` | `[oid => [needle => font_name]]` for matches both active font + fallback couldn't encode | +| `cid_split_mismatch` | `array>` | `[oid => [needle => cid_position]]` for matches that would split a CID interior | +| `rejected_substitutions` | `array` | `[needle => reason]` for input-validation rejections | + +#### Scenario: All keys present even when zero values + +- WHEN `replaceTextInDocument` is called with a single valid substitution that produces no matches +- THEN the returned array MUST contain ALL 9 keys +- AND counters MUST be 0 / empty arrays where there's no data + +### REQ-003: Idempotency across repeat calls + +Calling `replaceTextInDocument` twice on the same `PDFDoc` instance MUST be safe: the second call MUST NOT re-inject a fallback font that's already present, MUST NOT match its own previously-inserted placeholders, and MUST return diagnostic counts reflecting only the second call's matches. + +#### Scenario: Repeat call doesn't re-inject fallback + +- WHEN `replaceTextInDocument` is called twice with substitutions that each trigger the Helvetica fallback +- THEN the page's `/Resources/Font/F-anonymisation-fallback` MUST be set exactly once (the first call's injection survives) +- AND the second call MUST NOT create a duplicate Helvetica font object + +#### Scenario: Repeat call doesn't double-replace + +- WHEN the first call replaces `Jan Jansen` → `[PERSON: 7]` (1 match) +- AND the second call uses the same substitutions +- THEN the second call's `replacements_per_needle['Jan Jansen']` MUST equal 0 (the source bytes are now the placeholder, not the needle) +- AND `streams_modified` MUST equal 0 for the second call + +## MODIFIED Requirements + +### REQ-001: replaceTextInDocument SHALL match the needle in text space, not byte space + +Unchanged contract from `feat-tounicode-cmap`. Listed here for completeness — this PR adds parameter validation and stable diagnostic shape on top of the existing text-space matching. + +#### Scenario: WinAnsi-only fixture (PoC regression) still produces clean output + +- WHEN `replaceTextInDocument(['Jan Jansen' => '[PERSON: 7]'])` is called on `examples/poc-fixture.pdf` +- THEN the returned diagnostic surface MUST satisfy: `streams_modified == 1`, `replacements_per_needle['Jan Jansen'] == 1`, `unmatched_needles == []`, `subset_font_fallbacks_used == 0`, `font_encoding_misses == []`, `cid_split_mismatch == []`, `rejected_substitutions == []`, `tj_arrays_modified == 0` +- AND the re-extracted output MUST have `residual_needles == 0` and `placeholder_hits == 1` + +### REQ-002: Placeholder SHALL be emitted via the active font's forward map + +Unchanged in the no-fallback path; this PR adds the fallback recovery (see `subset-font-fallback` capability). Listed here so the full contract is reviewable in one place at archive time. + +#### Scenario: Active font encoding works → no fallback + +- WHEN the active font's forward map can encode every character of the placeholder +- THEN the placeholder MUST be emitted via the active font (NOT the fallback) +- AND the page's `/Resources/Font` MUST NOT be modified diff --git a/openspec/changes/feat-text-replacement-api/tasks.md b/openspec/changes/feat-text-replacement-api/tasks.md new file mode 100644 index 0000000..29a820f --- /dev/null +++ b/openspec/changes/feat-text-replacement-api/tasks.md @@ -0,0 +1,62 @@ +## 1. Pre-flight + +- [ ] 1.1 Confirm `feat-tounicode-cmap` and `feat-tj-flattening` are merged into `work/text-replacement` +- [ ] 1.2 Branch off as `feat/text-replacement-api` +- [ ] 1.3 Capture a real Woo-style fixture with a subset font that's missing `[`, `]`, or digits — needed for fallback regression testing. Check in as `examples/poc-fixture-subset-missing-chars.pdf` + +## 2. Helvetica fallback injection + +- [ ] 2.1 Add `private function injectFallbackFontResource(int $pageOid): string` in `PDFDoc` — returns the chosen resource name (default `/F-anonymisation-fallback`; collision-detected variant if taken) +- [ ] 2.2 Synthesise the standard `/Helvetica` Type1 dictionary, create the object via `$this->create_object($value)` (idempotent — track per-page injection state) +- [ ] 2.3 Add the resource entry to the page's `/Resources/Font` +- [ ] 2.4 Implement the inherited-resources promotion (D3) — copy parent's Resources to page-level before adding + +## 3. Splicer fallback path + +- [ ] 3.1 Refactor the placeholder-emit step in the splicer (`replaceTextInDocument`'s inner loop): first try the active font's forward map; on encoding miss, fall through to the fallback +- [ ] 3.2 Emit `q\n/F-anonymisation-fallback 12 Tf\n() Tj\nQ` (D2) — wrap the placeholder in graphics-state save/restore +- [ ] 3.3 If BOTH the active font AND `/WinAnsiEncoding` Helvetica can't encode every character, fall through to the existing `font_encoding_misses` skip path + +## 4. Input validation + +- [ ] 4.1 Validate substitutions before processing: reject empty needles, placeholders with control characters (`\x00`–`\x1F`), placeholders with PDF-escape-significant characters (`(`, `)`, `\`) +- [ ] 4.2 Record rejections in `rejected_substitutions` diagnostic key +- [ ] 4.3 Other substitutions in the same call proceed normally + +## 5. API polish + +- [ ] 5.1 Full PHPDoc on `replaceTextInDocument` — every parameter, every diagnostic key, the spec-section references +- [ ] 5.2 Add the worked example from the PHPDoc to `examples/upstream-poc.php` — the upstream-PR demo script +- [ ] 5.3 Stabilise the diagnostic-key naming (this is the upstream-submission shape — frozen at this point) +- [ ] 5.4 Add a `replaceTextInDocument` test that asserts ALL 9 diagnostic keys exist on the return value, including the zero/empty cases + +## 6. Tests / verification + +- [ ] 6.1 Round-trip on `examples/poc-fixture-subset-missing-chars.pdf`: needle replaced via fallback, placeholder renders in Helvetica +- [ ] 6.2 Idempotency test: two consecutive `replaceTextInDocument` calls on the same `PDFDoc` (D2 / OQ1 from upstream-PR #06) +- [ ] 6.3 Collision test: pre-existing `/F-anonymisation-fallback` resource → injector picks the next variant +- [ ] 6.4 Inherited-resources promotion test: page without own `/Resources` → fallback fires correctly without breaking siblings +- [ ] 6.5 Verify ALL prior PoC fixtures still exit 0 (no fallback used on WinAnsi or Identity-H-with-encodable-placeholder cases) +- [ ] 6.6 Negative test: placeholder containing `\` → `rejected_substitutions` populated, substitution skipped +- [ ] 6.7 Real-reader spot-check: open the redacted output in Adobe Reader, Firefox (pdf.js), poppler (`pdftotext`) — placeholder visible in all three + +## 7. Upstream-PR draft + +- [ ] 7.1 Update `docs/upstream-prs/08-text-replacement-api/{proposal,design,tasks}.md` — this is the upstream-submission package +- [ ] 7.2 Cross-reference the prior 7 upstream-PR drafts (link the dependency chain) +- [ ] 7.3 Add the worked-example snippet to `docs/upstream-prs/08-text-replacement-api/issue.md` +- [ ] 7.4 Leave `Posted at: ` placeholder +- [ ] 7.5 Write the upstream-PR description draft separately (not posted yet — user decides timing) + +## 8. Quality gate + +- [ ] 8.1 PHP 7.4 compatibility +- [ ] 8.2 No new composer dependencies +- [ ] 8.3 snake_case + PSR-12 discipline; full PHPDoc on every new method +- [ ] 8.4 Confirm the upstream-PR series (PRs #05, #01–#04, #06, #07, this) leaves no orphan code paths + +## 9. Commit + PR + +- [ ] 9.1 Commit on `feat/text-replacement-api` — split into a few commits to keep review-able (fallback injector, splicer fallback, API polish) +- [ ] 9.2 Open PR `feat/text-replacement-api → work/text-replacement` — note in the description that this completes the 8-PR series +- [ ] 9.3 Once merged into `work/text-replacement`, fast-forward `work/text-replacement` onto a release tag for OpenRegister's composer VCS reference to point at diff --git a/openspec/changes/feat-tj-flattening/.openspec.yaml b/openspec/changes/feat-tj-flattening/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-tj-flattening/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-tj-flattening/design.md b/openspec/changes/feat-tj-flattening/design.md new file mode 100644 index 0000000..d3d3ebf --- /dev/null +++ b/openspec/changes/feat-tj-flattening/design.md @@ -0,0 +1,70 @@ +## Context + +`TJ` operator syntax: `[ ... ] TJ`. Each fragment is a string literal `(...)` or hex string `<...>`. Each kern is a number (units of 1/1000 of a font em; negative values reduce spacing, positive values increase it). + +For text-space matching purposes the kern values are irrelevant — they affect rendering position, not the text the user reads. Word/Acrobat split text into many `TJ` fragments for sub-pixel kerning; a needle like "Jan Jansen" is typically emitted as something like `[(J) 2 (a) -1 (n) 3 ( ) -2 (J) 1 (a) -1 (n) 2 (s) 1 (e) -1 (n)] TJ`. + +The matching layer from upstream-PR #06 already handles `Tj`. We extend the tokeniser to recognise the `TJ` array shape, expose its fragments to the matcher as individual text-showing operands, then splice the array on match. + +## Goals / Non-Goals + +**Goals:** + +- Match needles that span TJ fragment boundaries. +- Preserve the kerning OUTSIDE the match span (text appearance before/after the placeholder stays visually identical). +- Drop the kerning INSIDE the match span (replaced by the placeholder's single `Tj`, kerning numbers were specific to the original characters and don't apply to the placeholder). + +**Non-Goals:** + +- Re-emit a `TJ` array around the placeholder for "perfect" rendering. The placeholder is a synthetic marker; preserving the original micro-kerning is not worth the implementation complexity. +- Preserving the original fragment-count cardinality. The spliced output has fewer operators than the input by design. + +## Decisions + +### D1 — Tokeniser exposes TJ fragments as virtual text-showing entries + +When the tokeniser hits a `TJ` operator, it emits per-fragment entries `(operator: 'TJ_fragment', operand_bytes, fragment_index, parent_tj_index, source_byte_start, source_byte_end)`. The matching layer treats `TJ_fragment` identically to `Tj` for text-space concatenation; the splicing layer uses the `parent_tj_index` to know which fragments to retain when splicing. + +### D2 — Match-span splice rules + +For a match span covering fragments `[m_start, m_end]` within a `TJ` array of fragments `[0, N-1]`: + +| Case | Splice result | +|------|---------------| +| `m_start == 0 && m_end == N-1` (full TJ matched) | Replace entire `[...] TJ` with `(placeholder) Tj` | +| `m_start == 0 && m_end < N-1` (matched prefix) | `(placeholder) Tj [] TJ` | +| `m_start > 0 && m_end == N-1` (matched suffix) | `[] TJ (placeholder) Tj` | +| `m_start > 0 && m_end < N-1` (matched middle) | `[] TJ (placeholder) Tj [] TJ` | + +Kerning numbers OUTSIDE the match span are preserved exactly. Kerning numbers INSIDE the match span are discarded. + +### D3 — Cross-`Tf` matches inside a single TJ are rare; same start-font rule applies + +A `TJ` operator is always emitted under a single `Tf`-set font scope (the operator doesn't internally switch fonts). So the cross-font matching concern from upstream-PR #06 doesn't apply inside a TJ. We still emit the placeholder via the active font's forward map. + +### D4 — Fragment-boundary alignment with CID interiors + +The CID-interior-split rule from upstream-PR #06 extends naturally: a match's start or end MUST align with a CID boundary inside the fragment it lands on. The fragment-level boundary itself is always CID-aligned (TJ fragments are whole-character sequences). + +### D5 — Hex-string fragments treated identically to literal-string fragments + +`TJ` operands can mix `(...)` literal strings and `<...>` hex strings (typical Identity-H pattern). The tokeniser normalises both to byte arrays; the matcher treats them identically; the splicer emits the placeholder as a hex string when the active font is composite (2-byte CID), as a literal string when it's simple (1-byte). Existing convention in upstream sapp. + +## Risks / Trade-offs + +- **Risk**: Discarding internal kerning may cause visible "tightening" of the surrounding text if the original `TJ` used large kerning adjustments. → **Mitigation**: in practice Word's per-character kerning is < 10 units (< 1% em), invisible to the eye. The placeholder's appearance is intentionally distinct anyway. + +- **Risk**: A `TJ` with zero fragments (`[] TJ` — spec-legal but pointless) could trip the tokeniser. → **Mitigation**: skip empty `TJ` arrays during tokenisation; emit a `p_debug` for observability. + +- **Trade-off**: Splicing logic for the "matched middle" case (Case 4 in D2) produces three operators where there was one. This grows the content stream slightly. → **Mitigation**: the growth is < 10 bytes per match for typical placeholder lengths. Re-encoding via FlateDecode usually shrinks it back below the original size. + +## Open Questions + +- **OQ1**: Should we collapse adjacent surviving `TJ` operators after splicing (e.g. when a match leaves two single-fragment `TJ` operators side by side)? **Provisional**: no. Acrobat handles consecutive `TJ` operators correctly; collapsing them would add complexity without observable benefit. + +- **OQ2**: Some PDFs emit `TJ` with array elements that are themselves arrays (deeply unusual but spec-legal per §9.4.3 ¶2 — "Each element ... shall be either a string or a number"). Reject or tolerate? **Provisional**: reject (per spec). Emit `p_error` if the tokeniser sees nested arrays. + +## Migration Plan + +Strictly additive change. No migration required for consumers — string-form callers (and unaware-of-this-change callers) observe byte-for-byte identical behaviour after merge. Rollback path is a clean revert of the commit. + diff --git a/openspec/changes/feat-tj-flattening/proposal.md b/openspec/changes/feat-tj-flattening/proposal.md new file mode 100644 index 0000000..08088e2 --- /dev/null +++ b/openspec/changes/feat-tj-flattening/proposal.md @@ -0,0 +1,30 @@ +## Why + +PDF text-showing operator `TJ` (PDF 1.7 §9.4.3) takes an ARRAY operand: alternating string fragments and numeric kerning adjustments, e.g. `[(Hello) -10 (World)] TJ`. Word and most modern PDF producers emit `TJ` (with kerning) much more often than the simpler `Tj` (single string). After upstream-PR #06 (ToUnicode CMap) lands, our matcher can walk Tj operators correctly — but matches that span fragment boundaries inside a `TJ` array still fail because the matcher only sees individual fragments. + +The Woo use case has many of these. A typical line of body text in a Word document is emitted as one `TJ` operator with dozens of kerning splits — needles like "Jan Jansen" almost always cross at least one fragment boundary. + +## What Changes + +- Recognise `TJ` operators in the tokeniser introduced by `feat-tounicode-cmap`. +- For each `TJ`, concatenate the string fragments (ignoring the kerning numbers for matching purposes) and run the same text-space matcher across the concatenated text. +- On match: replace the ENTIRE `TJ` array (all fragments + all kerning numbers within the match span) with a single `Tj` carrying the placeholder. Kerning that lies OUTSIDE the match span (preceding fragments or following fragments) is preserved as a smaller `TJ` operator, or hoisted to standalone `Tj` operators if the match consumes the array's middle. +- The diagnostic surface gains `tj_arrays_modified: int` (counter for matched `TJ` operations, separate from `streams_modified` which still counts decoded content streams). + +## Capabilities + +### New Capabilities + +- `tj-flattening`: TJ kerning-array text-space matching with selective fragment elision. + +### Modified Capabilities + +- `text-replacement`: extend the operator tokeniser + matcher to handle `TJ` array operands; extend the diagnostic surface with `tj_arrays_modified`. + +## Impact + +- **Touched files**: `src/PDFObject.php` (tokeniser TJ support, ~40 LOC), `src/PDFDoc.php` (match-and-splice logic for TJ arrays, ~80 LOC). +- **Public API**: `replaceTextInDocument` shape unchanged except for the additive `tj_arrays_modified` key. +- **Depends on**: `feat-tounicode-cmap` (text-space matching foundation). +- **Unblocks**: production Word-generated Woo PDFs at the typical text-emission shape. +- **Upstream-PR draft**: `docs/upstream-prs/07-tj-flattening/`. diff --git a/openspec/changes/feat-tj-flattening/specs/tj-flattening/spec.md b/openspec/changes/feat-tj-flattening/specs/tj-flattening/spec.md new file mode 100644 index 0000000..12ba814 --- /dev/null +++ b/openspec/changes/feat-tj-flattening/specs/tj-flattening/spec.md @@ -0,0 +1,133 @@ +**Status**: planned +**Scope**: change `feat-tj-flattening` (delta spec) +**OpenSpec changes**: +- `feat-tj-flattening` (in-progress) + +## Purpose + +Capability contract for `tj-flattening` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/tj-flattening/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: Tokeniser SHALL emit per-fragment entries for TJ operators + +When the content-stream tokeniser encounters a `TJ` operator, it MUST emit one virtual text-showing entry per string fragment in the array. Numeric kerning entries MUST be preserved but exposed as non-text-showing entries (so they don't participate in the text-space concatenation). + +#### Scenario: TJ with multiple fragments + +- GIVEN the operator is `[(Hello) -10 (World)] TJ` +- WHEN `processTjArray` is invoked +- THEN the tokeniser MUST emit 2 text-showing fragments: `(Hello)` and `(World)` +- AND preserve the kerning value `-10` between them +- AND mark both fragments with the same `parent_tj_index` + +#### Scenario: TJ with mixed literal + hex strings + +- GIVEN the operator is `[<0041> 0 (B) -5 <0043>] TJ` +- WHEN `processTjArray` is invoked +- THEN the tokeniser MUST emit 3 fragments and normalise both literal and hex strings to byte arrays +- AND preserve kerning values 0 and -5 between them + +#### Scenario: Empty TJ array + +- GIVEN the operator is `[] TJ` +- WHEN `processTjArray` is invoked +- THEN the tokeniser MUST emit no fragments and SHOULD emit a `p_debug` log line + +### REQ-002: Matching across TJ fragment boundaries SHALL succeed + +The text-space matcher MUST treat a TJ operator's fragments as a single contiguous text run for matching purposes. A needle that spans 2+ fragments MUST match. + +#### Scenario: Needle spans 4 fragments of a TJ + +- WHEN the content stream contains `[(J) 2 (a) -1 (n) 3 ( ) -2 (Jansen)] TJ` and the needle is `"Jan Jansen"` +- THEN the matcher MUST identify the match as covering all 5 fragments (J, a, n, space, Jansen) +- AND report this in the matched-fragments span for the splicer + +### REQ-003: Splicer SHALL produce the correct shape per match position + +Given a match span covering fragments `[m_start, m_end]` within a `TJ` array of fragments `[0, N-1]`, the splicer MUST produce one of four output shapes per Decision D2: + +| Case | Shape | +|------|-------| +| Full TJ matched (`m_start == 0 && m_end == N-1`) | `(placeholder) Tj` | +| Prefix matched (`m_start == 0`) | `(placeholder) Tj [] TJ` | +| Suffix matched (`m_end == N-1`) | `[] TJ (placeholder) Tj` | +| Middle matched | `[] TJ (placeholder) Tj [] TJ` | + +#### Scenario: Full TJ match becomes Tj + +- GIVEN the operator is `[(Jan ) -2 (Jansen)] TJ` and the entire array matches the needle +- WHEN `processTjArray` is invoked +- THEN the spliced output MUST be `() Tj` with no surrounding TJ + +#### Scenario: Prefix match preserves trailing TJ + +- GIVEN the operator is `[(Jan Jansen) -5 ( voor het loket.)] TJ` and the needle matches only `"Jan Jansen"` (fragment 0) +- WHEN `processTjArray` is invoked +- THEN the spliced output MUST be `() Tj [()] TJ` with kerning `-5` preserved inside the trailing TJ + +#### Scenario: Middle match produces three operators + +- GIVEN the operator is `[(Voor ) 2 (Jan Jansen) -3 ( voor het loket.)] TJ` and the needle matches the middle fragment only +- WHEN `processTjArray` is invoked +- THEN the spliced output MUST be `[(Voor )] TJ () Tj [( voor het loket.)] TJ` +- AND the kerning `2` between fragments 0 and 1 MUST be discarded (it was inside the match span boundary) +- AND the kerning `-3` between fragments 1 and 2 MUST be discarded + +### REQ-004: TJ matching SHALL honour CID-boundary alignment + +If a needle's match would start or end in the interior of a multi-codepoint CID inside a TJ fragment, the substitution MUST be skipped and `cid_split_mismatch` MUST be recorded (same rule as `feat-tounicode-cmap`). + +#### Scenario: CID-split inside a TJ fragment + +- WHEN a TJ fragment contains a single CID resolving to `"fi"` and the needle is `"f"` (would split the CID) +- THEN the substitution MUST NOT fire and a `cid_split_mismatch` diagnostic MUST be added + +## MODIFIED Requirements + +### REQ-001: replaceTextInDocument diagnostic surface + +The diagnostic surface returned by `replaceTextInDocument` MUST grow with a new key `tj_arrays_modified: int` counting the number of TJ operators that had at least one match-driven splice. + +- `streams_scanned: int` +- `streams_modified: int` +- `replacements_per_needle: array` +- `unmatched_needles: string[]` +- `font_encoding_misses: array>` +- `cid_split_mismatch: array>` +- `tj_arrays_modified: int` (new) + +#### Scenario: TJ flattening modifies one operator + +- WHEN a single TJ operator matches one needle and is spliced +- THEN `tj_arrays_modified` MUST equal 1 +- AND `streams_modified` MUST equal 1 +- AND `replacements_per_needle[needle]` MUST equal 1 diff --git a/openspec/changes/feat-tj-flattening/tasks.md b/openspec/changes/feat-tj-flattening/tasks.md new file mode 100644 index 0000000..4a9d073 --- /dev/null +++ b/openspec/changes/feat-tj-flattening/tasks.md @@ -0,0 +1,49 @@ +## 1. Pre-flight + +- [ ] 1.1 Confirm `feat-tounicode-cmap` is merged into `work/text-replacement` +- [ ] 1.2 Branch off as `feat/tj-flattening` +- [ ] 1.3 Capture a Word-generated fixture that emits the target name across multiple TJ fragments (check via inspection of the decoded content stream) and check in as `examples/poc-fixture-tj-fragmented.pdf` + +## 2. Tokeniser TJ support + +- [ ] 2.1 Extend `PDFObject::tokeniseOperators` (introduced in upstream-PR #06) to recognise `TJ` array operands +- [ ] 2.2 For each TJ, emit per-fragment text-showing entries with `parent_tj_index` link (D1) +- [ ] 2.3 Preserve kerning numbers as non-text-showing entries between fragments +- [ ] 2.4 Reject nested-array operands via `p_error` (OQ2) +- [ ] 2.5 Skip empty TJ arrays with a `p_debug` log line + +## 3. Match-and-splice + +- [ ] 3.1 Update the matcher in `PDFDoc::replaceTextInDocument` to consume per-fragment entries as if they were `Tj` operands (the concatenation logic doesn't care about origin) +- [ ] 3.2 When a match's source range covers TJ fragments, compute `m_start` and `m_end` fragment indices within the TJ +- [ ] 3.3 Implement the four splice shapes per D2 (full / prefix / suffix / middle) +- [ ] 3.4 Preserve OUTSIDE-the-match kerning numbers; drop INSIDE-the-match kerning +- [ ] 3.5 Splice the new operator(s) into the source byte range + +## 4. Diagnostic surface + +- [ ] 4.1 Add `tj_arrays_modified: int` to the returned array + +## 5. Tests / verification + +- [ ] 5.1 Round-trip on `examples/poc-fixture-tj-fragmented.pdf`: assert needle replaced, placeholder rendered correctly in a real viewer +- [ ] 5.2 Unit fixture: hand-crafted TJ with the four splice positions (full / prefix / suffix / middle) +- [ ] 5.3 Verify `examples/poc-replace-text.php` (Tj-only fixture) still exits 0 with `tj_arrays_modified == 0` +- [ ] 5.4 Negative test: TJ with nested array operand → `p_error` + stream unchanged +- [ ] 5.5 Negative test: TJ with CID-split needle alignment → `cid_split_mismatch` recorded, source unchanged + +## 6. Upstream-PR draft + +- [ ] 6.1 Update `docs/upstream-prs/07-tj-flattening/{proposal,design,tasks}.md` +- [ ] 6.2 Leave `Posted at: ` placeholder + +## 7. Quality gate + +- [ ] 7.1 PHP 7.4 compatibility +- [ ] 7.2 No new composer dependencies +- [ ] 7.3 snake_case discipline + +## 8. Commit + PR + +- [ ] 8.1 Commit on `feat/tj-flattening` +- [ ] 8.2 Open PR `feat/tj-flattening → work/text-replacement` — include before/after operator-sequence diff in the description diff --git a/openspec/changes/feat-tounicode-cmap/.openspec.yaml b/openspec/changes/feat-tounicode-cmap/.openspec.yaml new file mode 100644 index 0000000..fd706ac --- /dev/null +++ b/openspec/changes/feat-tounicode-cmap/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-26 diff --git a/openspec/changes/feat-tounicode-cmap/design.md b/openspec/changes/feat-tounicode-cmap/design.md new file mode 100644 index 0000000..9e36c36 --- /dev/null +++ b/openspec/changes/feat-tounicode-cmap/design.md @@ -0,0 +1,96 @@ +## Context + +A PDF's content stream emits text via operators like `Tj` (show string), `'` (next-line + show), `"` (next-line + show with spacing), and `TJ` (show string array with kerning). Each takes a PDF string (one or more bytes) as its operand. To know what the user sees, you must resolve the bytes through the currently active font's encoding: + +- **Simple fonts** (Type1, TrueType non-Identity): the encoding is a 1-byte-to-glyph-name map, then glyph name to Unicode via the font's `/ToUnicode` CMap (if present) or the glyph-name-to-Unicode mapping in Adobe Glyph List (AGL). +- **Composite fonts (Type0)** with `/Encoding /Identity-H` or `/Identity-V`: the encoding is "the 2-byte CID is identity-mapped to a 16-bit glyph index in the descendant font"; Unicode comes from `/ToUnicode` CMap exclusively. + +ToUnicode CMaps (Adobe Tech Note 5411) are PostScript-syntax streams declaring: + +- `bfchar ` blocks — direct 1-to-1 mappings. +- `bfrange ` blocks — contiguous code ranges. +- `bfrange [ ...]` blocks — contiguous codes with explicit per-code Unicode targets. +- Multi-codepoint Unicode targets (ligatures, decomposed characters) are encoded as hex strings of even length > 4. + +The forward direction (Unicode → CID sequence) is needed to emit the placeholder. ToUnicode CMaps only declare the reverse direction explicitly, but they're 1-to-1 mappings on the typical Word-emitted shapes, so the reverse is trivial to invert. Where the inversion is not 1-to-1 (different CIDs mapping to the same Unicode codepoint via combining-mark sequences), we pick the lowest-CID mapping and document the choice. + +## Goals / Non-Goals + +**Goals:** + +- Parse ToUnicode CMap streams correctly for the Adobe-Distiller / Word-emitted shape (covers >99% of in-the-wild Identity-H subset fonts). +- Build forward + reverse maps per font referenced by the page being processed. +- Match the needle in text space: concatenate the resolved Unicode characters from text-showing operators in source order, find the needle, identify the byte ranges in the content stream that contributed to the match. +- Replace the matched byte ranges with a placeholder encoded via the active font's forward map. If the forward map can't encode a placeholder character, emit a `font_encoding_misses` diagnostic and SKIP the substitution (no corruption — upstream-PR #08 handles the recovery). + +**Non-Goals:** + +- TJ kerning-array flattening — upstream-PR #07. +- Subset-font Helvetica fallback for unencodable placeholders — upstream-PR #08. +- Vertical writing mode (`Identity-V`) — same code path as `Identity-H` modulo glyph-position semantics; we handle the encoding identically. +- Right-to-left / bidi text — out of scope for the Woo use case. +- CIDFont-to-GID maps via `/CIDToGIDMap` — relevant for direct glyph-ID emission but not for our matching layer. + +## Decisions + +### D1 — Two new files, named after their domain + +- `src/CMap.php` — parser for the PostScript-syntax CMap streams. Builds both forward and reverse maps. Public methods: `CMap::fromStream(string $bytes): CMap`, `CMap::cidToUnicode(string $cidBytes): string`, `CMap::unicodeToCid(string $unicode): string|null` (null = unencodable). +- `src/FontEncoding.php` — implicit-encoding tables for `/WinAnsiEncoding`, `/MacRomanEncoding`, `/StandardEncoding`, plus Identity-H/V passthrough. Public methods: `FontEncoding::forName(string $name): FontEncoding`, `FontEncoding::byteToUnicode(int $byte): string`, `FontEncoding::unicodeToByte(string $unicode): int|null`. + +Both classes are pure value objects with no dependencies on `PDFDoc` / `PDFObject` — they take bytes in and return Unicode (or vice versa). Upstream-friendly shape. + +### D2 — Font resolver lives on `PDFDoc` + +Add `PDFDoc::resolveFontMap(int $oid, string $fontResourceName): array{forward: callable, reverse: callable, name: string}|null`. Caller passes the active page's font resource name (e.g. `F1` from `/F1 12 Tf`) plus the page object's OID; the resolver walks `/Resources/Font/`, locates the `/ToUnicode` stream (if any), parses it via `CMap::fromStream`, and falls back to `FontEncoding::forName($baseEncoding)` for simple fonts without a CMap. + +### D3 — Matching algorithm + +For each content stream we want to replace text in: + +1. Tokenise the stream's operator sequence (simple state machine: split on whitespace + operator names; preserve the operand sequences). This is the same parser depth we already need for `Tf` tracking in the PoC. +2. Walk the operator list, tracking the current font (`Tf` sets it). For each text-showing operator (`Tj`, `'`, `"`, and the string fragments inside `TJ`), record `(operator_index, source_byte_start, source_byte_end, resolved_unicode)`. +3. Concatenate the resolved Unicode in order. Find the needle's start + end position in the concatenated string. +4. Map the position back to a (start_operator_index, start_op_byte_offset, end_operator_index, end_op_byte_offset) span. +5. Emit the placeholder: encode the placeholder Unicode through the active font (resolved from the operator at the match's start). If the font's forward map can't encode every character of the placeholder, record `font_encoding_misses` and skip. +6. Splice the new operator sequence into the content stream, replacing all source bytes that contributed to the matched span. + +### D4 — Multi-font matches choose the start-font as authoritative + +If the matched text spans multiple `Tf` switches (rare but possible — "Jan" in one font, " " in another, "Jansen" in a third), we use the FIRST font in the span to encode the placeholder, emit a single `Tj` operator, and elide the intermediate `Tf` switches inside the match range. The remaining `Tf` switches outside the match are preserved. + +This choice is conservative: it always produces a syntactically valid content stream. Visual quality may degrade slightly (the placeholder uses one font even if the original text used several), but our placeholder format `[PERSON: 7]` is intentionally synthetic and doesn't pretend to match the document's typography. + +### D5 — Multi-codepoint ToUnicode targets are flattened to NFC + +ToUnicode CMaps can map a CID to a hex string of even length > 4, denoting a sequence of Unicode codepoints (ligatures: `fi` → `fi`; decomposed accented characters: `é` → `é`). We flatten these to NFC (Unicode Normalisation Form C) before matching. The needle is also NFC-normalised before the search. This handles the case where `fi` appears in the PDF but the operator searched for `fi`. + +### D6 — Diagnostic surface is purely additive + +`replaceTextInDocument` returns the same array shape as the PoC plus a new optional key `font_encoding_misses: array>`. Callers that don't inspect the new key see no behavioural change. + +## Risks / Trade-offs + +- **Risk**: ToUnicode CMap syntax has corners (nested `beginbfrange` blocks, comment lines, `usecmap` references to other CMaps). The Word-emitted shape uses only `beginbfchar` and `beginbfrange`. → **Mitigation**: implement only the Word-emitted shape; reject other shapes with `p_error` and a stream-unchanged failure. Document the gap. PRs `#07` / `#08` don't need the corners; we can fill them in later if production PDFs surface them. + +- **Risk**: Multi-codepoint ToUnicode targets and NFC normalisation interact badly with naive substring search (a 1-CID source could expand to N Unicode codepoints; a match boundary can split a CID). → **Mitigation**: index every text-showing operator output with `(source_byte_offset, unicode_codepoint_offset)` pairs; only allow matches that align with CID boundaries. Skip and diagnose a `cid_split_mismatch` if the match would cross a CID interior. + +- **Risk**: Subset fonts often omit Unicode characters that aren't in the original document. The placeholder `[PERSON: 7]` uses `[`, `]`, `:`, digits, and capital letters — common but not guaranteed. → **Mitigation**: emit `font_encoding_misses` for the unencodable characters; upstream-PR #08 adds the fallback. This PR ships a usable feature for documents where the subset font covers the placeholder. + +- **Trade-off**: Building per-page font resolution maps is non-trivial in PDF terms — fonts live in inheritable resources that can come from `/Resources` on the page itself, the page tree's `/Resources`, or a `/Form` XObject's `/Resources`. → **Mitigation**: walk the inheritance chain (page → parent pages → catalog) in `resolveFontMap`. Restrict to direct page-level resources for the first pass; widen if real-world fixtures need it. + +## Migration Plan + +The PoC's byte-level matcher continues to work on WinAnsi-encoded streams as a degenerate case (the FontEncoding tables for `/WinAnsiEncoding` are 1-byte = 1-codepoint, so text-space matching collapses to byte-space matching). No consumer code changes required. + +For OpenRegister: the `pdf-anonymisation` change's existing test fixture (`testdoc.pdf`, which uses Identity-H subset Helvetica) starts producing correct redacted output after this PR lands. + +Rollback: revert the commit. The PoC byte-level path is preserved as the fallback when the matched stream has no `Tf` operators (unusual but spec-valid; we treat the whole stream as byte-space). + +## Open Questions + +- **OQ1**: When a font has no `/ToUnicode` CMap AND no recognisable implicit encoding (custom `/Differences` array, custom CIDFont), should we ATTEMPT a best-effort match by treating bytes as Latin-1 (matches the original PoC behaviour) or skip the stream entirely? **Provisional**: skip the stream and emit a `font_encoding_unknown` diagnostic. False matches on unknown encodings could redact innocent bytes; safer to skip. + +- **OQ2**: Should we cache parsed CMaps across `replaceTextInDocument` calls? **Provisional**: yes, inside `PDFDoc` (per-document cache). CMaps are immutable for the document's lifetime; parsing them is O(N) in CMap stream size and a single Word document can reference 5-10 fonts. + +- **OQ3**: Identity-V — same code path as Identity-H? **Provisional**: yes, identical for the matching layer (the V/H difference is glyph-positioning, which we don't touch). Confirm with a real-world V fixture if one appears. diff --git a/openspec/changes/feat-tounicode-cmap/proposal.md b/openspec/changes/feat-tounicode-cmap/proposal.md new file mode 100644 index 0000000..9f6f569 --- /dev/null +++ b/openspec/changes/feat-tounicode-cmap/proposal.md @@ -0,0 +1,31 @@ +## Why + +The current PoC `replaceTextInDocument()` does byte-level literal matching on the decoded content stream. This works for the synthesised WinAnsi fixture but fails on every real-world Word-generated PDF, because Word emits subset embedded fonts with `/Encoding /Identity-H` and a `/ToUnicode` CMap. In those streams the bytes `Jan Jansen` never appear — what's emitted is a sequence of glyph IDs (CIDs) that only mean "Jan Jansen" once you resolve them through the font's `/ToUnicode` CMap. Without CMap resolution, the entire Woo use case (Word-generated Dutch government documents) is dead. + +This is the largest single piece of the text-replacement feature and the hardest to get right. It's also the gating dependency for any production rollout in OpenRegister. + +## What Changes + +- Add CMap parsing for `/ToUnicode` streams (PDF 1.7 §9.10.3 + Adobe ToUnicode CMap Tech Note 5411) plus the implicit encodings (`/WinAnsiEncoding`, `/MacRomanEncoding`, `/StandardEncoding`, `/Identity-H`, `/Identity-V`) for simple fonts. +- Build forward (text → CID-byte-sequence) and reverse (CID-byte-sequence → text) maps per font referenced by a page's `/Resources/Font`. +- Walk the content stream's text-showing operators (`Tj`, `'`, `"`, and `TJ` partially — full `TJ` is upstream-PR #07), track the current font from the `Tf` operator, and match the needle in TEXT SPACE (after CID→Unicode resolution), not byte space. +- Emit the placeholder in CID space using the FORWARD map of the currently active font. If the placeholder's Unicode characters can't be encoded via the active font's forward map, emit a `font_encoding_misses` diagnostic per substitution per stream (upstream-PR #08 adds the Helvetica fallback that recovers from this). +- Public API on `PDFDoc` stays the same shape (`replaceTextInDocument(array $substitutions): array`); the diagnostic surface gains `font_encoding_misses: array>`. + +## Capabilities + +### New Capabilities + +- `tounicode-cmap-resolution`: ToUnicode CMap parsing, font-encoding-aware glyph-text mapping for both Identity-H and simple-font encodings. + +### Modified Capabilities + +- `text-replacement`: switch the matching layer from byte-space to text-space; track active font across the content stream; emit placeholders via the active font's forward map. + +## Impact + +- **Touched files**: new `src/CMap.php` (~250 LOC for CMap parser + map builder), new `src/FontEncoding.php` (~150 LOC for implicit-encoding + Identity-H/V handling), `src/PDFObject.php` (helpers to fetch a font's CMap), `src/PDFDoc.php` (refactor `replaceTextInDocument` to walk text-showing operators and consult per-font maps). +- **Public API**: `replaceTextInDocument` signature unchanged; diagnostic shape grows additively. +- **Depends on**: `feat-filter-chain-dispatch` (CMap streams are usually FlateDecode-encoded — already supported on the PoC path). +- **Unblocks**: production Woo PDFs (Word-generated Identity-H subset fonts). +- **Upstream-PR draft**: `docs/upstream-prs/06-tounicode-cmap/`. diff --git a/openspec/changes/feat-tounicode-cmap/specs/text-replacement/spec.md b/openspec/changes/feat-tounicode-cmap/specs/text-replacement/spec.md new file mode 100644 index 0000000..fec03fc --- /dev/null +++ b/openspec/changes/feat-tounicode-cmap/specs/text-replacement/spec.md @@ -0,0 +1,124 @@ +**Status**: planned +**Scope**: change `feat-tounicode-cmap` (delta spec) +**OpenSpec changes**: +- `feat-tounicode-cmap` (in-progress) + +## Purpose + +Capability contract for `text-replacement` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/text-replacement/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: Replacement SHALL match the needle in text space, not byte space + +`PDFDoc::replaceTextInDocument(array $substitutions)` MUST resolve every text-showing operator's operand bytes through the font active at that operator's position, NFC-normalise both the resolved text and the needles, and search for matches in text space. A byte sequence whose Unicode resolution matches a needle MUST be replaced; a byte sequence that happens to contain the needle's literal bytes but resolves to different Unicode MUST NOT be replaced. + +#### Scenario: Identity-H subset font with a real-world Tj operator + +- WHEN the content stream contains `[(<0001000200030004>) Tj]` and the active font's `/ToUnicode` CMap maps CIDs 1..4 to `"J", "a", "n", " "` and 5..9 to "J", "a", "n", "s", "e", "n" +- AND the surrounding stream emits CIDs 5..9 as `<00050006000700080009>` immediately after +- AND the substitution is `{"Jan Jansen": "[PERSON: 7]"}` +- THEN `replaceTextInDocument` MUST replace the matched CID range with CID bytes encoding `[PERSON: 7]` via the active font's forward map + +#### Scenario: Byte-coincidence MUST NOT trigger a match + +- WHEN the content stream's raw bytes happen to contain the literal substring `"Jan Jansen"` but those bytes are CIDs that resolve to entirely different Unicode (e.g. "abcdefghij" in some custom encoding) +- AND the substitution is `{"Jan Jansen": "[PERSON: 7]"}` +- THEN `replaceTextInDocument` MUST NOT replace those bytes + +#### Scenario: Whitespace + NFC normalisation + +- WHEN the content stream emits `"Jan Jansen"` (non-breaking space U+00A0 between the words) and the substitution needle is `"Jan Jansen"` (regular space) +- THEN the match MUST NOT fire (NFC does not equate U+00A0 to U+0020). Operators wanting that match must supply the variant explicitly. + +#### Scenario: Ligature flattening via multi-codepoint ToUnicode target + +- WHEN the content stream emits a CID whose ToUnicode mapping is the "fi" ligature → `"fi"` and the surrounding text completes the word `"office"` +- AND the needle is `"office"` +- THEN the match MUST fire (the ligature CID is treated as the two-character sequence after NFC) + +### REQ-002: Placeholder SHALL be emitted via the active font's forward map + +The placeholder string for a successful match MUST be encoded through the font that was active at the START of the match. The resulting CID byte sequence MUST be wrapped in a `Tj` operator and spliced in place of the source operator(s) that contributed to the match. Intermediate `Tf` switches inside the match span MUST be elided. + +#### Scenario: Single-font match + +- WHEN the match span lies entirely within a single `Tf` scope and the active font's forward map can encode every character of the placeholder +- THEN the spliced content stream MUST contain a single `() Tj` operator at the match position + +#### Scenario: Multi-font match uses the start font + +- WHEN the match span crosses one or more `Tf` operators +- THEN the placeholder MUST be encoded via the font active at the START of the match +- AND the intermediate `Tf` operators inside the match span MUST be elided from the spliced stream +- AND `Tf` operators outside the match span MUST be preserved + +### REQ-003: Unencodable placeholders SHALL be diagnosed, not corrupted + +If the active font's forward map cannot encode every character of a placeholder, the substitution for that match MUST be skipped (source bytes left unchanged) and an entry MUST be added to the returned diagnostic surface under `font_encoding_misses[$oid][$needle] = $font_base_name`. The other substitutions in the same call MUST proceed normally. + +#### Scenario: Subset font can't encode `[` + +- WHEN the active font is a subset font whose forward map has no entry for `[` (U+005B) and the placeholder is `[PERSON: 7]` +- THEN the substitution at this match position MUST be SKIPPED (source bytes unchanged) +- AND the returned diagnostic surface MUST include `font_encoding_misses[$oid]["Jan Jansen"] = ""` +- AND `streams_modified` MUST NOT count this stream +- AND other matches in other streams MUST be unaffected + +### REQ-004: Cross-CID-boundary matches SHALL NOT corrupt the stream + +If a needle's match would start or end in the INTERIOR of a multi-codepoint ToUnicode mapping (i.e. the match boundary splits a single CID's resolved text), the substitution at that match position MUST be skipped and a `cid_split_mismatch` diagnostic MUST be added to the returned surface. + +#### Scenario: Match would split a ligature CID + +- WHEN a CID resolves to `"fi"` (two codepoints) and the needle is `"f"` (matches only the first codepoint of the CID) +- THEN the substitution MUST NOT fire +- AND the returned diagnostic surface MUST include `cid_split_mismatch[$oid]["f"]` with the offending CID position + +## MODIFIED Requirements + +### REQ-001: replaceTextInDocument diagnostic surface + +`PDFDoc::replaceTextInDocument(array $substitutions)` MUST return an array with these keys (additive growth from the PoC's shape): + +- `streams_scanned: int` +- `streams_modified: int` +- `replacements_per_needle: array` +- `unmatched_needles: string[]` +- `font_encoding_misses: array>` (new) +- `cid_split_mismatch: array>` (new) + +#### Scenario: WinAnsi-only fixture (PoC) still emits the original shape with empty additions + +- WHEN `replaceTextInDocument(['Jan Jansen' => '[PERSON: 7]'])` is called on the WinAnsi PoC fixture +- THEN `streams_modified` MUST equal 1 +- AND `replacements_per_needle['Jan Jansen']` MUST equal 1 +- AND `font_encoding_misses` MUST be an empty array (no encoding issues on WinAnsi) +- AND `cid_split_mismatch` MUST be an empty array diff --git a/openspec/changes/feat-tounicode-cmap/specs/tounicode-cmap-resolution/spec.md b/openspec/changes/feat-tounicode-cmap/specs/tounicode-cmap-resolution/spec.md new file mode 100644 index 0000000..6fdddc8 --- /dev/null +++ b/openspec/changes/feat-tounicode-cmap/specs/tounicode-cmap-resolution/spec.md @@ -0,0 +1,117 @@ +**Status**: planned +**Scope**: change `feat-tounicode-cmap` (delta spec) +**OpenSpec changes**: +- `feat-tounicode-cmap` (in-progress) + +## Purpose + +Capability contract for `tounicode-cmap-resolution` — the normative SHALL/MUST +behaviour the change must deliver. Scenarios below are the +acceptance criteria; tasks under the change's `tasks.md` reference +these requirements by REQ-NNN id. + +## Non-Functional Requirements + +- PHP >= 7.4 compatibility per upstream sapp's composer constraint. +- Zero new composer dependencies. +- snake_case method names on new utility helpers; PascalCase on + filter names (matches the existing `FlateDecode` convention). +- All round-trip scenarios MUST be lossless byte-for-byte unless + the spec explicitly documents a deviation. + +## Acceptance Criteria + +- Every requirement below MUST be exercised by at least one scenario + in the change's verification gate under `examples/`. +- The change's `tasks.md` MUST cite each REQ-NNN it implements. +- Existing verification gates (PoC and prior changes) MUST remain + green after the change lands. + +## Notes + +This is a delta spec — the canonical spec will be at +`openspec/specs/tounicode-cmap-resolution/spec.md` after `/opsx-archive`. The +delta operations below (`## ADDED Requirements`, `## MODIFIED +Requirements`) are merged into the canonical spec by the archiver. + +## ADDED Requirements + +### REQ-001: CMap parser SHALL handle Word-emitted ToUnicode CMap shape + +The parser MUST recognise `beginbfchar`/`endbfchar` blocks and `beginbfrange`/`endbfrange` blocks per Adobe Tech Note 5411. It MUST produce a forward map (Unicode → CID-byte-sequence) and a reverse map (CID-byte-sequence → Unicode) from the parsed entries. + +#### Scenario: bfchar block + +- WHEN the CMap stream contains `2 beginbfchar <0041> <0041> <0042> <0042> endbfchar` (CID 0x41 → U+0041 'A', CID 0x42 → U+0042 'B') +- THEN the reverse map MUST return `"A"` for input `"\x00\x41"` and `"B"` for input `"\x00\x42"` +- AND the forward map MUST return `"\x00\x41"` for input `"A"` and `"\x00\x42"` for input `"B"` + +#### Scenario: bfrange block with starting Unicode + +- WHEN the CMap stream contains `1 beginbfrange <0041> <005A> <0041> endbfrange` (CIDs 0x41..0x5A → U+0041..U+005A, the ASCII uppercase range) +- THEN the reverse map MUST return `"M"` for input `"\x00\x4D"` +- AND the forward map MUST return `"\x00\x4D"` for input `"M"` + +#### Scenario: bfrange block with explicit Unicode array + +- WHEN the CMap stream contains `1 beginbfrange <0001> <0003> [<0041> <0042> <0043>] endbfrange` (CIDs 1..3 → "A", "B", "C") +- THEN the reverse map MUST return `"B"` for input `"\x00\x02"` + +#### Scenario: Multi-codepoint Unicode target + +- WHEN the CMap stream contains `1 beginbfchar <0050> <00660069> endbfchar` (CID 0x50 → "fi" ligature, encoded as U+0066 U+0069) +- THEN the reverse map MUST return the NFC-normalised string `"fi"` for input `"\x00\x50"` + +### REQ-002: CMap parser SHALL fail safely on unsupported syntax + +Inputs containing unsupported constructs (nested CMap references via `usecmap`, `begincidrange` for CIDFont selection, etc.) MUST cause `p_error()` and the parser MUST return a null/empty CMap. Callers MUST be able to detect the failure mode without exception handling. + +#### Scenario: Unsupported usecmap directive + +- WHEN the CMap stream contains `/Identity usecmap` +- THEN the parser MUST call `p_error()` identifying the unsupported directive +- AND the resulting CMap MUST treat all inputs as unencodable + +### REQ-003: Implicit font encodings SHALL be honoured for simple fonts + +When a font has no `/ToUnicode` CMap, `FontEncoding::forName($encodingName)` MUST return a working encoding for `/WinAnsiEncoding`, `/MacRomanEncoding`, `/StandardEncoding`, `/Identity-H`, `/Identity-V`. Unrecognised names MUST return a null/empty encoding (callers diagnose). + +#### Scenario: WinAnsiEncoding byte-to-unicode + +- WHEN the encoding is `/WinAnsiEncoding` and the input byte is 0x41 +- THEN `byteToUnicode(0x41)` MUST return `"A"` + +#### Scenario: WinAnsiEncoding round-trip on common characters + +- WHEN the encoding is `/WinAnsiEncoding` and the input character is any printable ASCII character (0x20..0x7E) +- THEN `unicodeToByte(byteToUnicode($b))` MUST equal `$b` for every byte in that range + +#### Scenario: Identity-H passthrough + +- WHEN the encoding is `/Identity-H` and the input byte sequence is `"\x00\x41"` (2-byte CID 0x41) +- THEN `byteToUnicode` is undefined (Identity-H requires a ToUnicode CMap) +- AND `FontEncoding::isIdentityH()` MUST return `true` + +### REQ-004: Font resolution SHALL walk the page-tree resource inheritance + +`PDFDoc::resolveFontMap(int $pageOid, string $resourceName)` MUST return the font's forward + reverse maps for the named font resource. It MUST search the page's `/Resources`, then the page's parent pages' `/Resources`, then the document catalog's `/Resources` (PDF 1.7 §7.5.4 inheritance). + +#### Scenario: Font on the page itself + +- WHEN the page object's `/Resources/Font/F1` references a font with a `/ToUnicode` CMap +- THEN `resolveFontMap($pageOid, 'F1')` MUST return a non-null result with `forward` + `reverse` callables and the font's `/BaseFont` value as `name` + +#### Scenario: Font inherited from a parent page node + +- WHEN a page does NOT have `/Resources` but its parent does, and the parent's `/Resources/Font/F1` is a valid font +- THEN `resolveFontMap($pageOid, 'F1')` MUST resolve via inheritance and return the font + +#### Scenario: Unknown font resource name + +- WHEN the page has no `/Resources/Font/F99` (and no ancestor has it either) +- THEN `resolveFontMap($pageOid, 'F99')` MUST return `null` + +#### Scenario: Parsed CMaps are cached + +- WHEN `resolveFontMap` is called twice with the same arguments +- THEN the second call MUST NOT re-parse the CMap stream (verifiable via spy on the parser's invocation count) diff --git a/openspec/changes/feat-tounicode-cmap/tasks.md b/openspec/changes/feat-tounicode-cmap/tasks.md new file mode 100644 index 0000000..75a71a2 --- /dev/null +++ b/openspec/changes/feat-tounicode-cmap/tasks.md @@ -0,0 +1,81 @@ +## 1. Pre-flight + +- [ ] 1.1 Confirm `feat-filter-chain-dispatch` is merged into `work/text-replacement` +- [ ] 1.2 Branch off as `feat/tounicode-cmap` +- [ ] 1.3 Capture a real-world Word-generated Woo-style fixture (Identity-H subset Helvetica, contains "Jan Jansen" or equivalent test name) and check it into `examples/poc-fixture-identity-h.pdf` +- [ ] 1.4 Verify the fixture's content stream is byte-level UN-matchable (today's PoC produces malformed output on it — confirm the failure mode for the regression-test baseline) +- [ ] 1.5 Re-read PDF 1.7 §9.10.3 (`/ToUnicode`) + Adobe Tech Note 5411 + +## 2. CMap parser — `src/CMap.php` + +- [ ] 2.1 Add a `CMap` class with `private $forward = []` and `private $reverse = []` arrays +- [ ] 2.2 Implement `static fromStream(string $bytes): CMap` — tokenise the PostScript syntax; handle whitespace + comments +- [ ] 2.3 Parse `beginbfchar`/`endbfchar` blocks (REQ "bfchar block") +- [ ] 2.4 Parse `beginbfrange`/`endbfrange` blocks with starting Unicode (REQ "bfrange ... starting Unicode") +- [ ] 2.5 Parse `beginbfrange`/`endbfrange` blocks with explicit Unicode array (REQ "bfrange ... explicit array") +- [ ] 2.6 Parse multi-codepoint Unicode targets, NFC-normalise on insert (REQ "Multi-codepoint") +- [ ] 2.7 `cidToUnicode(string $cidBytes): string` lookup +- [ ] 2.8 `unicodeToCid(string $unicode): string|null` lookup (null = unencodable) +- [ ] 2.9 Fail-safe on unsupported directives (`usecmap`, `begincidrange`, etc.) — `p_error` + return null CMap (REQ "fail safely") + +## 3. Font encoding — `src/FontEncoding.php` + +- [ ] 3.1 Add a `FontEncoding` class with `byteToUnicode(int $byte): string` and `unicodeToByte(string $unicode): int|null` +- [ ] 3.2 Bake in `/WinAnsiEncoding`, `/MacRomanEncoding`, `/StandardEncoding` tables (per Adobe spec Appendix D) +- [ ] 3.3 Add `isIdentityH(): bool` + `isIdentityV(): bool` discriminators +- [ ] 3.4 `static forName(string $encodingName): FontEncoding` factory; null/empty encoding on unknown name + +## 4. Font resolution — `PDFDoc::resolveFontMap` + +- [ ] 4.1 Add `private array $fontMapCache = []` for the per-document parsed-CMap cache (OQ2) +- [ ] 4.2 Add `public function resolveFontMap(int $pageOid, string $resourceName): ?array` returning `['forward' => callable, 'reverse' => callable, 'name' => string]` +- [ ] 4.3 Walk the page-tree `/Resources` inheritance chain (REQ "inheritance") +- [ ] 4.4 If the font has `/ToUnicode`, decode the stream via `get_stream(false)` (uses the existing filter chain dispatch) and parse via `CMap::fromStream` +- [ ] 4.5 If no `/ToUnicode`, fall back to `FontEncoding::forName($font['Encoding'])` +- [ ] 4.6 Compose the `forward` + `reverse` callables that paper over the CMap-vs-encoding API difference + +## 5. Content-stream operator tokeniser + +- [ ] 5.1 Add `protected static function tokeniseOperators(string $stream): array` in `PDFObject` — split a content stream into `(operator_name, operand_bytes, source_byte_start, source_byte_end)` tuples +- [ ] 5.2 Handle string literals (`(...)` with escape sequences), hex strings (`<...>`), arrays (`[...]`), and operators +- [ ] 5.3 Track current font from `Tf` operators +- [ ] 5.4 Test against the existing PoC fixture and a small hand-crafted multi-`Tf` fixture + +## 6. Text-space matching — refactor `PDFDoc::replaceTextInDocument` + +- [ ] 6.1 For each FlateDecode-or-chain-decode-able content stream containing text-showing operators: tokenise, build a (resolved_unicode, source_byte_range) index +- [ ] 6.2 NFC-normalise the concatenated text and each needle +- [ ] 6.3 Search for matches per substitution; map back to source byte ranges +- [ ] 6.4 Reject matches that cross CID interiors (REQ "Cross-CID-boundary") +- [ ] 6.5 For each accepted match, encode the placeholder via the start-font's forward map; if any character is unencodable, skip + record `font_encoding_misses` (REQ "Unencodable placeholders") +- [ ] 6.6 Elide intermediate `Tf` operators inside the match span (D4) +- [ ] 6.7 Splice the new content stream and register the modified object via `$this->_pdf_objects[$oid] = $obj` (same write-back fix as the PoC) +- [ ] 6.8 Grow the diagnostic surface with `font_encoding_misses` + `cid_split_mismatch` + +## 7. Tests / verification + +- [ ] 7.1 Unit tests for `CMap` covering each bfchar / bfrange shape from the spec +- [ ] 7.2 Unit tests for `FontEncoding` round-trip on WinAnsi printable ASCII range +- [ ] 7.3 Round-trip test: load `examples/poc-fixture-identity-h.pdf`, call `replaceTextInDocument`, re-extract via SAPP, assert no residual needle + ≥1 placeholder hit +- [ ] 7.4 PoC regression: `examples/poc-replace-text.php` (WinAnsi fixture) still exits 0 with no `font_encoding_misses` and no `cid_split_mismatch` +- [ ] 7.5 Negative test: subset font with placeholder character missing → `font_encoding_misses` populated, source bytes unchanged +- [ ] 7.6 Negative test: ligature-CID needle alignment → `cid_split_mismatch` populated +- [ ] 7.7 Spot-check the output PDF in a real viewer (Adobe Reader, Firefox, pdf.js) + +## 8. Upstream-PR draft + +- [ ] 8.1 Update `docs/upstream-prs/06-tounicode-cmap/{proposal,design,tasks}.md` +- [ ] 8.2 Document the new files (`CMap.php`, `FontEncoding.php`) explicitly — upstream will scrutinise new top-level types +- [ ] 8.3 Leave `Posted at: ` placeholder + +## 9. Quality gate + +- [ ] 9.1 PHP 7.4 compatibility — no enums, no typed properties (use `@var` PHPDoc on private fields) +- [ ] 9.2 No new composer dependencies +- [ ] 9.3 PSR-12 + snake_case discipline on new methods (PascalCase on class names) +- [ ] 9.4 Confirm `CMap` and `FontEncoding` have no dependencies on `PDFDoc` / `PDFObject` (clean value objects, easier upstream review) + +## 10. Commit + PR + +- [ ] 10.1 Commit on `feat/tounicode-cmap` — likely 2-3 commits to keep review-able (parser, encoding, integration) +- [ ] 10.2 Open PR `feat/tounicode-cmap → work/text-replacement` — flag the two new files and the matching-layer refactor in the description diff --git a/openspec/config.yaml b/openspec/config.yaml new file mode 100644 index 0000000..6860c55 --- /dev/null +++ b/openspec/config.yaml @@ -0,0 +1,49 @@ +schema: spec-driven + +context: | + Project: SAPP (Simple and Agnostic PDF Parser) + Repo: ConductionNL/sapp + Upstream: dealfonso/sapp + Type: PHP library (NOT a Nextcloud app) + Long-standing integration branch: work/text-replacement + License: LGPL-3.0-or-later + + This fork exists to add text-replacement primitives that upstream + doesn't currently expose. Each change in this folder corresponds to + one PR that we intend to contribute back to dealfonso/sapp once the + text-replacement feature is stable in our consumer (OpenRegister's + PDF anonymisation pipeline). The upstream-PR drafts staged under + `docs/upstream-prs/01-asciihex-decode/` ... `08-text-replacement-api/` + are the eventual-upstream-submission artefacts; the OpenSpec changes + in this folder are the implementation contracts that drive the work + on the fork first. + + Time-pressure decision: we ship in this fork first (consumed by + OpenRegister via a composer VCS repository pointing at the + `work/text-replacement` branch), then create issues + PRs to + dealfonso/sapp after the feature stabilises. The fork-and-give-back + ordering matters because the OpenRegister anonymisation feature has + external deadlines that upstream review cycles cannot meet. + +rules: + proposal: + - State the PDF 1.7 specification section the change implements (e.g. §7.4.2 for ASCIIHexDecode) + - Document deviations from the spec explicitly with rationale (none expected — this is a spec-faithful library) + - Flag any cross-PR dependencies in the upstream-PR series ordering (e.g. #01 depends on #05 filter-chain dispatch) + - Note the matching upstream-PR-draft folder under `docs/upstream-prs/` + specs: + - Frame requirements with the upstream-acceptance lens — anything a SAPP maintainer would reject blocks the proposal + - Include normative roundtrip scenarios (encode→decode→encode must be lossless within the spec's defined tolerance) + - Cover predictor / EOD / variable-width / chained-filter edge cases explicitly when the filter supports them + design: + - PSR-12 coding style — matches upstream sapp's existing conventions + - PHP >= 7.4 compatible — matches upstream's `composer.json` constraint (do NOT raise the minimum) + - No external composer dependencies — the library is intentionally dependency-free; any new dep would block the upstream PR + - Method signatures follow upstream's snake_case convention (e.g. `to_pdf_file_b`, `get_stream`) — do not introduce camelCase + - PDFObject / PDFDoc / PDFValue* class boundaries from upstream are load-bearing — do not refactor across them in any single PR + - Each change SHOULD correspond to exactly one upstream PR — keeps the eventual contribution series clean + tasks: + - Include round-trip fixture tests where applicable (encode→decode against a hand-crafted reference output) + - Verify changes don't break existing test fixtures in `examples/` (testdoc.pdf, poc-fixture.pdf) + - Update the matching `docs/upstream-prs/NN-*/` draft folder with implementation notes worth surfacing in the eventual upstream PR + - Run `php examples/poc-replace-text.php` (when applicable) — must exit 0 with `residual_needles=0` diff --git a/openspec/specs/.gitkeep b/openspec/specs/.gitkeep new file mode 100644 index 0000000..e69de29