Summary
On macOS, running /graphify --update repeatedly re-extracts already-processed Office files (.docx, .xlsx) even when no content has changed. Each
run creates new .md sidecar files with different hash suffixes in /graphify-out/converted/, making --update treat them as new files and triggering a
full re-extraction every time.
Environment
- graphifyy version: 0.5.4
- Python: 3.13.9
- OS: macOS 15.6.1 Sequoia, APFS filesystem
Steps to Reproduce
1. Initial build with a folder containing .docx files
/graphify raw
2. Run --update without changing any files
/graphify raw --update
3. Run --update again
/graphify raw --update
Expected: Step 3 prints "No files changed since last run. Nothing to update."
Actual: Every run treats existing .docx files as new and re-extracts all of them.
Root Cause
The issue is in convert_office_file() in graphify/detect.py:
current code
name_hash = hashlib.sha256(str(path.resolve()).encode()).hexdigest()[:8]
out_path = out_dir / f"{path.stem}_{name_hash}.md"
On macOS (HFS+/APFS), path.resolve() returns paths in different Unicode normalization forms depending on how the Path object was constructed:
┌──────────────────────┬───────── ┬───────┐
│ How the path was created │ Unicode form │ Example hash │
├──────────────────────┼──────────┬───────┤
│ Via root.rglob('*') — OS filesystem enumeration │ NFD (macOS default) │ df70b1b3 │
├──────────────────────┼──────────┼───────┤
│ Via Path('/absolute/path') — direct construction │ NFC (Python default) │ a54fe208 │
└──────────────────────┴──────────┴───────┘
detect() enumerates files via rglob(), so it uses NFD paths. However, the normalization form of str(path.resolve()) can vary between runs depending on the
Python version, interpreter context, or OS state. The same .docx file can therefore produce a different hash on every run.
Observed result — duplicate sidecars accumulate:
raw/graphify-out/converted/
report_b2b2ed3e.md ← run 1
report_df70b1b3.md ← run 2 (same source file, different hash)
report_a54fe208.md ← run 3 (same source file, yet another hash)
Secondary issue — manifest mismatch:
detect_incremental() compares the manifest (which stores the previous run's file paths) against the current detect() output. Since the hash-based
filenames differ between runs:
- 200 old-hash files in the manifest → reported as "deleted" (they still exist on disk)
- 147 new-hash files from the current run → reported as "new" → full re-extraction triggered
Confirmed with a concrete test — hashing the same absolute path with NFC vs NFD:
import hashlib, unicodedata
path = "/Users/.../raw/docs/2024/report_한글파일명.docx"
hashlib.sha256(unicodedata.normalize('NFC', path).encode()).hexdigest()[:8]
→ 'a54fe208'
hashlib.sha256(unicodedata.normalize('NFD', path).encode()).hexdigest()[:8]
→ 'df70b1b3' ← matches today's converted file
Proposed Fix
Normalize the path to NFC before hashing in convert_office_file():
graphify/detect.py
import unicodedata
def convert_office_file(path: Path, out_dir: Path) -> Path | None:
...
# Normalize to NFC before hashing to ensure consistent results
# across macOS NFC/NFD Unicode path encoding variations
normalized_path = unicodedata.normalize('NFC', str(path.resolve()))
name_hash = hashlib.sha256(normalized_path.encode()).hexdigest()[:8]
out_path = out_dir / f"{path.stem}_{name_hash}.md"
# Early return if sidecar already exists — avoids re-conversion entirely
if out_path.exists():
return out_path
...
Two changes:
- Normalize to NFC before hashing — makes the hash deterministic across all macOS path construction methods.
- Early return if the sidecar already exists — once the hash is stable, this prevents unnecessary re-conversion even if the manifest drifts.
Workaround
Until a fix is released, after each --update run that processes new files (i.e. when new_total > 0), overwrite the manifest with a far-future mtime so
existing files are never flagged as changed again:
import json
from graphify.detect import detect
from pathlib import Path
PERMANENT = 253402300799.0 # Unix timestamp for year 9999
full = detect(Path('raw'))
all_files = [f for files in full['files'].values() for f in files]
Path('graphify-out/manifest.json').write_text(
json.dumps({f: PERMANENT for f in all_files}, indent=2)
)
This is safe for append-only raw/ folders: new files not yet in the manifest will still be detected correctly (their stored_mtime will be None), while
existing files will never be re-extracted.
Impact
- Affects all macOS users running --update on corpora with .docx or .xlsx files
- Every --update run triggers a full re-extraction regardless of actual changes
- Duplicate sidecar files accumulate in graphify-out/converted/ on every run
- Unnecessary LLM API token consumption and significantly increased run time
Summary
On macOS, running /graphify --update repeatedly re-extracts already-processed Office files (.docx, .xlsx) even when no content has changed. Each
run creates new .md sidecar files with different hash suffixes in /graphify-out/converted/, making --update treat them as new files and triggering a
full re-extraction every time.
Environment
Steps to Reproduce
1. Initial build with a folder containing .docx files
/graphify raw
2. Run --update without changing any files
/graphify raw --update
3. Run --update again
/graphify raw --update
Expected: Step 3 prints "No files changed since last run. Nothing to update."
Actual: Every run treats existing .docx files as new and re-extracts all of them.
Root Cause
The issue is in convert_office_file() in graphify/detect.py:
current code
name_hash = hashlib.sha256(str(path.resolve()).encode()).hexdigest()[:8]
out_path = out_dir / f"{path.stem}_{name_hash}.md"
On macOS (HFS+/APFS), path.resolve() returns paths in different Unicode normalization forms depending on how the Path object was constructed:
┌──────────────────────┬───────── ┬───────┐
│ How the path was created │ Unicode form │ Example hash │
├──────────────────────┼──────────┬───────┤
│ Via root.rglob('*') — OS filesystem enumeration │ NFD (macOS default) │ df70b1b3 │
├──────────────────────┼──────────┼───────┤
│ Via Path('/absolute/path') — direct construction │ NFC (Python default) │ a54fe208 │
└──────────────────────┴──────────┴───────┘
detect() enumerates files via rglob(), so it uses NFD paths. However, the normalization form of str(path.resolve()) can vary between runs depending on the
Python version, interpreter context, or OS state. The same .docx file can therefore produce a different hash on every run.
Observed result — duplicate sidecars accumulate:
raw/graphify-out/converted/
report_b2b2ed3e.md ← run 1
report_df70b1b3.md ← run 2 (same source file, different hash)
report_a54fe208.md ← run 3 (same source file, yet another hash)
Secondary issue — manifest mismatch:
detect_incremental() compares the manifest (which stores the previous run's file paths) against the current detect() output. Since the hash-based
filenames differ between runs:
Confirmed with a concrete test — hashing the same absolute path with NFC vs NFD:
import hashlib, unicodedata
path = "/Users/.../raw/docs/2024/report_한글파일명.docx"
hashlib.sha256(unicodedata.normalize('NFC', path).encode()).hexdigest()[:8]
→ 'a54fe208'
hashlib.sha256(unicodedata.normalize('NFD', path).encode()).hexdigest()[:8]
→ 'df70b1b3' ← matches today's converted file
Proposed Fix
Normalize the path to NFC before hashing in convert_office_file():
graphify/detect.py
import unicodedata
def convert_office_file(path: Path, out_dir: Path) -> Path | None:
...
# Normalize to NFC before hashing to ensure consistent results
# across macOS NFC/NFD Unicode path encoding variations
normalized_path = unicodedata.normalize('NFC', str(path.resolve()))
name_hash = hashlib.sha256(normalized_path.encode()).hexdigest()[:8]
out_path = out_dir / f"{path.stem}_{name_hash}.md"
Two changes:
Workaround
Until a fix is released, after each --update run that processes new files (i.e. when new_total > 0), overwrite the manifest with a far-future mtime so
existing files are never flagged as changed again:
import json
from graphify.detect import detect
from pathlib import Path
PERMANENT = 253402300799.0 # Unix timestamp for year 9999
full = detect(Path('raw'))
all_files = [f for files in full['files'].values() for f in files]
Path('graphify-out/manifest.json').write_text(
json.dumps({f: PERMANENT for f in all_files}, indent=2)
)
This is safe for append-only raw/ folders: new files not yet in the manifest will still be detected correctly (their stored_mtime will be None), while
existing files will never be re-extracted.
Impact