Skip to content

Optimize incremental sync and harden cache logic#108

Merged
wesm merged 5 commits intomainfrom
misc-fixes
Feb 8, 2026
Merged

Optimize incremental sync and harden cache logic#108
wesm merged 5 commits intomainfrom
misc-fixes

Conversation

@wesm
Copy link
Owner

@wesm wesm commented Feb 8, 2026

Summary

  • Optimize incremental sync for mass deletions: Apply label diffs directly from history records instead of re-fetching full messages from Gmail API, batch message fetches, and batch deletion markers. Eliminates per-message API calls for label changes on existing messages (the main bottleneck when deleting large numbers of emails).
  • Set maxResults=500 on History API: Reduces pagination round-trips from ~482 to ~97 for large history gaps, cutting scan time from ~54s to ~11s.
  • Fix cacheNeedsBuild for zero-message accounts: Prevent rebuild loop when DB and sync state both agree there are 0 messages.
  • Add comprehensive TestCacheNeedsBuild: 10 table-driven test cases covering all branches (zero-message, missing state, new messages, up-to-date, empty cache, soft-deleted, invalid state, DB failure, missing required tables).
  • Add .claude/ to .gitignore

Test plan

  • All existing incremental sync tests pass
  • New tests: label add/remove on existing messages (zero API calls), batch deletions, batch new messages, mixed operations
  • New store tests: AddMessageLabels, RemoveMessageLabels, MarkMessagesDeletedBatch
  • New TestCacheNeedsBuild with 10 cases covering all branches
  • go test ./internal/sync/ ./internal/store/ ./cmd/msgvault/cmd/ -count=1
  • Manual test with real Gmail account (48K history records)

🤖 Generated with Claude Code

wesm and others added 5 commits February 8, 2026 06:54
…assertion

Make cacheNeedsBuild() zero-message aware: when both the DB and sync
state report 0 messages, skip the rebuild instead of triggering one on
every TUI start (zero-message accounts never produce message parquet).

Relax the attachment filename test to assert valid UTF-8 + prefix/suffix
rather than an exact replacement character, matching the defense-in-depth
intent without coupling to enmime's specific sanitization behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate per-message API calls for label changes on existing messages by
applying label diffs directly from history records (AddMessageLabels /
RemoveMessageLabels). Batch new message fetches via GetMessagesRawBatch
and deletions via MarkMessagesDeletedBatch. For 450K deleted emails this
changes the bottleneck from ~0.3 msg/s (one API call per label change)
to local-only DB operations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add table-driven TestCacheNeedsBuild covering all branches: zero-message
accounts, missing state files, new messages, up-to-date cache, empty
parquet directories, and soft-deleted messages. Also add .claude/ to
.gitignore.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cover invalid sync state JSON, DB open failure, and missing required
parquet tables branches that were untested.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ListHistory was using Gmail's default page size of 100, requiring ~482
HTTP calls for 48K history records. Setting maxResults=500 (same as
ListMessages) reduces this to ~97 calls, cutting scan time from ~54s
to ~11s.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@wesm wesm changed the title Fix zero-message cache rebuild loop and relax brittle test assertion Optimize incremental sync and harden cache logic Feb 8, 2026
@wesm
Copy link
Owner Author

wesm commented Feb 8, 2026

This speeds up incremental sync a LOT after bulk deletions (I deleted more than 500K e-mails so stumbled on the need to do this)

@wesm wesm merged commit fc27abd into main Feb 8, 2026
3 checks passed
wesm added a commit that referenced this pull request Feb 8, 2026
Re-apply the NotFoundError handling in GetMessagesRawBatch that was lost
during the squash merge of PR #108: messages deleted between history scan
and fetch are expected during incremental sync, so log at debug instead
of warn. Add TestGetMessagesRawBatch_LogLevels to verify 404s produce
debug logs while other errors produce warn logs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant