fix: harden adaptive replay evidence - #2
Closed
hhh2210 wants to merge 2 commits into
Closed
Conversation
There was a problem hiding this comment.
Pull request overview
This PR hardens the fork-only adaptive replay harness by (1) segmenting legacy traces to avoid synthesizing timer ticks across long unobserved gaps, (2) auditing recorded live schedule evaluations separately from counterfactual replay advances, and (3) adding an offline “coding activity cap” candidate plus metrics/tests to support evidence gathering without changing production policy behavior.
Changes:
- Add legacy gap segmentation and a segmented replay path that reports excluded wall time and segment counts.
- Introduce
timerAdvanceEvaluatedtrace records +RecordedScheduleAuditorto validate live schedule comparisons (payload, menu linkage, multiplicity, ambiguity, sub-second lead). - Add coding-activity signal plumbed into replay input/metrics and a fork-only
CodingActivityAdaptivePolicy, with focused regression tests and updated docs/CLI output.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| Tests/CodexBarTests/ReplayTraceSegmentationTests.swift | Adds regression coverage for gap segmentation behavior and segmented rate math. |
| Tests/CodexBarTests/RecordedScheduleAuditTests.swift | Adds coverage for live schedule audit rules (legacy vs evaluated, linkage, multiplicity, fractional lead). |
| Tests/CodexBarTests/CodingActivityReplayPolicyTests.swift | Adds replay-only coverage for the activity-cap policy and compliance counters. |
| Tests/CodexBarTests/ClaudeOAuthCredentialsStoreCLIStorageOwnershipTests.swift | Updates test overrides to isolate Claude keychain ownership deterministically. |
| Tests/CodexBarTests/AdaptiveRefreshTraceRecordingTests.swift | Updates app-side trace recording expectations to include evaluations + accepted-only advances. |
| Sources/CodexBar/UsageStore.swift | Records every schedule evaluation and only records timerAdvanced when accepted. |
| Sources/CodexBar/AdaptiveRefreshTraceRecording.swift | Adds recordTimerAdvanceEvaluation(...) writer entrypoint. |
| Sources/AdaptiveReplayKit/ReplayTraceSegmentation.swift | Implements deadline-overrun gap segmentation + segmented replay aggregation. |
| Sources/AdaptiveReplayKit/ReplayPolicy.swift | Extends replay inputs with lastCodingActivityAt and derives activity age. |
| Sources/AdaptiveReplayKit/ReplayMetrics.swift | Adds activity compliance and segmentation-related metrics + staleness helper init. |
| Sources/AdaptiveReplayKit/ReplayEngine.swift | Adds activity signal reconstruction, compliance counting, and staleness sample plumbing. |
| Sources/AdaptiveReplayKit/RecordedScheduleAudit.swift | Adds the standalone recorded-schedule audit and mismatch accounting. |
| Sources/AdaptiveReplayKit/README.md | Documents segmentation, evaluation vs advances, and recorded-schedule auditing. |
| Sources/AdaptiveReplayKit/CandidatePolicies.swift | Adds the fork-only coding-activity capped adaptive candidate. |
| Sources/AdaptiveReplayKit/AdaptiveRefreshTrace.swift | Extends trace schema with timerAdvanceEvaluated and lead/acceptance metadata. |
| Sources/AdaptiveReplayCLI/main.swift | Adds CLI segmentation controls, activity/comparison reporting, and audit output. |
| docs/predictive-refresh-policy.md | Updates decision record status and appends local replay evidence/limitations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+67
to
+69
| if let last = currentRecords.last { | ||
| segments.append(ReplayTraceSegment(records: currentRecords, start: currentStart, end: last.timestamp)) | ||
| } |
Comment on lines
+5
to
+7
| public struct CodingActivityAdaptivePolicy: ReplayPolicy, Sendable { | ||
| public let name = "adaptive-activity-cap" | ||
| public let advancesOnInteraction = true |
Comment on lines
+241
to
244
| private static func lastActivity(_ samples: [ActivityObservation], at time: Date) -> Date? { | ||
| guard let index = self.lastIndex(samples.map(\.timestamp), atOrBefore: time) else { return nil } | ||
| return samples[index].lastCodingActivityAt | ||
| } |
Owner
Author
|
Superseded by the upstream Draft RFC: steipete#2029. Closing this fork-internal PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scope and non-goals
This PR targets the fork's
adaptive-replay-harnessbranch. It does not change the productionAdaptiveRefreshPolicy, make Adaptive the default, or feed coding activity into the running app's policy. The activity candidate exists only in offline replay.The raw trace stays local. The PR contains aggregate results, the trace record count, and its SHA-256, but no transcript content, paths, project names, or account data.
Replay evidence
Frozen input: 1,780 records, SHA-256
b1e4aa33180b7c177293eb9ed16b45e24e026d259600fba2b1b67b931b904f0b.The legacy heuristic produced 28 observed segments and excluded 26.10 hours after recorded timer deadlines. That excluded time is only "unobserved"; the heuristic cannot distinguish sleep or reboot from a long refresh or event-loop stall.
On this snapshot, current adaptive schedules 49.8% fewer batches than fixed 5m. The candidate adds two simulated refreshes and removes four active-delay violations. It does not improve p95 menu staleness.
The frozen trace predates
timerAdvanceEvaluated: it contains 46 legacytimerAdvancedevents and zero evaluation records, so it cannot validate accepted/rejected live comparisons. Simulated advances are counterfactual and are not compared by count with recorded advances.After packaging commit
105a9f16, the live trace recorded two schedule evaluations: one accepted and one rejected. The recorded-schedule audit reported zero payload, decision, menu-link, multiplicity, or ambiguity errors. The accepted evaluation had one matchingtimerAdvanced; the rejected evaluation had none.Privacy and runtime boundary
Validation
make checkDYLD_FRAMEWORK_PATH="$PWD/.build/out/Products/Debug" OS_ACTIVITY_MODE=disable make test(553 selections, 47/47 groups)DYLD_FRAMEWORK_PATH="$PWD/.build/out/Products/Debug" swift test --filter 'RecordedScheduleAuditTests|CodingActivityReplayPolicyTests|ReplayTraceSegmentationTests|AdaptiveRefreshTraceRecordingTests|AdaptiveReplayEngineTests'(36 tests in 5 suites)swift build --product AdaptiveReplayCLI./Scripts/package_app.sh debugcodesign --verify --deep --strict CodexBar.appCodexGitCommit=105a9f16; installed process stayed running andCodexBarCLI config validatereturnedConfig: OKLimitations
Related
Follow-up to steipete/CodexBar#1861. Updates the decision record introduced in steipete/CodexBar#1739.