From 9ed04e27be87b502eda182266f9eb6e21a5708f5 Mon Sep 17 00:00:00 2001 From: friendsa Date: Sat, 1 Aug 2026 14:05:48 +0000 Subject: [PATCH 1/2] SHA-23: Stage 5 capability matrix and release-note rollup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Align README Supported/Partial/Not-supported claims and design-doc status with main after SHA-12 C–J. Add §11 residual checklist, epic release-note narrative, Towncrier fragment, and refresh fixture gap inventory. No protocol features — public claims only; still not full Garmin FIT conformance. Co-authored-by: multica-agent --- README.md | 40 +++++++-- docs/EPIC_SHA12_RELEASE_NOTES.md | 83 +++++++++++++++++++ docs/FIT_CONFORMANCE_DESIGN.md | 136 +++++++++++++++++++------------ fit_tool/tests/data/README.md | 29 ++++--- fit_tool/validation.py | 9 +- news/SHA-23.doc | 6 ++ 6 files changed, 225 insertions(+), 78 deletions(-) create mode 100644 docs/EPIC_SHA12_RELEASE_NOTES.md create mode 100644 news/SHA-23.doc diff --git a/README.md b/README.md index fde406e..fe06100 100644 --- a/README.md +++ b/README.md @@ -23,15 +23,30 @@ package already implements. Profile version in use: `21.205.0` (see | Status | Capabilities | | --- | --- | -| **Supported** | Common Activity and Workout read/write via typed profile messages; `FitFileBuilder` encode path; **header CRC** (14-byte headers) and **file-level CRC** on load / stream exhaustion (`check_crc=True` default); developer fields for common declaration patterns; streaming iterators (`FitFile.iter_file` / `iter_stream`); CSV export (`to_csv` / `to_rows`); Course and similar message types when you construct them yourself; **chained multi-segment** FIT decode via `from_bytes` / `from_file` (all segments projected into `records`); **compressed timestamp** reconstruction into field 253; **subfield resolution** (ref-field match → type / scale / offset / units; multi-ref AND; first match wins); **component expansion** for all Profile main-field sources (generated registry) **and** components on the active subfield; nested expansion + accumulator rollover; **unknown field ids** on known messages as `UnknownField` (decoded values + `raw_bytes`); **encode modes** `EncodeMode.PRESERVE` (default; unedited bit-identical + post-edit dirty re-project) and `EncodeMode.CANONICAL` (full re-project, normalized sizes/CRCs; optional `strict=True` precheck) — see [Encode policies](#encode-policies) | -| **Partial** | Unknown global messages via `GenericMessage` (readable; unedited preserve keeps wire bytes; post-edit re-encodes dirty records only); composable validation API (`validate_fit_file` / `FitFile.validate`) with WIRE + PROFILE + **Activity, Workout, and Course** FILE_TYPE levels — **PROFILE validation is CORE today** (developer-field subset + **ambiguous subfield** ERROR); opt-in **PRESERVATION** level reports unknown-field `raw_bytes` loss after edits; architecture decision **O1** keeps bundled `Profile.xlsx` as the full metadata source of truth with future DOMAIN/FULL scopes (FULL opt-in, not default strict) — see [`docs/FIT_CONFORMANCE_DESIGN.md`](docs/FIT_CONFORMANCE_DESIGN.md) §3.1; Builder `strict=True` is a thin wrapper over the same checks (WIRE+PROFILE+FILE_TYPE only) | -| **Not supported / incomplete** | Full PROFILE semantics (native field requirements, enums, units beyond subfield scale/units); file-type rules for types other than Activity/Workout/Course (FILE_TYPE fails closed for remaining types); intentional `repair()` API (strict path never silent-repairs); bit-identical rewrite of compressed-timestamp dirty records when field 253 is not on the definition (strict raises; non-strict keeps compressed header) | +| **Supported** | Common Activity / Workout / Course read-write via typed profile messages; `FitFileBuilder` encode path; **header CRC** (14-byte headers) and **file-level CRC** on load / stream exhaustion (`check_crc=True` default); developer fields for common declaration patterns; streaming iterators (`FitFile.iter_file` / `iter_stream`); CSV export (`to_csv` / `to_rows`); **chained multi-segment** FIT decode via `from_bytes` / `from_file` (all segments projected into `records`); **compressed timestamp** reconstruction into field 253; **subfield resolution** (ref-field match → type / scale / offset / units; multi-ref AND; first match wins); **component expansion** for all Profile main-field sources (generated registry) **and** components on the active subfield; nested expansion + accumulator rollover; **unknown field ids** on known messages as `UnknownField` (decoded values + `raw_bytes`); **encode modes** `EncodeMode.PRESERVE` (default; unedited bit-identical + post-edit dirty re-project) and `EncodeMode.CANONICAL` (full re-project, normalized sizes/CRCs; optional `strict=True` precheck) — see [Encode policies](#encode-policies) | +| **Partial** | Unknown global messages via `GenericMessage` (readable; unedited preserve keeps wire bytes; post-edit re-encodes dirty records only); composable validation (`validate_fit_file` / `FitFile.validate`) with **WIRE** + **PROFILE** + **FILE_TYPE** (Activity / Workout / Course) and opt-in **PRESERVATION**; **PROFILE scopes** under O1 (`ProfileScope.CORE` default for `strict` / `DEFAULT_LEVELS`: developer-field rules + ambiguous-subfield ERROR; **DOMAIN** / **FULL** opt-in: native base-type + closed-enum checks from gen-exported `field_catalog` for high-frequency vs entire Profile.xlsx `21.205.0` catalog) — see [Validate FIT files](#validate-fit-files) and design doc §3.1; Builder `strict=True` wraps WIRE+PROFILE+FILE_TYPE at **CORE** only | +| **Not supported / incomplete** | Remaining PROFILE rule families (native **required** fields, units/scale consistency beyond subfield scale/units, open/ranged enums as ERROR); FILE_TYPE for types other than Activity/Workout/Course (fails closed); intentional `repair()` API (strict path never silent-repairs); bit-identical rewrite of compressed-timestamp dirty records when field 253 is not on the definition (strict raises; non-strict keeps compressed header); public `FitDocument` / multi-segment encode API; full Garmin SDK cross-validation as a release gate | + +### What this library still does not claim + +Until design-doc [§11 Definition of Done](docs/FIT_CONFORMANCE_DESIGN.md#11-definition-of-done) is met with evidence, do **not** describe the package as “full Garmin FIT / Profile conformant.” Prefer: + +- “Supports common Activity / Workout / Course workflows” +- “PROFILE validation defaults to CORE; DOMAIN/FULL are opt-in” +- “FILE_TYPE validates Activity, Workout, and Course only” + +**Non-goals** (by design, not just unfinished work): + +- Reproducing undocumented Garmin Connect acceptance heuristics +- Silent repair of corrupt files on the strict path +- Treating every Garmin best-practice note as a wire ERROR +- Replacing generated typed message classes as the ergonomic public API + +**Residuals** that still block §11 marketing claims (honest backlog, not this matrix’s “Supported” row): other standard file types, remaining PROFILE rule families, broader interop/golden corpus, compressed-timestamp encode parity, and performance/API migration items in design-doc Phases 1–5. See the design doc **Remaining gaps** / §11 residual checklist. If you need a construct listed as incomplete, prefer an official Garmin SDK or -wait for the phased work in the design doc (including the **Remaining gaps** -table in [`docs/FIT_CONFORMANCE_DESIGN.md`](docs/FIT_CONFORMANCE_DESIGN.md)). -Architecture reviews should judge this package against the matrix above, not -against full protocol conformance. +track follow-up work against the design doc. Architecture reviews should judge +this package against the matrix above, not against full protocol conformance. Installation ================== @@ -243,6 +258,10 @@ fit_file.validate(raise_on_error=True) # Wire-only (e.g. after decode, without file-type rules) validate_fit_file(fit_file, levels={ConformanceLevel.WIRE}) +# Opt-in PROFILE DOMAIN / FULL (native base-type + closed-enum; never default strict) +from fit_tool import ProfileScope +validate_fit_file(fit_file, profile_scope=ProfileScope.FULL) + # Opt-in post-edit rewrite-loss checks (unknown field raw_bytes, etc.) validate_fit_file(fit_file, levels={ConformanceLevel.PRESERVATION}) ``` @@ -324,10 +343,13 @@ Committed samples live under [`fit_tool/tests/data/`](fit_tool/tests/data/). See [`fit_tool/tests/data/README.md`](fit_tool/tests/data/README.md) for: - layout of `sdk/`, `interop/`, and device smokes; -- a **gap inventory** mapping Stage-2 topics (components, subfields, unknown fields, - multi-segment) to constructive helpers or fixtures; +- a **gap inventory** mapping protocol topics (components, subfields, unknown + fields, encode/FILE_TYPE/PROFILE scopes) to constructive helpers or fixtures; - how to obtain additional Garmin SDK samples when licensing allows. +Epic-level release narrative (next version bump): +[`docs/EPIC_SHA12_RELEASE_NOTES.md`](docs/EPIC_SHA12_RELEASE_NOTES.md). + Prefer constructive builders in `fit_tool/tests/protocol_fixture_helpers.py` over new large binary dumps. Known incomplete semantics are pinned with explicit `xfail` in `fit_tool/tests/test_protocol_gap_fixtures.py` (no silent skips). diff --git a/docs/EPIC_SHA12_RELEASE_NOTES.md b/docs/EPIC_SHA12_RELEASE_NOTES.md new file mode 100644 index 0000000..9ed28f8 --- /dev/null +++ b/docs/EPIC_SHA12_RELEASE_NOTES.md @@ -0,0 +1,83 @@ +# Protocol epic (SHA-12) — release-note rollup + +Draft narrative for the next version bump. Towncrier fragments under `news/` +(`SHA-13`…`SHA-22`, related SHA-3/9 wire work) are the machine-readable source; +this document is the human rollup for product claims. + +**Do not** market “full Garmin FIT / Profile conformance.” See +[`FIT_CONFORMANCE_DESIGN.md`](FIT_CONFORMANCE_DESIGN.md) §11 residual checklist +and the README [capability boundary](../README.md#capability-boundary). + +## Epic outcome + +Stages 1–4 closed Multica children **A–J** (docs/fixtures, components, +subfields, unknown fields, post-edit PRESERVATION, encode policies, PROFILE +scopes CORE/DOMAIN/FULL, Workout + Course FILE_TYPE). Stage 5 (**L**) publishes +honest Supported/Partial/Not-supported claims. + +Profile / SDK version: **`21.205.0`** (`SDK_VERSION`, `fit_tool/gen/`). + +## User-visible capabilities (rollup) + +### Wire and decode + +- Header CRC (14-byte) and file CRC on load / stream exhaustion (`check_crc`) +- Chained multi-segment FIT decode; optional trailing-byte allowance +- Compressed timestamp reconstruction into field 253 (decode path) +- Unified wire / facade decode path (earlier SHA-7/8 hygiene) + +### Profile runtime + +- Full main-field component registry (generated) with nested expansion and + accumulator rollover +- Subfield resolution (ref-field match, multi-ref AND, scale/offset/units) and + active-subfield components +- Unknown field ids on known messages as `UnknownField` with `raw_bytes` +- Unknown global messages via `GenericMessage` (partial preserve semantics) + +### Encode + +- `EncodeMode.PRESERVE` (default): unedited bit-identical; post-edit re-encodes + dirty records only +- `EncodeMode.CANONICAL`: full re-project; `strict=True` validates first, never + silent-repairs + +### Validation + +- Levels: WIRE, PROFILE, FILE_TYPE, opt-in PRESERVATION +- PROFILE scopes: **CORE** (default strict), **DOMAIN**, **FULL** (opt-in; + native base-type + closed-enum from gen field catalog) +- FILE_TYPE: **Activity**, **Workout**, **Course**; other types fail closed + +## Explicit non-goals / residuals + +| Item | Notes | +| --- | --- | +| Full PROFILE required-field / units rules | Not in CORE/DOMAIN/FULL today | +| FILE_TYPE beyond A/W/C | Fail closed | +| `repair()` API | Future; strict path never auto-repairs | +| Public `FitDocument` API | Design target; facade + wire only today | +| Compressed timestamp encode parity | Decode supported; encode edges remain | +| Garmin Connect heuristics | Non-goal | +| “Full conformance” marketing | Blocked by §11 residual checklist | + +## Towncrier / CHANGELOG + +Unreleased fragments for this epic include (non-exhaustive): + +- `news/SHA-13.doc` … `news/SHA-22.feature` (stages 1–4) +- `news/SHA-23.doc` (this matrix / claims pass) +- Earlier wire/validation fragments (`SHA-7`…`SHA-9`, `sha3-*`, etc.) + +At release time, run Towncrier to fold `news/*` into `CHANGELOG.md` under the +chosen version. No version bump is implied by Stage 5 alone. + +## Verification + +```bash +uv run pytest fit_tool/tests/test_validation.py fit_tool/tests/test_profile_scope.py \ + fit_tool/tests/test_encode_policies.py fit_tool/tests/test_post_edit_preservation.py \ + fit_tool/tests/test_workout_files.py fit_tool/tests/test_course_files.py -q +``` + +Capability claims must match `README.md` and design-doc **Current status**. diff --git a/docs/FIT_CONFORMANCE_DESIGN.md b/docs/FIT_CONFORMANCE_DESIGN.md index 786df86..1285090 100644 --- a/docs/FIT_CONFORMANCE_DESIGN.md +++ b/docs/FIT_CONFORMANCE_DESIGN.md @@ -9,11 +9,13 @@ For the library as shipped today, use the capability matrix in [`README.md`](../README.md#capability-boundary). That matrix is the source of truth for user-facing claims; this section must stay aligned with it. -Baseline after wire-layer work (#44 / #45 and follow-ups on `main`): +Baseline after Multica epic **SHA-12** stages 1–4 on `main` (wire #44/#45, +hygiene, then C–J). **User-facing claims must match** +[`README.md`](../README.md#capability-boundary); update both together. | Area | Today | This design | | --- | --- | --- | -| Common Activity / Workout read-write, `FitFileBuilder`, file-level CRC, streaming iterators, developer fields (common declaration patterns), CSV export | Supported | Baseline to preserve | +| Common Activity / Workout / Course read-write, `FitFileBuilder`, file-level CRC, streaming iterators, developer fields (common declaration patterns), CSV export | Supported | Baseline to preserve | | **Header CRC** (14-byte headers, gated by `check_crc`) | Supported | Keep strict; no silent repair | | **Chained multi-segment** FIT decode via `FitFile.from_bytes` / `from_file` (all segments projected into `records`) | Supported | Full `FitDocument` / segment API remains the long-term surface (§4–5) | | **Compressed timestamp** reconstruction into field 253 (wire decoder + projection) | Supported | Encode compressed headers still optional / canonical path | @@ -22,36 +24,38 @@ Baseline after wire-layer work (#44 / #45 and follow-ups on `main`): | **Ambiguous subfields** (more than one match) | PROFILE ERROR (decode still uses first match) | Same policy | | **Preservation encode** (`to_bytes(mode=EncodeMode.PRESERVE)` / `preserve=True`, default) for buffer-decoded files: unedited path bit-identical; **post-edit** path re-encodes dirty records and copies `source_bytes` for the rest | Supported | Aligns with design §6.1 | | **Canonical encode** (`to_bytes(mode=EncodeMode.CANONICAL)` / `preserve=False`); optional `strict=True` precheck | Supported | Compressed-header expansion only when field 253 is on the definition; otherwise keep compressed (non-strict) or raise (strict) | -| Unknown global messages (`GenericMessage`); composable validation (`validate_fit_file` / `FitFile.validate`) with WIRE + PROFILE (developer fields + ambiguous-subfield ERROR) + Activity/Workout FILE_TYPE; opt-in **PRESERVATION** level; Builder `strict=True` wraps default levels only | Partial | Full Profile field/enum/units rules; Course FILE_TYPE | -| Full PROFILE semantics (native required fields, enums, units beyond subfield scale/units); Course and other non-Activity/non-Workout FILE_TYPE rules | Not supported / incomplete | Phases 3–4 and remaining-gaps table below | +| Unknown global messages (`GenericMessage`); composable validation with WIRE + PROFILE scopes (CORE default; DOMAIN/FULL opt-in native base-type + closed-enum) + Activity/Workout/Course FILE_TYPE; opt-in **PRESERVATION**; Builder `strict=True` wraps default levels at **CORE** only | Partial | Remaining PROFILE rule families (required fields, units); other FILE_TYPE values | +| Remaining PROFILE rule families (native required fields, units/scale beyond subfield scale/units); FILE_TYPE for types other than Activity/Workout/Course; `repair()` API; public `FitDocument` | Not supported / incomplete | Follow-ups outside closed SHA-12 stages C–J; see residual checklist under §11 | | Unknown field ids on known messages (`UnknownField` + `raw_bytes` on decode; survive post-edit when not mutated) | Supported | Mutating an unknown field clears `raw_bytes` (PRESERVATION ERROR if that level is selected) | Until §11 Definition of Done is met, do not describe the library as fully -protocol-conformant. Prefer “supports common Activity/Workout workflows” and -point readers at the README matrix. +protocol-conformant. Prefer “supports common Activity/Workout/Course workflows” +and point readers at the README matrix (including **What this library still +does not claim**). ### Remaining gaps (roadmap ↔ Multica) -Tracked under Multica epic **SHA-12** (protocol / capability extension). Stage 1 -(A docs sync · B fixtures) is characterization only; implementation work is -stage 2+. Identifiers below are Multica issue keys (SHA-N); filed children have -stable keys, later letters are stage placeholders until children are created. +Tracked under Multica epic **SHA-12** (protocol / capability extension). Stages +1–4 children **C–J** are **done** on `main`; Stage 5 (**L** / SHA-23) is the +public-claims pass that keeps this table and the README matrix honest. | Gap | Design doc | Multica | Notes | | --- | --- | --- | --- | -| Fixture / golden corpus for remaining protocol edges | Phase 0 | **B** SHA-14 | Stage 1; coordinate with this status table | +| Docs / status truth vs main | Phase 0 | **A** SHA-13 | **Done** (re-synced at Stage 5 / L) | +| Fixture / golden corpus for protocol edges | Phase 0 | **B** SHA-14 | **Done** inventory + constructive goldens; expand as follow-up only | | Full component / accumulator coverage beyond `_KNOWN_COMPONENTS` | Phase 3 | **C** SHA-15 | **Done** main-field registry + nested + rollover; active-subfield components via D | | Subfield resolution (type / scale / units / components) | Phase 3 | **D** SHA-16 | **Done** runtime match + PROFILE ambiguity ERROR | -| Unknown field ids on known messages (decode retain + raw bytes) | Phase 3 | **E** SHA-17 | **Done** on main path: `UnknownField` + `raw_bytes`; prerequisite for post-edit preserve | -| Post-edit PRESERVATION (edited files, dirty records) | Phase 4 / PRESERVATION level | **F** SHA-18 | **Done**: per-record dirty + mixed encode; opt-in PRESERVATION findings | -| Encode policies (canonical vs preserve, strict vs repair) | Phase 4 §6 | **G** SHA-19 | **Done**: `EncodeMode` + policy matrix; no silent invalid clamp; `repair()` API still future | -| Full PROFILE validation from bundled Profile.xlsx `21.205.0` | Phase 3 PROFILE | **H** (SHA-12 stage 4; child TBD) | Slice by message family / rule kind | -| Workout FILE_TYPE rules | Phase 4 §7 | **I** SHA-21 | **Done**: required messages/fields for Workout; SDK Workout*.fit fixtures pass FILE_TYPE | -| Course FILE_TYPE rules | Phase 4 §7 | **J** (SHA-12 stage 4; child TBD) | Not in first batch; still fail-closed | -| Public capability matrix / release-note pass at claim flip | Phase 5 | **L** (SHA-12 stage 5; child TBD) | Update README + this section together | - -Letter **A** is this status-truth pass (Multica **SHA-13**). Do not claim gaps -closed until the matching child is `done` and README status rows flip. +| Unknown field ids on known messages (decode retain + raw bytes) | Phase 3 | **E** SHA-17 | **Done** `UnknownField` + `raw_bytes` on main path | +| Post-edit PRESERVATION (edited files, dirty records) | Phase 4 / PRESERVATION level | **F** SHA-18 | **Done** per-record dirty + mixed encode; opt-in PRESERVATION findings | +| Encode policies (canonical vs preserve, strict vs repair) | Phase 4 §6 | **G** SHA-19 | **Done** `EncodeMode` + policy matrix; no silent invalid clamp; `repair()` still future | +| PROFILE scopes from bundled Profile.xlsx `21.205.0` | Phase 3 PROFILE / §3.1 | **H** SHA-20 | **Done** CORE/DOMAIN/FULL + gen `field_catalog`; default remains CORE; native required fields / units still residual | +| Workout FILE_TYPE rules | Phase 4 §7 | **I** SHA-21 | **Done** required messages/fields; SDK Workout*.fit pass FILE_TYPE | +| Course FILE_TYPE rules | Phase 4 §7 | **J** SHA-22 | **Done** Course required messages/fields; Activity + Workout unchanged | +| Public capability matrix / release-note pass | Phase 5 | **L** SHA-23 | **This pass** — README + this section + §11 residual checklist + Towncrier epic notes | + +Letter **A** (SHA-13) was the first status-truth pass. After Stage 5, gaps +C–J are closed for the epic’s staged scope; residuals below §11 are **follow-up +work**, not silent “Supported” claims. ## 1. Objective @@ -141,16 +145,16 @@ Three layers must not be conflated: | **B. Runtime semantics** | Subfields, components, accumulators, native overrides | Design for full metadata; implement incrementally | | **C. `ConformanceLevel.PROFILE` validation** | Required fields, base types, enums, units consistency | Full catalog is an **opt-in scope**, not the default strict set | -#### Scopes (target API shape) +#### Scopes (shipped API) -When `ConformanceLevel.PROFILE` is selected, validation applies a **scope** -(name may land as `profile_scope`, `ProfileRuleSet`, or equivalent): +When `ConformanceLevel.PROFILE` is selected, validation applies +`profile_scope=` / `ProfileScope`: -| Scope | Intent | Default for `strict=True` / DEFAULT_LEVELS | -| --- | --- | --- | -| **CORE** | Developer-field rules + structural Profile consistency that does not depend on full field catalogs | **Yes** (current behavior grows into CORE) | -| **DOMAIN** | High-frequency Activity / Workout native rules (required fields, common enums) | Optional later inclusion after coverage is solid | -| **FULL** | Rules derived from the entire bundled Profile.xlsx | **No** — explicit opt-in (or primarily WARNING until stable) | +| Scope | Intent | Default for `strict=True` / DEFAULT_LEVELS | Status | +| --- | --- | --- | --- | +| **CORE** | Developer-field rules + ambiguous native subfield ERROR | **Yes** | Shipped | +| **DOMAIN** | CORE + native base-type and closed-enum checks on high-frequency Activity/Workout messages | No | Shipped (required-field / units families still residual) | +| **FULL** | CORE + same native base-type / closed-enum rules for the entire gen-exported catalog from Profile.xlsx | **No** — explicit opt-in | Shipped for those rule families only; not “full Profile semantics” | Principles: @@ -168,14 +172,14 @@ Principles: #### Implementation milestones (for Multica H / related stages) -| Milestone | Meaning | -| --- | --- | -| M1 | Gen exports subfield / component / required / enum tables from xlsx | -| M2 | Runtime applies those tables (semantics) | -| M3 | PROFILE CORE validation (today’s developer subset, evolved) | -| M4 | PROFILE DOMAIN (Activity/Workout high-frequency messages) | -| M5 | PROFILE FULL catalog; default remains CORE (and maybe DOMAIN) | -| M6 | Marketing / §11 DoD only with WIRE + scoped PROFILE + FILE_TYPE + PRESERVATION | +| Milestone | Meaning | Status | +| --- | --- | --- | +| M1 | Gen exports field / enum catalog (and related) from xlsx | **Done** (`field_catalog`, component registry; required-field table still residual) | +| M2 | Runtime applies those tables (semantics) | **Partial** (components, subfields, accumulators; not all rule families) | +| M3 | PROFILE CORE validation (developer subset + ambiguous subfields) | **Done** | +| M4 | PROFILE DOMAIN (Activity/Workout high-frequency messages) | **Done** for base-type + closed-enum | +| M5 | PROFILE FULL catalog; default remains CORE | **Done** for base-type + closed-enum on full catalog; default still CORE | +| M6 | Marketing / §11 DoD only with WIRE + scoped PROFILE + FILE_TYPE + PRESERVATION | **Blocked** by residual checklist under §11 (honest Partial claims only) | ## 4. Architectural principle: separate wire data from semantic projection @@ -685,8 +689,9 @@ allocation based solely on untrusted lengths. Exit: every known conformance gap has a fixture and issue. -**Progress:** status table and README matrix aligned with `main` after #44/#45; -fixture expansion continues under SHA-14. +**Progress:** status table and README matrix aligned after #44/#45 (SHA-13) and +re-synced at Stage 5 (SHA-23). Fixture inventory and constructive goldens +landed under SHA-14; further corpus growth is optional follow-up. ### Phase 1: lossless wire layer @@ -698,9 +703,9 @@ Exit: all structural fixtures preserve exact bytes. **Progress (partial):** `fit_tool/wire` decoder/encoder models, immutable definition snapshots, header + file CRC, chained multi-segment decode, and -unedited preservation rewrite of source segment bytes are on `main`. Public -`FitDocument` API and full structural golden corpus remain incomplete relative -to the exit criteria above. +unedited / post-edit preservation rewrite of source segment bytes are on +`main`. Public `FitDocument` API and a full structural golden corpus remain +incomplete relative to the exit criteria above. ### Phase 2: compressed timestamps @@ -721,11 +726,12 @@ Compressed encode and full Garmin bidirectional golden coverage remain. Exit: Profile-level golden corpus and Garmin cross-validation pass. -**Progress (partial):** generated Profile main-field component registry (37/37 -sources) expands packed fields with nested components and accumulator rollover -at decode time. Unknown field ids on known messages are retained as -`UnknownField` with `raw_bytes` (Stage 2 E). Subfield-gated components, -subfields, and full PROFILE validation remain Multica D / H (stages 2 and 4). +**Progress (partial):** main-field component registry (37/37 sources) with +nested expansion and accumulator rollover; active-subfield components via +subfield resolution (C/D); unknown field ids retained as `UnknownField` + +`raw_bytes` (E); PROFILE scopes CORE/DOMAIN/FULL with gen `field_catalog` for +native base-type + closed-enum (H / O1). Remaining: native required-field and +units rule families, broader Profile golden / Garmin cross-validation. ### Phase 4: strict encoder and file validators @@ -736,19 +742,25 @@ subfields, and full PROFILE validation remain Multica D / H (stages 2 and 4). Exit: all produced standard files pass the selected Garmin and repository validators without repair. -**Progress (partial):** unedited and **post-edit** PRESERVE paths exist; explicit -`EncodeMode` / `strict` / policy matrix landed (G / SHA-19). Activity and -Workout FILE_TYPE exist (I / SHA-21); other types (including Course / J) fail -closed. `repair()` API is still future. +**Progress (partial):** unedited and **post-edit** PRESERVE paths; explicit +`EncodeMode` / `strict` / policy matrix (G / SHA-19). FILE_TYPE for +**Activity**, **Workout**, and **Course** (I / SHA-21, J / SHA-22); other +`file_id.type` values still fail closed. `repair()` API is still future. ### Phase 5: API migration and performance - Route legacy APIs through the conformance core. - Add compatibility shims and deprecation warnings. - Re-establish parse, encode, and streaming-memory budgets. +- Publish accurate public capability claims (README matrix + this status table). Exit: no undocumented breaking changes and performance is no worse than the -current optimized branch by more than the agreed budget. +current optimized branch by more than the agreed budget; public claims match +code. + +**Progress (partial):** Stage 5 / **L** (SHA-23) lands the public capability +matrix and release-note rollup for SHA-12. Full API migration and performance +budgets remain follow-up work outside the protocol epic’s staged children. ## 11. Definition of done @@ -763,3 +775,23 @@ The project may claim strict Garmin FIT conformance only when: invalid values, and chained files have non-`xfail` tests; - strict mode performs no silent repair; - the exact supported Garmin SDK/Profile version is published. + +### §11 residual checklist (after SHA-12 stages 1–4) + +Honest status vs the bullets above. **None** of these residuals re-open C–J as +“undone”; they bound marketing claims until a future epic closes them. + +| §11 requirement | Status after SHA-12 C–J | Evidence / residual | +| --- | --- | --- | +| All published FIT wire constructs | **Partial** | Header/file CRC, chained decode, compressed-ts **decode**, definitions/data path on `main`; public `FitDocument`, full structural goldens, compressed-ts **encode** parity incomplete | +| Bundled Profile semantics | **Partial** | Subfields, main-field + active-subfield components, accumulators, developer fields (common patterns), PROFILE CORE/DOMAIN/FULL (base-type + closed-enum). Missing: native required fields, units/scale consistency rules, full developer native-override matrix | +| Every legal unknown field preserved | **Mostly met** | `UnknownField` + `raw_bytes`; unedited + post-edit PRESERVE when not mutated; PRESERVATION level reports loss. Structural edit / full re-project still drops wire snapshot (documented) | +| Standard file-type rules (caller-selected) | **Partial** | Activity + Workout + Course implemented; other types fail closed intentionally | +| Garmin cross-validation | **Incomplete** | SDK samples used as smokes/fixtures; not a release gate for full interop | +| Non-`xfail` tests for listed constructs | **Partial** | Strong constructive coverage for C–J; expand goldens / remove any remaining known-gap xfails as follow-up | +| Strict mode no silent repair | **Met** | `strict=True` / DEFAULT_LEVELS never clamp or auto-repair; no `repair()` on strict path | +| Published SDK/Profile version | **Met** | Profile / SDK `21.205.0` (`fit_tool/gen/`, `SDK_VERSION`) | + +**Bottom line:** SHA-12 may move to `done` / `in_review` with the residuals +above explicit. Do **not** flip public marketing to “full Garmin FIT +conformance” until this checklist is actually green. diff --git a/fit_tool/tests/data/README.md b/fit_tool/tests/data/README.md index c07cac7..2cf9630 100644 --- a/fit_tool/tests/data/README.md +++ b/fit_tool/tests/data/README.md @@ -53,14 +53,17 @@ FIT_JS_SDK_PATH=../fit-javascript-sdk \ See also the README section **Run Garmin SDK interoperability tests**. -## Gap inventory (protocol / Stage 2 topics) +## Gap inventory (protocol / SHA-12 topics) -Maps known conformance gaps to fixtures or constructive tests. Stage-2 work -(C/D/E) should promote `xfail` cases rather than invent new silent skips. +Maps protocol topics to fixtures or constructive tests after Stages 1–4 +(C–J). Prefer promoting known-gap `xfail` cases over inventing silent skips. +User-facing Supported/Partial claims live in the repository `README.md` +capability matrix; residual §11 blockers are in +[`docs/FIT_CONFORMANCE_DESIGN.md`](../../../docs/FIT_CONFORMANCE_DESIGN.md) §11. | Gap / topic | Status today | Fixture or generator | Primary tests | Stage | | --- | --- | --- | --- | --- | -| Compressed timestamp offset + rollover | Supported | Constructive + unit helpers | `test_protocol_high_severity.py` | done | +| Compressed timestamp offset + rollover | Supported (decode → field 253) | Constructive + unit helpers | `test_protocol_high_severity.py` | done | | Chained multi-segment FIT | Supported | Constructive (`segment + segment`) | `test_protocol_high_severity.TestChainedAndTrailing` | done | | Trailing bytes after last segment | Supported (`allow_trailing_bytes`) | Constructive | same | done | | Component: `compressed_speed_distance` | Supported (Profile registry) | Constructive wire + in-memory | `test_protocol_high_severity.TestComponents`, `test_components.py`, `test_protocol_gap_fixtures` | C done | @@ -68,20 +71,20 @@ Maps known conformance gaps to fixtures or constructive tests. Stage-2 work | Component: 16-bit / 8-bit / 12-bit accumulator **rollover** | Supported | Constructive expansion helper | `test_components.py`, `test_protocol_gap_fixtures` | C done | | Nested components (e.g. speed → enhanced_speed) | Supported | Constructive | `test_components.py` | C done | | Full Profile **main-field** component set | Supported (37/37 sources, generated registry) | `fit_tool/profile/component_registry.py` | `test_components.TestRegistryCoverage` | C done | -| Subfield-gated components (event sport_point, etc.) | **Gap** (needs D) | deferred | Stage 2 D | D | +| Subfield-gated components (event sport_point, etc.) | **Supported** (active subfield via D) | Constructive / unit | `test_components`, `test_subfields` | D done | | Unknown **global** messages | Partial (`GenericMessage`) | Device/SDK files with odd IDs; wire decode | `test_wire.py`, SDK smokes | E | -| Unknown **field ids** on known messages | **Supported** (`UnknownField` + `raw_bytes` on decode; unedited preserve ok) | Constructive definition + data | `test_protocol_gap_fixtures` | E | -| Subfields (e.g. `workout_step.duration_value`) | **Supported** (ref match + scale/units; multi-ref AND; PROFILE ERROR on ambiguity) | Constructive workout step | `test_subfields`, `test_protocol_gap_fixtures` | D | +| Unknown **field ids** on known messages | **Supported** (`UnknownField` + `raw_bytes`; unedited + post-edit preserve when not mutated) | Constructive definition + data | `test_protocol_gap_fixtures`, `test_post_edit_preservation` | E/F done | +| Subfields (e.g. `workout_step.duration_value`) | **Supported** (ref match + scale/units; multi-ref AND; PROFILE ERROR on ambiguity) | Constructive workout step | `test_subfields`, `test_protocol_gap_fixtures` | D done | | Developer fields (common patterns) | Supported | `sdk/DeveloperData.fit`, `activity_developerdata.fit` | `test_sdk_files.py`, `test_developer_fields.py` | — | | Header CRC (14-byte / extended) | Supported | Constructive | `test_protocol_high_severity.TestHeaderCRC` | done | | File CRC | Supported | All CRC-checked loads | suite-wide | done | -| Post-edit PRESERVATION | **Gap** | Untouched preserve tests only | `test_protocol_high_severity.TestPreservationEncode` | F | -| FILE_TYPE Workout / Course rules | **Gap** (not Stage 1) | Builder examples / later | deferred | I/J | -| Full PROFILE validation | **Gap** | deferred | deferred | H | +| Post-edit PRESERVATION | **Supported** (dirty records + mixed encode; opt-in PRESERVATION level) | Constructive | `test_post_edit_preservation`, encode policy tests | F done | +| Encode modes PRESERVE / CANONICAL | **Supported** | Constructive | `test_encode_policies` | G done | +| FILE_TYPE Activity / Workout / Course | **Supported** for those three; other types fail closed | SDK Activity/Workout + Course builders | `test_validation`, `test_workout_files`, `test_course_files` | I/J done | +| PROFILE scopes CORE / DOMAIN / FULL | **Partial** (CORE default; DOMAIN/FULL = base-type + closed-enum; no native required/units yet) | Gen `field_catalog` + constructive | `test_profile_scope`, `test_validation` | H done (scoped) | -Legend: **Gap** = behavior incomplete or incorrect relative to design doc; -constructive tests may assert current behavior and mark the desired semantics -with `@pytest.mark.xfail(strict=True, reason='… Stage N …')`. +Legend: rows marked **Partial** or residual §11 items are incomplete relative to +full design-doc DoD, not relative to the closed Multica stage letter. ## Adding fixtures diff --git a/fit_tool/validation.py b/fit_tool/validation.py index cad5cc8..5393881 100644 --- a/fit_tool/validation.py +++ b/fit_tool/validation.py @@ -11,13 +11,14 @@ ambiguous native subfields). DOMAIN and FULL add data-driven native base-type and closed-enum checks from the gen-exported field catalog (design doc §3.1 O1). FULL is **opt-in**, never the default for ``strict=True`` / :data:`DEFAULT_LEVELS`. -* **FILE_TYPE** — ``file_id`` rules and Activity required messages/fields +* **FILE_TYPE** — ``file_id`` rules plus required messages/fields for + **Activity**, **Workout**, and **Course** * **PRESERVATION** — opt-in checks for post-edit rewrite loss (e.g. unknown field ``raw_bytes`` cleared). Not part of default / strict levels. -File-type rules are implemented only for **Activity**. Other ``file_id.type`` -values fail closed at the FILE_TYPE level (intentional until more validators -exist). +File-type rules are implemented for Activity, Workout, and Course +(:data:`IMPLEMENTED_FILE_TYPES`). Other ``file_id.type`` values fail closed at +the FILE_TYPE level (intentional until more validators exist). """ from __future__ import annotations diff --git a/news/SHA-23.doc b/news/SHA-23.doc new file mode 100644 index 0000000..a6d8f81 --- /dev/null +++ b/news/SHA-23.doc @@ -0,0 +1,6 @@ +Stage 5 capability-matrix and release-note pass (Multica SHA-12 / L): README +Supported/Partial/Not-supported rows and “what we still do not claim” aligned +with code after C–J; design-doc current status, remaining-gaps table, phase +progress, and §11 residual checklist re-synced; protocol fixture gap inventory +updated. Towncrier fragments for stages 1–4 remain the epic’s user-visible +capability changelog; do not claim full Garmin FIT conformance until §11 is met. From 385692885afb23f64a06c16aed97e2ec3377bbd5 Mon Sep 17 00:00:00 2001 From: friendsa Date: Sat, 1 Aug 2026 14:19:27 +0000 Subject: [PATCH 2/2] docs: rewrite SHA-23 Towncrier fragment for release readers Codex P1: user-facing capability-matrix wording instead of Multica stage/issue planning language. Co-authored-by: multica-agent --- news/SHA-23.doc | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/news/SHA-23.doc b/news/SHA-23.doc index a6d8f81..81c2ead 100644 --- a/news/SHA-23.doc +++ b/news/SHA-23.doc @@ -1,6 +1,7 @@ -Stage 5 capability-matrix and release-note pass (Multica SHA-12 / L): README -Supported/Partial/Not-supported rows and “what we still do not claim” aligned -with code after C–J; design-doc current status, remaining-gaps table, phase -progress, and §11 residual checklist re-synced; protocol fixture gap inventory -updated. Towncrier fragments for stages 1–4 remain the epic’s user-visible -capability changelog; do not claim full Garmin FIT conformance until §11 is met. +Document the current capability boundary for readers: README Supported / +Partial / Not-supported matrix and a short “what this library still does not +claim” section now match the shipped code (PROFILE scopes CORE/DOMAIN/FULL, +Activity/Workout/Course FILE_TYPE validation, post-edit PRESERVE/CANONICAL +encode). The conformance design doc and residual checklist are re-synced so +public claims stay honest — full Garmin FIT / Profile conformance is still +not advertised until every Definition of Done item is met.