Add opt-in strict and version-aware DataSet JSON deserialization

[PierreRaybaut]

Summary

dataset_to_json() stores a DataSet's module/class identity and serialized item values, while json_to_dataset() imports the class, constructs it with current defaults and calls deserialize(). This permissive behavior is convenient for ordinary settings, but it can silently hide schema drift: a renamed field is ignored as unknown and the replacement field receives its current default.

Provide an opt-in strict, version-aware deserialization path that lets applications detect renamed, missing, unknown or future fields before a DataSet instance is constructed. Preserve the current permissive behavior by default for backward compatibility.

Motivating failure mode

A persisted scientific parameter named a is renamed to amplitude because its physical meaning changes. Loading the old JSON through the current json_to_dataset() does not report that a is unknown or that amplitude is missing; the resulting object contains the new default amplitude. The application can then regenerate data or execute a computation with a plausible but unrelated value.

This is not specific to peak fitting. The same risk applies to renamed calibration fields, changed units, altered enum domains and parameters whose semantics change while their Python type remains compatible.

Current implementation

guidata.dataset.conv.json_to_dataset(json_str) currently:

1. Creates JSONReader(json_str).
2. Reads class_module and class_name.
3. Imports and instantiates the current DataSet class.
4. Calls param.deserialize(reader).

There is no public validation step exposing the raw keys against the class's serializable items, no unknown/missing-field report and no schema-version hook before defaults are applied.

Proposed minimal API

Keep json_to_dataset(json_str) unchanged by default and add keyword-only strict/version options, or add a separate structured entry point if that is easier to evolve without ambiguity. One possible shape is:

def json_to_dataset(
    json_str: str,
    *,
    strict: bool = False,
    expected_version: int | None = None,
    migrate: Callable[[dict[str, Any], int | None], dict[str, Any]] | None = None,
) -> DataSet:
    ...

The exact API is open for maintainer design. The required capability is a pre-construction phase that can inspect and transform a plain decoded mapping before DataSet defaults obscure the original schema.

Strict-mode behavior

• Validate that the top-level payload is a JSON object.
• Validate class_module and class_name, with an optional expected class constraint to prevent loading a different importable DataSet.
• Compute the set of serializable DataItem names for the resolved class, excluding structural group markers and computed/read-only values that are not expected in input.
• Report unknown persisted fields.
• Report missing required persisted fields. The API should distinguish fields intentionally allowed to default from fields that signal schema drift; this may require a per-item property or an explicit allowed-missing set.
• Report malformed values through a structured exception that includes missing, unknown and invalid field names.
• If a payload version is newer than supported, fail before constructing the DataSet.
• If a migration callback is supplied, run it on a copy of the decoded mapping and validate the migrated result strictly before deserialization.
• Never mutate the caller's decoded mapping.

Version ownership

Two designs are worth considering:

• Application-owned envelope: guidata exposes raw decode and strict class-field validation helpers, while each application stores and migrates its own schema_version outside dataset_to_json().
• DataSet-owned version: DataSet classes may declare a class-level schema version and migration hook, and dataset_to_json() writes that version automatically.

The application-owned envelope is smaller and avoids imposing global versioning on existing DataSets. DataSet-owned hooks provide stronger reuse but need clear inheritance, plugin and downgrade semantics. This issue should not introduce a mandatory global container version without maintainer agreement.

Compatibility

• Existing calls to json_to_dataset(json_str) remain permissive.
• Existing serialized JSON remains readable in permissive mode.
• Strict mode is opt-in and must provide actionable diagnostics rather than silently defaulting.
• Callers that need scientific or security-sensitive persistence can validate raw payloads before object construction and implement explicit migrations.

Acceptance criteria

• A test DataSet renamed from a to amplitude causes strict loading of historical {"a": ...} data to report both unknown a and missing amplitude before defaults are applied.
• Permissive loading retains current behavior.
• Unknown fields, missing required fields, wrong class identity and future versions have distinct test coverage.
• A migration callback can rename/convert values non-destructively and the migrated mapping is revalidated.
• Nested/grouped DataSets, computed items, choice values and existing tuple serialization remain covered.
• Public documentation explains when strict mode is appropriate and who owns schema versions.

Related issues

• Source scientific contract and conversion API: DataLab-Platform/Sigima#38.
• DataLab desktop integration and migration UX: DataLab-Platform/DataLab#350.
• DataLab-Web runtime integration: DataLab-Platform/DataLab-Web#11.