Research: Strict schema validation — requiring ALL declared fields in observation results

[Sam-Bolling]

Finding

The server strictly validates that ALL fields declared in a datastream's schema are present in every observation result. If any declared field is missing from an observation payload, the server rejects it. This is a research spike to determine whether strict-vs-lenient schema validation is specified by OGC and what the recommended behavior should be.

Review Source: Live integration testing from OSHConnect-Python publishers against both connected-systems-go and OSH SensorHub
Severity: P3-Minor (research spike)
Category: API Design / Validation
Ownership: connected-systems-go

---

Problem Statement

When a datastream's schema declares a set of observation fields (e.g., timestamp, latitude, longitude, altitude, heading, velocity), the Go server requires that EVERY observation submitted to that datastream includes ALL of those fields. If a publisher omits a field that it considers optional or redundant (e.g., timestamp when resultTime already carries the same information), the server rejects the observation.

Observed behavior:

Datastream schema declares 6 fields including timestamp. Publisher sends:

{
  "resultTime": "2026-04-17T00:00:00Z",
  "result": {
    "latitude": 40.0,
    "longitude": -74.0,
    "altitude": 10500.0,
    "heading": 180.0,
    "velocity": 250.0
  }
}

→ Rejected because timestamp field is missing from result.

Comparative behavior:

Server	All fields present	Subset of fields	Extra fields
OSH SensorHub	✅ Accepted	✅ Accepted (lenient)	✅ Accepted
connected-systems-go	✅ Accepted	❌ Rejected	Unknown

Workaround applied: Our publishers now include ALL declared schema fields in every observation, even when some are redundant with top-level fields (e.g., duplicating resultTime as result.timestamp).

Research Questions

This spike should produce analysis, findings, and recommendations that may result in follow-on issues:

1. What does OGC 23-002 say about observation result validation against the datastream schema? Must all declared fields be present?
2. Does SWE Common distinguish required vs. optional fields in a DataRecord? Is there a minOccurs="0" concept for individual fields within an observation result?
3. Should the datastream schema support marking fields as optional? If so, what would the JSON encoding look like?
4. What's the performance implication of strict validation? Does validating every field of every observation create overhead?
5. How do other CSAPI implementations handle partial observations? Is SensorHub's leniency intentional or accidental?
6. Should extra (undeclared) fields be accepted, ignored, or rejected? What about schema evolution — adding a field to a live datastream?
7. Is there a distinction between "schema defines what CAN appear" vs. "schema defines what MUST appear"?

Expected Deliverables

• Analysis of OGC spec requirements for observation result validation
• Review of SWE Common DataRecord optionality mechanisms
• Comparison table: strict vs. lenient validation across CSAPI servers
• Recommendation: keep strict, add optional field support, or make configurable
• If changes are recommended, file follow-on implementation issue(s)

Scope

• ❌ Do NOT implement changes in this spike — research and recommend only
• ❌ Do NOT change client-side workarounds — they ensure compatibility with both servers

References

#	Document	What It Provides
1	OGC 23-002 §9.7	Observation result structure
2	OGC 23-002 §9.3	Datastream schema definition
3	SWE Common 3.0 §7.4	DataRecord component — field optionality
4	OSHConnect-Python opensky_publisher.py	Publisher that includes all fields as workaround
5	OSH SensorHub	Comparative server with lenient validation

[Sam-Bolling]

Deep evaluation completed (2026-04-30)

Full report: docs/research/issue-evaluations/issue-005.md. Evidence: docs/research/evidence/issue-005/.

Headline finding — the issue's premise is mistaken

The server does not strictly require all declared fields. It requires fields that are not marked "optional": true — which is exactly what SWE Common 3.0 §7.4 specifies. The mechanism is fully wired:

• DatastreamDataComponent.Optional *bool exists in the model (internal/model/domains/datastream.go:225).
• validateDataComponentValue consumes it correctly in the datarecord branch (line 92) and vector branch (line 115):

  fieldVal, exists := obj[field.Name]
  if !exists {
      if field.Optional != nil && *field.Optional { continue }
      return fmt.Errorf("%s.%s is required by datastream schema", path, field.Name)
  }

• It round-trips through GET datastream.

The OSHConnect-Python publisher's workaround (duplicating resultTime into result.timestamp on every observation) was unnecessary. The right fix is a one-line schema change at datastream-creation time.

Empirical proof (HEAD 4b994212)

Two datastreams with identical 6-field schemas; only difference is whether timestamp carries "optional": true. Same observation payload (timestamp omitted) sent to both:

#	Datastream	Result
T1	timestamp required (default)	400 result.timestamp is required by datastream schema
T2	timestamp "optional": true	201 Created ✓
T3	GET DS_TS_OPT	optional: true round-trips intact
T4	full-fields workaround on T1 schema	201
T5	extra undeclared:"x" field	201, persisted verbatim (open schema)
T6	typo latitudee instead of latitude	400 result.latitude is required — typo silently kept as extra; declared field reports missing

Answers to the 7 research questions

#	Question	Answer
Q1	OGC spec on result validation?	Validate against SWE Common DataRecord; required-by-default with per-field optional opt-out. cs-go matches.
Q2	SWE Common required vs optional?	Yes — optional boolean per field, default false.
Q3	Should schema support marking optional?	Already does. Confirmed by T2/T3.
Q4	Performance implication?	None measurable; O(declared_fields) per observation.
Q5	Other servers?	SensorHub leniency is publisher-friendly but not the spec mechanism. The interop gap is publisher-side: declare optionality.
Q6	Extra fields accepted/ignored/rejected?	Currently accepted-and-persisted (open schema, undocumented). Already tracked under #23.
Q7	"CAN appear" vs "MUST appear"?	Canonical SWE Common reading is "what the datastream emits"; required-by-default with opt-outs.

Sibling-finding audit (per the methodology pattern from #3/#4)

Strict-vs-lenient asymmetries surfaced in this area are already filed:

• [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21 (P2) — nilValues table is plumbed but unused by validator.
• [P3 bug] null on optional:true numeric field is rejected, while omission is accepted — asymmetric absence #22 (P3) — null on optional:true is rejected while omission is accepted.
• [P3 docs] Document result-schema validation contract: required/optional, null semantics, nilValues, open-schema policy #23 (P3 docs) — document the result-schema validation contract (required/optional, null semantics, nilValues, open-schema policy).

T6 (typo case) gets folded into #23 with a recommendation to either offer ?strict=true or surface a warning header listing unknown keys.

Recommendation

Close as research-resolved with no code changes required and no new follow-up issues.

Action items are entirely downstream:

1. (Publisher) Update OSHConnect-Python's datastream definitions to mark timestamp (redundant with top-level resultTime) as "optional": true. Drop the per-observation duplication workaround.
2. (Server docs) Cover the optional-field mechanism in the result-schema documentation deliverable already tracked under [P3 docs] Document result-schema validation contract: required/optional, null semantics, nilValues, open-schema policy #23.

Deferred audit candidates (not blocking)

• datachoice / dataarray / matrix branches do not consume optional on their subcomponents (grep shows the only consumers are in the datarecord and vector branches). May matter for advanced schemas; no observed publisher hits this. Will surface a separate issue only if/when needed.

[Sam-Bolling]

Closing as research-resolved with no code changes, in agreement with the maintainer.

Resolution

@SomethingCreativeStudios on the upstream triage pass: "Correct, as per rec from this issue, not doing anything." — i.e. the maintainer is accepting the recommendation in §6 of the evaluation that no server-side change is warranted.

Posting the rationale chain so the closure is auditable.

Why no code change

The issue framed cs-go as "strictly requiring every declared field in every observation". Live testing on HEAD 4b994212 refuted that framing:

• The validator at internal/api/observation_schema_validation.go (the datarecord branch, ~line 92, and the vector branch, ~line 115) already consumes field.Optional / coord.Optional and skips the required-check when it is true.
• The data model at internal/model/domains/datastream.go:225 declares Optional *bool \json:"optional,omitempty"``, and that field round-trips through GET (T3 of §2.3).
• End-to-end test T2 confirms: the same payload that returns 400 against an all-required schema returns 201 against a schema where timestamp is declared "optional": true. The mechanism works.

So the spec-correct behaviour the issue was implicitly asking for already exists. The publisher's symptom — having to duplicate resultTime into result.timestamp — was a publisher-side schema-declaration mistake (declaring a redundant field as required), not a server defect.

Spec alignment

OGC 23-002 Part 2 §9.7 defers result validation to the SWE Common Data Model. SWE Common 3.0 §7.4 (DataRecord) defines field membership via per-field optional boolean (default false). cs-go's Optional *bool field implements exactly this mechanism. There is no XML-schema minOccurs="0" analogue in the JSON encoding — the optional boolean is the mechanism.

Mapping rationale → original claims

Claim from issue body	Outcome
C1 — Server requires every declared field	Partially refuted. Server requires every non-optional field, per spec. optional: true skips the check (T2).
C2 — Publisher had to duplicate resultTime into result.timestamp	Workaround was unnecessary. Correct fix is publisher-side: declare timestamp with "optional": true in the datastream schema, then drop the duplication.
C3 — SensorHub silently accepts subset payloads	Out of scope; cs-go is spec-correct when the publisher uses the optionality mechanism appropriately. The interop gap is on the publisher side.
C4 — "P3 minor research spike"	Validated. Headline question was a misunderstanding of the existing mechanism; the genuine sibling defects in this area (null-vs-omit asymmetry, open-schema policy) were already filed elsewhere — see below.

Genuine sibling defects (already addressed elsewhere)

The §2.5 silent-failure audit during the evaluation surfaced three adjacent defects that are not part of #5's scope but are worth mentioning so this closure doesn't paper over them:

Sibling defect	Tracked under
null on an optional: true field is rejected while omission is accepted	#21 / batched into the 1b2b614 validator overhaul (closed)
Declared nilValues table never consulted by validator	#4 / batched into the same 1b2b614 (closed)
Open-schema policy: extra undeclared fields silently persisted; typo'd field name produces confusing diagnostics (T5, T6)	#23 — but the maintainer's note on #23 is "Not sure what the ask is, Claude did not either", so that thread needs a clearer follow-up rather than a fix here

So the genuine defects the optional-field exploration uncovered have either been fixed already (in the 1b2b614 validator overhaul that addressed #4 / #21) or are tracked separately on #23 pending a clarification round.

Action items (downstream, not this repo)

• Publisher (OSHConnect-Python / similar) — mark timestamp and any other resultTime-redundant fields as "optional": true in datastream schema declarations; drop the result-payload duplication workaround.
• Server docs — folded into [P3 docs] Document result-schema validation contract: required/optional, null semantics, nilValues, open-schema policy #23: document the result-schema validation contract (required-by-default + optional: true opt-out + open-schema-for-extras + nilValues lookup) in one place.

Cross-references

• Original evaluation: docs/research/issue-evaluations/issue-005.md (T1–T6 transcripts proving the optional mechanism works, source attestation showing the two Optional consumption sites, sibling-defect audit).
• Related fixes that did land: Research: NaN handling for numeric observation fields — rejection vs. nilValue support #4 / [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21 (validator overhaul in 1b2b614, now in this fork via merge 22e8bcd).
• Open thread for the open-schema documentation question: [P3 docs] Document result-schema validation contract: required/optional, null semantics, nilValues, open-schema policy #23.

Closing as not planned (won't fix), in agreement with the maintainer's triage.