Research: NaN handling for numeric observation fields — rejection vs. nilValue support

[Sam-Bolling]

Finding

The server rejects observation payloads containing "NaN" as a string value for numeric (float/double) fields. This is a research spike to determine whether NaN handling for numeric observation fields is specified by OGC, what other servers do, and what the recommended behavior should be.

Review Source: Live integration testing from OSHConnect-Python OpenSky ADS-B publisher against both connected-systems-go and OSH SensorHub
Severity: P3-Minor (research spike)
Category: API Design / Data Quality
Ownership: connected-systems-go

---

Problem Statement

Real-world sensor data frequently contains missing or undefined values. For example, ADS-B transponders may not report barometric altitude for all aircraft, resulting in null or NaN values in the source data. When our OpenSky publisher encounters these, the Python json.dumps() emits "NaN" (a string) since JSON has no native NaN representation.

Observed behavior:

POST /api/datastreams/<uuid>/observations
{
  "result": {
    "baro_altitude": "NaN",
    "geo_altitude": 10500.0,
    ...
  }
}

→ connected-systems-go rejects this with a validation error: the field is declared as numeric in the schema but "NaN" is a string.

Comparative behavior:

Server	"NaN" string for numeric field	null for numeric field	0.0 substitute
OSH SensorHub	✅ Accepted (leniently)	Unknown	✅ Accepted
connected-systems-go	❌ Rejected	Unknown	✅ Accepted

Workaround applied: Our OpenSky publisher detects the Go server and replaces NaN values with 0.0 before publishing. This loses the semantic meaning of "no data" vs "zero."

Research Questions

This spike should produce analysis, findings, and recommendations that may result in follow-on issues:

1. How does JSON handle NaN? JSON spec (RFC 8259) does not include NaN. What are the common approaches for representing it?
2. What does OGC SWE Common say about missing/no-data values? Is there a noDataValue or nilValue mechanism in the datastream schema?
3. What does OGC 23-002 say about handling missing observation values? Are there provisions for partial results?
4. How should the server handle null for a required numeric field? Is this a valid way to express "no data"?
5. Should the server support "NaN" as a special sentinel? If so, how should it be stored in PostGIS?
6. Is 0.0 a reasonable substitute, or does it corrupt data integrity? (For altitude, 0.0 means sea level, which is semantically different from "no reading.")
7. Should the schema support declaring a nilValue per field? Then clients could use { "baro_altitude": -999.0 } as an explicit no-data sentinel.

Expected Deliverables

• Analysis of JSON NaN handling approaches across the ecosystem
• Review of OGC SWE Common nilValue / noData mechanisms
• Comparison of how CSAPI server implementations handle missing numeric data
• Recommendation: reject, accept, convert, or implement nilValue support
• If changes are recommended, file follow-on implementation issue(s)

Scope

• ❌ Do NOT implement changes in this spike — research and recommend only
• ❌ Do NOT change client-side workarounds — they are functional

References

#	Document	What It Provides
1	RFC 8259	JSON specification — no NaN support
2	OGC SWE Common 3.0 §7.3	Quantity component, nilValues
3	OGC 23-002 §9.7	Observation result encoding
4	IEEE 754	Floating-point NaN definition
5	OSHConnect-Python opensky_publisher.py	Publisher with NaN→0.0 workaround

[Sam-Bolling]

Deep evaluation completed (2026-04-30)

Full report: docs/research/issue-evaluations/issue-004.md. Evidence (live transcript, static-analysis snapshot, spec extract): docs/research/evidence/issue-004/.

Headline finding

This was filed as a research spike — "should we accept NaN / null / sentinel?" — but the live tests turned up a concrete defect that reframes the question:

DatastreamDataComponent.NilValues is plumbed through the data model and round-trips correctly on GET, but the result validator never reads it. internal/api/observation_schema_validation.go enforces if !isNumber(value) { return error } for every Quantity field with no fallback to the field's declared nilValues table.

So the SWE Common nilValues mechanism — which the OGC spec OAS31 example shows explicitly with value: NaN / value: '-Infinity' / value: +Infinity — is half-built: declared, persisted, returned, ignored.

Spec position is unambiguous

OGC 23-002 / SWE Common 3.0 use a per-field nilValues: [(reason, value), …] table, where the value is encoded per the data stream's encoding rule. RFC 8259 forbids bare NaN/Infinity numeric literals, so the JSON-encoding publisher convention is the string form "NaN", "-Infinity", "+Infinity" — exactly what OSHConnect-Python emits and what OSH SensorHub accepts. The cs-go behaviour is therefore non-conformant whenever a publisher follows the spec idiom.

Empirical confirmation (HEAD 4b994212, 16 test cases)

#	Input	DS schema	HTTP	Verdict
T1	"NaN" (string)	no nilValues	400 must be a number	reject
T2	"NaN" (string)	nilValues=[{reason:missing,value:"NaN"}]	400 same	defect — declared, ignored
T3/T4	null	either	400 must be a number	reject
T5	omitted (required)	—	400 is required	OK
T6	-999.0	no nilValues	201	sentinel passes by type coincidence
T7	0.0 (current workaround)	—	201	works but corrupts semantics
T8/T9	bare NaN/Infinity literals	—	400 invalid character	correct (RFC 8259)
T11	extra undeclared:"hi" field	—	201, persisted verbatim	open schema — undocumented
T12	-999	nilValues=[{value:-999}]	201	passes Quantity type check; nilValues still not consulted
T13	GET DS_NIL	—	nilValues round-trips	data model works
T14	omitted	optional:true	201	optional honoured for absence
T15	null	optional:true	400	optional does NOT cover null — asymmetric

Same defect ships in the parent fork (SomethingCreativeStudios/connected-systems-go, sha 2bbb6202 matches).

Verdict on the original questions

Original question	Answer
Accept "NaN" for numeric?	Yes — but only when the datastream's nilValues table declares it. This matches the spec's mechanism instead of inventing a new convention.
Accept null?	Treat as absence iff optional: true; reject otherwise. Resolves the T14/T15 asymmetry.
Use 0.0 workaround?	No — silently corrupts the data. Already the OSHConnect symptom.
"Implement nilValue"?	Yes — and the schema model is already in place. The work is just in validateDataComponentValue.

Recommendation

Strict-with-nilValue-fallback. For each leaf scalar branch (quantity/count/time/category/text/boolean):

if !isNumber(value) {
    for _, nv := range component.NilValues {
        if matchesNilValue(nv.Value, value) { return nil } // accept
    }
    return fmt.Errorf("%s must be a number (or a declared nilValue)", path)
}

Plus: in the datarecord branch, treat value == nil as absence when the field is optional: true.

Optional follow-on (v1.1): record the matched reason URI alongside the stored observation so downstream consumers can distinguish missing from real-zero.

Follow-ups filed

• F1 (P2 bug) — nilValues ignored by result validator (the headline finding).
• F2 (P3 bug) — null vs. omitted asymmetry on optional:true.
• F3 (P3 docs) — Document the result-schema validation contract (required/optional, null semantics, nilValues, open-schema policy for extra fields per T11).

Will close this issue once F1-F3 are filed and the eval doc is committed.

[Sam-Bolling]

Closing as resolved upstream. Posting full evidence so the closure is auditable.

Resolution

This issue began as a research spike about whether observation numeric fields should accept "NaN" / null / a nilValue sentinel, but live testing surfaced a material defect that the issue body had not anticipated:

• nilValues was already plumbed through the data model (DatastreamDataComponent.NilValues, round-trips on GET — confirmed by T13) — but the validator never consulted it. A datastream declaring nilValues: [{reason: missing, value: "NaN"}] per OGC / SWE Common still rejected "NaN" with must be a number (T2). grep -rn NilValues internal/api/ returned zero hits.
• Asymmetric strictness on optional: true: a missing field passed (T14), but a null field on the same optional: true quantity was rejected (T15).

Both are addressed by upstream commit 1b2b614 ("Adding support for 'latest' for TimeRange / Better error handling for times"), which despite its name added internal/api/observation_schema_validation.go (the validator that issue-004.md §2.3 attests was missing nilValue awareness) and observation_schema_validation_test.go (+104 LOC of coverage). Now in this fork via merge 22e8bcd.

Maintainer's read

@SomethingCreativeStudios on the upstream triage pass: "Better number handling" (this issue) and on the closely-related #21: "during validation, look at nilValues; they can be anything even a string like 'NaN'" — i.e. the maintainer consciously chose the spec-conformant path: nilValues are looked up first, before the type switch, and the matched sentinel is accepted regardless of its JSON type. That is exactly the missing piece §2.1 of the evaluation called for.

What changed

1. nilValues is now consulted, first, before any type check. observation_schema_validation.go:82 (top of validateDataComponentValue):

// A value matching a declared nilValue is valid regardless of type.
if matchesNilValue(component, value) {
    return nil
}
switch componentType {
    case "datarecord": ...
    case "quantity":   if !isNumber(value) { ... }
    ...
}

The implementation (line 368) does an encoding/json decode of each declared nilValue.Value and compares with reflect.DeepEqual, so a declared "NaN" string nilValue matches an incoming "NaN" string, a declared -999 numeric nilValue matches an incoming -999, and the comparison short-circuits the type rejection. This is exactly what §2.1 / spec-idiomatic SWE Common decoding requires:

func matchesNilValue(component *domains.DatastreamDataComponent, value any) bool {
    if component == nil || len(component.NilValues) == 0 { return false }
    for _, nv := range component.NilValues {
        var decoded any
        if err := json.Unmarshal(nv.Value, &decoded); err != nil { continue }
        if reflect.DeepEqual(decoded, value) { return true }
    }
    return false
}

2. The optional / null asymmetry (T14 vs T15) is fixed. Inside the datarecord and vector branches:

fieldVal, exists := obj[field.Name]
if !exists {
    if field.Optional != nil && *field.Optional { continue }
    return fmt.Errorf("%s.%s is required by datastream schema", ...)
}
// null JSON value for an optional field is acceptable (treat as absent).
if fieldVal == nil && field.Optional != nil && *field.Optional {
    continue
}

null on an optional: true field is now treated as absent — the symmetric outcome the evaluation called for in §2.4.

3. Coverage was added at the unit-test level (observation_schema_validation_test.go, +104 LOC) and at the e2e level (the e2e/observations_test.go additions in 1b2b614). The previously silent T2/T15 paths are now part of the regression net.

Mapping fix → original claims

Claim from issue body	Fixed how
C1 — Server rejects "NaN" (string) for a numeric field	Now: rejection only when no matching nilValues declared. With a declared "NaN" nilValue, matchesNilValue accepts before the type switch.
C2 — OSH SensorHub accepts "NaN" leniently	Out of scope; cs-go now supports the spec-idiomatic path (declare nilValues, then send the sentinel).
C3 — Publisher workaround substitutes 0.0 and loses meaning	No longer necessary; the spec-correct carrier (string sentinel against declared nilValues) is now honoured.
C4 — "Minor / research spike" framing	Refuted by the evaluation; the actual fix is functional, not just docs.
§2.2 T2 — declared nilValues ignored by validator	Fixed: matchesNilValue consulted first.
§2.4 T15 — null on optional:true rejected	Fixed: null treated as absent on optional fields in datarecord and vector.
§2.7 — sibling "schema fields plumbed-but-not-consumed" anti-pattern	At least NilValues is now consumed; Constraint and Updatable remain open as a future audit item, but that's a separate concern from this issue.

Items not addressed by this fix (and why that's fine for closure)

• T11 — open-schema "extra fields silently kept on observations" is the scope of [P3 docs] Document result-schema validation contract: required/optional, null semantics, nilValues, open-schema policy #23. The maintainer's note on [P3 docs] Document result-schema validation contract: required/optional, null semantics, nilValues, open-schema policy #23 was "Not sure what the ask is, Claude did not either" — that's a separate clarification thread, not in scope here.
• null for non-optional quantity is still a 400 (correct: spec doesn't allow bare null for required scalars).
• Bare NaN / Infinity literals still 400 with invalid character — correct per RFC 8259.

Verification

Against current main (post-merge 22e8bcd):

• internal/api/observation_schema_validation.go exists; matchesNilValue(component, value) is the first thing validateDataComponentValue does, before the switch componentType.
• null-on-optional short-circuit is present in both datarecord and vector branches.
• internal/api/observation_schema_validation_test.go exists.

Cross-references

• Original evaluation: docs/research/issue-evaluations/issue-004.md (T1–T16 transcripts, source attestation showing zero NilValues references in internal/api/ at filing time, spec extract for the SWE Common nilValues idiom).
• Adjacent fix (validator hardening): [P2 bug] Result validator ignores nilValues table — spec-idiomatic "NaN" / "-Infinity" strings rejected as wrong-type #21 ("during validation, look at nilValues; they can be anything even a string like 'NaN'") — same 1b2b614 / c2ab201 commit window.

Closing as completed.