Research: Time field encoding — strict ISO 8601 string requirement vs. numeric timestamp acceptance

[Sam-Bolling]

Finding

The server requires all time/datetime fields in observation payloads to be ISO 8601 strings (e.g., "2026-04-17T00:00:00Z"). Numeric timestamps (e.g., 1773100000.0 as epoch seconds) are rejected. This is a research spike to determine whether this is the correct behavior per the OGC spec and whether any flexibility should be added for interoperability with existing clients.

Review Source: Live integration testing from OSHConnect-Python publishers against both connected-systems-go and OSH SensorHub
Severity: P3-Minor (research spike)
Category: API Design / Interoperability
Ownership: connected-systems-go

---

Problem Statement

When publishing observations, our publishers initially sent time fields as numeric epoch values (e.g., 1773100000.0). The Go CSAPI server rejects these with a validation error, requiring ISO 8601 string format instead.

Observed behavior:

POST /api/datastreams/<uuid>/observations
Content-Type: application/json

{
  "resultTime": 1773100000.0,
  "result": { ... }
}

→ Server returns 400 — time field must be a string.

Comparative behavior:

Server	Numeric time	String time
OSH SensorHub	✅ Accepted	✅ Accepted
connected-systems-go	❌ Rejected	✅ Accepted

Workaround applied: Our publishers detect the Go server via URL and conditionally coerce time fields to ISO 8601 strings with str() wrapper. This works but means the publisher must know which server variant it's talking to.

Research Questions

This spike should produce analysis, findings, and recommendations that may result in follow-on issues:

1. What does OGC 23-002 say about time field encoding in JSON? Is ISO 8601 string the only valid encoding, or are numeric timestamps also permitted?
2. What does SWE Common / ISO 8601 say? The Time component in SWE Common Data Model — does it mandate string encoding in JSON context?
3. What does the broader OGC API ecosystem do? Do other OGC APIs (Features, EDR, Records) accept numeric timestamps?
4. Is SensorHub's acceptance of numeric timestamps a deliberate feature or an accidental leniency? Would other CSAPI servers also reject numerics?
5. Should connected-systems-go accept numeric timestamps as a convenience? What are the tradeoffs of strict vs. lenient parsing?
6. Should the rejection error message be improved? Currently it may not clearly indicate what format is expected.

Expected Deliverables

• Analysis document summarizing spec requirements for time field encoding
• Comparison table of how multiple CSAPI server implementations handle time fields
• Recommendation: keep strict, add lenient parsing, or improve error messaging
• If changes are recommended, file follow-on implementation issue(s)

Scope

• ❌ Do NOT implement changes in this spike — research and recommend only
• ❌ Do NOT change client-side workarounds — they work and should remain as defense-in-depth

References

#	Document	What It Provides
1	OGC 23-002 §9.7	Observation resource JSON encoding
2	OGC 23-002 §16	JSON encoding rules for Part 2 resources
3	SWE Common Data Model 3.0	Time component encoding rules
4	OSHConnect-Python usgs_eq_publisher.py	Publisher with _coerce_time_to_str workaround
5	OSH SensorHub API	Comparative server that accepts both formats

[Sam-Bolling]

Evaluation outcome — strict rejection is spec-conformant; the real defects are misleading errors + sibling silent-failure paths

Full eval: docs/research/issue-evaluations/issue-003.md. Evidence (transcripts, source extracts, spec excerpt): docs/research/evidence/issue-003/.

Spec verdict

OGC 23-002 (CSAPI Part 2) bundled OAS31 defines phenomenonTime and resultTime as type: string, format: date-time. JSON Schema format: date-time ⇒ RFC 3339. Numeric epoch is not a conformant encoding. cs-go's rejection is correct per spec; SensorHub's leniency is a non-conformant convenience extension.

Live verification on HEAD (4b994212) — 13 cases

#	Input	HTTP	Body / Note
T1	resultTime:"2026-04-30T12:00:00Z"	201	baseline
T2	resultTime: 1773100000.0 (numeric)	400	{"error":"resultTime is required"} ← misleading; field IS present
T3-T4	numeric int / epoch ms	400	same misleading body
T5	"2026-04-30" (date only)	400	"Invalid resultTime format" (correct)
T6-T8	empty / null / missing	400	"resultTime is required" (correct)
T9	valid resultTime + numeric phenomenonTime: 1773100000	201	silent drop — observation created, no error, phenomenonTime field discarded
T10-T11	RFC 3339 with ms / TZ offset	201	accepted
T12	Datastream POST with phenomenonTime: [1773100000.0, null], resultTime: [1773100000.0, null]	201	silent data loss — datastream stored with NO phenomenonTime/resultTime at all
T13	same shape, string array	201	accepted

Mechanism (internal/api/observation_handler.go + internal/model/common_shared/time_range.go)

// observation_handler.go
if resultTimeRaw, ok := raw["resultTime"].(string); ok && resultTimeRaw != "" {
    // … parse …
}
// no else branch — wrong-type input falls through silently to the
// "resultTime is required" guard at the bottom of the function

// time_range.go
if s, ok := arr[0].(string); ok {
    if t, err := time.Parse(time.RFC3339, s); err == nil {
        tr.Start = &t
    }
} // both branches drop silently, no error returned

The strict reject and the silent drop come from the same "type-assert-and-discard" pattern — applied inconsistently across Observation.resultTime (rejects via the late IsZero() check), Observation.phenomenonTime (no late check, silent drop), and Datastream.{phenomenonTime,resultTime} via TimeRange.UnmarshalJSON (silent drop).

Verdicts

Surface	Behaviour	Spec?	Defect
Observation resultTime numeric	reject 400	✓	UX (misleading message)
Observation phenomenonTime numeric	silent drop, 201	✗	P2 data loss
Datastream TimeRange numeric/malformed	silent drop, 201	✗	P2 data loss
RFC 3339 with ms / TZ offset	accept	✓	none

Recommendation

Strict-with-clear-errors, not leniency:

1. Reject wrong-type time inputs explicitly in decodeObservationPayload (both resultTime and phenomenonTime) — branch on v.(string) failure with "resultTime must be an RFC 3339 string". Symmetric for phenomenonTime.
2. Make TimeRange.UnmarshalJSON return errors on type-assertion failure and on parse failure — currently swallows both.
3. Document encoding (numeric epoch not accepted; one-liner conversion recipes for Python/Go/JS publishers).
4. Defer numeric leniency. If SWG later wants it, gate behind config + deprecation header. Not required by spec.

Follow-up issues being filed

• F1 (P2 bug) — TimeRange.UnmarshalJSON silently swallows non-string and parse-failed times → Datastream POST with numeric/malformed times silently stores empty TimeRange.
• F2 (P2 bug) — decodeObservationPayload silently drops numeric phenomenonTime; should reject symmetrically with resultTime.
• F3 (P3 UX) — Error "resultTime is required" is misleading when the field is present but wrong type.

Recommendation for #3 itself

Close as research-resolved (no leniency change) once F1/F2/F3 are tracked. Or convert to a documentation issue tracking the encoding policy + publisher recipes. The original framing ("should we accept numeric?") has a clear spec answer (no); the productive work surfaces are F1–F3.

[Sam-Bolling]

Closing as resolved upstream. Posting full evidence so the closure is auditable.

Resolution

This issue began as a research spike about whether resultTime / phenomenonTime should accept numeric epoch in addition to RFC 3339 strings, but live verification surfaced two material defects worth fixing regardless of which way the spec spike landed:

1. Misleading error — numeric resultTime produced {"error":"resultTime is required"} even when the field was clearly present (it failed a silent string type assertion in decodeObservationPayload, fell through to the zero-value guard, and reported "missing" — see issue-003.md §2.3).
2. Silent data loss — numeric or wrong-shape phenomenonTime / resultTime on Observation and Datastream POSTs were silently dropped by TimeRange.UnmarshalJSON's un-checked arr[0].(string) assertion + ignored time.Parse error, producing a 201 with empty time fields (§2.4 + T9, T12).

Both defects, plus a comprehensive overhaul of the time-range surface, were addressed across two upstream commits, both now in this fork via merge 22e8bcd:

• 1b2b614 — "Adding support for 'latest' for TimeRange / Better error handling for times"
• c2ab201 — "time range and better 400 (now includes more info and field info)"

Maintainer's read

@SomethingCreativeStudios on the upstream triage pass: "Better date handling (the ogish time range bug)". Several adjacent issues — #11 ("mostly just a dup from time range related bug"), #18 ("fixed, but like a lot of the time related ones, it's mostly a dup"), #19 ("PhenomenonTime guard, mostly dup of time range bug"), and #20 ("ResultTime needs better error handling — overhauled most error handling not just time") — were collapsed onto this same fix.

What changed

1. TimeRange is now a proper, errorful unmarshaller. internal/model/common_shared/time_range.go gained ~144 LOC. Key changes:

• UnmarshalJSON now rejects non-string, non-null, non-".." elements explicitly instead of silently dropping them. From the file header comment:

  Non-null, non-".." elements that are not valid RFC3339 strings are rejected.

• The struct now also supports object form {"start":"...","end":"..."} and the "start/end" string format, and a new Latest bool flag to support ?resultTime=latest query semantics (paired with 1b2b614).
• A new test suite time_range_test.go (+137 LOC) covers the previously silent-failure surface.

2. New strict decoder that preserves field context. internal/model/common_shared/decode.go (new, +271 LOC) introduces a typed UnknownFieldError with a JSON-path field, plus DecodeWithFieldErrors[T] that runs strict JSON decoding and preserves field context for TimeRange errors that would otherwise lose their location:

type UnknownFieldError struct {
    Field string
    Path  string  // e.g. "observedProperties[0]"
}

// Also produces named field errors for any TimeRange fields, since
// custom UnmarshalJSON errors from TimeRange otherwise lose their
// field context.
func DecodeWithFieldErrors[T any](data []byte) (T, error) { ... }

This is the missing ingredient that made the decodeObservationPayload error misleading: with DecodeWithFieldErrors, the body now reports the actual field and reason rather than a downstream "required" guard.

3. New internal/api/errors.go (+56 LOC, included in c2ab201) standardizes the 400-response shape to carry field and reason, replacing the bare {"error":"..."} strings with structured payloads — addressing C4 of the issue body.

4. Comprehensive e2e coverage — 1b2b614 added e2e/observations_test.go (+170 LOC), e2e/system_events_test.go (+123 LOC), e2e/control_streams_test.go (+93 LOC), plus six other test files. The previously passing-but-wrong shapes from T9/T12 are now part of the regression net.

Mapping fix → original claims

Claim from issue body	Fixed how
C1 — Numeric resultTime returns 400	Spec-correct rejection preserved; error body now says why (wrong type, named field) instead of misleading "required"
C2 — OSH SensorHub accepts numeric epoch	Out of scope for cs-go; spec finding (RFC 3339 only) governs cs-go behaviour
C3 — Publishers coerce numeric → string	No change needed; cs-go remains spec-conformant on input
C4 — Server error message is unhelpful	Resolved: structured error payloads in internal/api/errors.go + field-aware DecodeWithFieldErrors
§2.2 T9 (hidden) — numeric phenomenonTime silently dropped on Observation	Fixed: TimeRange.UnmarshalJSON now rejects non-string elements
§2.2 T12 (hidden, severe) — numeric [start, end] array silently dropped on Datastream POST → silent data loss	Fixed: same UnmarshalJSON overhaul rejects the malformed shape with a field-tagged 400

Verification

Against current main (post-merge 22e8bcd):

• internal/model/common_shared/time_range.go exists with the new UnmarshalJSON shape and Latest field, header comment confirms non-RFC3339 rejection.
• internal/model/common_shared/decode.go exists with UnknownFieldError + DecodeWithFieldErrors[T].
• internal/api/errors.go exists for structured 400 responses.
• internal/model/common_shared/time_range_test.go and e2e/observations_test.go are present.

Cross-references

• Original evaluation: docs/research/issue-evaluations/issue-003.md (T1–T13 transcripts, source attestation for both silent-failure mechanisms, spec analysis).
• Issues collapsed onto the same fix per maintainer triage: Temporal query parameters (resultTime, phenomenonTime, datetime) silently ignored — all values discarded, including latest #11, TimeRange.UnmarshalJSON silently swallows non-string and parse-failed times — Datastream POST stores empty TimeRange without error #18, decodeObservationPayload silently drops numeric phenomenonTime; should reject symmetrically with resultTime #19, Misleading error 'resultTime is required' when field is present but wrong type #20.

Closing as completed.