[AUTO-2361] Document agent vs legacy confidence scores in get-run-results

[rosalie-byrnes]

Context

AUTO-2361 (from Slack) reported that as part of the agent mode rollout, the v2 results API started having only one confidence score instead of model confidence and OCR confidence, that v1 still defaulted OCR confidence to 1 for agent mode, and that the change was never documented or communicated.

The published Get run results docs still listed a confidence object with both model and ocr, with no indication that behavior depends on processing mode. The same gap affects the include_confidence_scores query param description (and was noted for the Postman collection).

I asked Cursor to verify current behavior against:

1. This OpenAPI spec (source of the Fern API docs)
2. The Instabase monorepo run-results / export / agent-mode extraction code
3. Related product docs on agent vs legacy mode
4. Git archaeology for when the agent-mode OCR stub and related skips landed

What Cursor found (important correction to the ticket premise)

Run results did not drop OCR confidence

For GET /api/v2/apps/runs/{run_id}/results with include_confidence_scores=true, the API still returns both keys:

• files/documents/fields/confidence/model
• files/documents/fields/confidence/ocr

plus document-level classification_confidence/model.

Verified in py/src/instabase/api_server/handler/api/v2/utils/results_utils.py (ResultsField._parse): when the flag is true, the response always builds:

{
  "model": field_result.get("model_confidence"),
  "ocr": field_result.get("ocr_confidence"),
}

Missing underlying scores become null, not omitted keys.

Agent mode stubs OCR to 1.0 (same idea as the v1 comms)

Agent-mode apps extract via the unified extractor / agent extract path. When building refined phrases, OCR/char_confidence are hardcoded to 1.0 because agent mode does not run a separate OCR step:

• py/src/instabase/flow_step_executors/unified_extractor/unified_extractor_step_executor.py — create_refined_phrase_from_pixels sets 'char_confidence': 1.0 and 'ocr_confidence': 1.0 # Agent mode doesn't do separate OCR
• Inline refiner path in the same executor resets both to 1.0 after refinement
• Export reads refined_phrase.get_char_confidence() into result.ocr_confidence (flow_export/export_results.py)
• v1 and v2 run results share that export pipeline; deployment integration formatting also uses ResultsAPIOptions(include_confidence_scores=True)

So the accurate statement is:

Mode	confidence.model	confidence.ocr
Agent mode (AI runtime 2.x)	Multimodal model confidence	Always 1.0 (placeholder; not measured)
Legacy mode (AI runtime 1.x)	Extraction model confidence	Measured OCR average, or null if no OCR provenance

This matches product docs: legacy mode alone offers OCR confidence scores (fern/docs/pages/apps/create.mdx, agent mode vs legacy mode section).

Related but different: Build extracted-fields skip

[EPD-5767] (2026-05-14; customer-facing around AI Hub 26.20) skips OCR population on the Build extracted-fields / validations path (aihub_build.py / Project.get_doc_extracted_values) for agent-mode projects. That is not the run-results serializer change. Do not use EPD-5767 as the “run results OCR removed” date.

When the run-results agent-mode stub came into effect

Event	Reference	Approx. date / release
Unified extractor introduced with char_confidence: 1.0	[AUTO-400] #73078	2025-07-30
Explicit ocr_confidence: 1.0 “so that the validations pass”	[AUTO-549] #74426 → 25.38.0-aihub	2025-09-08
Agent mode public preview	AI runtime 2.0.0	~2025-11 (25.44)
Agent mode GA	Release notes	25.48

Best date for the delayed-docs acknowledgment: agent mode GA (25.48), with the stub present from AUTO-549 / 25.38.

Extra signal

SDK system tests previously asserted confidence.ocr was non-null, then commented that assertion out in [JIRA-000] Update SDK to version 0.9.0 (sdk/system-tests/test_runs.py) — consistent with OCR being unreliable / placeholder-dependent depending on run shape, not with the key being removed from the schema.

Fixes in this PR

Docs-only OpenAPI updates for Get run results:

1. include_confidence_scores query param — lists classification_confidence/model, clarifies keys are always returned when the flag is true (values may be null), and documents agent vs legacy meaning, linking to /automate/creating#agent-mode-and-legacy-mode.
2. documentField.confidence — object-level description for when the block appears; model and ocr descriptions updated for mode-conditional meaning, null cases, and the agent-mode 1.0 stub.
3. classification_confidence — notes when returned and when model may be null.
4. Response examples — ocr example values changed from fabricated measured scores (0.69… / 0.79…) to 1.0 to match agent-mode default behavior (new apps are agent mode).

No server/API behavior change. Schema shape unchanged (still confidence.model + confidence.ocr).

Testing / verification

• Traced v2 results handler → flow_binary.get_flow_results → export_results → ResultsAPIResponse / ResultsField to confirm both confidence keys are always emitted when the flag is true.
• Traced agent-mode field creation through build_refined_phrases_from_fields → create_refined_phrase_from_pixels and confirmed OCR/char_confidence stub to 1.0.
• Traced legacy path through apply_refiner_fns.get_provenance_info_for_text (OCR-derived char_confidence) for contrast.
• Confirmed v1 flow binary results and v2 run results share the export path (same underlying OCR stub in agent mode).
• Confirmed product docs already state OCR confidence is a legacy-mode feature.
• Git archaeology dated AUTO-549 / 25.38 stub and distinguished EPD-5767 Build-path skip from run results.
• Link target /automate/creating#agent-mode-and-legacy-mode verified (slug + heading already used elsewhere in Fern docs).
• Diff review of openapi.yaml only; no Fern temp copy in this PR (docs site sync is a follow-up, same as last time).

Next steps (outside this PR)

1. Copy these OpenAPI description updates into the Fern docs site / regenerate API reference as usual.
2. Draft a release note acknowledging the delayed API-docs clarification against agent mode GA (25.48), correcting the “OCR removed from v2” framing if needed.
3. Update the AI Hub Postman collection query-param description the same way (called out in the original Slack thread; not in this repo).
4. Optionally regenerate the Python SDK so generated model field descriptions pick up the new OpenAPI text (document_field_confidence.py etc.). Shape/types do not need a hand edit.

What to review

• Query param description accurately reflects agent vs legacy behavior
• confidence.model / confidence.ocr / classification_confidence.model descriptions match server reality
• Example ocr: 1.0 is acceptable for the default (agent mode) response shape
• Ticket correction (not removed; stubbed to 1.0) is acceptable messaging for GTM/support

Made with Cursor

Made with Cursor