Eval span input/output are double-wrapped ({"input": ...} / {"output": ...}) instead of bare values; task span carries no input/output at all

[one-klink-to-rule-them-all]

Summary

Two adjacent bugs in Eval.evalOne(...) (braintrust-sdk/src/main/java/dev/braintrust/eval/Eval.java) cause Java-driven experiments to render incorrectly in the Braintrust UI and to be incompatible with the input/output conventions used by the Python, JavaScript, and Go SDKs:

1. The root eval span's braintrust.input_json and braintrust.output_json attributes are written as {"input": <value>} and {"output": <value>} — a one-level wrapper around the actual values — instead of the bare values the rest of the Braintrust ecosystem writes.
2. The task span is created and ended without braintrust.input_json or braintrust.output_json ever being set. In the Python SDK the task span receives both via span.log(input=task_args[0], output=output).

All line references below are pinned to master at e110611.

Locations

Root eval span — input double-wrap and (likely) expected key mismatch: Eval.java#L110-L113

braintrust-sdk-java/braintrust-sdk/src/main/java/dev/braintrust/eval/Eval.java

Lines 110 to 113 in e110611

	.setAttribute(
	"braintrust.input_json",
	toJson(Map.of("input", datasetCase.input())))
	.setAttribute("braintrust.expected", toJson(datasetCase.expected()))

Root eval span — output double-wrap on the success path: Eval.java#L141-L143

braintrust-sdk-java/braintrust-sdk/src/main/java/dev/braintrust/eval/Eval.java

Lines 141 to 143 in e110611

	rootSpan.setAttribute(
	"braintrust.output_json",
	toJson(Map.of("output", taskResult.result())));

Root eval span — output double-wrap on the task-exception path: Eval.java#L149-L151

braintrust-sdk-java/braintrust-sdk/src/main/java/dev/braintrust/eval/Eval.java

Lines 149 to 151 in e110611

	rootSpan.setAttribute(
	"braintrust.output_json",
	toJson(Collections.singletonMap("output", null)));

task span — created and ended without ever setting braintrust.input_json / braintrust.output_json: Eval.java#L129-L160

braintrust-sdk-java/braintrust-sdk/src/main/java/dev/braintrust/eval/Eval.java

Lines 129 to 160 in e110611

	{ // run task
	var taskSpan =
	tracer.spanBuilder("task")
	.setAttribute(PARENT, "experiment_id:" + experimentId)
	.setAttribute(
	"braintrust.span_attributes",
	toJson(Map.of("type", "task")))
	.startSpan();
	taskSpanId = taskSpan.getSpanContext().getSpanId();
	try (var unused =
	BraintrustContext.ofExperiment(experimentId, taskSpan).makeCurrent()) {
	taskResult = task.apply(datasetCase, parameters);
	rootSpan.setAttribute(
	"braintrust.output_json",
	toJson(Map.of("output", taskResult.result())));
	} catch (Exception e) {
	taskSpan.setStatus(StatusCode.ERROR, e.getMessage());
	taskSpan.recordException(e);
	taskSpan.end();
	rootSpan.setStatus(StatusCode.ERROR, e.getMessage());
	rootSpan.setAttribute(
	"braintrust.output_json",
	toJson(Collections.singletonMap("output", null)));
	log.debug("Task threw exception for input: " + datasetCase.input(), e);
	// run scoreForTaskException on each scorer
	for (var scorer : scorers) {
	runScoreForTaskException(experimentId, rootSpan, scorer, e, datasetCase);
	}
	return;
	}
	taskSpan.end();
	}

Cross-SDK comparison

Python (braintrust-sdk-python/py/src/braintrust/framework.py, _run_evaluator_internal) — root eval span gets bare input / expected, then the inner task span carries its own bare input / output:

base_event = dict(
    name="eval",
    span_attributes={"type": SpanTypeAttribute.EVAL},
    input=datum.input,        # bare value, no {"input": ...} wrapper
    expected=datum.expected,
    ...
)
...
with root_span.start_span("task", span_attributes={"type": SpanTypeAttribute.TASK}) as span:
    output = await await_or_run(event_loop, evaluator.task, *task_args)
    span.log(input=task_args[0], output=output)              # task span carries I/O
root_span.log(output=output, metadata=metadata, tags=tags)   # eval span: bare output

Go (braintrust-sdk-go/examples/manual-llm-logging/main.go) — the official manual-OTLP-attribute example writes bare values directly to braintrust.input_json / braintrust.output_json:

setJSONAttr(span, "braintrust.input_json",  messages)  // bare value
setJSONAttr(span, "braintrust.output_json", output)    // bare value

Effect in the Braintrust UI

• Input column on every experiment row shows {"input": …} instead of ….
• Output column shows {"output": …} instead of ….
• Dataset-row ↔ experiment-row comparison (which keys off the bare input shape) is broken — the input recorded for the experiment no longer matches the input recorded for the dataset row it originated from.
• The Task span in the experiment trace view shows empty Input/Output cells.

Screenshots

Suggested fix

1. In Eval.evalOne(...), set braintrust.input_json to toJson(datasetCase.input()) (drop the Map.of("input", ...) wrapper) on the root eval span.
2. Same for braintrust.output_json → toJson(taskResult.result()) (drop the Map.of("output", ...) wrapper). Also applies to the exception path where Collections.singletonMap("output", null) becomes a bare null JSON value.
3. On the task span, additionally set braintrust.input_json = toJson(datasetCase.input()) and, after the task returns, braintrust.output_json = toJson(taskResult.result()) — matching Python's span.log(input=..., output=...) on the task span.
4. While here: confirm the attribute name braintrust.expected (vs braintrust.expected_json) is correct for a JSON-string attribute. The rest of the SDK uses the _json suffix when the value is a JSON-encoded string.

Environment

• SDK versions affected: v0.3.7 → v0.3.10, unchanged on master @ e110611.
• JDK: 25.

[one-klink-to-rule-them-all]

Still reproducible on master @ 23d9fe9 / v0.3.14. Refreshed permalinks (line numbers shifted by ~2 due to an unrelated field addition in PR #114):

• Root eval span — input double-wrap: Eval.java#L112-L115
• Root eval span — output double-wrap on the success path: Eval.java#L143-L145
• Root eval span — output double-wrap on the task-exception path: Eval.java#L151-L153
• task span created and ended without ever setting braintrust.input_json / braintrust.output_json: Eval.java#L131-L162

PR #114 (classifier support) added strictly additive code at the bottom of evalOne but did not alter the lines this issue is about.

[realark]

Note: changes are in main, but this won't be fully complete until my changes to braintrust ingest are deployed and we release another version of the java sdk. leaving this open until that happens.

[realark]

fixed in 0.3.17