Model output leaks into JSON validation errors despite DONT_LOG_MODEL_DATA

[abhay-codes07]

Please read this first

• Have you read the docs? Yes, including the OPENAI_AGENTS_DONT_LOG_MODEL_DATA flag, which defaults to not logging LLM inputs and outputs.
• Have you searched for related issues? Yes. fix: redact invalid tool argument errors #4182 redacted invalid tool argument errors and fix: nested MCP cleanup error redaction #4049, fix: redact MCP prompt and resource transport errors #4067, fix: redact direct MCP cleanup transport errors #4078 covered MCP transport errors. Nothing covers the model output path through validate_json.

Describe the bug

agents.util._json.validate_json puts the offending model output into its ModelBehaviorError message and chains the pydantic ValidationError, which repeats the same payload again in its own input_value:

raise ModelBehaviorError(
    f"Invalid JSON when parsing {json_str} for {type_adapter}; {e}"
) from e

Neither is gated on _debug.DONT_LOG_MODEL_DATA, which is True by default. Any application that logs the exception, or lets it reach a traceback, records the raw model output.

This is the model side of the redaction added for tool arguments in #4182. It affects the three callers that pass model generated JSON:

• agents/agent_output.py output schema validation, so any agent with an output_type
• agents/handoffs/__init__.py handoff input validation
• agents/realtime/handoffs.py the realtime equivalent

The SpanError raised alongside it already uses data={}, so tracing is redacted correctly. Only the exception leaks.

Debug information

• Agents SDK version: 0.19.4, reproduced on main at f6a32fee
• Related library versions: pydantic 2.x
• Python version: 3.12
• Operating system: Windows 11, not platform specific
• Model and model provider: none needed, reproduced with tests/fake_model.py
• Does the issue reproduce with the latest Agents SDK release? Yes.
• Does the issue occur consistently or intermittently? Consistently and deterministically.

Repro steps

import asyncio
import sys

from pydantic import BaseModel

from agents import Agent, Runner

sys.path.insert(1, "tests")  # run from the repo root
from fake_model import FakeModel
from test_responses import get_text_message

SECRET = "SUPER_SECRET_VALUE_9f3a"


class Structured(BaseModel):
    answer: str
    count: int


async def main() -> None:
    model = FakeModel()
    agent = Agent(name="A", model=model, output_type=Structured)
    # The model omits a required field, so validation fails.
    model.set_next_output([get_text_message(f'{{"answer": "{SECRET}"}}')])

    try:
        await Runner.run(agent, "go")
    except Exception as error:
        print("message:", error)
        print("cause  :", type(error.__cause__).__name__ if error.__cause__ else None)
        print("leaked :", SECRET in str(error))


asyncio.run(main())

Actual behavior

message: Invalid JSON when parsing {"answer": "SUPER_SECRET_VALUE_9f3a"} for TypeAdapter(Structured); 1 validation error for Structured
count
  Field required [type=missing, input_value={'answer': 'SUPER_SECRET_VALUE_9f3a'}, input_type=dict]
cause  : ValidationError
leaked : True

The payload appears twice, once as the raw JSON and once inside the chained ValidationError.

Expected behavior

With the default flag, the error names the failure without carrying the payload, and nothing reachable from it carries the payload either:

message: Invalid JSON when parsing model output for TypeAdapter(Structured)
cause  : None
leaked : False

With OPENAI_AGENTS_DONT_LOG_MODEL_DATA=0 the current detailed error is still useful and should be preserved.

Root-cause hypothesis

(hypothesis) validate_json predates the data logging policy and formats the payload unconditionally. Gating the detailed message on _debug.DONT_LOG_MODEL_DATA handles the message, but the chained ValidationError has to be dropped as well, since it embeds the same input. Raising the redacted error outside the except block, the way #4182 does, keeps both __cause__ and __context__ clear.

Proposed scope

Gate the detailed error in validate_json on the model data flag and raise the redacted error outside the except block. One change covers all three callers. No public API change, and no change to the already redacted span error.

I have a fix with regression tests ready and will open a PR referencing this issue.