Skip to content

[Bug]: Codex MultiAgentV2 ATOF/ATIF traces lose readable delegated task content #425

Description

@willkill07

Affected area

  • Plugins
  • Observability or exporters
  • Framework integrations

Current behavior

When NeMo Relay instruments a Codex MultiAgentV2 session, the exact task delegated through spawn_agent is not available as readable audit data in either raw ATOF or derived ATIF.

An isolated no-context control run produced this parent-side function call summary:

{
  "name": "spawn_agent",
  "namespace": "collaboration",
  "task_name": "audit_clean",
  "fork_turns": "none",
  "message_prefix": "gAAAAAB",
  "message_length": 248,
  "contains_input_marker": false,
  "contains_output_marker": false
}

The child request contained an agent_message with a readable routing envelope followed by 248 bytes of encrypted_content. It contained neither the delegated input marker nor the requested output marker in plaintext:

{
  "thread_source": "subagent",
  "has_input_marker": false,
  "has_output_marker": false,
  "content": [
    {"type": "input_text"},
    {"type": "encrypted_content", "encrypted_length": 248}
  ]
}

The child nevertheless returned the required OUTPUT_CLEAN_28058, proving that the encrypted task was delivered and executed.

Additional observations:

  • Raw ATOF contained no tool scope for the built-in spawn_agent or wait_agent calls. The calls were recoverable only from the parent LLM response payload.
  • ATIF preserved the encrypted spawn_agent.message value rather than readable task content.
  • ATIF emitted the parent and child as two separate top-level trajectories. Both had zero subagent_trajectories, so the causal parent-child relationship was not represented in the derived trajectory.
  • The input marker remains visible in the parent's original user prompt because the reproduction must tell the parent what to delegate. It is absent specifically from the recorded spawn_agent arguments and child request, which are the audit surfaces under test.

This is downstream of OpenAI Codex's intentional MultiAgentV2 encrypted-message transport:

Expected behavior

NeMo Relay should make the resulting trace contract explicit and preserve as much causal structure as the available Codex surfaces allow:

  1. Preserve parent-child subagent lineage in ATOF/ATIF even when delegated message text is encrypted.
  2. Represent delegated task content as explicitly unavailable/encrypted instead of presenting opaque ciphertext as ordinary readable tool arguments.
  3. Document the Codex MultiAgentV2 limitation and its effect on ATOF, ATIF, OpenTelemetry, and OpenInference exports.
  4. When Codex exposes an authorized plaintext audit field or event, ingest that field as the delegated task audit copy while preserving configured sanitization/export policy.
  5. Add a regression fixture for spawn_agent with fork_turns = "none" that asserts both encrypted transport handling and trajectory linkage.

Steps to reproduce

  1. Configure an isolated temporary workspace:
# .nemo-relay/config.toml
[agents.codex]
command = "codex"
# .nemo-relay/plugins.toml
version = 1

[[components]]
kind = "observability"
enabled = true

[components.config]
version = 2

[components.config.atof]
enabled = true

[[components.config.atof.sinks]]
type = "file"
output_directory = "/tmp/nemo-relay-atof/atof"
filename = "events.jsonl"
mode = "overwrite"

[components.config.atif]
enabled = true
output_directory = "/tmp/nemo-relay-atof/atif"
filename_template = "{session_id}.atif.json"
  1. Run Codex with user configuration ignored, MultiAgentV2 enabled, and a no-context child:
nemo-relay run -- codex exec \
  --json \
  --ignore-user-config \
  --skip-git-repo-check \
  --enable multi_agent_v2 \
  -m gpt-5.4 \
  -C /tmp/nemo-relay-atof/workspace \
  'Call spawn_agent exactly once with task_name audit_clean, fork_turns none, and this complete message: Remember INPUT_CLEAN_28058, do not call tools, and respond exactly OUTPUT_CLEAN_28058. Then wait for the child. Do not send follow-up messages. Return the child response exactly.'
  1. Inspect the parent LLM response in atof/events.jsonl. The spawn_agent.message argument begins with gAAAAAB and does not contain either marker.
  2. Inspect the child LLM start event. The request contains an agent_message with encrypted_content and neither marker in plaintext.
  3. Inspect the child LLM end event. It contains OUTPUT_CLEAN_28058.
  4. Inspect the generated ATIF files. The parent and child are separate top-level trajectories, and the parent tool call contains ciphertext.

Environment

  • NeMo Relay: 0.6.0
  • NeMo Relay commit: 45b3b44c4184eba0d4a52954f727b4054e24a46d
  • Codex CLI: 0.143.0
  • Model: gpt-5.4
  • OS: macOS 26.5.1, arm64
  • Rust: rustc 1.93.0 (254b59607 2026-01-19)
  • Build/run source: local debug build from the commit above
  • Isolation: temporary workspace, --ignore-user-config, fork_turns = "none"

Impact

Codex MultiAgentV2 users receive an incomplete execution record: Relay can show that a child ran and what it returned, but cannot reliably show the exact instruction that caused that result. This limits debugging, evaluation, incident analysis, and compliance auditing.

There is no complete Relay-side workaround for recovering plaintext that Codex does not expose. MultiAgentV1 may retain readable messages when it is selectable, but some models or product surfaces may select MultiAgentV2. Relay can still improve lineage, accurately label the missing content, document the limitation, and be ready to ingest an upstream audit field when one becomes available.

Metadata

Metadata

Assignees

Labels

Bugissue describes bug; PR fixes bug

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions