[MLOB-7510] session-level documentation 2

[jennm]

What does this PR do? What is the motivation?

• Addresses the feedback given here.
• Adds session-level templates in prompt_templates.md
• Reorders prompt_templates.md to discussion session, trace, and then span.
• Reorders the menu to display Session-Level Evals before Trace-Level Evals

Merge instructions

Merge readiness:

• Ready for merge

For Datadog employees:

Your branch name MUST follow the <name>/<description> convention and include the forward slash (/). Without this format, your pull request will not pass CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.

If your branch doesn't follow this format, rename it or create a new branch and PR.

[6/5/2025] Merge queue has been disabled on the documentation repo. If you have write access to the repo, the PR has been reviewed by a Documentation team member, and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #documentation channel in Slack.

AI assistance

Additional notes

[github-actions]

Preview links (active after the build_preview check completes)

Modified Files

• https://docs-staging.datadoghq.com/jenn/MLOB-7510-part-2/llm_observability/evaluations/custom_llm_as_a_judge_evaluations/prompt_templating
• https://docs-staging.datadoghq.com/jenn/MLOB-7510-part-2/llm_observability/evaluations/custom_llm_as_a_judge_evaluations/session_level_evaluations

[domalessi]

Left some feedback/suggestions! Let me know if you have any Qs, and hit me up when ready for another look

[domalessi]

Minor — "Which paths are available depends on..." is a bit indirect. Suggestion:

Suggested change

Custom LLM-as-a-judge prompts inject session, trace, or span data into the {{< ui >}}User{{< /ui >}} message by wrapping a field path in `{{ ... }}`. The System Prompt holds the static instructions to the LLM judge and does not resolve placeholders. The same syntax works in both the test pane and at evaluation time. Which paths are available depends on the evaluation scope you choose—session, trace, or span.

Custom LLM-as-a-judge prompts inject session, trace, or span data into the {{< ui >}}User{{< /ui >}} message by wrapping a field path in `{{ ... }}`. The System Prompt holds the static instructions to the LLM judge and does not resolve placeholders. The same syntax works in both the test pane and at evaluation time. The available paths depend on the evaluation scope: session, trace, or span.

[domalessi]

Worth a quick sanity check: the (scope) labels in parentheses add real value for ambiguous rows like {{name}} or {{spans}} (where the scope isn't obvious from the pattern). For self-evident rows like {{traces[0].spans[0].meta.input.value}}, the label is arguably noise. I don't think it's a blocker, but could slim the table down by dropping the labels from rows where the pattern already makes the scope obvious.

[domalessi]

Two rows break the Description (scope) pattern used by the rest of the table. Fixing for consistency.

Suggested change

	| `{{span_input}}`, `{{span_output}}` | Span-scope aliases |
	| `{{*}}` | Entire session, trace, or span payload as JSON |
	| `{{span_input}}`, `{{span_output}}` | Aliases for span input and output fields (span scope) |
	| `{{*}}` | Entire payload as JSON (session, trace, or span scope) |

[domalessi]

The bracket notation {{traces...].spans...}} has a mismatched ] — there's no matching [ opening it. Based on the examples in this section (e.g., {{traces[0].spans[0].meta.input.value}}), the intended shorthand is {{traces[...].spans[...]}}.

Suggested change

	Session-scope evaluations expose every trace in the [user session][1] under the `traces` array. Each trace includes its own `spans` array, so you can read across traces and spans in one prompt. Use `{{traces...}}` paths (and nested `{{traces...].spans...}}` paths) to build session-level judges. The `{{span_input}}` and `{{span_output}}` aliases are not available in session scope.
	Session-scope evaluations expose every trace in the [user session][1] under the `traces` array. Each trace includes its own `spans` array, so you can read across traces and spans in one prompt. Use `{{traces[...]}}` paths (and nested `{{traces[...].spans[...]}}` paths) to build session-level judges. The `{{span_input}}` and `{{span_output}}` aliases are not available in session scope.

[domalessi]

The session lifecycle mechanics (30-min timeout, late-span behavior) are conceptual details that belong on the Session-Level Evaluations page, not a syntax reference. Suggest trimming to just the prerequisite and the cross-ref:

Suggested change

Session-level evaluations require spans to be tagged with a `session_id`. See [Tracking user sessions][1] to instrument your application. A session is considered complete after **30 minutes** of inactivity (no new spans for that session, measured from the most recent span); the evaluation runs once at that point with every trace and span from the session. Spans that arrive more than 30 minutes after the previous span are not included. See [Session-Level Evaluations][2] for configuration, example prompts, and when to choose session scope over trace or span scope.

Session-level evaluations require spans to be tagged with a `session_id`. See [Tracking user sessions][1] to instrument your application. See [Session-Level Evaluations][2] for configuration, example prompts, and when to choose session scope over trace or span scope.

[domalessi]

A few subsections have their explanatory text in the wrong place or missing:

1. "Filter traces or spans by attribute" (here) and "Filter spans by attribute" (trace scope, line 99) — the explanatory paragraph currently sits after the code block. Moving it before the code frames the examples before the reader hits them, consistent with how "Fan-out across traces" is structured.

2. "Direct field paths" (line 122) and "Array access" (line 131) — both go straight to code with no framing. Suggested lead-ins:

   • Direct field paths: "Reference any span field by its JSON path."
   • Array access: "Use bracket notation to index into, slice, or fan out over array fields."

The self-evident sections — "Reference the whole session/trace" and "Pick by index" — read fine without lead-ins.

[domalessi]

"The same way as in trace scope" is a backwards reference — session-scope now appears before trace-scope, so the reader hasn't seen it yet. The sentence also immediately explains the behavior anyway, so the phrase adds nothing.

Suggested change

	Use `[*]` on `traces` or `spans` the same way as in trace scope: values from every matching trace or span are collected and joined with newlines (`\n`), or serialized as JSON when the resolved values are objects.
	Use `[*]` on `traces` or `spans` to fan out: values from every matching element are joined with newlines (`\n`), or serialized as JSON when the resolved values are objects.

[domalessi]

The session panel name is written as prose while the other two panel names in the same list use the {{< ui >}} shortcode. Can you verify the exact UI label for the session panel and wrap it consistently?

Suggested change

	- Pick a sample in the panel on the right—the sample session pane listing traces in the session (session scope), {{< ui >}}Spans in Selected Trace{{< /ui >}} (trace scope), or {{< ui >}}Filtered Spans{{< /ui >}} (span scope)—then click {{< ui >}}Test Evaluation{{< /ui >}} to preview how each placeholder resolves on real data before saving the configuration.
	- Pick a sample in the panel on the right—{{< ui >}}Sample Session{{< /ui >}} listing traces in the session (session scope), {{< ui >}}Spans in Selected Trace{{< /ui >}} (trace scope), or {{< ui >}}Filtered Spans{{< /ui >}} (span scope)—then click {{< ui >}}Test Evaluation{{< /ui >}} to preview how each placeholder resolves on real data before saving the configuration.

[domalessi]

Same mismatched bracket as line 42 — {{traces...].spans...}} should be {{traces[...].spans[...]}}.

Suggested change

	- Prefer `{{traces}}` or targeted `{{traces...].spans...}}` paths for session judges when you need cross-turn context; use `{{spans}}` when a single trace is enough. See [Session-Level Evaluations][2] for scope guidance and example prompts.
	- Prefer `{{traces}}` or targeted `{{traces[...].spans[...]}}` paths for session judges when you need cross-turn context; use `{{spans}}` when a single trace is enough. See [Session-Level Evaluations][2] for scope guidance and example prompts.