fix(eval): use function calling for tool/input mocking so non-OpenAI models work (AE-1646)

[Chibionos]

Summary

Tool simulation and input generation in Studio Debug and Evaluation Set runs failed with AGENT_RUNTIME.UNEXPECTED_ERROR for non-OpenAI models (Anthropic Claude via Bedrock, Gemini), but worked with OpenAI/GPT.

Both eval mockers requested structured output via OpenAI-only response_format json_schema and parsed response.choices[0].message.content. On the normalized LLM Gateway, response_format structured output is only honored for OpenAI models; for Claude the content comes back empty/None, so json.loads(None) raised → wrapped as UiPathMockResponseGenerationError → AGENT_RUNTIME.UNEXPECTED_ERROR.

Fixes AE-1646 (customer: Sarasota Memorial Health Care System).

Root cause / regression

Regression from #1555, which started routing the agent's model into simulations. Before that, simulation always used a fixed OpenAI model (gpt_4_1_mini), so non-OpenAI providers were never exercised on this path — which is why Claude "worked before."

Fix

Switch both mockers to provider-agnostic function calling, mirroring llm_as_judge_evaluator (whose docstring already states function calling is the way to get structured output across OpenAI/Claude/Gemini):

• Build a forced tool that wraps the output/input schema under a response property, force it via tool_choice=required, and read tool_calls[0].arguments["response"] (already a parsed dict).
• Hoist nested $defs to the tool-parameters root so $refs from nested Pydantic models still resolve once the schema is wrapped.
• The normalized gateway's chat_completions now accepts raw-dict tools (pass-through) so arbitrary nested schemas survive — the ToolDefinition converter only emits flat properties.

New shared helper eval/mocks/_structured_output.py keeps both mockers DRY.

Tests

• test_llm_mockable_structured_output_via_tool_call — parametrized over gpt-4.1-mini, anthropic.claude-sonnet-4-5, gemini-2.5-pro; reproduces AE-1646 (content=None + tool_calls) and asserts the new contract.
• test_build_response_tool_hoists_defs_to_root + helper error-branch unit tests.
• test_raw_dict_tool_passthrough_mocked (platform) — asserts a nested array schema is forwarded byte-for-byte.
• Existing mocker/input/span tests updated to the function-calling contract (behavior assertions preserved).
• Full tests/cli/eval suite + platform mocked LLM tests green; ruff + mypy clean.

Note for reviewers

The OpenAI path also moves to function-calling (no longer response_format), matching the judge. Worth a live check that a Claude/Bedrock agent simulating a nested-model tool output round-trips through the gateway, since nested $defs in tool parameters has no prior precedent in this repo (the judge only used flat schemas).

🤖 Generated with Claude Code

[github-actions]

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

[Chibionos]

Re: the uipath-langchain cross-test heads-up above — that was from an earlier commit. After the adaptive fix (prefer response_format, fall back to function calling only when content is empty), the cross-tests pass on the latest commit: langchain-cross / {alpha,cloud,staging} and test-uipath-langchain (3.11/3.12/3.13 × ubuntu/windows) are all green. The heads-up comment is stale and can be disregarded.

Full suite is green: all eval test-cases (calculator-evals, simulation-testcase, tools-evals, etc.) pass across alpha/cloud/staging; lint, SonarCloud, build all pass.

[mjnovice]

Can we have separate classes for how we are doing Anthropic, OpenAI, Gemini etc. ?

[mjnovice]

Minor comment about making the generate_structured_output more modular.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Copilot]

Pull request overview

This PR addresses eval tool/input simulation failures for non-OpenAI models by introducing a provider-agnostic structured-output helper and updating the eval mockers and LLM gateway integration to support function-calling style structured responses (including nested schemas via raw-dict tool passthrough).

Changes:

• Add generate_structured_output() helper to prefer response_format when available and fall back to a forced tool call when content is empty/unsupported.
• Update eval LLM mocker and input mocker to use the shared structured-output helper, and adjust/extend unit tests for the new behavior (including non-OpenAI fallback).
• Update UiPathLlmChatService.chat_completions() to accept raw-dict tools (pass-through) so nested JSON-schema tool parameters are preserved; bump package versions accordingly.

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/uipath/uv.lock	Bumps uipath and uipath-platform locked versions.
packages/uipath/pyproject.toml	Bumps uipath version and raises minimum uipath-platform dependency.
packages/uipath-platform/uv.lock	Bumps uipath-platform locked version.
packages/uipath-platform/pyproject.toml	Bumps uipath-platform version.
packages/uipath-platform/src/uipath/platform/chat/_llm_gateway_service.py	Allows passing raw-dict tools through to the normalized gateway request body.
packages/uipath-platform/tests/services/test_uipath_llm_integration.py	Adds coverage ensuring raw-dict tool schemas are forwarded unchanged and tool_choice serialization works.
packages/uipath/src/uipath/eval/mocks/_structured_output.py	New shared helper for structured output with response_format-first + tool-call fallback; includes schema $defs inlining logic.
packages/uipath/src/uipath/eval/mocks/_llm_mocker.py	Switches mock response generation to the shared structured-output helper; improves error propagation.
packages/uipath/src/uipath/eval/mocks/_input_mocker.py	Switches input generation to the shared structured-output helper.
packages/uipath/tests/cli/eval/mocks/test_structured_output.py	New unit tests validating schema wrapping/inlining and response extraction/fallback behavior.
packages/uipath/tests/cli/eval/mocks/test_mocks.py	Updates existing mocks and adds a non-OpenAI fallback regression test (AE-1646).
packages/uipath/tests/cli/eval/mocks/test_input_mocker.py	Adds assertion that OpenAI path uses response_format without tool fallback.

Comments suppressed due to low confidence (1)

packages/uipath/src/uipath/eval/mocks/_input_mocker.py:158

• generate_llm_input() no longer surfaces a clear JSON-parsing error when the LLM returns invalid JSON (the previous code raised UiPathInputMockingError("Failed to parse LLM response as JSON: ...")). Now a json.JSONDecodeError from generate_structured_output() is wrapped as a generic "Failed to generate input" error, which makes debugging structured-output failures harder.

    except UiPathInputMockingError:
        raise
    except Exception as e:
        raise UiPathInputMockingError(f"Failed to generate input: {str(e)}") from e

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.