-
Notifications
You must be signed in to change notification settings - Fork 4.4k
examples: add ModelBehaviorError handling patterns and self-healing agent #3669
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
tahirkhan05
wants to merge
2
commits into
openai:main
from
tahirkhan05:examples/model-behavior-error-patterns
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
158 changes: 158 additions & 0 deletions
158
examples/agent_patterns/model_behavior_error_handling.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| """Handling ModelBehaviorError — three patterns. | ||
|
|
||
| ``ModelBehaviorError`` is raised when the model does something the SDK cannot | ||
| handle: calling a tool that does not exist, returning malformed JSON for a | ||
| structured output, or producing an output that fails schema validation. | ||
|
|
||
| This file demonstrates three practical ways to deal with it, in increasing | ||
| order of complexity. | ||
|
|
||
| Pattern 1 — Ignore and return a fallback value. | ||
| Useful when the task is best-effort and a graceful default is acceptable. | ||
|
|
||
| Pattern 2 — Log and re-raise with context. | ||
| Useful in production pipelines where you want structured error reporting | ||
| without losing the original exception. | ||
|
|
||
| Pattern 3 — Retry with a fallback agent. | ||
| When the primary model misbehaves, hand the same task to a more reliable | ||
| (often larger) fallback model. | ||
|
|
||
| Run with: | ||
| python -m examples.agent_patterns.model_behavior_error_handling | ||
| """ | ||
|
|
||
| import asyncio | ||
| from dataclasses import dataclass | ||
|
|
||
| from agents import Agent, ModelBehaviorError, Runner, function_tool | ||
|
|
||
|
|
||
| @function_tool | ||
| def get_stock_price(ticker: str) -> str: | ||
| """Return a (mocked) stock price for the given ticker symbol. | ||
|
|
||
| Args: | ||
| ticker: Stock ticker symbol, e.g. AAPL. | ||
| """ | ||
| prices = {"AAPL": "$189.30", "GOOGL": "$175.20", "MSFT": "$415.50"} | ||
| return prices.get(ticker.upper(), f"Ticker '{ticker}' not found.") | ||
|
|
||
|
|
||
| primary_agent = Agent( | ||
| name="Stock Assistant", | ||
| instructions=( | ||
| "You are a stock price assistant. Use the get_stock_price tool to answer questions. " | ||
| "Do not call any other tools." | ||
| ), | ||
| tools=[get_stock_price], | ||
| ) | ||
|
|
||
| # A more conservative fallback agent using explicit instructions to reduce errors. | ||
| fallback_agent = Agent( | ||
| name="Stock Assistant (fallback)", | ||
| instructions=( | ||
| "You are a stock price assistant. You have ONE tool: get_stock_price(ticker). " | ||
| "Call it exactly once with the ticker symbol the user mentioned. " | ||
| "Do not call any other tool. Do not make up data." | ||
| ), | ||
| tools=[get_stock_price], | ||
| ) | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Pattern 1 — Return a safe default on error | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| async def pattern_1_fallback_value(task: str) -> str: | ||
| """Run the agent; return a safe default string if ModelBehaviorError occurs.""" | ||
| try: | ||
| result = await Runner.run(primary_agent, task) | ||
| return result.final_output | ||
| except ModelBehaviorError as exc: | ||
| print(f"[pattern-1] ModelBehaviorError: {exc.message}") | ||
| return "Sorry, I could not retrieve that information right now." | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Pattern 2 — Log and re-raise with enriched context | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| @dataclass | ||
| class EnrichedModelError(Exception): | ||
| original: ModelBehaviorError | ||
| task: str | ||
| agent_name: str | ||
|
|
||
| def __str__(self) -> str: | ||
| return ( | ||
| f"Agent '{self.agent_name}' failed on task: {self.task!r}\n" | ||
| f"Reason: {self.original.message}" | ||
| ) | ||
|
|
||
|
|
||
| async def pattern_2_log_and_reraise(task: str) -> str: | ||
| """Run the agent; re-raise ModelBehaviorError with structured context.""" | ||
| try: | ||
| result = await Runner.run(primary_agent, task) | ||
| return result.final_output | ||
| except ModelBehaviorError as exc: | ||
| raise EnrichedModelError( | ||
| original=exc, | ||
| task=task, | ||
| agent_name=primary_agent.name, | ||
| ) from exc | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Pattern 3 — Retry with a fallback agent | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| async def pattern_3_fallback_agent(task: str) -> str: | ||
| """Try the primary agent; fall back to a more reliable agent on error.""" | ||
| try: | ||
| result = await Runner.run(primary_agent, task) | ||
| print("[pattern-3] Primary agent succeeded.") | ||
| return result.final_output | ||
| except ModelBehaviorError as exc: | ||
| print(f"[pattern-3] Primary agent failed ({exc.message}), trying fallback agent...") | ||
| result = await Runner.run(fallback_agent, task) | ||
| print("[pattern-3] Fallback agent succeeded.") | ||
| return result.final_output | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Demo | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| async def main() -> None: | ||
| task = "What is the current price of AAPL stock?" | ||
|
|
||
| print("=" * 60) | ||
| print("Pattern 1 — fallback value") | ||
| print("=" * 60) | ||
| output = await pattern_1_fallback_value(task) | ||
| print(f"Output: {output}\n") | ||
|
|
||
| print("=" * 60) | ||
| print("Pattern 2 — log and re-raise") | ||
| print("=" * 60) | ||
| try: | ||
| output = await pattern_2_log_and_reraise(task) | ||
| print(f"Output: {output}\n") | ||
| except EnrichedModelError as exc: | ||
| print(f"Caught enriched error:\n{exc}\n") | ||
|
|
||
| print("=" * 60) | ||
| print("Pattern 3 — fallback agent") | ||
| print("=" * 60) | ||
| output = await pattern_3_fallback_agent(task) | ||
| print(f"Output: {output}\n") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| asyncio.run(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| """Self-healing agent pattern. | ||
|
|
||
| When a model calls a tool that does not exist, or produces malformed JSON for | ||
| a structured output, the SDK raises ``ModelBehaviorError``. The default | ||
| behaviour is to let that exception propagate and crash the run. | ||
|
|
||
| This example shows a loop that catches ``ModelBehaviorError``, appends the | ||
| error text to the conversation as a correction, and retries — giving the | ||
| model a chance to fix its own mistake before giving up. | ||
|
|
||
| Crucially, the retry resumes from ``exc.run_data.new_items``, which contains | ||
| all the turns the agent completed before the error occurred. This avoids | ||
| re-running prior work (and re-triggering tool side effects) on every heal. | ||
|
|
||
| The pattern is useful when: | ||
| - You are using a smaller/cheaper model that occasionally calls wrong tools. | ||
| - You want resilience without replacing the model or hard-coding fallbacks. | ||
| - The task is long-running and restarting from scratch is expensive. | ||
|
|
||
| Run with: | ||
| python -m examples.agent_patterns.self_healing_agent | ||
| """ | ||
|
|
||
| import asyncio | ||
| from typing import Union | ||
|
|
||
| from agents import Agent, ModelBehaviorError, Runner, TResponseInputItem, function_tool | ||
|
|
||
| MAX_SELF_HEALS = 3 | ||
|
|
||
|
|
||
| @function_tool | ||
| def add(a: int, b: int) -> int: | ||
| """Return the sum of two integers. | ||
|
|
||
| Args: | ||
| a: First integer. | ||
| b: Second integer. | ||
| """ | ||
| return a + b | ||
|
|
||
|
|
||
| @function_tool | ||
| def multiply(a: int, b: int) -> int: | ||
| """Return the product of two integers. | ||
|
|
||
| Args: | ||
| a: First integer. | ||
| b: Second integer. | ||
| """ | ||
| return a * b | ||
|
|
||
|
|
||
| agent = Agent( | ||
| name="Math Assistant", | ||
| instructions=( | ||
| "You are a math assistant. Use the available tools to answer questions. " | ||
| "Only call tools that exist: add, multiply." | ||
| ), | ||
| tools=[add, multiply], | ||
| ) | ||
|
|
||
|
|
||
| async def run_with_self_healing(task: str) -> str: | ||
| """Run an agent task, retrying up to MAX_SELF_HEALS times on ModelBehaviorError. | ||
|
|
||
| On each retry the completed turns from ``exc.run_data.new_items`` are | ||
| preserved and a correction note is appended, so the model continues from | ||
| where it left off rather than restarting the full task. | ||
|
|
||
| Args: | ||
| task: The initial user message to send to the agent. | ||
|
|
||
| Returns: | ||
| The final output string from the agent. | ||
|
|
||
| Raises: | ||
| ModelBehaviorError: If the model keeps misbehaving after all retries. | ||
| """ | ||
| input: Union[str, list[TResponseInputItem]] = task | ||
| heals_remaining = MAX_SELF_HEALS | ||
|
|
||
| while True: | ||
| try: | ||
| result = await Runner.run(agent, input) | ||
| return result.final_output | ||
| except ModelBehaviorError as exc: | ||
| if heals_remaining <= 0: | ||
| print(f"[self-heal] Giving up after {MAX_SELF_HEALS} attempts.") | ||
| raise | ||
|
|
||
| heals_remaining -= 1 | ||
| print( | ||
| f"[self-heal] ModelBehaviorError caught " | ||
| f"({MAX_SELF_HEALS - heals_remaining}/{MAX_SELF_HEALS}): {exc.message}" | ||
| ) | ||
|
|
||
| # Build the correction message to append. | ||
| correction: TResponseInputItem = { | ||
| "role": "user", | ||
| "content": ( | ||
| f"Your previous response caused an error: {exc.message}\n" | ||
| "Please correct your approach and try again using only the tools available." | ||
| ), | ||
| } | ||
|
|
||
| # Resume from the completed turns rather than restarting from scratch. | ||
| # exc.run_data.new_items holds every RunItem produced before the error, | ||
| # so prior tool calls and their results are preserved and not re-run. | ||
| completed_items: list[TResponseInputItem] = ( | ||
| [item.to_input_item() for item in exc.run_data.new_items] | ||
| if exc.run_data is not None | ||
| else [] | ||
| ) | ||
|
|
||
| if completed_items: | ||
| input = completed_items + [correction] | ||
| print( | ||
| f"[self-heal] Resuming from {len(completed_items)} completed turn(s) " | ||
| "with correction appended..." | ||
| ) | ||
| else: | ||
| # No completed turns to preserve — fall back to the original task. | ||
| input = f"{task}\n\n{correction['content']}" | ||
| print("[self-heal] No prior turns to resume from. Retrying from scratch...") | ||
|
|
||
|
|
||
| async def main() -> None: | ||
| task = "What is (7 + 3) multiplied by 4?" | ||
|
|
||
| print(f"Task: {task}") | ||
| print("-" * 60) | ||
|
|
||
| output = await run_with_self_healing(task) | ||
|
|
||
| print("-" * 60) | ||
| print(f"Answer: {output}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| asyncio.run(main()) | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a failure happens after at least one completed turn,
new_itemscontains only the items produced by this run whileRunErrorDetails.inputholds the user input separately, so this retry starts the next run with assistant/tool items plus the correction but without the original task. In this version the fresh issue is that the attempted fix now preserves generated turns but drops the run input, and on later heals can also drop prior retry context; build the retry fromexc.run_data.inputplus the convertednew_itemsbefore appending the correction.Useful? React with 👍 / 👎.