Custom Evaluator Failing with server Side Error

[vvshankar78]

• Package Name: azure-ai-projects
• Package Version: 2.1.0
• Operating System: windows
• Python Version: 3.12.10

Describe the bug
Tried running this code https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py.

I am getting this error on the foundry.
There was an error with this evaluation run.Evaluation system error: (InternalError) AOAI evaluation run eval_69e899704cfc8191950ea6ee82203d49/evalrun_69e899732fe8819192460c960c9256ac failed with status failed.

Sharing the logs:
2026-04-22 09:48:31 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | Creating PythonGrader instance with 5 parameters
2026-04-22 09:48:31 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | Initializing Azure OpenAI evaluator: azureai://built-in/evaluators/azure-openai/python_grader
2026-04-22 09:48:31 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | Successfully created PythonGrader instance for evaluator: my_custom_evaluator_code_1776851292
2026-04-22 09:48:31 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | Execution path: foundry | Foundry evaluators: 1 | OpenAI graders: 0
2026-04-22 09:48:31 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | evaluatorDefinition provided: {'type': 'code', 'metrics': {'result': {'type': 'continuous', 'desirable_direction': 'increase', 'min_value': 0.0, 'max_value': 1.0}}}
2026-04-22 09:48:31 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | evaluatorId provided: azureai://accounts/Foundry-agentic-workshop/projects/proj-agentic-workshop/evaluators/my_custom_evaluator_code_1776851292/versions/1
2026-04-22 09:48:31 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | Execute evaluators with evaluators: ['my_custom_evaluator_code_1776851292']
2026-04-22 09:48:50 | ERROR | evalrun_586372ed7c09450ab99db65bbf02e41f | Error during evaluation for run ID evalrun_586372ed7c09450ab99db65bbf02e41f: (InternalError) AOAI evaluation run eval_69e899704cfc8191950ea6ee82203d49/evalrun_69e899732fe8819192460c960c9256ac failed with status failed.
Traceback (most recent call last):
File "/app/evaluation/evaluation.py", line 82, in run_evaluation
results, execution_path = await execute_evaluators_async(evaluation, input_path, output_path, logger, log_path, client=client)
File "/app/evaluation/eval_utils.py", line 1059, in execute_evaluators_async
results = await asyncio.to_thread(
File "/opt/miniconda/lib/python3.10/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/opt/miniconda/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/app/evaluation/eval_utils.py", line 986, in _execute_evaluators_blocking
results = evaluate(
File "/opt/venv/lib/python3.10/site-packages/azure/ai/evaluation/_evaluate/_evaluate.py", line 907, in evaluate
raise e
File "/opt/venv/lib/python3.10/site-packages/azure/ai/evaluation/_evaluate/_evaluate.py", line 863, in evaluate
results = _evaluate(
File "/opt/venv/lib/python3.10/site-packages/azure/ai/evaluation/_evaluate/_evaluate.py", line 1036, in _evaluate
raise e
File "/opt/venv/lib/python3.10/site-packages/azure/ai/evaluation/_evaluate/_evaluate.py", line 1019, in _evaluate
aoai_results, aoai_metrics = _get_evaluation_run_results(eval_run_info_list) # type: ignore
File "/opt/venv/lib/python3.10/site-packages/azure/ai/evaluation/_evaluate/_evaluate_aoai.py", line 281, in _get_evaluation_run_results
cur_output_df, cur_run_metrics = _get_single_run_results(run_info)
File "/opt/venv/lib/python3.10/site-packages/azure/ai/evaluation/_evaluate/_evaluate_aoai.py", line 310, in _get_single_run_results
raise EvaluationException(
azure.ai.evaluation._exceptions.EvaluationException: (InternalError) AOAI evaluation run eval_69e899704cfc8191950ea6ee82203d49/evalrun_69e899732fe8819192460c960c9256ac failed with status failed.
2026-04-22 09:48:50 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | Start pending upload for evaluationResult
2026-04-22 09:48:50 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | start_upload_url: https://foundry-agentic-workshop.services.ai.azure.com/api/projects/proj-agentic-workshop/evaluationresults/eval-result-evalrun_586372ed7c09450ab99db65bbf02e41f-GVHK/versions/1/startPendingUpload?api-version=2025-05-01
2026-04-22 09:48:50 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | token_length: 2948, last 4 characters: HXyg
2026-04-22 09:48:50 | INFO | evalrun_586372ed7c09450ab99db65bbf02e41f | Successful application/json response received with status: 200

To Reproduce

1. Ran it multiple times

Expected behavior
Expect the custom evaluation results to be run on foundry runtime and display the metrics.

Screenshots

Additional context

[YoYoJa]

Hello @vvshankar78 , custom code evaluators are not supported in eastus, could you try another supported region per this wiki?
https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-regions-limits-virtual-network#unsupported-regions-for-azure-openai-graders

[github-actions]

Hi @vvshankar78. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.