From ecb81bb7bbeb54a70e63149d8378fcd4978c8741 Mon Sep 17 00:00:00 2001 From: Sydney Lister Date: Wed, 16 Apr 2025 17:40:45 -0700 Subject: [PATCH 1/3] preliminary redteam tsg --- sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md b/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md index 023c75084dd4..b88969745993 100644 --- a/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md +++ b/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md @@ -51,6 +51,11 @@ The Adversarial simulator does not support selecting individual harms, instead w Identify the type of simulations being run (adversarial or non-adversarial). Adjust parameters such as `api_call_retry_sleep_sec`, `api_call_delay_sec`, and `concurrent_async_task`. Please note that rate limits to llm calls can be both tokens per minute and requests per minute. +## Handle RedTeam errors + +### Target resource not found +When initializing an Azure OpenAI model directly as `target` for a `RedTeam` scan, ensure `azure_endpoint` is specified in the format `https://.openai.azure.com/openai/deployments//chat/completions?api-version=2025-01-01-preview`. If using `AzureOpenAI`, `endpoint` should be specified in the format `https://.openai.azure.com/`. + ## Logging You can set logging level via environment variable `PF_LOGGING_LEVEL`, valid values includes `CRITICAL`, `ERROR`, `WARNING`, `INFO`, `DEBUG`, default to `INFO`. From d3c4368371c5fe07fbe8e1705dd24415ecb92e83 Mon Sep 17 00:00:00 2001 From: Sydney Lister Date: Thu, 17 Apr 2025 12:54:27 -0700 Subject: [PATCH 2/3] add column mapping bullet to evaluate tsg --- sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md b/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md index b88969745993..5685690d68e0 100644 --- a/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md +++ b/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md @@ -30,6 +30,10 @@ This guide walks you through how to investigate failures, common errors in the ` - Additionally, if you're using a virtual network or private link, and your evaluation run upload fails because of that, check out this [guide](https://docs.microsoft.com/azure/machine-learning/how-to-enable-studio-virtual-network#access-data-using-the-studio). +### Troubleshoot Column Mapping Issues + +- When using `column_mapping` parameter in evaluators, ensure all keys and values are non-empty strings and contain only alphanumeric characters. Empty strings, non-string values, or non-alphanumeric characters can cause serialization errors and issues in downstream applications. Example of valid mapping: `{"query": "${data.query}", "response": "${data.response}"}`. + ### Troubleshoot Safety Evaluator Issues - Risk and safety evaluators depend on the Azure AI Studio safety evaluation backend service. For a list of supported regions, please refer to the documentation [here](https://aka.ms/azureaisafetyeval-regionsupport). From c79aae77fbf365c9616609a07b758f46ac12272c Mon Sep 17 00:00:00 2001 From: Sydney Lister Date: Mon, 21 Apr 2025 10:18:48 -0700 Subject: [PATCH 3/3] add storage permissions warning section --- .../azure-ai-evaluation/TROUBLESHOOTING.md | 28 ++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md b/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md index c6d03bb3c7a6..7d8a17a549cd 100644 --- a/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md +++ b/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md @@ -6,11 +6,18 @@ This guide walks you through how to investigate failures, common errors in the ` - [Handle Evaluate API Errors](#handle-evaluate-api-errors) - [Troubleshoot Remote Tracking Issues](#troubleshoot-remote-tracking-issues) + - [Troubleshoot Column Mapping Issues](#troubleshoot-column-mapping-issues) - [Troubleshoot Safety Evaluator Issues](#troubleshoot-safety-evaluator-issues) + - [Troubleshoot Quality Evaluator Issues](#troubleshoot-quality-evaluator-issues) - [Handle Simulation Errors](#handle-simulation-errors) - [Adversarial Simulation Supported Regions](#adversarial-simulation-supported-regions) + - [Need to generate simulations for specific harm type](#need-to-generate-simulations-for-specific-harm-type) + - [Simulator is slow](#simulator-is-slow) +- [Handle RedTeam Errors](#handle-redteam-errors) + - [Target resource not found](#target-resource-not-found) + - [Insufficient Storage Permissions](#insufficient-storage-permissions) - [Logging](#logging) -- [Get additional help](#get-additional-help) +- [Get Additional Help](#get-additional-help) ## Handle Evaluate API Errors @@ -63,6 +70,25 @@ Adjust parameters such as `api_call_retry_sleep_sec`, `api_call_delay_sec`, and ### Target resource not found When initializing an Azure OpenAI model directly as `target` for a `RedTeam` scan, ensure `azure_endpoint` is specified in the format `https://.openai.azure.com/openai/deployments//chat/completions?api-version=2025-01-01-preview`. If using `AzureOpenAI`, `endpoint` should be specified in the format `https://.openai.azure.com/`. +### Insufficient Storage Permissions +If you see an error like `WARNING: Failed to log artifacts to MLFlow: (UserError) Failed to upload evaluation run to the cloud due to insufficient permission to access the storage`, you need to ensure that proper permissions are assigned to the storage account linked to your Azure AI Project. + +To fix this issue: +1. Open the associated resource group being used in your Azure AI Project in the Azure Portal +2. Look up the storage accounts associated with that resource group +3. Open each storage account and click on "Access control (IAM)" on the left side navigation +4. Add permissions for the desired users with the "Storage Blob Data Contributor" role + +If you have Azure CLI, you can use the following command: + +```Shell +# : Subscription ID of the Azure AI Studio hub's linked storage account (available in Azure AI hub resource view in Azure Portal). +# : Resource group of the Azure AI Studio hub's linked storage account. +# : User object ID for role assignment (retrieve with "az ad user show" command). + +az role assignment create --role "Storage Blob Data Contributor" --scope /subscriptions//resourceGroups/ --assignee-principal-type User --assignee-object-id "" +``` + ## Logging You can set logging level via environment variable `PF_LOGGING_LEVEL`, valid values includes `CRITICAL`, `ERROR`, `WARNING`, `INFO`, `DEBUG`, default to `INFO`.