From d2884d0b1103e226982463010134360571e20d4e Mon Sep 17 00:00:00 2001 From: April Kwong Date: Thu, 21 May 2026 18:33:20 -0700 Subject: [PATCH 1/9] Add 4 data generation job samples to azure-ai-projects Add four new samples to sdk/ai/azure-ai-projects/samples/datasets/ that cover the four hero scenarios of the 2.2.0a data generation API (`pc.beta.datasets.create_generation_job`), plus a README index: * sample_dataset_generation_job_traces_for_evaluation.py Traces -> Evaluation dataset. The Traces source consumes existing telemetry, so no model_options are required. * sample_dataset_generation_job_traces_for_finetuning.py Traces -> Supervised fine-tuning JSONL files (train_split=0.8 -> two file outputs). Resolves each FileDataGenerationJobOutput's real filename via openai_client.files.retrieve, because the service returns output.filename=None on FT outputs today. * sample_dataset_generation_job_simpleqna_with_dataset_source.py Multi-source SimpleQnA: a seed Dataset combined with an inline Prompt -> Evaluation dataset. Shows that caller-supplied output description and tags propagate onto the generated dataset (the service also auto-adds a data_generation_job_id tag). * sample_dataset_generation_job_simpleqna_for_finetuning.py Azure OpenAI File source -> short-answer + long-answer fine-tuning files (train_split=0.8). Polls the uploaded file until it reaches status=processed before submitting the job, otherwise the service rejects the reference with `must point to a completed file import`. All four match the style of sample_dataset_generation_job_with_evaluation.py: MIT header, parenthesized multi-with, dot-style polling, TERMINAL_STATUSES set, deletes the data generation job at the end but leaves the generated output (dataset or files) for the user to inspect. The SimpleQnA samples also delete the temporary inputs they upload (seed dataset / Azure OpenAI file). Each sample preflights `len(output_name) <= 50` because the service rejects longer output names with a 400 invalidPayload, and uses a short unique run id suffix so repeated runs do not collide. All four were live-validated end-to-end against a real Foundry project. README.md indexes the datasets folder, mirrors evaluations/README.md structure, and calls out per-sample environment variable requirements (FOUNDRY_AGENT_NAME for traces samples, FOUNDRY_MODEL_NAME for simple_qna samples, openai for samples that interact with Azure OpenAI files). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../samples/datasets/README.md | 53 ++++ ...generation_job_simpleqna_for_finetuning.py | 221 +++++++++++++++++ ...ation_job_simpleqna_with_dataset_source.py | 226 ++++++++++++++++++ ...et_generation_job_traces_for_evaluation.py | 156 ++++++++++++ ...et_generation_job_traces_for_finetuning.py | 165 +++++++++++++ 5 files changed, 821 insertions(+) create mode 100644 sdk/ai/azure-ai-projects/samples/datasets/README.md create mode 100644 sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py create mode 100644 sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py create mode 100644 sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py create mode 100644 sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md new file mode 100644 index 000000000000..9f9d18f4163b --- /dev/null +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -0,0 +1,53 @@ +# Azure AI Projects - Dataset Samples + +This folder contains samples demonstrating how to work with versioned Datasets and data generation jobs in Azure AI Foundry using the `azure-ai-projects` SDK. + +## Prerequisites + +Before running any sample: + +```bash +pip install "azure-ai-projects>=2.0.0" azure-identity python-dotenv +``` + +To run asynchronous samples, you will also need to install `aiohttp`. The data generation samples that interact with Azure OpenAI files (any sample that emits or consumes Azure OpenAI File outputs/inputs) also require `openai`. + +Set these environment variables: +- `FOUNDRY_PROJECT_ENDPOINT` - Required for all samples. Your Azure AI Project endpoint (e.g., `https://.services.ai.azure.com/api/projects/`). +- `FOUNDRY_MODEL_NAME` - Required for `simple_qna` data generation samples (`sample_dataset_generation_job_with_evaluation.py`, `sample_dataset_generation_job_simpleqna_with_dataset_source.py`, `sample_dataset_generation_job_simpleqna_for_finetuning.py`). The model deployment name (e.g., `gpt-4o-mini`). +- `FOUNDRY_AGENT_NAME` - Required for traces-based data generation samples (`sample_dataset_generation_job_traces_for_evaluation.py`, `sample_dataset_generation_job_traces_for_finetuning.py`). The name of a Foundry agent with recent traces in Application Insights. + +Most samples accept additional optional environment variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY_TRACES_WINDOW_DAYS`, etc.) — see each sample's docstring for details. + +## Running a Sample + +```bash +# Set environment variables +export FOUNDRY_PROJECT_ENDPOINT="https://.services.ai.azure.com/api/projects/" +export FOUNDRY_MODEL_NAME="gpt-4o-mini" # Replace with your model + +# Run a sample. For example: +python sample_datasets.py +``` + +## Sample Index + +### Dataset Basics + +| Sample | Description | +|--------|-------------| +| [sample_datasets.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_datasets.py) | Upload files, create, list, and delete versioned Datasets | +| [sample_datasets_async.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_datasets_async.py) | Async version of the dataset CRUD sample | +| [sample_datasets_download.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_datasets_download.py) | Upload a folder as a Dataset and download its files via an Azure storage ContainerClient | + +### Data Generation Jobs + +Data generation jobs synthesize evaluation datasets or supervised fine-tuning files from different kinds of sources (a Foundry agent's traces, an inline prompt, an existing Dataset, or an Azure OpenAI File). The job runs server-side; the samples below show how to submit a job, poll it to completion, and locate the generated artifacts. + +| Sample | Source(s) | Scenario | Description | +|--------|-----------|----------|-------------| +| [sample_dataset_generation_job_with_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_with_evaluation.py) | Prompt | Evaluation | Generate a QnA dataset from an inline prompt and run an evaluation against it end-to-end | +| [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate an evaluation dataset from a Foundry agent's recent conversation traces | +| [sample_dataset_generation_job_traces_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py) | Traces | Supervised fine-tuning | Generate ready-to-use training + validation JSONL files from a Foundry agent's recent traces | +| [sample_dataset_generation_job_simpleqna_with_dataset_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py) | Dataset + Prompt | Evaluation | Combine a seed Dataset with an inline Prompt to steer multi-source SimpleQnA generation, and confirm output metadata propagation | +| [sample_dataset_generation_job_simpleqna_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py) | File (Azure OpenAI) | Supervised fine-tuning | Upload a reference document as an Azure OpenAI File and generate short- and long-answer fine-tuning files from it | diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py new file mode 100644 index 000000000000..d29d7fcd017a --- /dev/null +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py @@ -0,0 +1,221 @@ +# pylint: disable=line-too-long,useless-suppression +# ------------------------------------ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# ------------------------------------ + +""" +DESCRIPTION: + Generates supervised fine-tuning data from a Markdown reference document + uploaded as an Azure OpenAI File. The sample: + + 1. Uploads a short reference document via the Azure OpenAI Files API + (`purpose=user_data`) so it can be referenced by file id. + 2. Creates a `DataGenerationJob` (scenario=SUPERVISED_FINETUNING, + type=simple_qna) that synthesizes short-answer and long-answer + question / answer pairs from the file content and emits them as + training and validation JSONL files. + 3. Polls the job to completion and prints every generated file output. + 4. Cleans up the Azure OpenAI input file and the data generation job. + + `simple_qna` REQUIRES `model_options` — the service uses the configured LLM + to synthesize the QnA pairs. Setting `train_split` triggers a split of + the generated samples into two Azure OpenAI output files. + +USAGE: + python sample_dataset_generation_job_simpleqna_for_finetuning.py + + Before running the sample: + + pip install "azure-ai-projects>=2.2.0" azure-identity openai python-dotenv + + Set these environment variables with your own values: + 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found + in the overview page of your Microsoft Foundry project. + 2) FOUNDRY_MODEL_NAME - Required. The name of an LLM model deployment used to + synthesize the QnA samples (e.g. `gpt-4o`, `gpt-5`). + 3) DATASET_NAME - Optional. Name to assign to the generated output files + (used as the file name prefix). Defaults to `simpleqna-finetuning-sample`. + The service caps the rendered output name at 50 characters, so keep + custom values short — the sample appends a unique run id suffix. + 4) POLL_INTERVAL_SECONDS - Optional. Number of seconds to sleep between status + polls for the data generation job. Defaults to 10. +""" + +import io +import os +import time +import uuid +from datetime import datetime, timezone + +from dotenv import load_dotenv + +from azure.identity import DefaultAzureCredential +from azure.ai.projects import AIProjectClient +from azure.ai.projects.models import ( + DataGenerationJob, + DataGenerationJobInputs, + DataGenerationJobOutputOptions, + DataGenerationJobScenario, + DataGenerationModelOptions, + FileDataGenerationJobOutput, + FileDataGenerationJobSource, + JobStatus, + SimpleQnADataGenerationJobOptions, + SimpleQnAFineTuningQuestionType, +) + +load_dotenv() + +endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] +model_name = os.environ["FOUNDRY_MODEL_NAME"] +dataset_name = os.environ.get("DATASET_NAME", "simpleqna-finetuning-sample") +poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) + +# Unique per-run output name so repeated runs do not collide. +# The service rejects output names longer than 50 characters. +run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" +output_name = f"{dataset_name}-{run_id}" +if len(output_name) > 50: + raise ValueError( + f"Output name `{output_name}` exceeds the 50-character service limit. " + f"Lower DATASET_NAME (currently `{dataset_name}`) so that `-` fits within 50 characters." + ) + +# Reference document the sample uploads as an Azure OpenAI file. The service +# requires the file to contain at least 1 KB of content to generate QnA from. +SEED_REFERENCE_DOCUMENT = """# Widgets and Gizmos Reference + +## Products +- Widget: blue, manufactured at Factory 7 in Acme, carbon-fiber, rated to 80 C, sold in packs of 4, 250 g each. +- Gizmo: red, manufactured at Factory 12 in Bedrock, carbon-fiber, rated to 80 C, sold individually, 1.2 kg each. +- Sprocket: green, manufactured at Factory 3 in Acme, stainless steel, rated to 200 C, sold individually, 500 g each. + +## Operations +- Factory operates weekdays 0700-1900 local time. +- Closed on public holidays, except for the annual maintenance run on December 27. +- ISO 9001 certified; audited annually by an independent third party. +- Quality control samples every 100th unit and runs full destructive testing on every 5000th unit. + +## Customer support +- Warranty claims: email support@example.com with the serial number printed on the underside of the product. +- Returns: accepted within 30 days if unopened; opened items are eligible for repair only. +- Bulk orders (50+ units): contact sales@example.com for volume pricing and an extended 90-day return window. +- Replacement parts: orderable directly from the support portal using the original order number. + +## Pricing and SLAs +- Widget pack: USD 24.99 per 4-pack; free shipping on orders over USD 75. +- Gizmo unit: USD 49.99; free shipping on orders over USD 75. +- Sprocket unit: USD 14.99; ships from regional warehouses in 1-2 business days. +- Standard support response: within one business day. Priority support response: within four hours. +""" + +TERMINAL_STATUSES = {JobStatus.SUCCEEDED, JobStatus.FAILED, JobStatus.CANCELLED} + +with ( + DefaultAzureCredential() as credential, + AIProjectClient(endpoint=endpoint, credential=credential) as project_client, + project_client.get_openai_client() as openai_client, +): + + # ------------------------------------------------------------------ + # 1. Upload the seed reference document as an Azure OpenAI file. + # ------------------------------------------------------------------ + seed_filename = f"widgets-gizmos-seed-{run_id}.md" + print(f"Upload the seed reference document as Azure OpenAI file `{seed_filename}`.") + seed_file = openai_client.files.create( + file=(seed_filename, io.BytesIO(SEED_REFERENCE_DOCUMENT.encode("utf-8"))), + purpose="user_data", + ) + print(f"Uploaded Azure OpenAI file (id: {seed_file.id}).") + + # Wait for the file to finish processing — the data generation service + # rejects references to files that are not yet in the `processed` state. + print("Wait for the Azure OpenAI file to be processed.", end="", flush=True) + while seed_file.status not in ("processed", "error"): + time.sleep(2) + seed_file = openai_client.files.retrieve(file_id=seed_file.id) + print(".", end="", flush=True) + print() + if seed_file.status != "processed": + raise RuntimeError(f"Azure OpenAI file `{seed_file.id}` failed to process: status=`{seed_file.status}`.") + + # ------------------------------------------------------------------ + # 2. Submit a fine-tuning data generation job that consumes the file. + # ------------------------------------------------------------------ + print("Create a fine-tuning data generation job from the Azure OpenAI file.") + job = DataGenerationJob( + inputs=DataGenerationJobInputs( + name=f"simpleqna-finetuning-{run_id}", + scenario=DataGenerationJobScenario.SUPERVISED_FINETUNING, + sources=[ + FileDataGenerationJobSource( + description="Widgets & Gizmos product / operations reference (Azure OpenAI file).", + id=seed_file.id, + ), + ], + options=SimpleQnADataGenerationJobOptions( + # Service requires max_samples to be between 15 and 1000. + max_samples=15, + # `simple_qna` REQUIRES model_options. + model_options=DataGenerationModelOptions(model=model_name), + # Split generated samples 80% training / 20% validation. + train_split=0.8, + # Ask for both short-answer and long-answer questions. + question_types=[ + SimpleQnAFineTuningQuestionType.SHORT_ANSWER, + SimpleQnAFineTuningQuestionType.LONG_ANSWER, + ], + ), + output_options=DataGenerationJobOutputOptions(name=output_name), + ), + ) + job = project_client.beta.datasets.create_generation_job(job=job) + print(f"Created data generation job `{job.id}` (status: `{job.status}`).") + + print(f"Poll job `{job.id}` until it reaches a terminal state.", end="", flush=True) + while True: + job = project_client.beta.datasets.get_generation_job(job_id=job.id) + if job.status in TERMINAL_STATUSES: + break + time.sleep(poll_interval_seconds) + print(".", end="", flush=True) + print() + print(f"Final job status: `{job.status}`.") + + if job.status != JobStatus.SUCCEEDED: + message = job.error.message if job.error is not None else "" + raise RuntimeError(f"Job `{job.id}` ended with status `{job.status}`: {message}") + + # ------------------------------------------------------------------ + # 3. Inspect the generated fine-tuning file outputs. + # ------------------------------------------------------------------ + # `train_split=0.8` produces two Azure OpenAI files: a training partition + # and a validation partition. Both are emitted as FileDataGenerationJobOutput + # entries in `job.result.outputs`. + file_outputs = [ + output + for output in ((job.result.outputs if job.result is not None else None) or []) + if isinstance(output, FileDataGenerationJobOutput) + ] + if not file_outputs: + raise RuntimeError(f"Job `{job.id}` did not produce any file outputs.") + + print(f"Generated {len(file_outputs)} fine-tuning file(s):") + for output in file_outputs: + if not output.id: + raise RuntimeError(f"Job `{job.id}` returned a file output without an id.") + # Resolve the Azure OpenAI file to surface its real filename and size. + file_info = openai_client.files.retrieve(file_id=output.id) + print(f" - filename=`{file_info.filename}` id=`{output.id}` bytes={file_info.bytes}") + if job.result is not None and job.result.generated_samples is not None: + print(f"Generated samples: {job.result.generated_samples}") + + # ------------------------------------------------------------------ + # 4. Clean up. + # ------------------------------------------------------------------ + print(f"Delete the Azure OpenAI input file `{seed_file.id}`.") + openai_client.files.delete(file_id=seed_file.id) + + print(f"Delete the data generation job `{job.id}`.") + project_client.beta.datasets.delete_generation_job(job_id=job.id) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py new file mode 100644 index 000000000000..9046b27aed32 --- /dev/null +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py @@ -0,0 +1,226 @@ +# pylint: disable=line-too-long,useless-suppression +# ------------------------------------ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# ------------------------------------ + +""" +DESCRIPTION: + Generates an evaluation dataset from a multi-source `simple_qna` job that + combines a seed Dataset with an inline Prompt. The sample: + + 1. Uploads a short Markdown reference document as a new versioned + Dataset that will act as the seed (product / operations reference). + 2. Creates a `DataGenerationJob` (scenario=EVALUATION, type=simple_qna) + with two sources: the seed `Dataset` and a `Prompt` that adds an + instruction to generate expert-level, high-difficulty questions. + 3. Polls the job to completion, resolves the generated `DatasetVersion`, + and shows that the caller-supplied output `description` and `tags` are + propagated onto the new dataset. + 4. Cleans up the seed dataset and the data generation job. + + `simple_qna` REQUIRES `model_options` — the service uses the configured LLM + to synthesize question / answer pairs from the combined sources. + +USAGE: + python sample_dataset_generation_job_simpleqna_with_dataset_source.py + + Before running the sample: + + pip install "azure-ai-projects>=2.2.0" azure-identity python-dotenv + + Set these environment variables with your own values: + 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found + in the overview page of your Microsoft Foundry project. + 2) FOUNDRY_MODEL_NAME - Required. The name of an LLM model deployment used to + synthesize the QnA samples (e.g. `gpt-4o`, `gpt-5`). + 3) DATASET_NAME - Optional. Name to assign to the generated output dataset. + Defaults to `simpleqna-multisource-sample`. The service caps the rendered + output name at 50 characters, so keep custom values short — the sample + appends a unique run id suffix. + 4) POLL_INTERVAL_SECONDS - Optional. Number of seconds to sleep between status + polls for the data generation job. Defaults to 10. +""" + +import os +import tempfile +import time +import uuid +from datetime import datetime, timezone + +from dotenv import load_dotenv + +from azure.identity import DefaultAzureCredential +from azure.ai.projects import AIProjectClient +from azure.ai.projects.models import ( + DataGenerationJob, + DataGenerationJobInputs, + DataGenerationJobOutputOptions, + DataGenerationJobScenario, + DataGenerationModelOptions, + DatasetDataGenerationJobOutput, + DatasetDataGenerationJobSource, + DatasetVersion, + JobStatus, + PromptDataGenerationJobSource, + SimpleQnADataGenerationJobOptions, +) + +load_dotenv() + +endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] +model_name = os.environ["FOUNDRY_MODEL_NAME"] +dataset_name = os.environ.get("DATASET_NAME", "simpleqna-multisource-sample") +poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) + +# Unique per-run resource names so repeated runs do not collide. +# The service rejects output names longer than 50 characters. +run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" +seed_dataset_name = f"widgets-gizmos-seed-{run_id}" +output_dataset_name = f"{dataset_name}-{run_id}" +if len(output_dataset_name) > 50: + raise ValueError( + f"Output dataset name `{output_dataset_name}` exceeds the 50-character service limit. " + f"Lower DATASET_NAME (currently `{dataset_name}`) so that `-` fits within 50 characters." + ) + +# Reference document the sample uploads as the seed Dataset. Keep this >= 1 KB +# so the service has enough material to synthesize meaningful QnA pairs. +SEED_REFERENCE_DOCUMENT = """# Widgets and Gizmos Reference + +## Products +- Widget: blue, manufactured at Factory 7 in Acme, carbon-fiber, rated to 80 C, sold in packs of 4, 250 g each. +- Gizmo: red, manufactured at Factory 12 in Bedrock, carbon-fiber, rated to 80 C, sold individually, 1.2 kg each. +- Sprocket: green, manufactured at Factory 3 in Acme, stainless steel, rated to 200 C, sold individually, 500 g each. + +## Operations +- Factory operates weekdays 0700-1900 local time. +- Closed on public holidays, except for the annual maintenance run on December 27. +- ISO 9001 certified; audited annually by an independent third party. +- Quality control samples every 100th unit and runs full destructive testing on every 5000th unit. + +## Customer support +- Warranty claims: email support@example.com with the serial number printed on the underside of the product. +- Returns: accepted within 30 days if unopened; opened items are eligible for repair only. +- Bulk orders (50+ units): contact sales@example.com for volume pricing and an extended 90-day return window. +- Replacement parts: orderable directly from the support portal using the original order number. + +## Pricing and SLAs +- Widget pack: USD 24.99 per 4-pack; free shipping on orders over USD 75. +- Gizmo unit: USD 49.99; free shipping on orders over USD 75. +- Sprocket unit: USD 14.99; ships from regional warehouses in 1-2 business days. +- Standard support response: within one business day. Priority support response: within four hours. +""" + +EXPECTED_OUTPUT_DESCRIPTION = "Expert-level QnA pairs generated from the Widgets & Gizmos reference." +EXPECTED_OUTPUT_TAGS = {"sample": "dataset-generation-simpleqna-with-dataset-source", "difficulty": "expert"} + +TERMINAL_STATUSES = {JobStatus.SUCCEEDED, JobStatus.FAILED, JobStatus.CANCELLED} + +with ( + DefaultAzureCredential() as credential, + AIProjectClient(endpoint=endpoint, credential=credential) as project_client, +): + + # ------------------------------------------------------------------ + # 1. Upload the seed reference document as a versioned Dataset. + # ------------------------------------------------------------------ + print(f"Upload the seed reference document as dataset `{seed_dataset_name}` v1.") + with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False, encoding="utf-8") as tmp: + tmp.write(SEED_REFERENCE_DOCUMENT) + seed_local_path = tmp.name + try: + seed_dataset = project_client.datasets.upload_file( + name=seed_dataset_name, + version="1", + file_path=seed_local_path, + ) + finally: + os.remove(seed_local_path) + print(f"Uploaded seed dataset (id: {seed_dataset.id}).") + + # ------------------------------------------------------------------ + # 2. Submit a multi-source SimpleQnA data generation job. + # ------------------------------------------------------------------ + # Two sources are combined for a single job: + # - The Dataset source contributes the source material (the reference + # document uploaded above). + # - The Prompt source contributes a steering instruction (difficulty). + print("Create a multi-source data generation job (Dataset + Prompt).") + job = DataGenerationJob( + inputs=DataGenerationJobInputs( + name=f"simpleqna-multisource-{run_id}", + scenario=DataGenerationJobScenario.EVALUATION, + sources=[ + DatasetDataGenerationJobSource( + description="Widgets & Gizmos product / operations reference.", + name=seed_dataset.name or "", + version=seed_dataset.version or "", + ), + PromptDataGenerationJobSource( + description="Specifies the question difficulty for SimpleQnA generation.", + prompt="Generate expert-level questions of high difficulty.", + ), + ], + options=SimpleQnADataGenerationJobOptions( + # Service requires max_samples to be between 15 and 1000. + max_samples=15, + # `simple_qna` REQUIRES model_options. + model_options=DataGenerationModelOptions(model=model_name), + ), + output_options=DataGenerationJobOutputOptions( + name=output_dataset_name, + description=EXPECTED_OUTPUT_DESCRIPTION, + tags=EXPECTED_OUTPUT_TAGS, + ), + ), + ) + job = project_client.beta.datasets.create_generation_job(job=job) + print(f"Created data generation job `{job.id}` (status: `{job.status}`).") + + print(f"Poll job `{job.id}` until it reaches a terminal state.", end="", flush=True) + while True: + job = project_client.beta.datasets.get_generation_job(job_id=job.id) + if job.status in TERMINAL_STATUSES: + break + time.sleep(poll_interval_seconds) + print(".", end="", flush=True) + print() + print(f"Final job status: `{job.status}`.") + + if job.status != JobStatus.SUCCEEDED: + message = job.error.message if job.error is not None else "" + raise RuntimeError(f"Job `{job.id}` ended with status `{job.status}`: {message}") + + # Locate the Dataset output produced by the job. + output_name: str = "" + output_version: str = "" + for output in (job.result.outputs if job.result is not None else None) or []: + if isinstance(output, DatasetDataGenerationJobOutput): + output_name = output.name or "" + output_version = output.version or "" + break + if not output_name or not output_version: + raise RuntimeError(f"Job `{job.id}` did not produce a dataset output.") + + # ------------------------------------------------------------------ + # 3. Inspect the generated dataset and show metadata propagation. + # ------------------------------------------------------------------ + # The caller-supplied output `description` and `tags` are persisted onto + # the generated dataset. The service also automatically adds a + # `data_generation_job_id` tag pointing back at this job. + dataset: DatasetVersion = project_client.datasets.get(name=output_name, version=output_version) + print(f"Generated dataset: name=`{dataset.name}` version=`{dataset.version}` id=`{dataset.id}`") + print(f" description: {dataset.description}") + print(f" tags: {dataset.tags}") + if job.result is not None and job.result.generated_samples is not None: + print(f"Generated samples: {job.result.generated_samples}") + + # ------------------------------------------------------------------ + # 4. Clean up. + # ------------------------------------------------------------------ + print(f"Delete the seed dataset `{seed_dataset.name}` v{seed_dataset.version}.") + project_client.datasets.delete(name=seed_dataset.name or "", version=seed_dataset.version or "") + + print(f"Delete the data generation job `{job.id}`.") + project_client.beta.datasets.delete_generation_job(job_id=job.id) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py new file mode 100644 index 000000000000..7c835145b48b --- /dev/null +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py @@ -0,0 +1,156 @@ +# pylint: disable=line-too-long,useless-suppression +# ------------------------------------ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# ------------------------------------ + +""" +DESCRIPTION: + Generates an evaluation dataset from a Foundry agent's recent conversation + traces. The sample: + + 1. Creates a `DataGenerationJob` (scenario=EVALUATION, type=traces) that + reads spans from Application Insights for an existing agent within a + time window and synthesizes question / answer pairs into a new + versioned Dataset. + 2. Polls the job to completion and resolves the resulting `DatasetVersion`. + 3. Cleans up the data generation job. + + The Traces source consumes existing telemetry, so no `model_options` are + required — the service derives samples directly from the agent's traces. + The agent must have at least one trace recorded within the configured + look-back window or the job will succeed with zero generated samples. + +USAGE: + python sample_dataset_generation_job_traces_for_evaluation.py + + Before running the sample: + + pip install "azure-ai-projects>=2.2.0" azure-identity python-dotenv + + Set these environment variables with your own values: + 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found + in the overview page of your Microsoft Foundry project. + 2) FOUNDRY_AGENT_NAME - Required. The name of a Foundry agent that has recent + conversation traces in Application Insights. + 3) DATASET_NAME - Optional. Name to assign to the generated output dataset. + Defaults to `traces-eval-sample`. The service caps the rendered output + name at 50 characters, so keep custom values short — the sample appends + a unique run id suffix. + 4) FOUNDRY_TRACES_WINDOW_DAYS - Optional. How far back, in days, to look for + agent traces. Defaults to 7. + 5) POLL_INTERVAL_SECONDS - Optional. Number of seconds to sleep between status + polls for the data generation job. Defaults to 10. +""" + +import os +import time +import uuid +from datetime import datetime, timedelta, timezone + +from dotenv import load_dotenv + +from azure.identity import DefaultAzureCredential +from azure.ai.projects import AIProjectClient +from azure.ai.projects.models import ( + DataGenerationJob, + DataGenerationJobInputs, + DataGenerationJobOutputOptions, + DataGenerationJobScenario, + DatasetDataGenerationJobOutput, + DatasetVersion, + JobStatus, + TracesDataGenerationJobOptions, + TracesDataGenerationJobSource, +) + +load_dotenv() + +endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] +agent_name = os.environ["FOUNDRY_AGENT_NAME"] +dataset_name = os.environ.get("DATASET_NAME", "traces-eval-sample") +traces_window_days = int(os.environ.get("FOUNDRY_TRACES_WINDOW_DAYS", "7")) +poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) + +# Unique per-run output dataset name so repeated runs do not collide. +# The service rejects output names longer than 50 characters. +run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" +output_dataset_name = f"{dataset_name}-{run_id}" +if len(output_dataset_name) > 50: + raise ValueError( + f"Output dataset name `{output_dataset_name}` exceeds the 50-character service limit. " + f"Lower DATASET_NAME (currently `{dataset_name}`) so that `-` fits within 50 characters." + ) + +# Trace look-back window: now - `traces_window_days` ... now. +end_time = datetime.now(tz=timezone.utc) +start_time = end_time - timedelta(days=traces_window_days) + +TERMINAL_STATUSES = {JobStatus.SUCCEEDED, JobStatus.FAILED, JobStatus.CANCELLED} + +with ( + DefaultAzureCredential() as credential, + AIProjectClient(endpoint=endpoint, credential=credential) as project_client, +): + + # ------------------------------------------------------------------ + # 1. Submit a data generation job that reads agent traces. + # ------------------------------------------------------------------ + print(f"Create a data generation job from traces for agent `{agent_name}` (window: {traces_window_days} day(s)).") + job = DataGenerationJob( + inputs=DataGenerationJobInputs( + name=f"traces-eval-{run_id}", + scenario=DataGenerationJobScenario.EVALUATION, + sources=[ + TracesDataGenerationJobSource( + description="Application Insights conversation traces for the Foundry agent.", + agent_name=agent_name, + start_time=start_time, + end_time=end_time, + ), + ], + options=TracesDataGenerationJobOptions( + # Service requires max_samples to be between 15 and 1000. + max_samples=15, + ), + output_options=DataGenerationJobOutputOptions(name=output_dataset_name), + ), + ) + job = project_client.beta.datasets.create_generation_job(job=job) + print(f"Created data generation job `{job.id}` (status: `{job.status}`).") + + print(f"Poll job `{job.id}` until it reaches a terminal state.", end="", flush=True) + while True: + job = project_client.beta.datasets.get_generation_job(job_id=job.id) + if job.status in TERMINAL_STATUSES: + break + time.sleep(poll_interval_seconds) + print(".", end="", flush=True) + print() + print(f"Final job status: `{job.status}`.") + + if job.status != JobStatus.SUCCEEDED: + message = job.error.message if job.error is not None else "" + raise RuntimeError(f"Job `{job.id}` ended with status `{job.status}`: {message}") + + # Locate the Dataset output produced by the job. + output_name: str = "" + output_version: str = "" + for output in (job.result.outputs if job.result is not None else None) or []: + if isinstance(output, DatasetDataGenerationJobOutput): + output_name = output.name or "" + output_version = output.version or "" + break + if not output_name or not output_version: + raise RuntimeError(f"Job `{job.id}` did not produce a dataset output.") + + dataset: DatasetVersion = project_client.datasets.get(name=output_name, version=output_version) + print(f"Generated dataset: name=`{dataset.name}` version=`{dataset.version}` id=`{dataset.id}`") + if job.result is not None and job.result.generated_samples is not None: + print(f"Generated samples: {job.result.generated_samples}") + + # ------------------------------------------------------------------ + # 2. Clean up. + # ------------------------------------------------------------------ + print(f"Delete the data generation job `{job.id}`.") + project_client.beta.datasets.delete_generation_job(job_id=job.id) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py new file mode 100644 index 000000000000..7d7899227975 --- /dev/null +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py @@ -0,0 +1,165 @@ +# pylint: disable=line-too-long,useless-suppression +# ------------------------------------ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# ------------------------------------ + +""" +DESCRIPTION: + Generates supervised fine-tuning data from a Foundry agent's recent + conversation traces. The sample: + + 1. Creates a `DataGenerationJob` (scenario=SUPERVISED_FINETUNING, + type=traces) that reads spans from Application Insights for an + existing agent within a time window and emits ready-to-use fine-tuning + JSONL files split into training and validation partitions. + 2. Polls the job to completion and prints every generated file output. + + Setting `train_split` triggers a split of the generated samples into two + Azure OpenAI files — a training partition and a validation partition. + The Traces source consumes existing telemetry, so no `model_options` are + required. + +USAGE: + python sample_dataset_generation_job_traces_for_finetuning.py + + Before running the sample: + + pip install "azure-ai-projects>=2.2.0" azure-identity python-dotenv + + Set these environment variables with your own values: + 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found + in the overview page of your Microsoft Foundry project. + 2) FOUNDRY_AGENT_NAME - Required. The name of a Foundry agent that has recent + conversation traces in Application Insights. + 3) DATASET_NAME - Optional. Name to assign to the generated output files + (used as the file name prefix). Defaults to `traces-finetuning-sample`. + The service caps the rendered output name at 50 characters, so keep + custom values short — the sample appends a unique run id suffix. + 4) FOUNDRY_TRACES_WINDOW_DAYS - Optional. How far back, in days, to look for + agent traces. Defaults to 7. + 5) POLL_INTERVAL_SECONDS - Optional. Number of seconds to sleep between status + polls for the data generation job. Defaults to 10. +""" + +import os +import time +import uuid +from datetime import datetime, timedelta, timezone + +from dotenv import load_dotenv + +from azure.identity import DefaultAzureCredential +from azure.ai.projects import AIProjectClient +from azure.ai.projects.models import ( + DataGenerationJob, + DataGenerationJobInputs, + DataGenerationJobOutputOptions, + DataGenerationJobScenario, + FileDataGenerationJobOutput, + JobStatus, + TracesDataGenerationJobOptions, + TracesDataGenerationJobSource, +) + +load_dotenv() + +endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] +agent_name = os.environ["FOUNDRY_AGENT_NAME"] +dataset_name = os.environ.get("DATASET_NAME", "traces-finetuning-sample") +traces_window_days = int(os.environ.get("FOUNDRY_TRACES_WINDOW_DAYS", "7")) +poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) + +# Unique per-run output name so repeated runs do not collide. +# The service rejects output names longer than 50 characters. +run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" +output_name = f"{dataset_name}-{run_id}" +if len(output_name) > 50: + raise ValueError( + f"Output name `{output_name}` exceeds the 50-character service limit. " + f"Lower DATASET_NAME (currently `{dataset_name}`) so that `-` fits within 50 characters." + ) + +# Trace look-back window: now - `traces_window_days` ... now. +end_time = datetime.now(tz=timezone.utc) +start_time = end_time - timedelta(days=traces_window_days) + +TERMINAL_STATUSES = {JobStatus.SUCCEEDED, JobStatus.FAILED, JobStatus.CANCELLED} + +with ( + DefaultAzureCredential() as credential, + AIProjectClient(endpoint=endpoint, credential=credential) as project_client, + project_client.get_openai_client() as openai_client, +): + + # ------------------------------------------------------------------ + # 1. Submit a fine-tuning data generation job that reads agent traces. + # ------------------------------------------------------------------ + print(f"Create a fine-tuning data generation job from traces for agent `{agent_name}` (window: {traces_window_days} day(s)).") + job = DataGenerationJob( + inputs=DataGenerationJobInputs( + name=f"traces-finetuning-{run_id}", + scenario=DataGenerationJobScenario.SUPERVISED_FINETUNING, + sources=[ + TracesDataGenerationJobSource( + description="Application Insights conversation traces for the Foundry agent.", + agent_name=agent_name, + start_time=start_time, + end_time=end_time, + ), + ], + options=TracesDataGenerationJobOptions( + # Service requires max_samples to be between 15 and 1000. + max_samples=15, + # Split generated samples 80% training / 20% validation. + train_split=0.8, + ), + output_options=DataGenerationJobOutputOptions(name=output_name), + ), + ) + job = project_client.beta.datasets.create_generation_job(job=job) + print(f"Created data generation job `{job.id}` (status: `{job.status}`).") + + print(f"Poll job `{job.id}` until it reaches a terminal state.", end="", flush=True) + while True: + job = project_client.beta.datasets.get_generation_job(job_id=job.id) + if job.status in TERMINAL_STATUSES: + break + time.sleep(poll_interval_seconds) + print(".", end="", flush=True) + print() + print(f"Final job status: `{job.status}`.") + + if job.status != JobStatus.SUCCEEDED: + message = job.error.message if job.error is not None else "" + raise RuntimeError(f"Job `{job.id}` ended with status `{job.status}`: {message}") + + # ------------------------------------------------------------------ + # 2. Inspect the generated fine-tuning file outputs. + # ------------------------------------------------------------------ + # `train_split=0.8` produces two Azure OpenAI files: a training partition + # and a validation partition. Both are emitted as FileDataGenerationJobOutput + # entries in `job.result.outputs`. + file_outputs = [ + output + for output in ((job.result.outputs if job.result is not None else None) or []) + if isinstance(output, FileDataGenerationJobOutput) + ] + if not file_outputs: + raise RuntimeError(f"Job `{job.id}` did not produce any file outputs.") + + print(f"Generated {len(file_outputs)} fine-tuning file(s):") + for output in file_outputs: + if not output.id: + raise RuntimeError(f"Job `{job.id}` returned a file output without an id.") + # Resolve the Azure OpenAI file to surface its real filename and size. + file_info = openai_client.files.retrieve(file_id=output.id) + print(f" - filename=`{file_info.filename}` id=`{output.id}` bytes={file_info.bytes}") + if job.result is not None and job.result.generated_samples is not None: + print(f"Generated samples: {job.result.generated_samples}") + + # ------------------------------------------------------------------ + # 3. Clean up. + # ------------------------------------------------------------------ + print(f"Delete the data generation job `{job.id}`.") + project_client.beta.datasets.delete_generation_job(job_id=job.id) From c6cb56ebfef9a21f37238f865f33e12c47da4e04 Mon Sep 17 00:00:00 2001 From: April Kwong Date: Thu, 21 May 2026 23:55:16 -0700 Subject: [PATCH 2/9] Address PR review feedback on data generation samples - Add sample_dataset_generation_job_simpleqna_with_agent_source.py: self-contained sample that creates a short-lived PromptAgentDefinition, uses it as an AgentDataGenerationJobSource for SimpleQnA evaluation, and cleans up the agent in a finally block. - Add sample_dataset_generation_job_simpleqna_with_file_source.py: replaces the removed dataset-source sample with a multi-source File + Prompt SimpleQnA evaluation job that uploads the seed via the Azure OpenAI Files API and verifies description/tag propagation. - Remove sample_dataset_generation_job_simpleqna_with_dataset_source.py: the service no longer supports reading dataset input files for QnA generation. - Rename sample_dataset_generation_job_with_evaluation.py to sample_dataset_generation_job_simpleqna_with_prompt_source.py for naming consistency with the new _with__source samples. - Add full cleanup of generated artifacts (datasets, fine-tuning files, prompt agents) across every data-generation sample so repeated runs do not accumulate artifacts. - README: clarify FOUNDRY_MODEL_NAME refers to a model deployment, add a link to the Responses-API supported-model list for evaluation jobs and a chat-completions hint for fine-tuning jobs, note that traces sources also work with third-party (OpenTelemetry-instrumented) agents, and document the self-cleaning behavior of the samples. - CHANGELOG: update reference to renamed sample. - tests/samples/test_samples.py: update references to renamed sample. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/ai/azure-ai-projects/CHANGELOG.md | 2 +- .../samples/datasets/README.md | 19 +- ...generation_job_simpleqna_for_finetuning.py | 13 +- ...eration_job_simpleqna_with_agent_source.py | 209 ++++++++++++++++++ ...eration_job_simpleqna_with_file_source.py} | 92 ++++---- ...ation_job_simpleqna_with_prompt_source.py} | 15 +- ...et_generation_job_traces_for_evaluation.py | 7 +- ...et_generation_job_traces_for_finetuning.py | 7 +- .../tests/samples/test_samples.py | 6 +- 9 files changed, 306 insertions(+), 64 deletions(-) create mode 100644 sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py rename sdk/ai/azure-ai-projects/samples/datasets/{sample_dataset_generation_job_simpleqna_with_dataset_source.py => sample_dataset_generation_job_simpleqna_with_file_source.py} (73%) rename sdk/ai/azure-ai-projects/samples/datasets/{sample_dataset_generation_job_with_evaluation.py => sample_dataset_generation_job_simpleqna_with_prompt_source.py} (93%) diff --git a/sdk/ai/azure-ai-projects/CHANGELOG.md b/sdk/ai/azure-ai-projects/CHANGELOG.md index b4e7fcf1ec65..8276cd056e0b 100644 --- a/sdk/ai/azure-ai-projects/CHANGELOG.md +++ b/sdk/ai/azure-ai-projects/CHANGELOG.md @@ -19,7 +19,7 @@ * New read-only property `content_hash` on `CodeConfiguration`, returning the SHA-256 hex digest of the uploaded code zip. * New optional `force` parameter on `agents.delete` and `agents.delete_version` methods. * New optional `blueprint_reference` parameters on `agents.create_version` method. -* New sample `sample_dataset_generation_job_with_evaluation.py` showing an end-to-end flow that generates a QnA dataset via `.beta.datasets.create_generation_job` and runs an OpenAI evaluation. +* New sample `sample_dataset_generation_job_simpleqna_with_prompt_source.py` showing an end-to-end flow that generates a QnA dataset via `.beta.datasets.create_generation_job` and runs an OpenAI evaluation. ### Breaking Changes diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md index 9f9d18f4163b..6af0376b7858 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/README.md +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -14,8 +14,8 @@ To run asynchronous samples, you will also need to install `aiohttp`. The data g Set these environment variables: - `FOUNDRY_PROJECT_ENDPOINT` - Required for all samples. Your Azure AI Project endpoint (e.g., `https://.services.ai.azure.com/api/projects/`). -- `FOUNDRY_MODEL_NAME` - Required for `simple_qna` data generation samples (`sample_dataset_generation_job_with_evaluation.py`, `sample_dataset_generation_job_simpleqna_with_dataset_source.py`, `sample_dataset_generation_job_simpleqna_for_finetuning.py`). The model deployment name (e.g., `gpt-4o-mini`). -- `FOUNDRY_AGENT_NAME` - Required for traces-based data generation samples (`sample_dataset_generation_job_traces_for_evaluation.py`, `sample_dataset_generation_job_traces_for_finetuning.py`). The name of a Foundry agent with recent traces in Application Insights. +- `FOUNDRY_MODEL_NAME` - Required for `simple_qna` data generation samples (`sample_dataset_generation_job_simpleqna_with_prompt_source.py`, `sample_dataset_generation_job_simpleqna_with_file_source.py`, `sample_dataset_generation_job_simpleqna_with_agent_source.py`, `sample_dataset_generation_job_simpleqna_for_finetuning.py`). The name of an Azure OpenAI model **deployment** in your project (matches the `FOUNDRY_MODEL_NAME` convention used elsewhere in this samples folder). For **evaluation** jobs the deployment must support the [Responses API](https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support); for **fine-tuning** jobs the deployment must support chat completions (e.g. `gpt-4o`, `gpt-4.1`). +- `FOUNDRY_AGENT_NAME` - Required for the two traces samples (`sample_dataset_generation_job_traces_for_evaluation.py`, `sample_dataset_generation_job_traces_for_finetuning.py`). The name of an agent that has recent traces in Application Insights. Traces sources support both Foundry Agents and third-party (OpenTelemetry instrumented) agents. The agent-source SimpleQnA sample (`sample_dataset_generation_job_simpleqna_with_agent_source.py`) does *not* read this variable — it creates its own short-lived prompt agent at runtime and cleans it up at the end. Most samples accept additional optional environment variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY_TRACES_WINDOW_DAYS`, etc.) — see each sample's docstring for details. @@ -24,7 +24,7 @@ Most samples accept additional optional environment variables (`DATASET_NAME`, ` ```bash # Set environment variables export FOUNDRY_PROJECT_ENDPOINT="https://.services.ai.azure.com/api/projects/" -export FOUNDRY_MODEL_NAME="gpt-4o-mini" # Replace with your model +export FOUNDRY_MODEL_NAME="gpt-4o-mini" # Replace with your model deployment # Run a sample. For example: python sample_datasets.py @@ -42,12 +42,15 @@ python sample_datasets.py ### Data Generation Jobs -Data generation jobs synthesize evaluation datasets or supervised fine-tuning files from different kinds of sources (a Foundry agent's traces, an inline prompt, an existing Dataset, or an Azure OpenAI File). The job runs server-side; the samples below show how to submit a job, poll it to completion, and locate the generated artifacts. +Data generation jobs synthesize evaluation datasets or supervised fine-tuning files from different kinds of sources (an agent's traces, an agent's definition, an inline prompt, or an Azure OpenAI File). The job runs server-side; the samples below show how to submit a job, poll it to completion, and locate the generated artifacts. + +To keep the project clean across repeated runs, each sample below also deletes every resource it creates (including the job record, any uploaded input files, any short-lived agent, and the generated dataset or fine-tuning files) before exiting. | Sample | Source(s) | Scenario | Description | |--------|-----------|----------|-------------| -| [sample_dataset_generation_job_with_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_with_evaluation.py) | Prompt | Evaluation | Generate a QnA dataset from an inline prompt and run an evaluation against it end-to-end | -| [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate an evaluation dataset from a Foundry agent's recent conversation traces | -| [sample_dataset_generation_job_traces_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py) | Traces | Supervised fine-tuning | Generate ready-to-use training + validation JSONL files from a Foundry agent's recent traces | -| [sample_dataset_generation_job_simpleqna_with_dataset_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py) | Dataset + Prompt | Evaluation | Combine a seed Dataset with an inline Prompt to steer multi-source SimpleQnA generation, and confirm output metadata propagation | +| [sample_dataset_generation_job_simpleqna_with_prompt_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py) | Prompt | Evaluation | Generate a QnA dataset from an inline prompt and run an evaluation against it end-to-end | +| [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate an evaluation dataset from an agent's recent conversation traces (traces recipe) | +| [sample_dataset_generation_job_traces_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py) | Traces | Supervised fine-tuning | Generate ready-to-use training + validation JSONL files from an agent's recent traces | +| [sample_dataset_generation_job_simpleqna_with_agent_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py) | Agent definition | Evaluation | Self-contained: creates a short-lived `PromptAgentDefinition`, then generates an evaluation dataset from the agent's instructions / prompt via the `simple_qna` recipe | +| [sample_dataset_generation_job_simpleqna_with_file_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py) | File (Azure OpenAI) + Prompt | Evaluation | Combine an uploaded reference document with an inline Prompt to steer multi-source SimpleQnA generation, and confirm output metadata propagation | | [sample_dataset_generation_job_simpleqna_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py) | File (Azure OpenAI) | Supervised fine-tuning | Upload a reference document as an Azure OpenAI File and generate short- and long-answer fine-tuning files from it | diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py index d29d7fcd017a..8a545c3dd6b3 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py @@ -16,7 +16,7 @@ question / answer pairs from the file content and emits them as training and validation JSONL files. 3. Polls the job to completion and prints every generated file output. - 4. Cleans up the Azure OpenAI input file and the data generation job. + 4. Cleans up the generated fine-tuning files, the Azure OpenAI input file, and the data generation job. `simple_qna` REQUIRES `model_options` — the service uses the configured LLM to synthesize the QnA pairs. Setting `train_split` triggers a split of @@ -32,8 +32,9 @@ Set these environment variables with your own values: 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found in the overview page of your Microsoft Foundry project. - 2) FOUNDRY_MODEL_NAME - Required. The name of an LLM model deployment used to - synthesize the QnA samples (e.g. `gpt-4o`, `gpt-5`). + 2) FOUNDRY_MODEL_NAME - Required. The name of an Azure OpenAI model + deployment used to synthesize the QnA samples. For `simple_qna` fine-tuning, + the deployment must support the chat completions API (e.g. `gpt-4o`, `gpt-4.1`). 3) DATASET_NAME - Optional. Name to assign to the generated output files (used as the file name prefix). Defaults to `simpleqna-finetuning-sample`. The service caps the rendered output name at 50 characters, so keep @@ -73,7 +74,7 @@ poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) # Unique per-run output name so repeated runs do not collide. -# The service rejects output names longer than 50 characters. +# Output names are capped at 50 characters by the service. run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" output_name = f"{dataset_name}-{run_id}" if len(output_name) > 50: @@ -214,6 +215,10 @@ # ------------------------------------------------------------------ # 4. Clean up. # ------------------------------------------------------------------ + for output in file_outputs: + print(f"Delete the generated Azure OpenAI file `{output.id}`.") + openai_client.files.delete(file_id=output.id) + print(f"Delete the Azure OpenAI input file `{seed_file.id}`.") openai_client.files.delete(file_id=seed_file.id) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py new file mode 100644 index 000000000000..70063d32da26 --- /dev/null +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py @@ -0,0 +1,209 @@ +# pylint: disable=line-too-long,useless-suppression +# ------------------------------------ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# ------------------------------------ + +""" +DESCRIPTION: + Generates an evaluation dataset from a prompt agent's definition using the + `simple_qna` recipe. The sample is fully self-contained — it creates a + short-lived `PromptAgentDefinition` whose `instructions` give the service + enough material to synthesize QnA pairs, then uses that agent as the + source for the data generation job. The sample: + + 1. Creates a `PromptAgentDefinition` agent with domain-specific + instructions (a small Widgets & Gizmos customer-support persona). + 2. Creates a `DataGenerationJob` (scenario=EVALUATION, type=simple_qna) + whose source is an `Agent` reference pointing at the new agent. The + service fetches the agent's instructions / prompt and uses the + configured LLM to synthesize question / answer pairs from them. + 3. Polls the job to completion and resolves the resulting `DatasetVersion`. + 4. Cleans up the data generation job, the generated dataset, and the agent version. + + `simple_qna` REQUIRES `model_options`. For `simple_qna` evaluation jobs the + deployed model must support the Azure OpenAI Responses API. See the + supported-model list: + https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support + + If you want to synthesize QnA from an agent's recorded conversation traces + instead of its definition, see + `sample_dataset_generation_job_traces_for_evaluation.py` (traces recipe). + +USAGE: + python sample_dataset_generation_job_simpleqna_with_agent_source.py + + Before running the sample: + + pip install "azure-ai-projects>=2.2.0" azure-identity python-dotenv + + Set these environment variables with your own values: + 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found + in the overview page of your Microsoft Foundry project. + 2) FOUNDRY_MODEL_NAME - Required. The name of an Azure OpenAI model + deployment used both as the prompt agent's backing model and to synthesize + the QnA samples. For `simple_qna` evaluation, choose a Responses-API + capable model (see the link in the description). + 3) DATASET_NAME - Optional. Name to assign to the generated output dataset. + Defaults to `simpleqna-agent-source-sample`. The service caps the + rendered output name at 50 characters, so keep custom values short — + the sample appends a unique run id suffix. + 4) POLL_INTERVAL_SECONDS - Optional. Number of seconds to sleep between status + polls for the data generation job. Defaults to 10. +""" + +import os +import time +import uuid +from datetime import datetime, timezone + +from dotenv import load_dotenv + +from azure.identity import DefaultAzureCredential +from azure.ai.projects import AIProjectClient +from azure.ai.projects.models import ( + AgentDataGenerationJobSource, + DataGenerationJob, + DataGenerationJobInputs, + DataGenerationJobOutputOptions, + DataGenerationJobScenario, + DataGenerationModelOptions, + DatasetDataGenerationJobOutput, + DatasetVersion, + JobStatus, + PromptAgentDefinition, + SimpleQnADataGenerationJobOptions, +) + +load_dotenv() + +endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] +model_name = os.environ["FOUNDRY_MODEL_NAME"] +dataset_name = os.environ.get("DATASET_NAME", "simpleqna-agent-source-sample") +poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) + +# Unique per-run names so repeated runs do not collide. +# Output names are capped at 50 characters by the service. +run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" +output_dataset_name = f"{dataset_name}-{run_id}" +if len(output_dataset_name) > 50: + raise ValueError( + f"Output dataset name `{output_dataset_name}` exceeds the 50-character service limit. " + f"Lower DATASET_NAME (currently `{dataset_name}`) so that `-` fits within 50 characters." + ) + +# The prompt agent's instructions seed the QnA generation. Make them concrete +# and domain-specific so the service has enough material to synthesize from. +AGENT_INSTRUCTIONS = """You are a customer support assistant for Acme's "Widgets & Gizmos" product line. + +Product catalog you can answer about: +- Widget: blue, manufactured at Factory 7 in Acme, carbon-fiber, rated to 80 C, sold in packs of 4, 250 g each, USD 24.99 per pack. +- Gizmo: red, manufactured at Factory 12 in Bedrock, carbon-fiber, rated to 80 C, sold individually, 1.2 kg each, USD 49.99. +- Sprocket: green, manufactured at Factory 3 in Acme, stainless steel, rated to 200 C, sold individually, 500 g each, USD 14.99. + +Operational policy: +- Factory operates weekdays 0700-1900 local time and is closed on public holidays except the annual maintenance run on December 27. +- Warranty claims must include the serial number printed on the underside of the product, emailed to support@example.com. +- Returns are accepted within 30 days if unopened; opened items are eligible for repair only. +- Bulk orders of 50 or more units use the sales@example.com channel and get an extended 90-day return window. +- Standard support responds within one business day; priority support responds within four hours. +- Free shipping on orders over USD 75. + +When asked about anything outside this catalog and policy, politely say you do not have that information. +""" + +agent_name = f"widgets-gizmos-support-{run_id}" + +TERMINAL_STATUSES = {JobStatus.SUCCEEDED, JobStatus.FAILED, JobStatus.CANCELLED} + +with ( + DefaultAzureCredential() as credential, + AIProjectClient(endpoint=endpoint, credential=credential) as project_client, +): + + # ------------------------------------------------------------------ + # 1. Create a short-lived prompt agent to act as the source. + # ------------------------------------------------------------------ + print(f"Create prompt agent `{agent_name}`.") + agent = project_client.agents.create_version( + agent_name=agent_name, + definition=PromptAgentDefinition( + model=model_name, + instructions=AGENT_INSTRUCTIONS, + ), + ) + print(f"Agent created (id: {agent.id}, name: {agent.name}, version: {agent.version}).") + + try: + # ------------------------------------------------------------------ + # 2. Submit a SimpleQnA data generation job sourced from the agent. + # ------------------------------------------------------------------ + # The service fetches the agent's instructions / prompt and uses + # `model_options.model` to synthesize QnA pairs from them. + print(f"Create a SimpleQnA evaluation job sourced from agent `{agent.name}` (version {agent.version}).") + job = DataGenerationJob( + inputs=DataGenerationJobInputs( + name=f"simpleqna-agent-{run_id}", + scenario=DataGenerationJobScenario.EVALUATION, + sources=[ + AgentDataGenerationJobSource( + description="Agent definition (instructions / prompt) used to seed QnA generation.", + agent_name=agent.name, + agent_version=agent.version, + ), + ], + options=SimpleQnADataGenerationJobOptions( + # Service requires max_samples to be between 15 and 1000. + max_samples=15, + # `simple_qna` REQUIRES model_options. + model_options=DataGenerationModelOptions(model=model_name), + ), + output_options=DataGenerationJobOutputOptions(name=output_dataset_name), + ), + ) + job = project_client.beta.datasets.create_generation_job(job=job) + print(f"Created data generation job `{job.id}` (status: `{job.status}`).") + + print(f"Poll job `{job.id}` until it reaches a terminal state.", end="", flush=True) + while True: + job = project_client.beta.datasets.get_generation_job(job_id=job.id) + if job.status in TERMINAL_STATUSES: + break + time.sleep(poll_interval_seconds) + print(".", end="", flush=True) + print() + print(f"Final job status: `{job.status}`.") + + if job.status != JobStatus.SUCCEEDED: + message = job.error.message if job.error is not None else "" + raise RuntimeError(f"Job `{job.id}` ended with status `{job.status}`: {message}") + + # Locate the Dataset output produced by the job. + output_name: str = "" + output_version: str = "" + for output in (job.result.outputs if job.result is not None else None) or []: + if isinstance(output, DatasetDataGenerationJobOutput): + output_name = output.name or "" + output_version = output.version or "" + break + if not output_name or not output_version: + raise RuntimeError(f"Job `{job.id}` did not produce a dataset output.") + + dataset: DatasetVersion = project_client.datasets.get(name=output_name, version=output_version) + print(f"Generated dataset: name=`{dataset.name}` version=`{dataset.version}` id=`{dataset.id}`") + if job.result is not None and job.result.generated_samples is not None: + print(f"Generated samples: {job.result.generated_samples}") + + # ------------------------------------------------------------------ + # 3. Clean up the generated dataset and the data generation job + # (the agent is deleted in the `finally` block below). + # ------------------------------------------------------------------ + print(f"Delete the generated dataset `{dataset.name}` v{dataset.version}.") + project_client.datasets.delete(name=dataset.name or "", version=dataset.version or "") + + print(f"Delete the data generation job `{job.id}`.") + project_client.beta.datasets.delete_generation_job(job_id=job.id) + finally: + # The agent is short-lived — always delete it, even if the job failed. + print(f"Delete the prompt agent `{agent.name}` (version {agent.version}).") + project_client.agents.delete_version(agent_name=agent.name, agent_version=agent.version) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py similarity index 73% rename from sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py rename to sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py index 9046b27aed32..f52d331e2603 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_dataset_source.py +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py @@ -7,43 +7,48 @@ """ DESCRIPTION: Generates an evaluation dataset from a multi-source `simple_qna` job that - combines a seed Dataset with an inline Prompt. The sample: + combines an Azure OpenAI File with an inline Prompt. The sample: - 1. Uploads a short Markdown reference document as a new versioned - Dataset that will act as the seed (product / operations reference). + 1. Uploads a short Markdown reference document via the Azure OpenAI Files + API (`purpose=user_data`) so it can be referenced by file id. 2. Creates a `DataGenerationJob` (scenario=EVALUATION, type=simple_qna) - with two sources: the seed `Dataset` and a `Prompt` that adds an + with two sources: the uploaded `File` and a `Prompt` that adds an instruction to generate expert-level, high-difficulty questions. 3. Polls the job to completion, resolves the generated `DatasetVersion`, and shows that the caller-supplied output `description` and `tags` are propagated onto the new dataset. - 4. Cleans up the seed dataset and the data generation job. + 4. Cleans up the generated dataset, the Azure OpenAI input file, and the data generation job. `simple_qna` REQUIRES `model_options` — the service uses the configured LLM to synthesize question / answer pairs from the combined sources. + For `simple_qna` evaluation jobs the deployed model must support the + Azure OpenAI Responses API. See the supported-model list: + https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support + USAGE: - python sample_dataset_generation_job_simpleqna_with_dataset_source.py + python sample_dataset_generation_job_simpleqna_with_file_source.py Before running the sample: - pip install "azure-ai-projects>=2.2.0" azure-identity python-dotenv + pip install "azure-ai-projects>=2.2.0" azure-identity openai python-dotenv Set these environment variables with your own values: 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found in the overview page of your Microsoft Foundry project. - 2) FOUNDRY_MODEL_NAME - Required. The name of an LLM model deployment used to - synthesize the QnA samples (e.g. `gpt-4o`, `gpt-5`). + 2) FOUNDRY_MODEL_NAME - Required. The name of an Azure OpenAI model + deployment used to synthesize the QnA samples. For `simple_qna` evaluation, + choose a Responses-API capable model (see the link in the description). 3) DATASET_NAME - Optional. Name to assign to the generated output dataset. - Defaults to `simpleqna-multisource-sample`. The service caps the rendered + Defaults to `simpleqna-file-source-sample`. The service caps the rendered output name at 50 characters, so keep custom values short — the sample appends a unique run id suffix. 4) POLL_INTERVAL_SECONDS - Optional. Number of seconds to sleep between status polls for the data generation job. Defaults to 10. """ +import io import os -import tempfile import time import uuid from datetime import datetime, timezone @@ -59,8 +64,8 @@ DataGenerationJobScenario, DataGenerationModelOptions, DatasetDataGenerationJobOutput, - DatasetDataGenerationJobSource, DatasetVersion, + FileDataGenerationJobSource, JobStatus, PromptDataGenerationJobSource, SimpleQnADataGenerationJobOptions, @@ -70,13 +75,12 @@ endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] model_name = os.environ["FOUNDRY_MODEL_NAME"] -dataset_name = os.environ.get("DATASET_NAME", "simpleqna-multisource-sample") +dataset_name = os.environ.get("DATASET_NAME", "simpleqna-file-source-sample") poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) # Unique per-run resource names so repeated runs do not collide. -# The service rejects output names longer than 50 characters. +# Output names are capped at 50 characters by the service. run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" -seed_dataset_name = f"widgets-gizmos-seed-{run_id}" output_dataset_name = f"{dataset_name}-{run_id}" if len(output_dataset_name) > 50: raise ValueError( @@ -84,8 +88,8 @@ f"Lower DATASET_NAME (currently `{dataset_name}`) so that `-` fits within 50 characters." ) -# Reference document the sample uploads as the seed Dataset. Keep this >= 1 KB -# so the service has enough material to synthesize meaningful QnA pairs. +# Reference document the sample uploads as an Azure OpenAI file. The service +# requires the file to contain at least 1 KB of content to generate QnA from. SEED_REFERENCE_DOCUMENT = """# Widgets and Gizmos Reference ## Products @@ -113,49 +117,54 @@ """ EXPECTED_OUTPUT_DESCRIPTION = "Expert-level QnA pairs generated from the Widgets & Gizmos reference." -EXPECTED_OUTPUT_TAGS = {"sample": "dataset-generation-simpleqna-with-dataset-source", "difficulty": "expert"} +EXPECTED_OUTPUT_TAGS = {"sample": "dataset-generation-simpleqna-with-file-source", "difficulty": "expert"} TERMINAL_STATUSES = {JobStatus.SUCCEEDED, JobStatus.FAILED, JobStatus.CANCELLED} with ( DefaultAzureCredential() as credential, AIProjectClient(endpoint=endpoint, credential=credential) as project_client, + project_client.get_openai_client() as openai_client, ): # ------------------------------------------------------------------ - # 1. Upload the seed reference document as a versioned Dataset. + # 1. Upload the seed reference document as an Azure OpenAI file. # ------------------------------------------------------------------ - print(f"Upload the seed reference document as dataset `{seed_dataset_name}` v1.") - with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False, encoding="utf-8") as tmp: - tmp.write(SEED_REFERENCE_DOCUMENT) - seed_local_path = tmp.name - try: - seed_dataset = project_client.datasets.upload_file( - name=seed_dataset_name, - version="1", - file_path=seed_local_path, - ) - finally: - os.remove(seed_local_path) - print(f"Uploaded seed dataset (id: {seed_dataset.id}).") + seed_filename = f"widgets-gizmos-seed-{run_id}.md" + print(f"Upload the seed reference document as Azure OpenAI file `{seed_filename}`.") + seed_file = openai_client.files.create( + file=(seed_filename, io.BytesIO(SEED_REFERENCE_DOCUMENT.encode("utf-8"))), + purpose="user_data", + ) + print(f"Uploaded Azure OpenAI file (id: {seed_file.id}).") + + # Wait for the file to finish processing — the data generation service + # rejects references to files that are not yet in the `processed` state. + print("Wait for the Azure OpenAI file to be processed.", end="", flush=True) + while seed_file.status not in ("processed", "error"): + time.sleep(2) + seed_file = openai_client.files.retrieve(file_id=seed_file.id) + print(".", end="", flush=True) + print() + if seed_file.status != "processed": + raise RuntimeError(f"Azure OpenAI file `{seed_file.id}` failed to process: status=`{seed_file.status}`.") # ------------------------------------------------------------------ # 2. Submit a multi-source SimpleQnA data generation job. # ------------------------------------------------------------------ # Two sources are combined for a single job: - # - The Dataset source contributes the source material (the reference + # - The File source contributes the source material (the reference # document uploaded above). # - The Prompt source contributes a steering instruction (difficulty). - print("Create a multi-source data generation job (Dataset + Prompt).") + print("Create a multi-source data generation job (File + Prompt).") job = DataGenerationJob( inputs=DataGenerationJobInputs( name=f"simpleqna-multisource-{run_id}", scenario=DataGenerationJobScenario.EVALUATION, sources=[ - DatasetDataGenerationJobSource( - description="Widgets & Gizmos product / operations reference.", - name=seed_dataset.name or "", - version=seed_dataset.version or "", + FileDataGenerationJobSource( + description="Widgets & Gizmos product / operations reference (Azure OpenAI file).", + id=seed_file.id, ), PromptDataGenerationJobSource( description="Specifies the question difficulty for SimpleQnA generation.", @@ -219,8 +228,11 @@ # ------------------------------------------------------------------ # 4. Clean up. # ------------------------------------------------------------------ - print(f"Delete the seed dataset `{seed_dataset.name}` v{seed_dataset.version}.") - project_client.datasets.delete(name=seed_dataset.name or "", version=seed_dataset.version or "") + print(f"Delete the generated dataset `{dataset.name}` v{dataset.version}.") + project_client.datasets.delete(name=dataset.name or "", version=dataset.version or "") + + print(f"Delete the Azure OpenAI input file `{seed_file.id}`.") + openai_client.files.delete(file_id=seed_file.id) print(f"Delete the data generation job `{job.id}`.") project_client.beta.datasets.delete_generation_job(job_id=job.id) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_with_evaluation.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py similarity index 93% rename from sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_with_evaluation.py rename to sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py index 95012ecddbc5..a4cb9bb00b68 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_with_evaluation.py +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py @@ -17,10 +17,10 @@ Azure AI evaluators. 4. Runs the evaluation against the generated dataset by passing the dataset's id as the run's `file_id`. - 5. Cleans up the evaluation and the data generation job. + 5. Cleans up the evaluation, the generated dataset, and the data generation job. USAGE: - python sample_dataset_generation_job_with_evaluation.py + python sample_dataset_generation_job_simpleqna_with_prompt_source.py Before running the sample: @@ -29,9 +29,11 @@ Set these environment variables with your own values: 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found in the overview page of your Microsoft Foundry project. - 2) FOUNDRY_MODEL_NAME - Required. The name of an LLM model deployment used both - to generate the QnA samples and as the judge model for builtin evaluators - (e.g. `gpt-4o`, `gpt-5`). + 2) FOUNDRY_MODEL_NAME - Required. The name of an Azure OpenAI model + deployment used both to generate the QnA samples and as the judge model + for builtin evaluators. For `simple_qna` evaluation jobs the deployed + model must support the Azure OpenAI Responses API. See the supported-model + list: https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support 3) DATASET_NAME - Optional. Name to assign to the generated output dataset. Defaults to `dataset-generation-eval-sample`. 4) POLL_INTERVAL_SECONDS - Optional. Number of seconds to sleep between status @@ -263,5 +265,8 @@ print(f"Delete evaluation `{eval_object.id}`.") openai_client.evals.delete(eval_id=eval_object.id) + print(f"Delete the generated dataset `{dataset.name}` v{dataset.version}.") + project_client.datasets.delete(name=dataset.name or "", version=dataset.version or "") + print(f"Delete the data generation job `{job.id}`.") project_client.beta.datasets.delete_generation_job(job_id=job.id) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py index 7c835145b48b..00d487c57aae 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py @@ -14,7 +14,7 @@ time window and synthesizes question / answer pairs into a new versioned Dataset. 2. Polls the job to completion and resolves the resulting `DatasetVersion`. - 3. Cleans up the data generation job. + 3. Cleans up the generated dataset and the data generation job. The Traces source consumes existing telemetry, so no `model_options` are required — the service derives samples directly from the agent's traces. @@ -73,7 +73,7 @@ poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) # Unique per-run output dataset name so repeated runs do not collide. -# The service rejects output names longer than 50 characters. +# Output names are capped at 50 characters by the service. run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" output_dataset_name = f"{dataset_name}-{run_id}" if len(output_dataset_name) > 50: @@ -152,5 +152,8 @@ # ------------------------------------------------------------------ # 2. Clean up. # ------------------------------------------------------------------ + print(f"Delete the generated dataset `{dataset.name}` v{dataset.version}.") + project_client.datasets.delete(name=dataset.name or "", version=dataset.version or "") + print(f"Delete the data generation job `{job.id}`.") project_client.beta.datasets.delete_generation_job(job_id=job.id) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py index 7d7899227975..9ca7dde7322e 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py @@ -14,6 +14,7 @@ existing agent within a time window and emits ready-to-use fine-tuning JSONL files split into training and validation partitions. 2. Polls the job to completion and prints every generated file output. + 3. Cleans up the generated fine-tuning files and the data generation job. Setting `train_split` triggers a split of the generated samples into two Azure OpenAI files — a training partition and a validation partition. @@ -71,7 +72,7 @@ poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10")) # Unique per-run output name so repeated runs do not collide. -# The service rejects output names longer than 50 characters. +# Output names are capped at 50 characters by the service. run_id = f"{datetime.now(tz=timezone.utc).strftime('%y%m%d%H%M%S')}-{uuid.uuid4().hex[:4]}" output_name = f"{dataset_name}-{run_id}" if len(output_name) > 50: @@ -161,5 +162,9 @@ # ------------------------------------------------------------------ # 3. Clean up. # ------------------------------------------------------------------ + for output in file_outputs: + print(f"Delete the generated Azure OpenAI file `{output.id}`.") + openai_client.files.delete(file_id=output.id) + print(f"Delete the data generation job `{job.id}`.") project_client.beta.datasets.delete_generation_job(job_id=job.id) diff --git a/sdk/ai/azure-ai-projects/tests/samples/test_samples.py b/sdk/ai/azure-ai-projects/tests/samples/test_samples.py index b6bc0593a102..6260e8b2ff8d 100644 --- a/sdk/ai/azure-ai-projects/tests/samples/test_samples.py +++ b/sdk/ai/azure-ai-projects/tests/samples/test_samples.py @@ -153,8 +153,8 @@ def test_deployments_samples(self, sample_path: str, **kwargs) -> None: @additionalSampleTests( [ AdditionalSampleTestDetail( - test_id="sample_dataset_generation_job_with_evaluation", - sample_filename="sample_dataset_generation_job_with_evaluation.py", + test_id="sample_dataset_generation_job_simpleqna_with_prompt_source", + sample_filename="sample_dataset_generation_job_simpleqna_with_prompt_source.py", env_vars={ "POLL_INTERVAL_SECONDS": "60", }, @@ -165,7 +165,7 @@ def test_deployments_samples(self, sample_path: str, **kwargs) -> None: "sample_path", get_sample_paths( "datasets", - samples_to_skip=["sample_dataset_generation_job_with_evaluation.py"], + samples_to_skip=["sample_dataset_generation_job_simpleqna_with_prompt_source.py"], ), ) @SamplePathPasser() From 8294b0ae39a3a17bef7efa77910b664a92c22fd6 Mon Sep 17 00:00:00 2001 From: April Kwong Date: Fri, 22 May 2026 00:34:18 -0700 Subject: [PATCH 3/9] Tighten datasets README: consistency pass - Bump pip line to azure-ai-projects>=2.2.0 to match all data-gen samples. - Drop redundant 'matches the FOUNDRY_MODEL_NAME convention' parenthetical in the FOUNDRY_MODEL_NAME description. - Reword the quickstart export line to use a placeholder '' and rephrase the inline comment. - Align FOUNDRY_PROJECT_ENDPOINT placeholder casing across the prerequisites bullet and the quickstart code block. - Standardize the agent-source sample reference as 'simple_qna' (snake_case, code-formatted) to match the surrounding usage. - Reorder and rewrite the Data Generation Jobs sample table so rows are grouped by scenario (Evaluation first, Supervised fine-tuning second), and unify the description voice across rows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../samples/datasets/README.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md index 6af0376b7858..57fb4e8c5a0e 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/README.md +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -7,15 +7,15 @@ This folder contains samples demonstrating how to work with versioned Datasets a Before running any sample: ```bash -pip install "azure-ai-projects>=2.0.0" azure-identity python-dotenv +pip install "azure-ai-projects>=2.2.0" azure-identity python-dotenv ``` To run asynchronous samples, you will also need to install `aiohttp`. The data generation samples that interact with Azure OpenAI files (any sample that emits or consumes Azure OpenAI File outputs/inputs) also require `openai`. Set these environment variables: -- `FOUNDRY_PROJECT_ENDPOINT` - Required for all samples. Your Azure AI Project endpoint (e.g., `https://.services.ai.azure.com/api/projects/`). -- `FOUNDRY_MODEL_NAME` - Required for `simple_qna` data generation samples (`sample_dataset_generation_job_simpleqna_with_prompt_source.py`, `sample_dataset_generation_job_simpleqna_with_file_source.py`, `sample_dataset_generation_job_simpleqna_with_agent_source.py`, `sample_dataset_generation_job_simpleqna_for_finetuning.py`). The name of an Azure OpenAI model **deployment** in your project (matches the `FOUNDRY_MODEL_NAME` convention used elsewhere in this samples folder). For **evaluation** jobs the deployment must support the [Responses API](https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support); for **fine-tuning** jobs the deployment must support chat completions (e.g. `gpt-4o`, `gpt-4.1`). -- `FOUNDRY_AGENT_NAME` - Required for the two traces samples (`sample_dataset_generation_job_traces_for_evaluation.py`, `sample_dataset_generation_job_traces_for_finetuning.py`). The name of an agent that has recent traces in Application Insights. Traces sources support both Foundry Agents and third-party (OpenTelemetry instrumented) agents. The agent-source SimpleQnA sample (`sample_dataset_generation_job_simpleqna_with_agent_source.py`) does *not* read this variable — it creates its own short-lived prompt agent at runtime and cleans it up at the end. +- `FOUNDRY_PROJECT_ENDPOINT` - Required for all samples. Your Azure AI Project endpoint (e.g., `https://.services.ai.azure.com/api/projects/`). +- `FOUNDRY_MODEL_NAME` - Required for `simple_qna` data generation samples (`sample_dataset_generation_job_simpleqna_with_prompt_source.py`, `sample_dataset_generation_job_simpleqna_with_file_source.py`, `sample_dataset_generation_job_simpleqna_with_agent_source.py`, `sample_dataset_generation_job_simpleqna_for_finetuning.py`). The name of an Azure OpenAI model **deployment** in your project. For **evaluation** jobs the deployment must support the [Responses API](https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support); for **fine-tuning** jobs the deployment must support chat completions (e.g. `gpt-4o`, `gpt-4.1`). +- `FOUNDRY_AGENT_NAME` - Required for the two traces samples (`sample_dataset_generation_job_traces_for_evaluation.py`, `sample_dataset_generation_job_traces_for_finetuning.py`). The name of an agent that has recent traces in Application Insights. Traces sources support both Foundry Agents and third-party (OpenTelemetry instrumented) agents. The agent-source `simple_qna` sample (`sample_dataset_generation_job_simpleqna_with_agent_source.py`) does *not* read this variable — it creates its own short-lived prompt agent at runtime and cleans it up at the end. Most samples accept additional optional environment variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY_TRACES_WINDOW_DAYS`, etc.) — see each sample's docstring for details. @@ -24,7 +24,7 @@ Most samples accept additional optional environment variables (`DATASET_NAME`, ` ```bash # Set environment variables export FOUNDRY_PROJECT_ENDPOINT="https://.services.ai.azure.com/api/projects/" -export FOUNDRY_MODEL_NAME="gpt-4o-mini" # Replace with your model deployment +export FOUNDRY_MODEL_NAME="" # Replace with your model name # Run a sample. For example: python sample_datasets.py @@ -49,8 +49,8 @@ To keep the project clean across repeated runs, each sample below also deletes e | Sample | Source(s) | Scenario | Description | |--------|-----------|----------|-------------| | [sample_dataset_generation_job_simpleqna_with_prompt_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py) | Prompt | Evaluation | Generate a QnA dataset from an inline prompt and run an evaluation against it end-to-end | -| [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate an evaluation dataset from an agent's recent conversation traces (traces recipe) | -| [sample_dataset_generation_job_traces_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py) | Traces | Supervised fine-tuning | Generate ready-to-use training + validation JSONL files from an agent's recent traces | -| [sample_dataset_generation_job_simpleqna_with_agent_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py) | Agent definition | Evaluation | Self-contained: creates a short-lived `PromptAgentDefinition`, then generates an evaluation dataset from the agent's instructions / prompt via the `simple_qna` recipe | -| [sample_dataset_generation_job_simpleqna_with_file_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py) | File (Azure OpenAI) + Prompt | Evaluation | Combine an uploaded reference document with an inline Prompt to steer multi-source SimpleQnA generation, and confirm output metadata propagation | -| [sample_dataset_generation_job_simpleqna_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py) | File (Azure OpenAI) | Supervised fine-tuning | Upload a reference document as an Azure OpenAI File and generate short- and long-answer fine-tuning files from it | +| [sample_dataset_generation_job_simpleqna_with_file_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py) | File (Azure OpenAI) + Prompt | Evaluation | Generate a QnA dataset from a multi-source job that combines an uploaded Azure OpenAI File with an inline Prompt, and verify that the caller-supplied description and tags propagate onto the dataset | +| [sample_dataset_generation_job_simpleqna_with_agent_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py) | Agent definition | Evaluation | Generate a QnA dataset by creating a short-lived `PromptAgentDefinition` and sourcing the job from the agent's instructions | +| [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate a QnA evaluation dataset from an agent's recent conversation traces | +| [sample_dataset_generation_job_simpleqna_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py) | File (Azure OpenAI) | Supervised fine-tuning | Generate supervised fine-tuning JSONL files (training and validation partitions) from an uploaded Azure OpenAI File | +| [sample_dataset_generation_job_traces_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py) | Traces | Supervised fine-tuning | Generate supervised fine-tuning JSONL files (training and validation partitions) from an agent's recent conversation traces | From 4befda7549dd8883ee2404cdc879339f99a77073 Mon Sep 17 00:00:00 2001 From: April Kwong Date: Fri, 22 May 2026 00:39:44 -0700 Subject: [PATCH 4/9] Make datasets README more concise - Replace the three-bullet env-var section with a compact table; drop the per-bullet sample-filename enumerations (the Sample Index already maps samples to their scenarios) and the agent-source FOUNDRY_AGENT_NAME exception note (covered in the sample's own docstring). - Trim the async/openai prerequisites sentence. - Collapse the two Data Generation Jobs intro paragraphs into one. - Trim the file-source row description and drop a redundant 'evaluation' qualifier from the traces-eval row (Scenario column already says it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../samples/datasets/README.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md index 57fb4e8c5a0e..679e68349716 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/README.md +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -10,14 +10,17 @@ Before running any sample: pip install "azure-ai-projects>=2.2.0" azure-identity python-dotenv ``` -To run asynchronous samples, you will also need to install `aiohttp`. The data generation samples that interact with Azure OpenAI files (any sample that emits or consumes Azure OpenAI File outputs/inputs) also require `openai`. +To run asynchronous samples, also install `aiohttp`. Samples that produce or consume Azure OpenAI files also require `openai`. Set these environment variables: -- `FOUNDRY_PROJECT_ENDPOINT` - Required for all samples. Your Azure AI Project endpoint (e.g., `https://.services.ai.azure.com/api/projects/`). -- `FOUNDRY_MODEL_NAME` - Required for `simple_qna` data generation samples (`sample_dataset_generation_job_simpleqna_with_prompt_source.py`, `sample_dataset_generation_job_simpleqna_with_file_source.py`, `sample_dataset_generation_job_simpleqna_with_agent_source.py`, `sample_dataset_generation_job_simpleqna_for_finetuning.py`). The name of an Azure OpenAI model **deployment** in your project. For **evaluation** jobs the deployment must support the [Responses API](https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support); for **fine-tuning** jobs the deployment must support chat completions (e.g. `gpt-4o`, `gpt-4.1`). -- `FOUNDRY_AGENT_NAME` - Required for the two traces samples (`sample_dataset_generation_job_traces_for_evaluation.py`, `sample_dataset_generation_job_traces_for_finetuning.py`). The name of an agent that has recent traces in Application Insights. Traces sources support both Foundry Agents and third-party (OpenTelemetry instrumented) agents. The agent-source `simple_qna` sample (`sample_dataset_generation_job_simpleqna_with_agent_source.py`) does *not* read this variable — it creates its own short-lived prompt agent at runtime and cleans it up at the end. -Most samples accept additional optional environment variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY_TRACES_WINDOW_DAYS`, etc.) — see each sample's docstring for details. +| Variable | Required by | Value | +|---|---|---| +| `FOUNDRY_PROJECT_ENDPOINT` | All samples | Your Azure AI Project endpoint, e.g. `https://.services.ai.azure.com/api/projects/` | +| `FOUNDRY_MODEL_NAME` | `simple_qna` data-generation samples | An Azure OpenAI model deployment in your project. For **evaluation** jobs use a [Responses API](https://learn.microsoft.com/azure/foundry/openai/how-to/responses?tabs=python-key#model-support) model; for **fine-tuning** jobs use a chat-completions model (e.g. `gpt-4o`, `gpt-4.1`). | +| `FOUNDRY_AGENT_NAME` | Traces data-generation samples | An agent with recent traces in Application Insights. Foundry Agents and OpenTelemetry-instrumented third-party agents are both supported. | + +Optional per-sample variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY_TRACES_WINDOW_DAYS`, etc.) are documented in each sample's docstring. ## Running a Sample @@ -42,15 +45,13 @@ python sample_datasets.py ### Data Generation Jobs -Data generation jobs synthesize evaluation datasets or supervised fine-tuning files from different kinds of sources (an agent's traces, an agent's definition, an inline prompt, or an Azure OpenAI File). The job runs server-side; the samples below show how to submit a job, poll it to completion, and locate the generated artifacts. - -To keep the project clean across repeated runs, each sample below also deletes every resource it creates (including the job record, any uploaded input files, any short-lived agent, and the generated dataset or fine-tuning files) before exiting. +Data generation jobs run server-side to synthesize evaluation datasets or supervised fine-tuning files from an agent's traces, an agent's definition, an inline prompt, or an Azure OpenAI File. Each sample below submits a job, polls it to completion, locates the generated artifacts, and then deletes everything it created (job record, uploaded input files, short-lived agents, and the generated outputs) so repeated runs don't accumulate artifacts. | Sample | Source(s) | Scenario | Description | |--------|-----------|----------|-------------| | [sample_dataset_generation_job_simpleqna_with_prompt_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py) | Prompt | Evaluation | Generate a QnA dataset from an inline prompt and run an evaluation against it end-to-end | -| [sample_dataset_generation_job_simpleqna_with_file_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py) | File (Azure OpenAI) + Prompt | Evaluation | Generate a QnA dataset from a multi-source job that combines an uploaded Azure OpenAI File with an inline Prompt, and verify that the caller-supplied description and tags propagate onto the dataset | +| [sample_dataset_generation_job_simpleqna_with_file_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py) | File (Azure OpenAI) + Prompt | Evaluation | Generate a QnA dataset from a multi-source job combining an Azure OpenAI File with an inline Prompt; verifies output metadata propagation | | [sample_dataset_generation_job_simpleqna_with_agent_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py) | Agent definition | Evaluation | Generate a QnA dataset by creating a short-lived `PromptAgentDefinition` and sourcing the job from the agent's instructions | -| [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate a QnA evaluation dataset from an agent's recent conversation traces | +| [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate a QnA dataset from an agent's recent conversation traces | | [sample_dataset_generation_job_simpleqna_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py) | File (Azure OpenAI) | Supervised fine-tuning | Generate supervised fine-tuning JSONL files (training and validation partitions) from an uploaded Azure OpenAI File | | [sample_dataset_generation_job_traces_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py) | Traces | Supervised fine-tuning | Generate supervised fine-tuning JSONL files (training and validation partitions) from an agent's recent conversation traces | From 12eead0e6dc9d9c84542741470a9c2d09e2c8947 Mon Sep 17 00:00:00 2001 From: April Kwong Date: Fri, 22 May 2026 00:50:30 -0700 Subject: [PATCH 5/9] Drop redundant 'Running a Sample' section from datasets README Prerequisites already documents the required environment variables, and 'python .py' is self-evident; the duplicate snippet only added noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/ai/azure-ai-projects/samples/datasets/README.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md index 679e68349716..8a812eb3c28f 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/README.md +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -22,17 +22,6 @@ Set these environment variables: Optional per-sample variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY_TRACES_WINDOW_DAYS`, etc.) are documented in each sample's docstring. -## Running a Sample - -```bash -# Set environment variables -export FOUNDRY_PROJECT_ENDPOINT="https://.services.ai.azure.com/api/projects/" -export FOUNDRY_MODEL_NAME="" # Replace with your model name - -# Run a sample. For example: -python sample_datasets.py -``` - ## Sample Index ### Dataset Basics From 93fe9112c1e944cb9fcc87cfcdb80397d1f92a9c Mon Sep 17 00:00:00 2001 From: April Kwong Date: Fri, 22 May 2026 01:00:26 -0700 Subject: [PATCH 6/9] Drop Data Generation Jobs intro paragraph from datasets README The table immediately below already shows the source, scenario, and description for each sample; the prose summary just restated it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/ai/azure-ai-projects/samples/datasets/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md index 8a812eb3c28f..eac863675229 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/README.md +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -34,8 +34,6 @@ Optional per-sample variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY ### Data Generation Jobs -Data generation jobs run server-side to synthesize evaluation datasets or supervised fine-tuning files from an agent's traces, an agent's definition, an inline prompt, or an Azure OpenAI File. Each sample below submits a job, polls it to completion, locates the generated artifacts, and then deletes everything it created (job record, uploaded input files, short-lived agents, and the generated outputs) so repeated runs don't accumulate artifacts. - | Sample | Source(s) | Scenario | Description | |--------|-----------|----------|-------------| | [sample_dataset_generation_job_simpleqna_with_prompt_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py) | Prompt | Evaluation | Generate a QnA dataset from an inline prompt and run an evaluation against it end-to-end | From 721ad5d5ff8540714a9525251915f1aac3eb502f Mon Sep 17 00:00:00 2001 From: April Kwong Date: Fri, 22 May 2026 01:04:28 -0700 Subject: [PATCH 7/9] Drop intro blurb from datasets README The title plus the Sample Index already make the scope obvious. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/ai/azure-ai-projects/samples/datasets/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md index eac863675229..065bec6f973c 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/README.md +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -1,7 +1,5 @@ # Azure AI Projects - Dataset Samples -This folder contains samples demonstrating how to work with versioned Datasets and data generation jobs in Azure AI Foundry using the `azure-ai-projects` SDK. - ## Prerequisites Before running any sample: From df91cbbaf4c6c9dc02e3e38aaec3e851f3e1f703 Mon Sep 17 00:00:00 2001 From: April Kwong Date: Fri, 22 May 2026 01:21:41 -0700 Subject: [PATCH 8/9] Smooth out two awkward Sample Index descriptions - file-source row: drop the 'verifies output metadata propagation' tail (it's a test detail, not a user-facing feature) and 'multi-source job combining' (the Source column already says File + Prompt). - agent-source row: replace 'short-lived \PromptAgentDefinition\' with 'prompt agent' so the description reads naturally without leaking a type name into the table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- sdk/ai/azure-ai-projects/samples/datasets/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/README.md b/sdk/ai/azure-ai-projects/samples/datasets/README.md index 065bec6f973c..68a7e111ff82 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/README.md +++ b/sdk/ai/azure-ai-projects/samples/datasets/README.md @@ -35,8 +35,8 @@ Optional per-sample variables (`DATASET_NAME`, `POLL_INTERVAL_SECONDS`, `FOUNDRY | Sample | Source(s) | Scenario | Description | |--------|-----------|----------|-------------| | [sample_dataset_generation_job_simpleqna_with_prompt_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_prompt_source.py) | Prompt | Evaluation | Generate a QnA dataset from an inline prompt and run an evaluation against it end-to-end | -| [sample_dataset_generation_job_simpleqna_with_file_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py) | File (Azure OpenAI) + Prompt | Evaluation | Generate a QnA dataset from a multi-source job combining an Azure OpenAI File with an inline Prompt; verifies output metadata propagation | -| [sample_dataset_generation_job_simpleqna_with_agent_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py) | Agent definition | Evaluation | Generate a QnA dataset by creating a short-lived `PromptAgentDefinition` and sourcing the job from the agent's instructions | +| [sample_dataset_generation_job_simpleqna_with_file_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_file_source.py) | File (Azure OpenAI) + Prompt | Evaluation | Generate a QnA dataset from an Azure OpenAI File combined with an inline Prompt | +| [sample_dataset_generation_job_simpleqna_with_agent_source.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_with_agent_source.py) | Agent definition | Evaluation | Generate a QnA dataset by creating a prompt agent and sourcing the job from the agent's instructions | | [sample_dataset_generation_job_traces_for_evaluation.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py) | Traces | Evaluation | Generate a QnA dataset from an agent's recent conversation traces | | [sample_dataset_generation_job_simpleqna_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_simpleqna_for_finetuning.py) | File (Azure OpenAI) | Supervised fine-tuning | Generate supervised fine-tuning JSONL files (training and validation partitions) from an uploaded Azure OpenAI File | | [sample_dataset_generation_job_traces_for_finetuning.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py) | Traces | Supervised fine-tuning | Generate supervised fine-tuning JSONL files (training and validation partitions) from an agent's recent conversation traces | From 2a03e9476f3c8c1bab8a48be41c9d7de9f03b605 Mon Sep 17 00:00:00 2001 From: April Kwong Date: Fri, 22 May 2026 01:26:26 -0700 Subject: [PATCH 9/9] Note third-party agent support in trace sample docstrings Aligns the trace sample docstrings with the FOUNDRY_AGENT_NAME row in the datasets README: both Foundry Agents and OpenTelemetry-instrumented third-party agents are valid sources. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../sample_dataset_generation_job_traces_for_evaluation.py | 5 +++-- .../sample_dataset_generation_job_traces_for_finetuning.py | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py index 00d487c57aae..9b2ca86a89bf 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_evaluation.py @@ -6,7 +6,7 @@ """ DESCRIPTION: - Generates an evaluation dataset from a Foundry agent's recent conversation + Generates an evaluation dataset from an agent's recent conversation traces. The sample: 1. Creates a `DataGenerationJob` (scenario=EVALUATION, type=traces) that @@ -31,7 +31,8 @@ Set these environment variables with your own values: 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found in the overview page of your Microsoft Foundry project. - 2) FOUNDRY_AGENT_NAME - Required. The name of a Foundry agent that has recent + 2) FOUNDRY_AGENT_NAME - Required. The name of an agent (Foundry Agent or + OpenTelemetry-instrumented third-party agent) that has recent conversation traces in Application Insights. 3) DATASET_NAME - Optional. Name to assign to the generated output dataset. Defaults to `traces-eval-sample`. The service caps the rendered output diff --git a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py index 9ca7dde7322e..a5e19780a0b3 100644 --- a/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py +++ b/sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_traces_for_finetuning.py @@ -6,7 +6,7 @@ """ DESCRIPTION: - Generates supervised fine-tuning data from a Foundry agent's recent + Generates supervised fine-tuning data from an agent's recent conversation traces. The sample: 1. Creates a `DataGenerationJob` (scenario=SUPERVISED_FINETUNING, @@ -31,7 +31,8 @@ Set these environment variables with your own values: 1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found in the overview page of your Microsoft Foundry project. - 2) FOUNDRY_AGENT_NAME - Required. The name of a Foundry agent that has recent + 2) FOUNDRY_AGENT_NAME - Required. The name of an agent (Foundry Agent or + OpenTelemetry-instrumented third-party agent) that has recent conversation traces in Application Insights. 3) DATASET_NAME - Optional. Name to assign to the generated output files (used as the file name prefix). Defaults to `traces-finetuning-sample`.