Skip to content

[fix] -e environment switch support + relative dataset path + not sta…#8566

Merged
Zyysurely merged 5 commits into
Azure:mainfrom
Zyysurely:zyying/eval_papercuts
Jun 9, 2026
Merged

[fix] -e environment switch support + relative dataset path + not sta…#8566
Zyysurely merged 5 commits into
Azure:mainfrom
Zyysurely:zyying/eval_papercuts

Conversation

@Zyysurely

@Zyysurely Zyysurely commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Rename azd ai agent eval initeval generate, and fix -e flag, dataset path resolution, and stale agent version

Problem

Four related issues across the azd ai agent optimize and eval command families:

  1. Unclear command name: eval init doesn't convey that the command generates eval assets (dataset/evaluators) and writes eval.yaml.
  2. -e flag not honored: The -e / --environment flag is registered as a persistent flag by the SDK (extCtx.Environment), but the optimize/eval commands ignore it — they call GetCurrent() and always use the project default environment.
  3. --dataset relative path confusion: When --dataset ./data.jsonl is passed from a directory other than the agent project root, the path is resolved against agentProject instead of the caller's working directory, producing "file not found" errors.
  4. Stale agent.version: After azd deploy creates a new agent version, eval.yaml still holds the old version. eval run had partial reconciliation, but optimize read the config version directly with no environment reconciliation, optimizing against an outdated version.

Changes

Command rename (initgenerate):

  • Rename the subcommand and update help text and user-facing messages to reference generate.
  • Keep a hidden, deprecated init command that points users to generate.

-e flag support:

  • Thread extCtx.Environment through all optimize and eval subcommand constructors.
  • Add endpointFromNamedEnv() to read FOUNDRY_PROJECT_ENDPOINT from a named environment.
  • Update optimizeConnectionFlags.resolve() to check the named env before falling back to the default cascade.
  • Use getExistingEnvironment() across optimize helpers for job ID persistence, portal links, and deployment reporting.

Dataset cwd resolution:

  • Add resolveCwdRelative() to convert relative paths to absolute based on cwd.
  • Call it at CLI entry points (--dataset flag and interactive prompt) before resolveLocalDatasetFile(), which still resolves against agentProject.
  • Add ResolveRelPath on the read side (eval_run, optimize submitJob) so relative paths stored in eval.yaml resolve correctly against agentProject.

Agent version reconciliation:

  • Merge reconcileConfigAgentName and reconcileConfigAgentVersion into a single reconcileConfigAgent(agent, envName, envVersion, configPath).
  • Add agentVersion to optimizeAgentContext so optimize resolves version from the environment alongside the agent name.
  • Stop writing agent.version to eval.yaml in newEvalConfig() — version is always resolved at runtime from the environment.
  • Clear a stale version from config when the environment has no version (meaning "use latest").

Testing

  • TestGetExistingEnvironment_* (3): empty name uses current, explicit name uses Get, not found returns nil.
  • TestResolveCwdRelative_* (2): absolute unchanged, relative resolved to cwd.
  • TestReconcileConfigAgent/* (6): includes "clears stale version" and "env version overrides config version".
  • TestNewEvalConfig/uses_default_name: updated to verify version is not persisted.
  • All existing call sites updated for the new function signatures.
  • Full suite passes: go test -short ./internal/cmd/....

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Azure AI Agents extension’s optimize and eval command families to correctly respect the -e/--environment selection, improve dataset path resolution behavior (cwd-relative at CLI entry, project-relative when read from config), and avoid using stale deployed agent versions by reconciling config against environment state at runtime.

Changes:

  • Thread the selected environment name through optimize/eval commands and helpers so environment-scoped reads (endpoints, agent name/version, model deployments, job ID persistence) honor -e.
  • Fix dataset path handling by resolving CLI-provided relative paths against the caller’s cwd, while still resolving config-stored relative paths against the agent project directory.
  • Consolidate agent name/version reconciliation into reconcileConfigAgent(...) and stop persisting agent.version into eval.yaml to prevent drift after redeployments.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize.go Threads env name into optimize flows; resolves dataset paths and reconciles agent version.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_test.go Updates tests for new reconcile helper and updated optimize output signature.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_status.go Honors -e for endpoint resolution and last-job lookup; passes env through output helpers.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_status_test.go Updates command construction to accept ExtensionContext.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_prompts.go Threads env name through model/deployment resolution helpers.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_prompts_test.go Updates tests for new prompt helper signatures.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_list.go Honors -e for endpoint resolution in list command.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_list_test.go Updates command construction to accept ExtensionContext.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_helpers.go Adds named-environment endpoint lookup and env-scoped persistence for last job ID + portal links.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_helpers_test.go Updates resolve signatures and adds tests for getExistingEnvironment.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_deploy.go Threads env name into deploy command; named-env endpoint resolution for deploy.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_deploy_test.go Updates command construction to accept ExtensionContext.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_cancel.go Honors -e for endpoint resolution in cancel command.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_cancel_test.go Updates command construction to accept ExtensionContext.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_apply.go Uses selected env when persisting candidate ID to the azd environment.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval.go Adds envName option and uses getExistingEnvironment for env-scoped reads.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_update.go Threads env name into context resolution for update.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_show.go Threads env name into context resolution for show.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_show_test.go Updates command construction to accept ExtensionContext.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_run.go Uses unified reconcile helper and resolves config-relative dataset paths against agent project dir.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_list.go Threads env name into context resolution for list.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_list_test.go Updates command construction to accept ExtensionContext.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init.go Threads env name into context resolution; resolves CLI dataset path against cwd before project resolution.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init_test.go Updates tests for non-persisted version and adds resolveCwdRelative tests.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init_prompts.go Threads env name through model selection prompt.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init_jobs.go Stops persisting agent version in new eval configs; introduces resolveCwdRelative.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_helpers.go Introduces reconcileConfigAgent and threads env name through model selection helpers.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_helpers_test.go Updates tests for reconcileConfigAgent and adds version reconciliation cases.

Comment thread cli/azd/extensions/azure.ai.agents/internal/cmd/eval_helpers.go Outdated
Comment thread cli/azd/extensions/azure.ai.agents/internal/cmd/eval_helpers.go
Comment thread cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_apply.go
@github-actions github-actions Bot added the ext-agents azure.ai.agents extension label Jun 8, 2026
@Zyysurely
Zyysurely merged commit 717b90f into Azure:main Jun 9, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ext-agents azure.ai.agents extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue] Fix -e flag, dataset path resolution, and stale agent version in optimize/eval commands

4 participants