[fix] -e environment switch support + relative dataset path + not sta…#8566
Merged
Conversation
Zyysurely
requested review from
JeffreyCA,
glharper,
therealjohn,
trangevi and
trrwilson
as code owners
June 8, 2026 20:36
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Azure AI Agents extension’s optimize and eval command families to correctly respect the -e/--environment selection, improve dataset path resolution behavior (cwd-relative at CLI entry, project-relative when read from config), and avoid using stale deployed agent versions by reconciling config against environment state at runtime.
Changes:
- Thread the selected environment name through optimize/eval commands and helpers so environment-scoped reads (endpoints, agent name/version, model deployments, job ID persistence) honor
-e. - Fix dataset path handling by resolving CLI-provided relative paths against the caller’s cwd, while still resolving config-stored relative paths against the agent project directory.
- Consolidate agent name/version reconciliation into
reconcileConfigAgent(...)and stop persistingagent.versionintoeval.yamlto prevent drift after redeployments.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize.go | Threads env name into optimize flows; resolves dataset paths and reconciles agent version. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_test.go | Updates tests for new reconcile helper and updated optimize output signature. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_status.go | Honors -e for endpoint resolution and last-job lookup; passes env through output helpers. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_status_test.go | Updates command construction to accept ExtensionContext. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_prompts.go | Threads env name through model/deployment resolution helpers. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_prompts_test.go | Updates tests for new prompt helper signatures. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_list.go | Honors -e for endpoint resolution in list command. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_list_test.go | Updates command construction to accept ExtensionContext. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_helpers.go | Adds named-environment endpoint lookup and env-scoped persistence for last job ID + portal links. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_helpers_test.go | Updates resolve signatures and adds tests for getExistingEnvironment. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_deploy.go | Threads env name into deploy command; named-env endpoint resolution for deploy. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_deploy_test.go | Updates command construction to accept ExtensionContext. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_cancel.go | Honors -e for endpoint resolution in cancel command. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_cancel_test.go | Updates command construction to accept ExtensionContext. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_apply.go | Uses selected env when persisting candidate ID to the azd environment. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval.go | Adds envName option and uses getExistingEnvironment for env-scoped reads. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_update.go | Threads env name into context resolution for update. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_show.go | Threads env name into context resolution for show. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_show_test.go | Updates command construction to accept ExtensionContext. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_run.go | Uses unified reconcile helper and resolves config-relative dataset paths against agent project dir. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_list.go | Threads env name into context resolution for list. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_list_test.go | Updates command construction to accept ExtensionContext. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init.go | Threads env name into context resolution; resolves CLI dataset path against cwd before project resolution. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init_test.go | Updates tests for non-persisted version and adds resolveCwdRelative tests. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init_prompts.go | Threads env name through model selection prompt. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_init_jobs.go | Stops persisting agent version in new eval configs; introduces resolveCwdRelative. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_helpers.go | Introduces reconcileConfigAgent and threads env name through model selection helpers. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/eval_helpers_test.go | Updates tests for reconcileConfigAgent and adds version reconciliation cases. |
therealjohn
approved these changes
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rename
azd ai agent eval init→eval generate, and fix-eflag, dataset path resolution, and stale agent versionProblem
Four related issues across the
azd ai agent optimizeandevalcommand families:eval initdoesn't convey that the command generates eval assets (dataset/evaluators) and writeseval.yaml.-eflag not honored: The-e/--environmentflag is registered as a persistent flag by the SDK (extCtx.Environment), but the optimize/eval commands ignore it — they callGetCurrent()and always use the project default environment.--datasetrelative path confusion: When--dataset ./data.jsonlis passed from a directory other than the agent project root, the path is resolved againstagentProjectinstead of the caller's working directory, producing "file not found" errors.agent.version: Afterazd deploycreates a new agent version,eval.yamlstill holds the old version.eval runhad partial reconciliation, butoptimizeread the config version directly with no environment reconciliation, optimizing against an outdated version.Changes
Command rename (
init→generate):generate.initcommand that points users togenerate.-eflag support:extCtx.Environmentthrough all optimize and eval subcommand constructors.endpointFromNamedEnv()to readFOUNDRY_PROJECT_ENDPOINTfrom a named environment.optimizeConnectionFlags.resolve()to check the named env before falling back to the default cascade.getExistingEnvironment()across optimize helpers for job ID persistence, portal links, and deployment reporting.Dataset cwd resolution:
resolveCwdRelative()to convert relative paths to absolute based on cwd.--datasetflag and interactive prompt) beforeresolveLocalDatasetFile(), which still resolves againstagentProject.ResolveRelPathon the read side (eval_run, optimizesubmitJob) so relative paths stored ineval.yamlresolve correctly againstagentProject.Agent version reconciliation:
reconcileConfigAgentNameandreconcileConfigAgentVersioninto a singlereconcileConfigAgent(agent, envName, envVersion, configPath).agentVersiontooptimizeAgentContextso optimize resolves version from the environment alongside the agent name.agent.versiontoeval.yamlinnewEvalConfig()— version is always resolved at runtime from the environment.Testing
TestGetExistingEnvironment_*(3): empty name uses current, explicit name usesGet, not found returns nil.TestResolveCwdRelative_*(2): absolute unchanged, relative resolved to cwd.TestReconcileConfigAgent/*(6): includes "clears stale version" and "env version overrides config version".TestNewEvalConfig/uses_default_name: updated to verify version is not persisted.go test -short ./internal/cmd/....