test(azure.ai.agents): validate azure.yaml examples in docs

[Glenn Harper (glharper)]

Fixes #9330

Problem

azure.yaml examples in this extension's docs were not validated, so they could drift away from both the extension and azd core:

1. README.md once omitted the required agent name; copying it failed with name cannot be empty (fix: name the azure.yaml key in the RAI policy validation error #9328).
2. A Learn article documented unsupported rai_config.rai_policy_name. azd ignored it, so the agent deployed with no guardrail and no error (MicrosoftDocs/azure-ai-docs-pr#13567).

Approach

TestDocExamplesAreValid walks the extension's markdown, extracts every fenced YAML block declaring an azure.ai.agent service, and applies three checks.

1. Runtime resolver

The snippet must survive AgentDefinitionFromService, the same entry point the extension uses at deploy time. This catches missing/invalid agent definition values.

2. Full extension JSON Schema

Extension-owned properties are validated against the compiled schemas/azure.ai.agent.json, including:

• required fields
• value types
• enum values
• patterns
• nested additionalProperties

The existing recursive vocabulary check remains because the schema's top-level additionalProperties is deliberately permissive so core service fields can coexist with extension properties. Our own docs still reject an unknown extension key that azd would silently ignore.

Inline and deprecated config properties are not validated as if they merge. activeDocAgentConfig mirrors runtime precedence: inline wins unless it omits kind and config declares it. If both locations contain extension properties, the inactive location is rejected because those values have no effect.

3. Core-owned YAML shapes

Core service properties are decoded through strict test-only mirrors of the checked-in azd YAML shapes:

• Docker
• AKS, Helm, and Kustomize
• infra, recursive layers, and deployment stacks
• hooks (single mapping or list), typed hook fields, and nil-entry validation
• the scalar/list/map fields on ServiceConfig

The top-level mirror retains yaml:",inline" AdditionalProperties so extension properties remain legal. Nested core typos are rejected instead of silently ignored. Provider-specific infra.config and hook config remain property bags intentionally.

Partial snippets

Some snippets are deliberately incomplete. The private-networking examples omit kind because the example is about the project network: block and azd falls back to on-disk agent.yaml.

They opt out explicitly:

<!-- azd:doc-example partial -->

The marker relaxes only the resolver's "must fully resolve" assertion. Schema, vocabulary, and core YAML checks still run. Default behavior remains strict, so accidentally dropping kind from a complete example is not silently excused.

Regression coverage

TestDocSchemaValidatesConstraints proves the schema catches:

• missing deployment model
• missing deployment sku
• wrong capacity type
• unsupported kind enum value
• invalid connection-name pattern

TestDecodeCoreServiceFieldsRejectsMalformedValues covers top-level and nested type failures, unknown nested fields, scalar hooks, malformed hook fields, null hook mappings/list entries, infra hooks, and deployment-stack typos/types.

TestActiveDocAgentConfig covers inline/config runtime precedence and rejects ignored properties in the inactive location.

TestExtractYAMLExamples covers language filtering, marker scoping, indented-fence dedenting, and CommonMark fence-length nesting.

Current docs contribute 7 validated agent snippets: 4 in README.md and 3 in docs/private-networking.md. New markdown examples are picked up automatically.

Validation

From cli/azd/extensions/azure.ai.agents:

• go build ./...
• go test ./... -count=1
• gofmt -s -l internal/project — clean
• golangci-lint run ./internal/project/... — 0 issues
• cspell — clean
• git diff --check — clean

The complete local diff was reviewed three times before push. Those reviews caught and fixed runtime shape precedence, null hook validation, nested core typos, and typed deployment-stack validation.

Notes

• Test-only plus doc markers; no production behavior or core cli/azd changes.
• jsonschema/v6 was already a transitive dependency and is now marked direct because the test imports it; go.sum is unchanged.
• No CHANGELOG.md entry, per the extension release process.
• AGENTS.md documents the convention and escape hatch.
• Approach 2 from the issue (make schemas/examples/*.azure.yaml the single source of truth) remains complementary but is deliberately outside this PR.

[github-actions]

📋 Prioritization Note

Thanks for the contribution! The linked issue isn't in the current milestone yet.
Thank you for logging this issue; our team is reviewing it. If you need urgent prioritization, tag @RickWinter and @kristenwomack to let us know.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
21 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds automated validation to prevent azure.ai.agent documentation examples from drifting out of sync.

Changes:

• Extracts and validates fenced YAML examples.
• Adds explicit markers for intentionally partial snippets.
• Documents the validation convention.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
internal/project/doc_examples_test.go	Adds extraction and validation tests.
docs/private-networking.md	Marks three examples as partial.
AGENTS.md	Documents validation requirements and opt-out usage.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:278

• [azd-code-reviewer] These map fields do not apply core's actual shape rules as the helper claims. For example, core decodes docker into project.DockerProjectOptions, where path is a string, but this mirror accepts docker: {path: [Dockerfile]}; checkVocabulary then skips docker as core-owned, so the broken snippet passes. Mirror the nested core types faithfully (also for K8s/Infra/Hooks), or use core's decoder.

	Docker               map[string]any    `yaml:"docker"`
	K8s                  map[string]any    `yaml:"k8s"`
	Module               string            `yaml:"module"`
	Infra                map[string]any    `yaml:"infra"`
	Hooks                map[string]any    `yaml:"hooks"`

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:407

• [azd-code-reviewer] This assertion only proves that one snippet was discovered. If any of the six existing examples changes its fence label, loses host, or misspells it as azure.ai.agents, that example silently drops out while the test remains green. Assert the current baseline count (and update it for intentional additions/removals), or track the expected snippets individually so losing coverage is visible.

	require.NotZero(t, checked, "no azure.ai.agent doc examples were found — is the extractor still working?")

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:130

• [azd-code-reviewer] Treating absent/true additionalProperties as runtime support leaves the same silent-no-op gap for typed objects. The shipped schema marks agentEndpoint and agentCard with additionalProperties: true, while runtime unmarshals them into fixed Go structs; therefore agentEndpoint: {rai_config: ...} is ignored by json.Unmarshal and still passes this vocabulary check. Tighten those schema nodes (while retaining genuinely free-form maps such as metadata/credentials) or validate them against the runtime types.

This issue also appears on line 274 of the same file.

			if allowed, ok := additional.(bool); !declared || (ok && allowed) {
				continue

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:402

• [azd-code-reviewer] The extractor has dedicated tests, but neither advertised validation rule has a committed negative regression test. With only valid live docs as fixtures, a change that makes checkVocabulary permissive or stops enforcing resolver failures can leave this suite green; the temporary defect injections described in the PR will not protect future changes. Add table-driven invalid snippets for missing name, unknown top-level/nested properties, and the partial-marker exception.

				checkVocabulary(t, e, name, svc, schema)

[Jon Gallant (jongio)]

One nit inline. Everything else checks out.

I ran the new test on the branch and mutated the docs to confirm each guard actually fires: dropping name from the README migration example reproduces the name cannot be empty failure from #9330, an undeclared key at the top level and one nested under policies[0] both fail with the file:line message, and project: [src] is a core parse error now instead of a silent coercion to "".

The extractor's nested-fence handling holds up too, so the example block inside AGENTS.md isn't picked up as its own snippet.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:349

• [azd-code-reviewer] Add negative regression fixtures for the validation rules. The current docs are already valid, so this test still passes if the missing-name check or the top-level/recursive vocabulary rejection is accidentally weakened; the manually injected defects listed in the PR description are not preserved in CI. Table-driven snippets covering missing name, an unknown service sibling, and an unknown nested property would lock down both defect classes this test is meant to catch.

func TestDocExamplesAreValid(t *testing.T) {

[Jon Gallant (jongio)]

One blocking item and one note.

Blocking: the new test fails CI on all three BuildExtension legs. It passes on this branch in isolation, which is why a local run looks clean. It only breaks against the merge with main. #9079 added a partial env: snippet to the extension README after this branch was cut, and strict-by-default rejects it for missing kind. Detail and the one-line fix are inline on the require.True assertion.

Note: the reply on my structpb thread posted as the literal text @C:\Users\glharper\AppData\Local\Temp\reply9368.md instead of the file contents, so whatever you wrote there isn't visible on the PR. Worth reposting.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:278

• [azd-code-reviewer] These untyped mirrors still do not apply core's nested shape rules. For example, a valid agent snippet containing hooks: {predeploy: "echo hi"} passes this map[string]any decode and the resolver, but core's HooksConfig.UnmarshalYAML rejects that scalar because each hook must be a mapping or sequence (pkg/ext/hooks_config.go:33-72). The test would therefore approve an example that cannot be copied into azure.yaml. Mirror the concrete nested field shapes (including their custom unmarshalling) or decode through core's actual ServiceConfig; the same gap applies to docker, k8s, and infra.

	Docker               map[string]any    `yaml:"docker"`
	K8s                  map[string]any    `yaml:"k8s"`
	Module               string            `yaml:"module"`
	Infra                map[string]any    `yaml:"infra"`
	Hooks                map[string]any    `yaml:"hooks"`

[Jon Gallant (jongio)]

The blocking item from my last review is cleared. I verified against the merge with main rather than the branch, since that's where it failed.

On 3ca6440 with main merged in, the suite validates 7 snippets and passes. Dropping the marker back out reproduces the exact CI signature I flagged:

README.md:63: service "my-agent" did not resolve to an agent definition (is `kind` missing?).

I also checked whether the marker buys more silence than intended, since the escape hatch is the part of this with the most room to go wrong later. It doesn't. With the marker still in place, an undeclared key on that same snippet fails:

README.md:64: service "my-agent" documents property "rai_config", which azd does not support.

and project: [src] fails with cannot be parsed by azd core. So partial drops only the "must resolve to an agent definition" assertion and leaves the vocabulary and core-parse checks running, which is what AGENTS.md describes.

The reply on the structpb thread that posted as a literal file path is showing real content now.

No new findings. My earlier approval went stale when these commits landed, so this still needs a current approval to merge.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:989

• [azd-code-reviewer] This rejects valid workflow examples even though the runtime accepts inline kind: workflow definitions (agent_yaml.ValidAgentKinds and TestLoadAgentDefinition_ResolvedKindValidation). The shipped schema currently permits only hosted and does not declare workflow's trigger, so a copyable workflow snippet would fail this docs test. Bring the schema/validator into parity with both supported agent kinds and add a valid workflow case.

	require.NoError(t, schema.validate(active),
		"%s: service %q %s properties do not satisfy schemas/azure.ai.agent.json. "+
			"Fix the example so it can be copied into azure.yaml as-is.", e, name, location)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:925

• [azd-code-reviewer] This is not the resolver used by deployment when a service has a root $ref: deployment calls AgentDefinitionFromResolvedService, while this raw variant sees no kind and returns found=false. The later vocabulary check also rejects $ref because it is not in the schema. As a result, a supported, copyable $ref example cannot pass this validator; validate through the resolved entry point (using the snippet's project root) and explicitly allow the include key.

				cfg := serviceConfigFromDoc(t, e, name, svc)
				_, _, found, _, err := AgentDefinitionFromService(cfg)