test(extensions): verify provider manifests (part 2/2) - #9122
Conversation
6223cc8 to
d44a551
Compare
d44a551 to
ef36889
Compare
There was a problem hiding this comment.
Pull request overview
Adds CI-enforced verification that first-party extensions register exactly the providers they declare in extension.yaml, rolling the canonical verification test + reusable workflow across the Azure AI extensions.
Changes:
- Added a central Go test (
pkg/extensions) that scans first-party extension manifests and enforces presence of the canonical provider-manifest verification test. - Added
TestConfigureExtensionHostMatchesManifestto each provider-bearing Azure AI extension to assertazdext.VerifyProvidersMatchManifest(configureExtensionHost, ...). - Added a reusable GitHub Actions workflow to run provider-manifest verification and wired it into the Azure AI extension lint workflows; updated the Go module template + extension modules to
github.com/azure/azure-dev/cli/azd v1.28.0.
Show a summary per file
| File | Description |
|---|---|
| cli/azd/pkg/extensions/extension_manifest_files_test.go | Adds central enforcement that provider-bearing extensions include the canonical verification test and validates the AST matcher behavior. |
| cli/azd/extensions/microsoft.azd.extensions/internal/resources/languages/go/go.sum | Updates Go language template sums for the bumped azd module and indirect deps. |
| cli/azd/extensions/microsoft.azd.extensions/internal/resources/languages/go/go.mod.tmpl | Updates Go language template to Go 1.26.4 and azd module v1.28.0 (plus indirect deps). |
| cli/azd/extensions/azure.ai.toolboxes/internal/cmd/providers_manifest_test.go | Adds canonical provider-manifest verification test for toolboxes extension. |
| cli/azd/extensions/azure.ai.toolboxes/go.sum | Updates toolboxes module sums for azd module bump. |
| cli/azd/extensions/azure.ai.toolboxes/go.mod | Bumps toolboxes dependency on azd module to v1.28.0. |
| cli/azd/extensions/azure.ai.skills/internal/cmd/providers_manifest_test.go | Adds canonical provider-manifest verification test for skills extension. |
| cli/azd/extensions/azure.ai.skills/go.sum | Updates skills module sums (including new indirect deps). |
| cli/azd/extensions/azure.ai.skills/go.mod | Bumps skills dependency on azd module to v1.28.0 and aligns indirect deps. |
| cli/azd/extensions/azure.ai.routines/internal/cmd/providers_manifest_test.go | Adds canonical provider-manifest verification test for routines extension. |
| cli/azd/extensions/azure.ai.routines/go.sum | Updates routines module sums for azd module bump. |
| cli/azd/extensions/azure.ai.routines/go.mod | Bumps routines dependency on azd module to v1.28.0. |
| cli/azd/extensions/azure.ai.projects/internal/cmd/providers_manifest_test.go | Adds canonical provider-manifest verification test for projects extension. |
| cli/azd/extensions/azure.ai.projects/go.sum | Updates projects module sums (including new indirect deps). |
| cli/azd/extensions/azure.ai.projects/go.mod | Bumps projects dependency on azd module to v1.28.0 and aligns indirect deps. |
| cli/azd/extensions/azure.ai.connections/internal/cmd/providers_manifest_test.go | Adds canonical provider-manifest verification test for connections extension. |
| cli/azd/extensions/azure.ai.connections/go.sum | Updates connections module sums for azd module bump. |
| cli/azd/extensions/azure.ai.connections/go.mod | Bumps connections dependency on azd module to v1.28.0. |
| cli/azd/extensions/azure.ai.agents/internal/cmd/providers_manifest_test.go | Adds canonical provider-manifest verification test for agents extension. |
| cli/azd/extensions/azure.ai.agents/go.sum | Updates agents module sums for azd module bump. |
| cli/azd/extensions/azure.ai.agents/go.mod | Bumps agents dependency on azd module to v1.28.0. |
| .github/workflows/verify-ext-providers.yml | Adds reusable workflow + a PR-triggered “coverage” job to enforce canonical test presence and execute provider verification. |
| .github/workflows/lint-ext-azure-ai-toolboxes.yml | Wires provider verification reusable workflow into toolboxes lint pipeline. |
| .github/workflows/lint-ext-azure-ai-skills.yml | Wires provider verification reusable workflow into skills lint pipeline. |
| .github/workflows/lint-ext-azure-ai-routines.yml | Wires provider verification reusable workflow into routines lint pipeline. |
| .github/workflows/lint-ext-azure-ai-projects.yml | Wires provider verification reusable workflow into projects lint pipeline. |
| .github/workflows/lint-ext-azure-ai-connections.yml | Wires provider verification reusable workflow into connections lint pipeline. |
| .github/workflows/lint-ext-azure-ai-agents.yml | Wires provider verification reusable workflow into agents lint pipeline. |
Review details
- Files reviewed: 21/28 changed files
- Comments generated: 3
- Review effort level: Low
5180f07 to
5b59d13
Compare
|
Azure Pipelines: Successfully started running 8 pipeline(s). 14 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
jongio
left a comment
There was a problem hiding this comment.
Reviewed the provider-manifest verification rollout. The AST coverage guard enforces the canonical asserted �zdext.VerifyProvidersMatchManifest(configureExtensionHost, ...) form, and the TestHasCanonical... table exercises the discarded-error, commented-out, wrong-callback, and wrong-test cases. The reusable workflow's inputs.working-directory gating correctly splits the central coverage job from the per-extension caller job. Deferred items (demo per-ext workflow wiring, control-flow analysis) are called out and acceptable for this scope.
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
Fixes #8992
Summary
This PR rolls out provider-manifest verification across every provider-bearing Azure AI extension and adds CI enforcement so future registration drift is rejected before publication.
Issue
The extension SDK can verify runtime provider registrations against
extension.yaml, but the Azure AI extensions must adopt that helper for the protection to cover the extensions affected by #8992. Without consistent tests and CI wiring, a manifest can still claim a provider that its extension does not register.Azure AI extension coverage
The agents, connections, projects, routines, skills, and toolboxes extensions now consume azd SDK
v1.28.0and carry the canonicalTestConfigureExtensionHostMatchesManifesttest.configureExtensionHostcallback.Central coverage guard
A central manifest scan requires every provider-bearing first-party extension to carry the canonical test. It parses the test source and requires
require.NoErrorto assert the result ofazdext.VerifyProvidersMatchManifest, rejecting comments, discarded errors, or calls placed in another test.The reusable workflow also checks that the canonical test file and exact test name exist before executing it because
go test -runotherwise succeeds when no tests match.Behavior at a glance
Scope and follow-ups
This PR completes the focused first-party CI guard for #8992.
Testing
Testing covers the canonical provider-manifest check for the demo, agents, connections, projects, routines, skills, and toolboxes extensions. Coverage-guard tests verify that only an asserted helper call in the canonical test satisfies enforcement, including protection against discarded errors, comments, calls in another test, and assertions followed by later calls.