Proposal: Allow Extensions to Define Local Preflight Checks

[vhvb1989]

Proposal: Allow Extensions to Define Local Preflight Checks

Summary

This proposal introduces a mechanism for azd extensions to register local preflight checks that azd core runs alongside its own checks before Bicep provisioning. Extension check results are merged into the unified PreflightReport, giving extensions access to the same warning/error/abort UX flow and the rich validation context (Bicep snapshot, resource properties) that core checks use.

This is a continuation of #7053, which established the local preflight validation infrastructure.

Problem

Today, extensions that need to validate deployment preconditions before provisioning have limited options:

1. preprovision event handler — Can only return an error (blocking failure) or succeed silently. There is no way to produce warnings that let the user decide whether to proceed.

2. Init-time validation — Extensions like azure.ai.agents validate manifests during azd agent init, but this happens long before provisioning and cannot inspect the deployment's predicted resource graph.

3. Mimicking the warning UX — Extensions can try to replicate the preflight warning-then-prompt pattern, but the result is inconsistent with core's UX, duplicates logic, and doesn't integrate with telemetry.

None of these approaches give extensions access to the validation context that azd core generates during preflight — the Bicep snapshot (fully resolved resource graph), deployment scope, resource properties, or the consolidated PreflightReport.

Benefits

Benefit	Description
Unified UX	Extension warnings and errors appear in the same PreflightReport as core checks — consistent user experience
Rich context access	Extensions can inspect the full predicted resource graph (Bicep snapshot), deployment scope, and derived resource properties
User control	Warnings prompt the user for continuation; errors block — same behavior as core checks
Concurrent execution	Core runs extension checks alongside its own checks — no sequential bottleneck
Extension composability	Multiple extensions can independently contribute checks without coordinating
Telemetry integration	Extension diagnostic IDs flow through the same telemetry pipeline as core diagnostics
Delegation to core	Extensions don't need to implement their own warning/prompt/abort logic — they declare findings and core handles the UX

Proposed Design

Scope

This proposal targets Bicep-based deployments exclusively. All current core extensions use Bicep, and the preflight infrastructure from #7053 is Bicep-specific.

New Extension Capability: preflight-checks

Extensions declare the capability in extension.yaml:

capabilities:
  - preflight-checks

Extension-Side API

Extensions register preflight checks through the extension host, similar to how they register service targets or event handlers:

host := azdext.NewExtensionHost(azdClient).
    WithPreflightCheck("agent-manifest-validation", func(ctx context.Context, args *azdext.PreflightCheckArgs) (*azdext.PreflightCheckResult, error) {
        // Access the Bicep snapshot resources
        for _, res := range args.SnapshotResources {
            if strings.EqualFold(res.Type, "Microsoft.MachineLearningServices/workspaces") {
                // Validate extension-specific requirements
                if !hasRequiredConfig(res) {
                    return &azdext.PreflightCheckResult{
                        Severity:     azdext.PreflightWarning,
                        DiagnosticID: "azure.ai.agents/missing-workspace-config",
                        Message:      "AI workspace is missing required identity configuration.",
                    }, nil
                }
            }
        }
        return nil, nil // all good
    })

gRPC Contract

New proto messages for the preflight request/response:

message PreflightCheckRequest {
  // Deployment scope information
  string subscription_id = 1;
  string resource_group = 2;
  string location = 3;

  // Bicep snapshot: fully resolved predicted resources
  bytes snapshot_resources_json = 4;

  // Derived resource properties
  ResourceProperties properties = 5;
}

message PreflightCheckResponse {
  // nil/empty = check passed
  repeated PreflightCheckResultItem results = 1;
}

message PreflightCheckResultItem {
  string diagnostic_id = 1;     // e.g. "azure.ai.agents/missing-config"
  PreflightSeverity severity = 2; // WARNING or ERROR
  string message = 3;
}

enum PreflightSeverity {
  PREFLIGHT_WARNING = 0;
  PREFLIGHT_ERROR = 1;
}

message ResourceProperties {
  bool has_role_assignments = 1;
  // Extensible for future derived properties
}

Core-Side Integration

In BicepProvider.validatePreflight():

1. Discover extensions that declare the preflight-checks capability
2. Send the PreflightCheckRequest with snapshot resources and deployment scope to each extension
3. Run extension checks concurrently with core checks (with a timeout)
4. Merge extension PreflightCheckResultItems into the PreflightReport alongside core results
5. Apply the same UX flow: warnings prompt, errors block

azd provision
  ├── Compile Bicep module → ARM template + parameters
  ├── ► Local preflight validation
  │     ├── Generate Bicep snapshot
  │     ├── Analyze resources (derive properties)
  │     ├── Run core checks (e.g., role_assignment_permissions)
  │     ├── Run extension checks (concurrent, gRPC)    ← NEW
  │     │     ├── Extension A: preflight check results
  │     │     └── Extension B: preflight check results
  │     └── Merge all results into PreflightReport
  │           ├── Warnings → prompt user
  │           └── Errors → abort
  ├── Server-side preflight (Azure ValidatePreflight API)
  └── Deploy

Example: AI Agents Extension

The azure.ai.agents extension could use this to validate:

• Agent manifest references match predicted resources in the Bicep snapshot
• Required identity/RBAC configuration is present for AI workspaces
• Required dependent services (e.g., Azure OpenAI, Bing Search) are included in the deployment

Instead of failing during deploy or relying on init-time checks, these validations would surface as warnings before the user commits to a potentially long provisioning operation.

Relationship to #7053

This proposal builds directly on the preflight infrastructure established in #7053:

• The PreflightCheck / PreflightCheckFn / validationContext pattern is the foundation
• The PreflightReport UX component handles the merged output
• The warning/error/abort flow and exit code behavior remain unchanged
• No breaking changes to existing preflight behavior — this is purely additive

The key change is extending the check registration from internal-only (localPreflight.AddCheck()) to include extension-contributed checks via gRPC.

[wbreza]

Design Review: Issue #7832 — Validation Framework Extensibility

Assessment: The current proposal solves a narrow, urgent problem (Bicep preflight validation) but lacks extensibility for the broader validation needs across the ACD lifecycle.

Core Issues with Current Design

1. Single-Phase Architecture
   Only covers pre-provisioning checks. Ignores init, pre-deployment, deployment, and post-deployment phases.

2. Infrastructure-Specific Coupling
   Tightly bound to Bicep. Terraform, ARM templates, and custom deployments get no story.

3. No Streaming Feedback Model
   Results only reported at end, not streamed back with progress/context.

4. Missing Use Cases

   • Extension authors pre-provision quota checks ✓ Covered
   • Pre-deployment test suite execution ✗ Not covered
   • Post-deployment smoke tests ✗ Not covered
   • Infrastructure-agnostic validation ✗ Not covered

What a Robust Solution Needs

Multi-Phase Validation Framework

• Registration points across all ACD lifecycle phases (init → provision → deploy → post-deploy)
• Consistent gRPC streaming contract
• Phase-aware context (what data is available at each phase)

Extensible Streaming Model

• Validations stream progress back to core during execution
• Unified UX (spinner state, warnings, errors, recovery suggestions)
• Cancellation and timeout support

Scope Clarity

• Explicit boundaries for each phase (what belongs where)
• Clear data access per phase (e.g., pre-provision gets Bicep snapshot; post-deploy gets deployed resource state)

Future-Proof Versioning

• Extension validation contracts evolve without breaking compatibility
• Clear deprecation path

Recommended Path Forward

1. Reframe the proposal — Position Bicep pre-provision checks as one instance of a multi-phase validation framework, not the full framework
2. Define phase boundaries — Map which validations belong in init, pre-provision, pre-deploy, post-deploy with concrete examples
3. Specify streaming contract — Design the gRPC shape and callback model once, reuse across all phases
4. Prototype extensibility — Validate that pre-deploy test suites and post-deploy smoke tests fit the model without modification

The current design is a band-aid that will require rework when these additional use cases emerge. A more general framework up-front prevents later refactoring and enables extension authors to build to a stable contract.

[jongio]

My read: a new preflight-checks extension capability that lets extensions register gRPC-invoked checks during BicepProvider.validatePreflight(). They receive the Bicep snapshot + deployment scope and return severity-tagged findings that flow into the existing PreflightReport. Bicep-only, additive on top of #7053.

The UX unification (warnings prompt, errors block, same PreflightReport as core) is the right instinct. The part worth pushing on is whether locking in a single-phase Bicep-shaped contract now is the right move given where @wbreza is pointing.

@wbreza's framing is right - the question is how far to take it

Three ways to land in roughly that direction, ordered by up-front cost:

Option 1 - Enrich the existing event model. Extend preprovision (and later predeploy / postdeploy / postprovision) with a structured findings response (severity + diagnostic id + message) and pass the snapshot in event data. No new capability surface. Same hook model extension authors already know. Trade-off: conflates side-effect events ("do a thing") with validation ("inspect and report"), and changing event return semantics is a contract bump for every existing handler.

Option 2 - Phase-aware validation capability (recommended starting point). Single capability. Handler registration takes a phase enum (pre_provision, pre_deploy, post_deploy, init). Phase-typed context: pre-provision gets a normalized resource graph (not Bicep-snapshot bytes), pre-deploy gets the deployment plan, post-deploy gets actual deployed resource state. Typed Finding[] response. Implementation only wires pre-provision in v1 - other phases are additive, not breaking. This gets to the multi-phase framework @wbreza described while keeping initial scope manageable.

Option 3 - Full lifecycle validation framework with streaming. Option 2 plus streaming progress, cancellation, and post-deploy smoke tests as first-class. The destination @wbreza laid out, taken end-to-end. Highest up-front cost, but extension authors build to a stable contract from day one and there's no v2 conversation to have.

Reasonable path: ship Option 2 now with the proto designed so Option 3's streaming + extra phases drop in additively. The capability name (validation not preflight-checks) and the phase-aware context shape are what need to be right today - the wire-up scope can stay narrow.

Contract issues to fix under any option

These bite regardless of which path is chosen:

• Drop snapshot_resources_json bytes. Define the resource shape in proto (or, ideally, a normalized infra-agnostic ResourceGraph so Terraform/ARM/custom slot in later). Raw JSON forces every extension to reimplement parsing and silently breaks when the snapshot format moves. Core's in-process checks already get a typed []armTemplateResource (per cli/azd/docs/design/local-preflight-validation.md) - extensions deserve at least the same.
• Drop ResourceProperties { has_role_assignments bool }. A bag of named booleans doesn't extend - it just grows. Let extensions derive their own views from the resource list, or make this an explicit core-owned struct not a shared extension surface.
• Specify failure semantics. Per-extension timeout? Fail-open or fail-closed on hang/crash/malformed response? Per-extension disable (today provision.preflight off is all-or-nothing)? A misbehaving third-party extension shouldn't be able to block every azd provision in a project.
• Namespace diagnostic ids + define the telemetry/PII contract for extension-authored Message strings. Core owns its diagnostic text today; third-party extensions emitting arbitrary content into the same telemetry stream is a different risk profile worth calling out.
• Resolve the single-result (*PreflightCheckResult) vs multi-result (repeated PreflightCheckResultItem) asymmetry between the in-process API and the gRPC surface. Pick one.
• CI / --no-prompt behavior for extension-emitted warnings needs to be in the spec.

[vhvb1989]

Summary & Decision

Thanks @wbreza and @jongio for the thorough feedback. After reviewing, we are closing this proposal in favor of a broader design.

Why not expose only Bicep preflight to extensions?

Concern	Detail
Single-phase lock-in	Only covers pre-provisioning. Extensions needing pre-deploy, post-deploy, or init-time validation would require a separate, incompatible contract later.
Bicep-specific coupling	The contract is shaped around Bicep snapshots (snapshot_resources_json bytes). Terraform, ARM, and custom providers get no story, forcing a redesign when they need validation.
Raw JSON fragility	Passing opaque JSON bytes forces every extension to reimplement parsing and silently breaks when the snapshot format changes. Core already has typed []armTemplateResource internally — extensions deserve at least the same.
Non-extensible surface	ResourceProperties { has_role_assignments bool } is a bag of booleans that only grows. Extensions should derive views from a typed resource graph instead.
Missing failure semantics	No per-extension timeout, no fail-open/fail-closed policy, no per-extension disable. A misbehaving third-party extension could block every azd provision.
Contract asymmetry	Single-result (*PreflightCheckResult) in-process vs multi-result (repeated PreflightCheckResultItem) over gRPC — inconsistent and confusing for extension authors.
No streaming or cancellation	Results only reported at the end, with no progress feedback or cancellation support.

Path forward

The right approach is a phase-aware validation capability (Option 2 from @jongio) where:

• The capability name is validation, not preflight-checks
• Handlers register with a phase enum (pre_provision, pre_deploy, post_deploy, init)
• Context is phase-typed (e.g., pre-provision gets a normalized resource graph, post-deploy gets deployed resource state)
• The proto is designed so streaming and additional phases can be added without breaking changes
• Implementation scope stays narrow in v1 (pre-provision only), but the contract shape supports the full lifecycle

Closing in favor of this broader design direction.