Skip to content

Add validate command for custom training#7407

Merged
saanikaguptamicrosoft merged 27 commits into
Azure:foundry-training-devfrom
saanikaguptamicrosoft:saanika/validate
Apr 21, 2026
Merged

Add validate command for custom training#7407
saanikaguptamicrosoft merged 27 commits into
Azure:foundry-training-devfrom
saanikaguptamicrosoft:saanika/validate

Conversation

@saanikaguptamicrosoft

@saanikaguptamicrosoft saanikaguptamicrosoft commented Mar 31, 2026

Copy link
Copy Markdown
Member

Notes

Added validation for-

  1. command field is required
  2. environment field is required
  3. compute field is required
  4. code if present, it should not start with git:// or git+
  5. The local path mentioned in code and inputs.<key>.path are existent
  6. Placeholder mapping is correct. Different cases:
    • The command field contains placeholder in the form of ${{inputs.xxx}}, the xxx should be present as key in inputs.
      • For inputs -> if key is present but its value is {} or None, throw error.
    • The command field contains placeholder in the form ${{outputs.xxx}}, the xxx may or may not be present as key in inputs. If it's not present, means user wants to go with default output values. If it's present means user wants to override output configs.
      • For outputs -> if key is present but its value is {} or None, show warning. Because user might have forgotten to override the value.
    • If ${{inputs.xxx}} is wrapped inside [ ], it's optional, so we don't validate its presence in inputs key.
    • Single bracket i.e. {inputs.xxx} or {outputs.xxx} or ${inputs.xxx} or ${outputs.xxx} should throw error

Added unit tests.

Testing

  • Local testing
image image image image image
  • Build ~ Successful
azd x build
  • UTs ~ Successful
go test ./internal/utils/ -v -count=1

@saanikaguptamicrosoft saanikaguptamicrosoft changed the title Saanika/validate Add validate command for custom training Mar 31, 2026

@jongio jongio left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review - #7407

Add validate command for custom training by @saanikaguptamicrosoft

Summary

What: Adds an offline job validate command that checks a YAML job definition for required fields, invalid git paths, local path existence, placeholder correctness, and empty input/output definitions - collecting all findings before reporting instead of failing on the first error.

Why: Lets users catch YAML mistakes locally before submitting to the backend, saving a round-trip.

Assessment: The validation logic is solid and well-structured, but the command can't actually run offline due to the parent command's PersistentPreRunE hook. This should be addressed before merge. A few edge cases in the validator produce confusing output (duplicate errors on git paths, silent pass on ${inputs.xxx} typos).

Findings

Category High Medium Low
Design 1 0 0
Logic 0 2 0
Error Handling 0 1 0
Consistency 0 1 1
Tests 0 0 1
Total 1 4 2

Key Findings

  1. [HIGH] Design: Parent job command's PersistentPreRunE requires Azure env setup, preventing offline use
  2. [MEDIUM] Logic: git:// code paths produce duplicate errors (git-not-supported + local-path-not-found)
  3. [MEDIUM] Logic: ${inputs.xxx} typos silently bypass single-brace detection
  4. [MEDIUM] Error Handling: os.Stat permission/IO errors silently ignored in path checks
  5. [MEDIUM] Consistency: Missing -f shorthand for --file flag (inconsistent with job submit)
  6. [LOW] Consistency: Missing Long description on cobra command (all sibling commands have one)
  7. [LOW] Tests: No test for the git:// double-error scenario (assert only 1 finding, not 2)

Test Coverage Estimate

  • Well covered: required fields, git path detection, local path checks, placeholder mapping, single-brace detection, empty definitions, multiline commands
  • Missing: command-layer test (cobra handler), ${inputs.xxx} edge case, git:// double-error assertion, os.Stat permission errors, empty/zero-byte YAML input

What's Done Well

  • Collects all findings instead of failing on the first error - much better UX than the existing ValidateJobDefinition
  • Good separation between the command handler and validation logic
  • Optional [...] block handling for input placeholders is a nice touch
  • Test file has clear YAML comments showing what each test case maps to

5 inline comments below.

Comment thread cli/azd/extensions/azure.ai.customtraining/internal/cmd/job.go

@jongio jongio left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous 5 findings addressed. New issues found:

  • job_validator_test.go:223 - single-brace test assertions search for "single-brace '...'" but the actual message is "Incorrect placeholder format...". Three assertions will fail (223, 226, 236)
  • job_validator.go:201 - duplicate findings when the same placeholder key appears twice in command
  • job_validator.go:106 - fmt.Sprintf with no format verbs (staticcheck SA1006)

Comment thread cli/azd/extensions/azure.ai.customtraining/internal/utils/job_validator_test.go Outdated
Comment thread cli/azd/extensions/azure.ai.customtraining/internal/utils/job_validator.go Outdated

@jongio jongio left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All 8 findings from my previous reviews are addressed - nice work. A few new items from this pass, mostly edge cases in the placeholder validation. Nothing blocking.

@jongio jongio left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All 8+ findings from my previous rounds are addressed - nice work.

One non-blocking edge case for a follow-up: malformed ${{inputs.data} (correct opening but only one closing brace) slips through both regex checks silently. placeholderRegex needs }} so it doesn't match, and singleBraceRegex matches but the ${ guard skips it. The backend would catch this, but it'd be good to detect offline too.

// Skip if this is already part of a ${{...}} (preceded by "${")
if start >= 2 && command[start-2:start] == "${" {
continue
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: ${{inputs.data} (missing one closing brace) bypasses both checks - placeholderRegex needs }}, and singleBraceRegex sees the ${ prefix and skips. Worth adding a malformed-placeholder check in a follow-up.

@saanikaguptamicrosoft
saanikaguptamicrosoft merged commit 35430ef into Azure:foundry-training-dev Apr 21, 2026
1 of 2 checks passed
saanikaguptamicrosoft added a commit that referenced this pull request May 21, 2026
* Custom training (#7125)

* adding design detaiils for command job CLI

* adding more details

* adding dedup details

* adding api details

* adding execution plan

* adding draft version of custom training commands

* feat: add job name auto-generation, fix endpoint URL, rename job get to show

- Make job name optional in YAML; auto-generate {adj}_{noun}_{suffix} (matching AML SDK)
- Fix buildProjectEndpoint to use services.ai.azure.com (not cognitiveservices.azure.com)
- Rename 'job get' to 'job show' to match models/finetune extensions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Custom training (#7180)

* adding design detaiils for command job CLI

* adding more details

* adding dedup details

* adding api details

* adding execution plan

* adding draft version of custom training commands

* integrating with API

* feat: enhance job list with pagination, filters, and systemData support (#7203)

- Add --skip-token flag for pagination with next-page UX message
- Add --tag and --properties flags for server-side filtering
- Add --include-archived flag for listViewType control
- Add SystemData (createdBy, createdAt) to job list output
- Update doDataPlane() to support variadic query params

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: add CODEOWNERS for azure.ai.customtraining extension (#7204)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: rename job create to submit and add resolver layer for compute, code, and input resolution (#7205)

- Rename job create command to job submit for consistency with finetune extension
- Add resolver interfaces: ComputeResolver, CodeResolver, InputResolver
- Add JobResolver orchestrator that resolves all references in JobDefinition
- Wire resolver into submit flow before buildJobResource()
- Stub implementations guide users to provide full ARM IDs / remote URIs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add artifact resolution for code and inputs in job create (#7153)

* Implement cancel job command for custom training (#7272)

* Impelement delete job command for custom training (#7273)

* Custom training clean (#7454)

* Custom training (#7180)

* adding design detaiils for command job CLI

* adding more details

* adding dedup details

* adding api details

* adding execution plan

* adding draft version of custom training commands

* integrating with API

* adding -e -s override

* fixing asset resolution

* custom training: enhance job show, fix asset resolution, add full resource config support

- Enhanced job show with rich output: run history, metrics, artifacts, timing, compute info
- Added client APIs for run history, metrics, and artifacts endpoints
- Fixed dataset version field: json:dataType -> json:type
- Fixed input/output mode mapping: ro_mount -> ReadOnlyMount, rw_mount -> ReadWriteMount
- Added full resource config support: instanceType, shmSize, dockerArgs, properties
- Added ResourceDefinition YAML struct with AISuperComputer properties pass-through
- Backward compatible: flat instance_count still works

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* custom training: add spinner progress to job show command

Show animated spinner with progress text while fetching job details.
Updates text as each parallel fetch (run history, metrics, artifacts)
completes, showing remaining items until all data is loaded.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Implement download command for custom training (#7453)

* Add validate command for custom training (#7407)

* Rename EnvironmentID to EnvironmentImageReference (#7891)

* Revert "Implement download command for custom training (#7453)" (#7892)

This reverts commit 5216202.

* Add userAssignedIdentityId support for command jobs (#7927)

* Implement stream command for training job (#7939)

* Add support for experiment name (#7961)

* Implement connect-ssh to job node (#7964)

* Add support for gpuCount for partial SKU scenario (#8067)

* Share offline validation between job validate and job submit commands (#8068)

* fix(azure.ai.customtraining): honor --subscription and --project-endpoint flags over stored env values (#8093)

* Implement download command for training job (#8029)

* Rename azure.ai.customtraining to azure.ai.training (#8106)

* Implement show services command (#8121)

* Add validations for UAMI requirement (#8122)

* Add template flag in init command (#8123)

* Bump armcognitiveservices SDK from v1.8.0 to v2.0.0 to fix NetworkInjections unmarshal error

* Bump go directive to 1.26.1 to align with repo standard and other extensions

* Use Subscription.UserTenantId for credential tenant to support guest/multi-tenant users

* Pin azd module to semver v1.24.3 instead of pseudo-version for stable dependency

* Add UTs

* Consolidate doUpload and doUploadWithTag into single method with optional tags parameter

* Use PromptSubscriptionResource for interactive Foundry project selection in init

* Surface azcopy scanner errors so truncated stdout doesn't mask upload failures

* Add schema header and requiredAzdVersion >=1.25.1 to extension.yaml

* Cap error and service-instance response body reads to prevent unbounded memory use

* Add retry policy (429/502/503/504 + net errors) to Foundry data plane client

* Use azd Confirm prompt and honor --no-prompt in job delete

* Escape user-supplied IDs in client URL paths

* Fix JSON tag mismatch for DataType

* Route client debug prints to stderr to keep stdout JSON parseable

* Adopt azdext.NewExtensionRootCommand and remove reserved-flag conflicts

* chore(ext/azure.ai.training): add CHANGELOG, README, cspell, golangci config and CI lint + release pipeline

* chore(ext/azure.ai.training): extend cspell dictionary to fix CI lint

* chore(ext/azure.ai.training): apply go fix modernization (interface{}→any, CutPrefix, drop loopvar capture)

* chore(ext/azure.ai.training): fix golangci-lint issues

* fix(ext/azure.ai.training): surface azcopy failure diagnostics

* refactor(ext/azure.ai.training): consolidate ServiceEndpoint helper in internal/utils

* test(ext/azure.ai.training): add hash + upload_service unit tests

* refactor(ext/azure.ai.training): rename job_get.go to job_show.go

* chore(ext/azure.ai.training): rename Design/ to design/ and link from README

* chore(ext/azure.ai.training): address PR feedback (ssh ProxyCommand % escape, design/ rename)

* Fix cspell error

* fix: apply go fix modernizations

* Move to APIM APIs and update API paths as per latest Typespec

* Temp: Print API request response for testing

* Fix API paths for metrics

* fix(ai.training): use delete operation result url to surface accurate job delete outcome

* fix(ai.training): fan out artifact contentinfo per unique root folder for job download

* Revert "Temp: Print API request response for testing"

This reverts commit 6d7f2d7.

* Support attaching remaining service types (jupyter_lab, tensor_board, vs_code, custom) to job

* Support distribution type (pytorch, tensorflow, mpi, ray) in job YAML

* Add polling for job deletion | Add a --no-wait flag

* Add polling for job cancel | Add a --no-wait flag | Refactor to share same poller as delete

* fix: surface all per-root errors in artifact contentinfo fan-out

* fix: redact query string from --debug URL logs to avoid leaking SAS tokens

* fix: harden redactSAS with SAS-marker fallback for URLs without '?'

* fix: validate ray distribution port and dashboard_port ranges in job validator

* fix: make root-folder fan-out semaphore acquire context-aware in job download

* fix: write full azcopy diagnostics to side-file when terminal output is truncated

* fix: resolve cspell (skoid/sktid) and gosec G104 (LRO body close) lint failures

* fix: redact request body for /credentials endpoints under --debug

---------

Co-authored-by: Amit Chauhan <70937115+achauhan-scc@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI pushed a commit that referenced this pull request May 21, 2026
* Custom training (#7125)

* adding design detaiils for command job CLI

* adding more details

* adding dedup details

* adding api details

* adding execution plan

* adding draft version of custom training commands

* feat: add job name auto-generation, fix endpoint URL, rename job get to show

- Make job name optional in YAML; auto-generate {adj}_{noun}_{suffix} (matching AML SDK)
- Fix buildProjectEndpoint to use services.ai.azure.com (not cognitiveservices.azure.com)
- Rename 'job get' to 'job show' to match models/finetune extensions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Custom training (#7180)

* adding design detaiils for command job CLI

* adding more details

* adding dedup details

* adding api details

* adding execution plan

* adding draft version of custom training commands

* integrating with API

* feat: enhance job list with pagination, filters, and systemData support (#7203)

- Add --skip-token flag for pagination with next-page UX message
- Add --tag and --properties flags for server-side filtering
- Add --include-archived flag for listViewType control
- Add SystemData (createdBy, createdAt) to job list output
- Update doDataPlane() to support variadic query params

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: add CODEOWNERS for azure.ai.customtraining extension (#7204)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: rename job create to submit and add resolver layer for compute, code, and input resolution (#7205)

- Rename job create command to job submit for consistency with finetune extension
- Add resolver interfaces: ComputeResolver, CodeResolver, InputResolver
- Add JobResolver orchestrator that resolves all references in JobDefinition
- Wire resolver into submit flow before buildJobResource()
- Stub implementations guide users to provide full ARM IDs / remote URIs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add artifact resolution for code and inputs in job create (#7153)

* Implement cancel job command for custom training (#7272)

* Impelement delete job command for custom training (#7273)

* Custom training clean (#7454)

* Custom training (#7180)

* adding design detaiils for command job CLI

* adding more details

* adding dedup details

* adding api details

* adding execution plan

* adding draft version of custom training commands

* integrating with API

* adding -e -s override

* fixing asset resolution

* custom training: enhance job show, fix asset resolution, add full resource config support

- Enhanced job show with rich output: run history, metrics, artifacts, timing, compute info
- Added client APIs for run history, metrics, and artifacts endpoints
- Fixed dataset version field: json:dataType -> json:type
- Fixed input/output mode mapping: ro_mount -> ReadOnlyMount, rw_mount -> ReadWriteMount
- Added full resource config support: instanceType, shmSize, dockerArgs, properties
- Added ResourceDefinition YAML struct with AISuperComputer properties pass-through
- Backward compatible: flat instance_count still works

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* custom training: add spinner progress to job show command

Show animated spinner with progress text while fetching job details.
Updates text as each parallel fetch (run history, metrics, artifacts)
completes, showing remaining items until all data is loaded.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Implement download command for custom training (#7453)

* Add validate command for custom training (#7407)

* Rename EnvironmentID to EnvironmentImageReference (#7891)

* Revert "Implement download command for custom training (#7453)" (#7892)

This reverts commit 5216202.

* Add userAssignedIdentityId support for command jobs (#7927)

* Implement stream command for training job (#7939)

* Add support for experiment name (#7961)

* Implement connect-ssh to job node (#7964)

* Add support for gpuCount for partial SKU scenario (#8067)

* Share offline validation between job validate and job submit commands (#8068)

* fix(azure.ai.customtraining): honor --subscription and --project-endpoint flags over stored env values (#8093)

* Implement download command for training job (#8029)

* Rename azure.ai.customtraining to azure.ai.training (#8106)

* Implement show services command (#8121)

* Add validations for UAMI requirement (#8122)

* Add template flag in init command (#8123)

* Bump armcognitiveservices SDK from v1.8.0 to v2.0.0 to fix NetworkInjections unmarshal error

* Bump go directive to 1.26.1 to align with repo standard and other extensions

* Use Subscription.UserTenantId for credential tenant to support guest/multi-tenant users

* Pin azd module to semver v1.24.3 instead of pseudo-version for stable dependency

* Add UTs

* Consolidate doUpload and doUploadWithTag into single method with optional tags parameter

* Use PromptSubscriptionResource for interactive Foundry project selection in init

* Surface azcopy scanner errors so truncated stdout doesn't mask upload failures

* Add schema header and requiredAzdVersion >=1.25.1 to extension.yaml

* Cap error and service-instance response body reads to prevent unbounded memory use

* Add retry policy (429/502/503/504 + net errors) to Foundry data plane client

* Use azd Confirm prompt and honor --no-prompt in job delete

* Escape user-supplied IDs in client URL paths

* Fix JSON tag mismatch for DataType

* Route client debug prints to stderr to keep stdout JSON parseable

* Adopt azdext.NewExtensionRootCommand and remove reserved-flag conflicts

* chore(ext/azure.ai.training): add CHANGELOG, README, cspell, golangci config and CI lint + release pipeline

* chore(ext/azure.ai.training): extend cspell dictionary to fix CI lint

* chore(ext/azure.ai.training): apply go fix modernization (interface{}→any, CutPrefix, drop loopvar capture)

* chore(ext/azure.ai.training): fix golangci-lint issues

* fix(ext/azure.ai.training): surface azcopy failure diagnostics

* refactor(ext/azure.ai.training): consolidate ServiceEndpoint helper in internal/utils

* test(ext/azure.ai.training): add hash + upload_service unit tests

* refactor(ext/azure.ai.training): rename job_get.go to job_show.go

* chore(ext/azure.ai.training): rename Design/ to design/ and link from README

* chore(ext/azure.ai.training): address PR feedback (ssh ProxyCommand % escape, design/ rename)

* Fix cspell error

* fix: apply go fix modernizations

* Move to APIM APIs and update API paths as per latest Typespec

* Temp: Print API request response for testing

* Fix API paths for metrics

* fix(ai.training): use delete operation result url to surface accurate job delete outcome

* fix(ai.training): fan out artifact contentinfo per unique root folder for job download

* Revert "Temp: Print API request response for testing"

This reverts commit 6d7f2d7.

* Support attaching remaining service types (jupyter_lab, tensor_board, vs_code, custom) to job

* Support distribution type (pytorch, tensorflow, mpi, ray) in job YAML

* Add polling for job deletion | Add a --no-wait flag

* Add polling for job cancel | Add a --no-wait flag | Refactor to share same poller as delete

* fix: surface all per-root errors in artifact contentinfo fan-out

* fix: redact query string from --debug URL logs to avoid leaking SAS tokens

* fix: harden redactSAS with SAS-marker fallback for URLs without '?'

* fix: validate ray distribution port and dashboard_port ranges in job validator

* fix: make root-folder fan-out semaphore acquire context-aware in job download

* fix: write full azcopy diagnostics to side-file when terminal output is truncated

* fix: resolve cspell (skoid/sktid) and gosec G104 (LRO body close) lint failures

* fix: redact request body for /credentials endpoints under --debug

---------

Co-authored-by: Amit Chauhan <70937115+achauhan-scc@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: therealjohn <1501196+therealjohn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants