Automated Pattern Adherence Tracking System

[humancompanion-usds]

Summary

Implements automated tracking of which VA.gov forms use which codified design patterns. This addresses issue #5446.

Status: ✅ Implementation complete - includes core script, workflow automation, tests, and documentation

What's Implemented

Core Script (scripts/collect-pattern-adherence.js)

• ✅ Pattern metadata parser - extracts code-links from pattern frontmatter
• ✅ Pattern discovery - recursively finds all pattern files
• ✅ Product directory integration - fetches form list via GitHub API
• ✅ Import analysis - searches vets-website for pattern usage via GitHub Code Search
• ✅ Pattern-to-forms mapping - cross-references imports with forms
• ✅ Report generation - outputs JSON + markdown reports

GitHub Actions Workflow (.github/workflows/pattern-adherence.yml)

• ✅ Weekly scheduled run (Mondays 6 AM UTC)
• ✅ Manual trigger support
• ✅ Sparse checkout of vets-website for performance
• ✅ Auto-creates PRs with updated data

Generated Reports

Analysis Results (9 codified patterns across 55 forms):

Pattern	Compliance	Forms Using
Phone numbers	31%	17/55
Addresses	27%	15/55
Dates	25%	14/55
SSN/VA file number	25%	14/55
Email address	24%	13/55
Signature	22%	12/55
Names	16%	9/55
Relationship	5%	3/55
Confirmation	0%	0/55 ⚠️

Output Files:

• src/assets/data/metrics/pattern-adherence.json (27KB)
• src/_data/metrics/pattern-adherence.json (27KB - Jekyll data)
• src/assets/data/metrics/pattern-adherence-report.md (10KB)

Documentation

• Implementation plan: docs/plans/2026-01-09-pattern-adherence-tracking.md
• Scripts README: scripts/README.md
• Tracking issue: Automated Pattern Adherence Tracking System #5446

Intentional behavior changes

metrics-dashboard.yml: pull_request trigger guarded

The pull_request trigger on the metrics-dashboard workflow previously caused infinite loops when the workflow committed updated metrics data back to the PR branch. A branches-ignore guard now prevents runs on automated metrics-update-* and pattern-adherence-update-* branches. A comment in the YAML explains this decision.

Notable Findings

⚠️ Confirmation pattern shows 0% usage - The "Keep a record of submitted information" pattern links to a README rather than a component file, so the import-based detection does not find matches. Future enhancement could add a dedicated detection strategy for this pattern.

Testing

• ✅ Jest unit tests: __tests__/collect-pattern-adherence.test.js (23 tests covering parsePatternMetadata, isFormProduct, generateMarkdownReport)
• ✅ Integration tests: scripts/integration-test-pattern-parser.js (requires gh CLI)
• ✅ Manual test run successful with local vets-website

Security

✅ All security issues addressed:

• Uses execFileSync (no shell) with argument validation
• Input validation on all external data
• Timeouts on all API calls
• Error handling prevents script crashes
  Open Preview Environment

[Copilot]

Pull request overview

This PR implements an automated pattern adherence tracking system to monitor which VA.gov forms use codified design patterns. The core implementation is complete with a Node.js script that parses pattern metadata, fetches the product directory, analyzes code imports via GitHub API, and generates compliance reports.

Key Changes:

• Pattern metadata parser with YAML frontmatter extraction
• GitHub API integration for product directory and code search
• Automated report generation (JSON and Markdown formats)
• Test suite for validation

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/collect-pattern-adherence.js	Core tracking script with pattern discovery, import analysis, and report generation
scripts/test-pattern-parser.js	Test suite validating parser, discovery, and product directory integration
src/assets/data/metrics/pattern-adherence.json	Generated JSON report with pattern-to-forms mapping data
src/_data/metrics/pattern-adherence.json	Jekyll data file (duplicate of above for template access)
src/assets/data/metrics/pattern-adherence-report.md	Human-readable markdown report with compliance matrix
docs/plans/2026-01-09-pattern-adherence-tracking.md	Detailed implementation plan with task breakdown and testing guidance

[humancompanion-usds]

Copilot Review Response

Thanks for the thorough review! All issues have been addressed:

✅ Fixed - Comments #2 & #3: Inconsistent path formatting

Issue: Product directory data has inconsistent leading slashes ("/src/applications/..." vs "src/applications/..."), causing regex matching failures.

Fix (commit 0469ec6): Updated all path extraction regex patterns to handle optional leading slashes:

// Before
/^([^\/]+)/  or  /^src\/applications\//

// After  
/^\/?([^\/]+)/  and  /^\/?src\/applications\//

Now correctly extracts app names from both formats.

ℹ️ Obsolete - Comment #1: filenameWithoutExt validation

This code no longer exists. The findImporters() function was completely rewritten to use PATTERN_EXPORT_MAPPING instead of filename-based searches (commits a7118ed, e95fb36).

The new approach maps pattern files to their exported names and searches for actual export usage via regex matching - no search query construction from filenames.

✅ Fixed - Comment #4: Missing scripts/README.md

Added (commit 8fa03c4): Comprehensive documentation at scripts/README.md covering:

• Script purpose, usage, and CLI options
• Pattern detection strategies (standard + special cases)
• Pattern export mapping and regex matching
• Performance comparison (local vs GitHub API)
• GitHub Actions workflow integration
• Troubleshooting guide and known limitations

Documentation provides complete context for future maintainers.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

[Copilot]

When code-link exists but the /blob/(main|master)/... path can’t be extracted, this returns hasCodeLink: true with codeFile: null. That pattern will still be counted as “codified” (via hasCodeLink) but skipped during import analysis, inflating total_patterns and producing misleading 0%-usage entries. Consider treating an unparseable code-link as not codified (set hasCodeLink: false) or exclude such patterns from codifiedPatterns.

Suggested change

	hasCodeLink: true,
	hasCodeLink: false,

[Copilot]

buildPatternAdherence processes all patterns in parallel, but each findImporters run scans thousands of files under src/applications. Running these scans concurrently can create significant I/O pressure (and longer runtimes / possible EMFILE errors) in CI. Consider limiting concurrency (e.g., a small worker pool) and/or reusing a single precomputed file list / content cache across patterns.

[Copilot]

The objects returned from getFormProducts() set fields to undefined when missing in the product directory, which causes those keys to be omitted in the generated JSON. This makes the forms schema inconsistent for consumers. Consider normalizing optional fields to null (e.g., platform_console_category: form.platform_console_category ?? null, github_product_label: form.github_product_label ?? null, etc.) so the output shape is stable.

Suggested change

	path_to_code: form.path_to_code,
	analytics_category: form.analytics_category,
	platform_console_category: form.platform_console_category,
	github_product_label: form.github_product_label
	path_to_code: form.path_to_code ?? null,
	analytics_category: form.analytics_category ?? null,
	platform_console_category: form.platform_console_category ?? null,
	github_product_label: form.github_product_label ?? null

[Copilot]

This workflow only runs on schedule and workflow_dispatch, but the job-level if references github.event.head_commit.*, which is not present for those event payloads. This can cause the expression evaluation to fail or behave unexpectedly, preventing the job from running. Consider removing this guard entirely (since there’s no push trigger) or rewriting it to use fields that exist for these events (e.g., github.actor, or gating on github.event_name == 'push').

Suggested change

	# Skip if last commit was made by GitHub Action
	if: |
	github.event.head_commit.author.email != 'action@github.com' &&
	!contains(github.event.head_commit.message, '📊 Update pattern adherence data')

[Copilot]

appId extraction here takes the first 2 path segments unconditionally when available (e.g., caregivers/config/form.js becomes caregivers/config). That won’t match product-directory path_to_code entries that point at the app root (e.g., src/applications/caregivers → caregivers), causing false negatives for single-segment apps. Consider normalizing importer IDs to the application root (e.g., use 2 segments only when the second segment is an actual app folder, not pages/config).

Suggested change

	// Extract app identifier from path using up to 2 segments
	// e.g., "simple-forms/20-10206/pages/..." -> "simple-forms/20-10206"
	const segments = filePath.replace(/^\//, '').split('/').filter(Boolean);
	const appId = segments.length >= 2
	? `${segments[0]}/${segments[1]}`
	: (segments[0] || null);
	// Extract app identifier from path, normalizing to application root
	// - For two-segment apps: "simple-forms/20-10206/pages/..." -> "simple-forms/20-10206"
	// - For single-segment apps: "caregivers/config/form.js" -> "caregivers"
	const segments = filePath.replace(/^\//, '').split('/').filter(Boolean);
	const [firstSegment, secondSegment] = segments;
	let appId = null;
	if (firstSegment) {
	if (!secondSegment) {
	// Single-segment app (e.g., "caregivers/...")
	appId = firstSegment;
	} else {
	// If the second segment is a non-root folder like "pages" or "config",
	// treat the app as a single-segment root. Otherwise, use the first two segments.
	const nonAppFolders = ['pages', 'config'];
	if (nonAppFolders.includes(secondSegment)) {
	appId = firstSegment;
	} else {
	appId = `${firstSegment}/${secondSegment}`;
	}
	}
	}

[Copilot]

Same as findImporters: for config paths like caregivers/config/form.js, this will record appId as caregivers/config instead of the app root caregivers, so Signature usage detection will miss single-segment apps. Consider extracting the application root consistently (e.g., first segment, or first two segments only when the second segment is not config/pages).

Suggested change

	// Extract app identifier from path using up to 2 segments
	const segments = filePath.replace(/^\//, '').split('/').filter(Boolean);
	const appId = segments.length >= 2
	// Extract app identifier from path, treating "config" and "pages" as structural
	const segments = filePath.replace(/^\//, '').split('/').filter(Boolean);
	const hasTwoSegments = segments.length >= 2;
	const secondIsStructural = hasTwoSegments && (segments[1] === 'config' || segments[1] === 'pages');
	const appId = hasTwoSegments && !secondIsStructural