[Schema Consistency] Workflow-First Analysis: reaction Field Missing, Polymorphic Pattern Gaps #5482
Closed
Replies: 2 comments 1 reply
-
|
/plan |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
⚓ Avast! This discussion be marked as outdated by Schema Consistency Checker. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
This analysis took a radically different approach by starting with 150 production workflows and reverse-engineering their patterns to validate against schema, parser, and documentation. This "workflow-first" methodology discovered critical gaps where implementation and documentation exist but schema representation is missing or incomplete.
Summary
Critical Issues Requiring Immediate Action
Critical Issues
1.⚠️
reactionField Missing from SchemaSeverity: Critical - Feature is fully implemented, tested, and documented but absent from JSON schema
Evidence:
docs/src/content/docs/reference/frontmatter.md:34listsreactionas trigger optiondocs/src/content/docs/reference/frontmatter-full.md:578-589provides complete specificationreactionfunctionalitypkg/workflow/activation_checkout_test.go:48-59pkg/workflow/comment_env_vars_conditional_test.go:22-57jq '.properties | has("reaction")' main_workflow_schema.jsonreturnsfalseSupported Values (from documentation):
Valid Reactions:
+1,-1,laugh,confused,heart,hooray,rocket,eyes,noneImpact:
reactionfieldRecommended Fix:
File to Update:
pkg/parser/schemas/main_workflow_schema.json2.
cache-memoryPolymorphic Patterns Incompletely CapturedSeverity: Critical - Schema may not capture full complexity of production usage patterns
Pattern A - Simple Boolean (5+ workflows):
audit-workflows.md,ci-doctor.md,cli-version-checker.mdPattern B - Array of Objects (10+ workflows):
daily-code-metrics.md,daily-copilot-token-report.md,developer-docs-consolidator.md${{ github.workflow }},${{ github.run_id }}Real-World Key Patterns Found:
cloclo-memory-${{ github.workflow }}-${{ github.run_id }}code-metrics-${{ github.workflow }}copilot-token-metrics-${{ github.workflow }}developer-docs-cachemcp-response-analysis-${{ github.workflow }}poem-memory-${{ github.workflow }}-${{ github.run_id }}quality-focus-${{ github.workflow }}schema-consistency-cache-${{ github.workflow }}stale-repos-analysis-${{ github.workflow }}-${{ github.run_id }}Schema Validation Needed:
booleanORarray of objectswithidandkeyfields?keyfield?idandkeyboth required in array pattern?Impact: If schema doesn't fully capture array pattern, external validators will reject 10+ production workflows.
Recommended Action: Audit
cache-memoryschema definition to ensure it captures:{ "oneOf": [ {"type": "boolean"}, { "type": "array", "items": { "type": "object", "properties": { "id": {"type": "string", "description": "Cache identifier"}, "key": {"type": "string", "description": "Cache key (supports GitHub Actions expressions)"} }, "required": ["id", "key"], "additionalProperties": false } } ] }3. Bash Command Allowlist Wildcard Patterns
Severity: Moderate-High - Undocumented pattern used in production
Found Pattern (5+ workflows):
Workflows Using This Pattern:
blog-auditor.mdclose-old-discussions.mdpoem-bot.mdQuestions Requiring Answers:
bash?*,**,?,[...])Current Schema Check: Verify
tools.bashschema supports:Impact: If wildcard syntax not documented, users may:
Recommended Action:
Moderate Issues and Documentation Gaps
Moderate Issues
4.
safe-outputs.messagesTemplate SystemFinding: Production workflows use sophisticated message customization with template variables
Example from
archie.md:Template Variables Discovered:
{workflow_name}- Workflow display name{run_url}- GitHub Actions run URL{event_type}- Triggering event type (issue, pull_request, etc.){status}- Workflow execution statusSchema Validation Needed:
messagesobject documented insafe-outputsschema?footer,run-started,run-success,run-failure?Impact: Powerful customization feature may be underdocumented. Users unaware of capability.
Recommendation:
messagesobject in schema with four required string properties5.
safe-outputs.close-older-discussionsCleanup PatternUsage Statistics: 33 workflows (22% of total) use this pattern
Common Combination:
Purpose: Automatically close older discussions in same category when creating new one. Prevents discussion proliferation in automated reporting workflows.
Related Pattern -
title-prefix: 62 workflows (41%) use title prefixesSchema Status: Likely in schema, but verify:
close-older-discussionsboolean documented?maxfield explained?Recommendation: Confirm schema completeness and add usage examples showing cleanup patterns.
6. GitHub
toolsetsEnum CompletenessFound 18+ Unique Toolset Combinations across 150 workflows:
Individual Toolsets:
default- Most common (appears in 40+ workflows)all- Used in 5+ workflowsrepos- Repository operationsissues- Issue managementpull_requests- PR operationsactions- Workflow/action queriesdiscussions- Discussion managementlabels- Label operationscode_security- Security scanningdependabot- Dependabot alertscontext- Contextual metadataExample Combinations:
Schema Audit Needed:
allspecial value (expands to all toolsets)?defaulta predefined collection (if so, what does it include)?Impact: If enum incomplete, valid workflows fail validation.
Recommendation:
all,default)Positive Findings and Best Practices
Positive Findings
✓ Excellent Schema Field Coverage
All 18 major fields discovered in workflows are properly in schema:
Validated Fields:
strict- Strict mode validation ✅tracker-id- Workflow tracking identifier ✅network- Network access control ✅imports- Shared configuration imports ✅sandbox- Sandbox runtime configuration ✅safe-inputs- Input validation ✅secret-masking- Secret redaction ✅concurrency- Concurrent execution control ✅features- Feature flags ✅roles- AI role configuration ✅post-steps- Post-execution steps ✅mcp-servers- MCP server configuration ✅source- Workflow source tracking ✅services- Container services ✅cache- Build cache configuration ✅name- Workflow display name ✅if- Conditional execution ✅jobs- Multi-job workflows ✅Impact: Core schema coverage is excellent. Issues are edge cases and nested configurations.
✓ Complex
command.eventsPattern Correctly CapturedThe sophisticated pattern from
archie.mdis properly supported in schema:Schema Structure: Uses
oneOfto support both string and object command formats:command: "name"(conflicts with comment events)command: {name: "archie", events: [...]}(allows event combinations)This hierarchical dependency pattern (Strategy-018 finding) is correctly captured.
✓ Network Configuration Well-Structured
Schema provides excellent examples for network access control:
Found in production workflows:
deep-report.md:[defaults, python, node]artifacts-summary.md:[defaults, node]Impact: Network security patterns well-documented and schema-aligned.
✓ Production Workflows are Conservative
Insight: Workflows copy proven patterns rather than exploring all schema features
Safe-Outputs Complexity:
issue-monster.md(12 ops),poem-bot.md(16 ops)Timeout Distribution:
Tool Usage:
github+bashmost common combinationcache-memorygrowing adoption (15% of workflows)playwrightspecialized use (docs testing, UI validation)Impact: Schema doesn't need to support every theoretical combination. Focus on patterns actually used in production.
Workflow-First Methodology Insights
Key Insights from Bottom-Up Analysis
Insight 1: Documentation Often Ahead of Schema
Pattern Observed: Features documented and implemented but schema lags behind
Example -
reactionField:frontmatter.md,frontmatter-full.md,command-triggers.mdWhy This Happens:
Impact: Schema becomes incomplete source of truth. External validators fail on valid workflows.
Recommendation: Establish process ensuring schema updates when documentation added.
Insight 2: Customization Concentrates in
safe-outputsMost Customization Found In:
messages- Custom status messages with templates (archie.md pattern)title-prefix- 62 workflows (41%) use prefixesclose-older-discussions- 33 workflows (22%) use cleanupfooter- Custom attribution and brandingWhy
safe-outputsis Customization Hub:Impact:
safe-outputsschema needs richest documentation and examples.Insight 3: Tool Configuration Patterns are Diverse
Three Tools with Polymorphic Patterns:
1.
cache-memory:cache-memory: true(simple)cache-memory: [{id: "x", key: "y"}](complex)timeout: 300(companion setting)2.
bash:bash: true(unrestricted)bash: ["date *", "echo *"](glob allowlist)3.
github.toolsets:[default]to[context, repos, code_security, pull_requests]Impact: Schema must capture polymorphic flexibility without sacrificing validation.
Insight 4: Workflow-First Reveals "Contract as Implemented"
Traditional Schema-First Analysis:
Workflow-First Analysis:
Complementary Approaches:
Result: Combined approaches provide complete consistency picture.
Insight 5: Import Patterns Show Modularity Success
High Import Count Workflows:
mcp-inspector.md: 16 imports (comprehensive analysis)scout.md: 8 imports (research workflow)prompt-clustering-analysis.md: 5 imports (specialized)Common Imports:
shared/jqschema.md- JSON Schema queryingshared/reporting.md- Report formattingshared/mcp/gh-aw.md- MCP integrationshared/weekly-issues-data-fetch.md- Data collectionshared/safe-output-app.md- Output helpersImpact:
importsfeature successfully promotes code reuse. Workflows compose well.Recommendations and Next Steps
Recommendations
Immediate Actions (Priority 1)
1. Add
reactionField to SchemaFile:
pkg/parser/schemas/main_workflow_schema.jsonAction: Add as top-level property:
Validation: Cross-reference with
docs/src/content/docs/reference/frontmatter-full.md:578-5892. Validate
cache-memoryArray PatternFile:
pkg/parser/schemas/main_workflow_schema.jsonAction: Verify schema captures array-of-objects pattern:
{ "oneOf": [ {"type": "boolean"}, { "type": "array", "items": { "type": "object", "properties": { "id": {"type": "string"}, "key": {"type": "string", "description": "Supports GitHub Actions expressions"} }, "required": ["id", "key"] } } ] }Test: Validate against
daily-code-metrics.mdanddaily-copilot-token-report.mdShort-Term Actions (Priority 2)
3. Audit GitHub
toolsetsEnumFile:
pkg/parser/schemas/main_workflow_schema.jsonAction: Extract enum values and compare with production usage:
jq '.properties.tools.properties.github.properties.toolsets.items.enum' main_workflow_schema.jsonRequired Values:
default,all,repos,issues,pull_requests,actions,discussions,labels,code_security,dependabot,contextAdd Missing: Document any missing values with descriptions
4. Document Bash Allowlist Wildcards
Files:
pkg/parser/schemas/main_workflow_schema.jsondocs/src/content/docs/reference/tools.mdAction:
bash: ["date *", "gh issue *"]Medium-Term Actions (Priority 3)
5. Document
safe-outputs.messagesTemplate SystemFiles:
pkg/parser/schemas/main_workflow_schema.jsondocs/src/content/docs/reference/safe-outputs.mdAction:
messagesobject schema with four properties{workflow_name},{run_url},{event_type},{status}6. Establish Schema-Documentation Sync Process
Process Improvement:
Problem: Documentation updates without schema updates cause drift
Solution:
Research Questions (Priority 4)
7. Investigate Expression Interpolation Support
Question: What GitHub Actions expressions are valid in which fields?
Known Valid:
cache-memory.key:${{ github.workflow }},${{ github.run_id }}github-token:${{ secrets.CUSTOM_PAT }}Need Documentation:
8. Polymorphic Type Pattern Strategy
Question: How to balance flexibility vs validation for polymorphic fields?
Examples:
cache-memory: boolean | arraybash: boolean | arraynetwork: string | objectreaction: string | integerResearch:
Strategy Performance and Methodology
Strategy Performance
Strategy-021: Workflow-First Reverse Engineering
Performance Metrics:
Why This Strategy Succeeded:
Complementary Perspective: Bottom-up vs top-down analysis
Reality Check: Production workflows are source of truth
Pattern Mining: Analyzes real-world usage diversity
Usage Statistics: Quantifies feature adoption
Unique Discoveries:
reactionfield missing from schema (Critical 1)cache-memorypolymorphic complexity (Critical 2)safe-outputs.messagestemplate system (Moderate 4)Methodology
Data Collection
Sources:
.github/workflows/*.mdpkg/parser/schemas/main_workflow_schema.jsondocs/src/content/docs/reference/*.mdpkg/parser/*.go,pkg/workflow/*.gopkg/workflow/*_test.goSampling Strategy:
Analysis Techniques
1. Field Enumeration (Reverse):
2. Pattern Extraction:
- itemsyntax)${{ ... }})3. Value Analysis:
4. Schema Validation:
5. Usage Statistics:
Key Files Analyzed in Detail
High-Complexity Workflows:
archie.md- Custom messages, command.events patternaudit-workflows.md- Cache-memory, imports, stepsdeep-report.md- Network, imports, multiple toolsetsmcp-inspector.md- 16 imports (maximum)poem-bot.md- 16 safe-outputs operations (maximum)Representative Workflows:
6.
daily-malicious-code-scan.md- Strict mode, security7.
blog-auditor.md- Bash allowlists, playwright8.
daily-code-metrics.md- Cache-memory array pattern9.
craft.md- Command workflow, tool composition10.
schema-consistency-checker.md- Meta-analysis workflowEdge Cases:
11.
smoke-srt.md- Features.sandbox-runtime12.
ai-triage-campaign.md- Complex toolsets, project integrationComparison with Previous Strategies
Strategy-021 vs Other Approaches
Complementary Pairs:
When to Use Strategy-021:
When NOT to Use Strategy-021:
Next Analysis Recommendation
For December 5, 2025 (Day 339, mod 10 = 9):
Day 339 % 10 = 9, which triggers NEW APPROACH mode (values 7-9).
Recommended New Strategy: "Schema Evolution Impact Analysis"
Alternative: "Cross-Repository Workflow Pattern Analysis"
Fallback: Use Strategy-004 (Cross-Schema Consistency) - proven high-effectiveness strategy complementary to today's workflow-first approach.
Files Requiring Updates
High Priority
pkg/parser/schemas/main_workflow_schema.json- Addreactionfieldpkg/parser/schemas/main_workflow_schema.json- Validatecache-memoryarray patternpkg/parser/schemas/main_workflow_schema.json- Audittoolsetsenum completenessMedium Priority
docs/src/content/docs/reference/tools.md- Document bash allowlist wildcardsdocs/src/content/docs/reference/safe-outputs.md- Documentmessagestemplate systemreactionfieldProcess Improvements
Beta Was this translation helpful? Give feedback.
All reactions