[file-diet] File Diet: Split pkg/workflow/compiler_yaml_main_job.go (1265 lines) into focused modules

[github-actions]

Overview

The file pkg/workflow/compiler_yaml_main_job.go has grown to 1265 lines, exceeding the healthy threshold of 800 lines. This task involves refactoring it into smaller, focused files aligned with the five compilation phases it already internally documents.

Current State

• File: pkg/workflow/compiler_yaml_main_job.go
• Size: 1265 lines
• Test file: pkg/workflow/compiler_yaml_main_job_test.go (1060 lines)
• Test-to-source ratio: ~0.84 (84%)
• Function count: 23 methods/functions across 5 logical phases
• Complexity: Most functions are 30–136 lines; the largest (generateInitialAndCheckoutSteps 136 lines, generateEngineInstallAndPreAgentSteps 118 lines, collectArtifactPaths 110 lines, generateAgentRunSteps 109 lines) represent distinct, separable concerns

Full File Analysis

Function Distribution by Phase

Phase	Functions	Approx. Lines
Orchestration	generateMainJobSteps	~34
Checkout & Imports	generateInitialAndCheckoutSteps, generateRepositoryImportCheckouts, generateLegacyAgentImportCheckout, parseRepositoryImportSpec, generateDevModeCLIBuildSteps	~270
Runtime & Custom Steps	generateRuntimeAndWorkspaceSetupSteps, prepareRuntimeSetupAndCheckoutInfo, emitRuntimeSetupPrelude, generateArcDindToolCacheRedirectStep, generateArcDindNodePathStep, generateActivationArtifactAndCommentMemorySteps, generateCommentMemoryEarlyConfigStep, generateCommentMemoryEarlyConfigLines, emitCustomSteps, addCustomStepsAsIs, addCustomStepsWithRuntimeInsertion, sanitizeAndWarnCustomSteps	~390
Engine & Agent Execution	generateEngineInstallAndPreAgentSteps, generateAgentRunSteps	~230
Post-Agent & Artifacts	collectArtifactPaths, generateSummarySteps, generatePostAgentCollectionAndUpload	~250

Complexity Hotspots

• generateInitialAndCheckoutSteps (136 lines, ~L55-191): handles OTLP masking, pre-steps, workspace checkout, dev-mode CLI build, additional checkouts, repo imports, legacy agent import, and .github folder merging -- multiple distinct concerns
• generateEngineInstallAndPreAgentSteps (118 lines, ~L493-611): wires git creds, PR-ready checkout, engine install, MCP token minting, lockdown detection, guard vars, DIFC proxy, and MCP gateway -- a long sequential pipeline
• collectArtifactPaths (110 lines, ~L732-842): accumulates paths from six different subsystems; already has strong internal cohesion and is a good split candidate
• addCustomStepsWithRuntimeInsertion (86 lines, ~L1009-1097): complex interleaving logic; directly related to custom-step handling in compiler_yaml_step_conversion.go

Existing Split Pattern

The repo already uses fine-grained splits under pkg/workflow/compiler_yaml_*.go: compiler_yaml_ai_execution.go (254 lines), compiler_yaml_artifacts.go (58 lines), compiler_yaml_step_generation.go (324 lines). The new splits below follow that convention.

Refactoring Strategy

Proposed File Splits

1. compiler_yaml_main_job.go (keep, reduce to orchestration only)

   • Functions: generateMainJobSteps
   • Responsibility: Top-level phase orchestrator; calls the four phase functions below
   • Estimated LOC: ~50

2. compiler_yaml_checkout.go (new)

   • Functions: generateInitialAndCheckoutSteps, generateRepositoryImportCheckouts, generateLegacyAgentImportCheckout, parseRepositoryImportSpec, generateDevModeCLIBuildSteps
   • Responsibility: Workspace checkout, dev-mode CLI build, repo import checkouts
   • Estimated LOC: ~280

3. compiler_yaml_runtime_setup.go (new)

   • Functions: generateRuntimeAndWorkspaceSetupSteps, prepareRuntimeSetupAndCheckoutInfo, emitRuntimeSetupPrelude, generateArcDindToolCacheRedirectStep, generateArcDindNodePathStep, generateActivationArtifactAndCommentMemorySteps, generateCommentMemoryEarlyConfigStep, generateCommentMemoryEarlyConfigLines, emitCustomSteps, addCustomStepsAsIs, addCustomStepsWithRuntimeInsertion, sanitizeAndWarnCustomSteps
   • Responsibility: Runtime detection, ARC/DIND adapters, memory setup, custom step emission and sanitization
   • Estimated LOC: ~400

4. compiler_yaml_post_agent.go (new)

   • Functions: collectArtifactPaths, generateSummarySteps, generatePostAgentCollectionAndUpload
   • Responsibility: Artifact path accumulation, step summary generation, post-agent upload and cleanup
   • Estimated LOC: ~260

Note: generateEngineInstallAndPreAgentSteps and generateAgentRunSteps (~230 lines combined) fit naturally in the existing compiler_yaml_ai_execution.go (currently 254 lines, would grow to ~480 lines -- still under 500 lines) or can form a dedicated compiler_yaml_engine_setup.go.

Shared Utilities

• sanitizeAndWarnCustomSteps + addCustomStepsAsIs could eventually consolidate with compiler_yaml_step_conversion.go if that file has capacity
• generateCommentMemoryEarlyConfigLines is already invoked by custom job paths; making it accessible from a well-named file improves discoverability

Interface Abstractions

• The five-phase orchestration in generateMainJobSteps is already well-factored; no new interfaces needed for this split
• Consider a CheckoutPhaseResult struct to replace the multi-return (*CheckoutManager, bool, error) tuple if adding more return values in future

Test Coverage Plan

Split the test file in parallel with the source:

1. compiler_yaml_checkout_test.go

   • Test cases: repo import spec parsing (all three formats), dev-mode build step injection, legacy agent import path, additional checkouts merge
   • Target coverage: >80%

2. compiler_yaml_runtime_setup_test.go

   • Test cases: ARC/DIND step injection, comment-memory early config (present/absent handlers), custom step sanitization (expression extraction warnings), addCustomStepsWithRuntimeInsertion runtime insertion logic
   • Target coverage: >80%

3. compiler_yaml_post_agent_test.go

   • Test cases: artifact path accumulation per subsystem, summary step generation, post-agent upload with/without staging and repo-memory
   • Target coverage: >80%

Existing tests in compiler_yaml_main_job_test.go (1060 lines) should be reorganized to match the new source files.

Implementation Guidelines

1. Preserve Behavior: All existing functionality must work identically after the split
2. Maintain Exports: Keep public API unchanged (all current functions are unexported methods on *Compiler)
3. Same package: All new files stay in package workflow -- no import changes needed
4. Incremental Changes: Move one phase group at a time; verify make test-unit passes after each
5. Run Tests Frequently: make test-unit after each file move
6. Run Linting: make lint must pass
7. No .lock.yml changes: This is a pure Go refactor; no workflow recompilation needed

Acceptance Criteria

• compiler_yaml_main_job.go is reduced to the orchestration function (~50 lines)
• Two or three new compiler_yaml_*.go files created, each under 500 lines
• All tests pass (make test-unit)
• Test coverage ≥80% for new files
• No breaking changes to public API
• Code passes linting (make lint)
• Build succeeds (make build)
• Golden files updated if console output changes (make update-golden)

Additional Context

• Existing split pattern: compiler_yaml_ai_execution.go, compiler_yaml_artifacts.go, compiler_yaml_step_generation.go, compiler_yaml_step_conversion.go -- follow these naming conventions
• Code Organization: See AGENTS.md and .github/skills/developer/SKILL.md for file-size guidelines (target <500 lines per file)
• Testing: Match existing test patterns in pkg/workflow/compiler_yaml_*_test.go

---

Priority: Medium
Effort: Medium (pure mechanical split -- no logic changes, same package, no import rewiring)
Expected Impact: Improved navigability, easier targeted testing, reduced cognitive load when editing any single compilation phase

Generated by 🧹 Daily File Diet · 54.5 AIC · ⌖ 20.4 AIC · ⊞ 6.8K · ◷

• expires on Jul 11, 2026, 5:45 AM UTC-08:00

[github-actions]

🍪 Issue Monster selected this for Copilot

I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · 2.32 AIC · ⌖ 6.28 AIC · ⊞ 8.4K · ◷