-
Notifications
You must be signed in to change notification settings - Fork 9
chore(ci): enhance ci performance profiling and add new scripts #197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| --- | ||
| name: profile-ci | ||
| description: Run and analyze Elements cold CI performance profiles and propose evidence-backed build, test, lint, and dependency-graph improvements. Use whenever asked to profile or benchmark pnpm run ci, rerun ci:profile, create a CI performance audit, compare CI timings, find bottlenecks or the completion path, investigate a CI performance regression, or recommend measured CI/build optimizations. | ||
| --- | ||
|
|
||
| # Profile CI | ||
|
|
||
| Produce a repeatable cold-CI profile, explain what controls wall-clock completion, and turn the evidence into prioritized, testable recommendations. | ||
|
|
||
| ## Required context | ||
|
|
||
| 1. Read the repository `AGENTS.md`. | ||
| 2. Read `projects/internals/BUILD.md`. | ||
| 3. Inspect the current root `ci:profile` script and `projects/internals/ci/ci-profile.js`. Treat the script as the profiling source of truth. | ||
| 4. Read a project's `DEVELOPMENT.md` before running project-specific commands when that file exists. | ||
|
|
||
| Do not copy profiling logic into this skill. Update the repository profiler when its behavior needs to change. | ||
|
|
||
| ## Choose the workflow | ||
|
|
||
| - Run a new profile when the user asks to rerun, benchmark current changes, refresh an audit, or verify an optimization. | ||
| - Analyze existing `.metrics/ci-profile.{json,md}` artifacts without rerunning when the user asks only for interpretation and the artifacts match the intended commit and worktree state. | ||
| - Create recommendations without implementing them unless the user also asks for the changes. | ||
|
|
||
| ## Profile safely | ||
|
|
||
| Run all repository commands through mise. | ||
|
|
||
| 1. Inspect `git status --short`. | ||
| 2. Preview ignored files that reset would delete with `git clean -ndX`. | ||
| 3. Stop and ask before profiling if that preview includes user data, local assets, secrets, or other non-reproducible files. The profiler runs `pnpm run ci:reset`, which deletes ignored files and reinstalls dependencies before every sample. | ||
| 4. Never stash, commit, discard, or clean tracked changes merely to make the profiler accept the worktree. | ||
| 5. Use the clean command when the worktree is clean: | ||
|
|
||
| ```shell | ||
| mise exec -- pnpm run ci:profile | ||
| ``` | ||
|
|
||
| 6. When the dirty changes are the intentional subject of the audit, preserve them and run: | ||
|
|
||
| ```shell | ||
| CI_PROFILE_ALLOW_DIRTY=1 mise exec -- pnpm run ci:profile | ||
| ``` | ||
|
|
||
| Record the dirty-worktree condition in the report. Do not use this override for unrelated or unexplained changes. | ||
|
|
||
| Allow all three cold samples to finish. Dependency installation or browser setup can require network access. If a run fails, inspect the copied `.metrics/ci-profile-run-*.log` and reset logs, report the incomplete profile, and do not invent missing samples. | ||
|
|
||
| ## Verify the artifacts | ||
|
|
||
| The profiler writes ignored artifacts under `.metrics/`: | ||
|
|
||
| - `ci-profile.json`: structured metadata, run durations, and per-script samples | ||
| - `ci-profile.md`: generated method and top-ten summary | ||
| - `ci-profile-run-{1,2,3}.log`: complete CI logs | ||
| - `ci-profile-reset-{1,2,3}.log`: reset and install logs | ||
|
|
||
| Before analysis, confirm: | ||
|
|
||
| - all three runs exist; | ||
| - every run has zero incomplete scripts; | ||
| - each ranked script has three samples; | ||
| - commit, dirty state, tool versions, CPU, and memory describe the intended environment; | ||
| - the generated Markdown agrees with the JSON. | ||
|
|
||
| Use JSON as the numeric source of truth. Keep full precision during calculations and round only for presentation. | ||
|
|
||
| ## Analyze wall-clock relevance | ||
|
|
||
| The slowest command is not automatically the critical path. | ||
|
|
||
| 1. Rank leaf scripts by median duration from `ci-profile.json`. | ||
| 2. Inspect the end of each run log to identify the scripts that consistently complete last. | ||
| 3. Trace those scripts through their Wireit `dependencies` in the relevant `package.json` files. | ||
| 4. Separate: | ||
| - final or near-final dependency branches; | ||
| - long parallel work that consumes CPU, memory, filesystem, or browser capacity; | ||
| - upstream work that delays a final branch; | ||
| - composite commands whose phases need separate timing. | ||
| 5. Inspect each candidate's command, configuration, file count, output, output consumers, and serialization settings before suggesting a change. | ||
| 6. Compare with the existing audit only when its environment and method are compatible. Describe before/after changes as concurrent cold-CI measurements, not isolated causal proof. | ||
|
|
||
| Read [the optimization playbook](references/optimization-playbook.md) when generating recommendations or designing follow-up experiments. | ||
|
|
||
| ## Create the audit | ||
|
|
||
| Create or refresh `projects/internals/ci/CI-PERFORMANCE.md`. Use `.metrics/ci-profile.md` as generated evidence, not as the finished audit. | ||
|
|
||
| Include: | ||
|
|
||
| 1. Frontmatter and generation context | ||
| 2. Executive summary with median CI time and comparison when available | ||
| 3. Method, environment, and all run results | ||
| 4. Current top-ten scripts | ||
| 5. Completion-path analysis | ||
| 6. Numbered open findings | ||
| 7. Verified completed changes when a prior audit exists | ||
| 8. Recommended experiment order | ||
| 9. Raw artifact location | ||
|
|
||
| For every finding, provide: | ||
|
|
||
| - **Evidence:** measurements plus current configuration or dependency facts | ||
| - **Recommendation:** one bounded change or experiment | ||
| - **Validation:** output-equivalence checks, metrics to compare, and resource checks | ||
| - **Confidence:** confidence in the diagnosis and in the proposed approach | ||
|
|
||
| Keep completed work out of the open priority list. Mark an item verified only when the configuration changed and a comparable full profile confirms the result. | ||
|
|
||
| ## Recommendation guardrails | ||
|
|
||
| - Do not add CI-share percentages together; scripts overlap. | ||
| - Do not claim a critical path from rankings alone. | ||
| - Do not recommend more concurrency without checking isolation, shared state, ports, output paths, report merging, memory, and machine-wide contention. | ||
| - Do not remove a quality check unless the same coverage remains in another required workflow. | ||
| - Prefer one-variable experiments and compare at least three samples when variance matters. | ||
| - Distinguish targeted project benchmarks from full concurrent CI results. | ||
| - Treat cache optimization separately from this cold profile because the profiler disables Wireit caching. | ||
| - Label estimates and inferred causes. Do not present them as measurements. | ||
| - Keep dependency upgrades separate from performance changes unless the upgrade is the explicit experiment. | ||
|
|
||
| ## Check the report | ||
|
|
||
| Run: | ||
|
|
||
| ```shell | ||
| mise exec -- pnpm exec prettier --write projects/internals/ci/CI-PERFORMANCE.md | ||
| mise exec -- pnpm exec vale --config .vale.ini projects/internals/ci/CI-PERFORMANCE.md | ||
| git diff --check | ||
| ``` | ||
|
|
||
| If `projects/internals/ci/ci-profile.test.js` exists, also run: | ||
|
|
||
| ```shell | ||
| mise exec -- node --test projects/internals/ci/ci-profile.test.js | ||
| ``` | ||
|
|
||
| Do not rerun the full CI merely to validate the report: the profiler already completed it three times. Report the median, comparison, top remaining opportunities, artifact paths, and validation results to the user. | ||
108 changes: 108 additions & 0 deletions
108
.agents/skills/profile-ci/references/optimization-playbook.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| # CI optimization playbook | ||
|
|
||
| Use this reference after collecting a valid profile. Select approaches from current evidence; do not list every approach in every audit. | ||
|
|
||
| ## Selection order | ||
|
|
||
| ### Remove unconsumed or duplicate work | ||
|
|
||
| Look for bundle visualizers, duplicate test suites, unused report formats, repeated downloads, redundant generation, and work repeated in both a main job and a dedicated job. | ||
|
|
||
| Verify all output consumers before removal. Preserve required test, release, documentation, and deployment coverage. | ||
|
|
||
| ### Improve the completion branch | ||
|
|
||
| Trace the scripts that finish last through Wireit dependencies. Consider: | ||
|
|
||
| - starting independent prerequisites earlier; | ||
| - removing dependencies that do not represent real input requirements; | ||
| - narrowing broad dependency fan-in; | ||
| - reducing work in the final command; | ||
| - avoiding a second copy, transform, or package pass. | ||
|
|
||
| Do not decouple a dependency until output and runtime behavior prove that the downstream task does not need it. | ||
|
|
||
| ### Split composite commands | ||
|
|
||
| Split sequential phases when one command hides attribution or forces broad cache invalidation. Give each phase accurate Wireit files, outputs, dependencies, and environment. | ||
|
|
||
| Splitting improves observability and caching. It does not reduce cold wall time by itself unless the graph can safely overlap phases. | ||
|
|
||
| ### Add controlled parallelism | ||
|
|
||
| Look for one-worker test suites, disabled file parallelism, sequential linting, and independent generated targets. | ||
|
|
||
| Prefer process-level shards when tools share browser state or globals. Give every shard separate ports, browser profiles, coverage directories, screenshots, JUnit/JSON output, and temporary files. Merge reports deterministically and verify identical totals and thresholds. | ||
|
|
||
| Measure peak memory and full CI time. A faster package command can slow the pipeline by starving sibling work. | ||
|
|
||
| ### Reduce transformed or generated inputs | ||
|
|
||
| Use module counts, page counts, plugin timings, and repeated transforms to find large input surfaces. Consider: | ||
|
|
||
| - deduplicating identical generated modules; | ||
| - externalizing safe shared assets; | ||
| - omitting development-only modules from production; | ||
| - avoiding repeated full-document scans; | ||
| - narrowing entry points and globs; | ||
| - generating shared metadata once. | ||
|
|
||
| Require output-equivalence checks for routes, bundles, screenshots, metadata, and runtime behavior. | ||
|
|
||
| ### Improve cache boundaries | ||
|
|
||
| Audit Wireit `files`, `output`, `dependencies`, `cascade`, environment variables, and package-lock inputs. Split tasks whose unrelated inputs invalidate expensive work. | ||
|
|
||
| Measure cache-hit behavior separately. The cold profiler sets `WIREIT_CACHE=none`, so cache changes cannot explain its results. | ||
|
|
||
| ### Use tool-specific diagnostics | ||
|
|
||
| Inspect the installed tool version and local configuration before choosing flags. | ||
|
|
||
| - Wireit: use command start/finish order and dependency declarations. | ||
| - Vite or Rolldown: inspect transformed module counts and plugin timing warnings. | ||
| - Eleventy: compare written files, templates, transforms, and Vite phases. | ||
| - Vitest: inspect file/test totals, workers, concurrency, isolation, retries, browser providers, and reporters. | ||
| - ESLint: inspect rule timing, file counts, cache state, and supported concurrency modes. | ||
| - Coverage: identify report consumers and the cost of collection, transformation, serialization, and report generation. | ||
|
|
||
| ## Experiment record | ||
|
|
||
| For each experiment, capture: | ||
|
|
||
| - hypothesis; | ||
| - exact configuration difference; | ||
| - baseline and candidate commands; | ||
| - sample count and environment; | ||
| - median, minimum, maximum, and variance; | ||
| - output-equivalence checks; | ||
| - test and coverage totals; | ||
| - peak memory or other resource constraints; | ||
| - targeted timing and full-CI timing; | ||
| - decision and confidence. | ||
|
|
||
| Change one performance variable at a time where practical. | ||
|
|
||
| ## Confidence rubric | ||
|
|
||
| - **High diagnosis confidence:** repeated measurements and configuration directly explain the serialization or work. | ||
| - **Medium diagnosis confidence:** evidence is consistent, but concurrent contention or hidden tool behavior remains. | ||
| - **Low diagnosis confidence:** the signal is noisy, appears only once, or lacks configuration support. | ||
| - **High approach confidence:** a low-risk removal or configuration change preserves verified outputs. | ||
| - **Medium approach confidence:** the approach needs isolation, report merging, or dependency validation. | ||
| - **Low approach confidence:** the expected gain is speculative or depends on undocumented behavior. | ||
|
|
||
| State diagnosis confidence separately from approach confidence when they differ. | ||
|
|
||
| ## Reject weak recommendations | ||
|
|
||
| Do not recommend: | ||
|
|
||
| - deleting caches to make a warm build faster; | ||
| - adding all script durations to estimate total CI time; | ||
| - increasing every worker count to the CPU count; | ||
| - removing tests solely because they are slow; | ||
| - moving a check to another workflow without confirming that required status checks include it; | ||
| - claiming an isolated benchmark equals full CI savings; | ||
| - upgrading dependencies and changing orchestration in the same performance experiment; | ||
| - optimizing a top-ten parallel script while ignoring the branch that completes last. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Require explicit approval before the destructive reset.
The ignored-file preview is not consent to delete every ignored path. Require confirmation after
git clean -ndX(or an explicit destructive-reset flag) before invokingci:reset; otherwise the skill can delete local data based on its own classification.🧰 Tools
🪛 SkillSpector (2.4.4)
[warning] 114: [EA2] Autonomous Decision Making: Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.
Remediation: Add human-in-the-loop confirmation for destructive, irreversible, or high-impact operations. Never auto-execute commands that modify files, send data, or alter system state.
(Excessive Agency (EA2))
🤖 Prompt for AI Agents
Source: Linters/SAST tools