Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions .agents/skills/profile-ci/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
---
name: profile-ci
description: Run and analyze Elements cold CI performance profiles and propose evidence-backed build, test, lint, and dependency-graph improvements. Use whenever asked to profile or benchmark pnpm run ci, rerun ci:profile, create a CI performance audit, compare CI timings, find bottlenecks or the completion path, investigate a CI performance regression, or recommend measured CI/build optimizations.
---

# Profile CI

Produce a repeatable cold-CI profile, explain what controls wall-clock completion, and turn the evidence into prioritized, testable recommendations.

## Required context

1. Read the repository `AGENTS.md`.
2. Read `projects/internals/BUILD.md`.
3. Inspect the current root `ci:profile` script and `projects/internals/ci/ci-profile.js`. Treat the script as the profiling source of truth.
4. Read a project's `DEVELOPMENT.md` before running project-specific commands when that file exists.

Do not copy profiling logic into this skill. Update the repository profiler when its behavior needs to change.

## Choose the workflow

- Run a new profile when the user asks to rerun, benchmark current changes, refresh an audit, or verify an optimization.
- Analyze existing `.metrics/ci-profile.{json,md}` artifacts without rerunning when the user asks only for interpretation and the artifacts match the intended commit and worktree state.
- Create recommendations without implementing them unless the user also asks for the changes.

## Profile safely

Run all repository commands through mise.

1. Inspect `git status --short`.
2. Preview ignored files that reset would delete with `git clean -ndX`.
3. Stop and ask before profiling if that preview includes user data, local assets, secrets, or other non-reproducible files. The profiler runs `pnpm run ci:reset`, which deletes ignored files and reinstalls dependencies before every sample.
4. Never stash, commit, discard, or clean tracked changes merely to make the profiler accept the worktree.
5. Use the clean command when the worktree is clean:

```shell
mise exec -- pnpm run ci:profile
Comment on lines +31 to +36

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Require explicit approval before the destructive reset.

The ignored-file preview is not consent to delete every ignored path. Require confirmation after git clean -ndX (or an explicit destructive-reset flag) before invoking ci:reset; otherwise the skill can delete local data based on its own classification.

🧰 Tools
🪛 SkillSpector (2.4.4)

[warning] 114: [EA2] Autonomous Decision Making: Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Remediation: Add human-in-the-loop confirmation for destructive, irreversible, or high-impact operations. Never auto-execute commands that modify files, send data, or alter system state.

(Excessive Agency (EA2))

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.agents/skills/profile-ci/SKILL.md around lines 31 - 36, Update the
profiling instructions around ci:profile and ci:reset to require explicit user
approval after reviewing git clean -ndX output, rather than treating the preview
as consent. Allow an explicit destructive-reset flag as an alternative, and do
not invoke ci:reset until one of these approvals is provided.

Source: Linters/SAST tools

```

6. When the dirty changes are the intentional subject of the audit, preserve them and run:

```shell
CI_PROFILE_ALLOW_DIRTY=1 mise exec -- pnpm run ci:profile
```

Record the dirty-worktree condition in the report. Do not use this override for unrelated or unexplained changes.

Allow all three cold samples to finish. Dependency installation or browser setup can require network access. If a run fails, inspect the copied `.metrics/ci-profile-run-*.log` and reset logs, report the incomplete profile, and do not invent missing samples.

## Verify the artifacts

The profiler writes ignored artifacts under `.metrics/`:

- `ci-profile.json`: structured metadata, run durations, and per-script samples
- `ci-profile.md`: generated method and top-ten summary
- `ci-profile-run-{1,2,3}.log`: complete CI logs
- `ci-profile-reset-{1,2,3}.log`: reset and install logs

Before analysis, confirm:

- all three runs exist;
- every run has zero incomplete scripts;
- each ranked script has three samples;
- commit, dirty state, tool versions, CPU, and memory describe the intended environment;
- the generated Markdown agrees with the JSON.

Use JSON as the numeric source of truth. Keep full precision during calculations and round only for presentation.

## Analyze wall-clock relevance

The slowest command is not automatically the critical path.

1. Rank leaf scripts by median duration from `ci-profile.json`.
2. Inspect the end of each run log to identify the scripts that consistently complete last.
3. Trace those scripts through their Wireit `dependencies` in the relevant `package.json` files.
4. Separate:
- final or near-final dependency branches;
- long parallel work that consumes CPU, memory, filesystem, or browser capacity;
- upstream work that delays a final branch;
- composite commands whose phases need separate timing.
5. Inspect each candidate's command, configuration, file count, output, output consumers, and serialization settings before suggesting a change.
6. Compare with the existing audit only when its environment and method are compatible. Describe before/after changes as concurrent cold-CI measurements, not isolated causal proof.

Read [the optimization playbook](references/optimization-playbook.md) when generating recommendations or designing follow-up experiments.

## Create the audit

Create or refresh `projects/internals/ci/CI-PERFORMANCE.md`. Use `.metrics/ci-profile.md` as generated evidence, not as the finished audit.

Include:

1. Frontmatter and generation context
2. Executive summary with median CI time and comparison when available
3. Method, environment, and all run results
4. Current top-ten scripts
5. Completion-path analysis
6. Numbered open findings
7. Verified completed changes when a prior audit exists
8. Recommended experiment order
9. Raw artifact location

For every finding, provide:

- **Evidence:** measurements plus current configuration or dependency facts
- **Recommendation:** one bounded change or experiment
- **Validation:** output-equivalence checks, metrics to compare, and resource checks
- **Confidence:** confidence in the diagnosis and in the proposed approach

Keep completed work out of the open priority list. Mark an item verified only when the configuration changed and a comparable full profile confirms the result.

## Recommendation guardrails

- Do not add CI-share percentages together; scripts overlap.
- Do not claim a critical path from rankings alone.
- Do not recommend more concurrency without checking isolation, shared state, ports, output paths, report merging, memory, and machine-wide contention.
- Do not remove a quality check unless the same coverage remains in another required workflow.
- Prefer one-variable experiments and compare at least three samples when variance matters.
- Distinguish targeted project benchmarks from full concurrent CI results.
- Treat cache optimization separately from this cold profile because the profiler disables Wireit caching.
- Label estimates and inferred causes. Do not present them as measurements.
- Keep dependency upgrades separate from performance changes unless the upgrade is the explicit experiment.

## Check the report

Run:

```shell
mise exec -- pnpm exec prettier --write projects/internals/ci/CI-PERFORMANCE.md
mise exec -- pnpm exec vale --config .vale.ini projects/internals/ci/CI-PERFORMANCE.md
git diff --check
```

If `projects/internals/ci/ci-profile.test.js` exists, also run:

```shell
mise exec -- node --test projects/internals/ci/ci-profile.test.js
```

Do not rerun the full CI merely to validate the report: the profiler already completed it three times. Report the median, comparison, top remaining opportunities, artifact paths, and validation results to the user.
108 changes: 108 additions & 0 deletions .agents/skills/profile-ci/references/optimization-playbook.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# CI optimization playbook

Use this reference after collecting a valid profile. Select approaches from current evidence; do not list every approach in every audit.

## Selection order

### Remove unconsumed or duplicate work

Look for bundle visualizers, duplicate test suites, unused report formats, repeated downloads, redundant generation, and work repeated in both a main job and a dedicated job.

Verify all output consumers before removal. Preserve required test, release, documentation, and deployment coverage.

### Improve the completion branch

Trace the scripts that finish last through Wireit dependencies. Consider:

- starting independent prerequisites earlier;
- removing dependencies that do not represent real input requirements;
- narrowing broad dependency fan-in;
- reducing work in the final command;
- avoiding a second copy, transform, or package pass.

Do not decouple a dependency until output and runtime behavior prove that the downstream task does not need it.

### Split composite commands

Split sequential phases when one command hides attribution or forces broad cache invalidation. Give each phase accurate Wireit files, outputs, dependencies, and environment.

Splitting improves observability and caching. It does not reduce cold wall time by itself unless the graph can safely overlap phases.

### Add controlled parallelism

Look for one-worker test suites, disabled file parallelism, sequential linting, and independent generated targets.

Prefer process-level shards when tools share browser state or globals. Give every shard separate ports, browser profiles, coverage directories, screenshots, JUnit/JSON output, and temporary files. Merge reports deterministically and verify identical totals and thresholds.

Measure peak memory and full CI time. A faster package command can slow the pipeline by starving sibling work.

### Reduce transformed or generated inputs

Use module counts, page counts, plugin timings, and repeated transforms to find large input surfaces. Consider:

- deduplicating identical generated modules;
- externalizing safe shared assets;
- omitting development-only modules from production;
- avoiding repeated full-document scans;
- narrowing entry points and globs;
- generating shared metadata once.

Require output-equivalence checks for routes, bundles, screenshots, metadata, and runtime behavior.

### Improve cache boundaries

Audit Wireit `files`, `output`, `dependencies`, `cascade`, environment variables, and package-lock inputs. Split tasks whose unrelated inputs invalidate expensive work.

Measure cache-hit behavior separately. The cold profiler sets `WIREIT_CACHE=none`, so cache changes cannot explain its results.

### Use tool-specific diagnostics

Inspect the installed tool version and local configuration before choosing flags.

- Wireit: use command start/finish order and dependency declarations.
- Vite or Rolldown: inspect transformed module counts and plugin timing warnings.
- Eleventy: compare written files, templates, transforms, and Vite phases.
- Vitest: inspect file/test totals, workers, concurrency, isolation, retries, browser providers, and reporters.
- ESLint: inspect rule timing, file counts, cache state, and supported concurrency modes.
- Coverage: identify report consumers and the cost of collection, transformation, serialization, and report generation.

## Experiment record

For each experiment, capture:

- hypothesis;
- exact configuration difference;
- baseline and candidate commands;
- sample count and environment;
- median, minimum, maximum, and variance;
- output-equivalence checks;
- test and coverage totals;
- peak memory or other resource constraints;
- targeted timing and full-CI timing;
- decision and confidence.

Change one performance variable at a time where practical.

## Confidence rubric

- **High diagnosis confidence:** repeated measurements and configuration directly explain the serialization or work.
- **Medium diagnosis confidence:** evidence is consistent, but concurrent contention or hidden tool behavior remains.
- **Low diagnosis confidence:** the signal is noisy, appears only once, or lacks configuration support.
- **High approach confidence:** a low-risk removal or configuration change preserves verified outputs.
- **Medium approach confidence:** the approach needs isolation, report merging, or dependency validation.
- **Low approach confidence:** the expected gain is speculative or depends on undocumented behavior.

State diagnosis confidence separately from approach confidence when they differ.

## Reject weak recommendations

Do not recommend:

- deleting caches to make a warm build faster;
- adding all script durations to estimate total CI time;
- increasing every worker count to the CPU count;
- removing tests solely because they are slow;
- moving a check to another workflow without confirming that required status checks include it;
- claiming an isolated benchmark equals full CI savings;
- upgrading dependencies and changing orchestration in the same performance experiment;
- optimizing a top-ten parallel script while ignoring the branch that completes last.
11 changes: 11 additions & 0 deletions config/vale/styles/config/vocabularies/Elements/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ ESLint
Wireit
pnpm
Vite
Rolldown
Vitest
WCAG
ARIA
Expand All @@ -30,8 +31,10 @@ Chromium
Playwright
OWASP
CDN
CPUs
polyfill
polyfills
profiler
entrypoint
entrypoints
postcss
Expand Down Expand Up @@ -72,15 +75,19 @@ popover
popovers
shortcode
shortcodes
passthrough
server-driven
single-column
multi-column
Heatmap
heatmap
heatmaps
hotspot
datalist
debounce
debounced
deduplicating
deduplication
dropdown
dropdowns
checkbox
Expand Down Expand Up @@ -124,8 +131,10 @@ dev
async
Rollup
Webpack
Turbopack
subheader
tsconfig
typecheck
tsx
TSX
jsx
Expand All @@ -147,6 +156,7 @@ svg
SVG
scrollable
parallelization
sharding
minimizable
datagrid
ESBuild
Expand Down Expand Up @@ -174,6 +184,7 @@ inlined
Inlined
globals
gitignored
untracked
exportparts
downleveling
downlevel
Expand Down
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"ci:all": "PAGES_BASE_URL=/elements/ pnpm run ci && PAGES_BASE_URL=/elements/ node ./projects/internals/ci/cache-validate.js ci",
"ci:reset": "pnpm run clean && pnpm i --frozen-lockfile --prefer-offline",
"ci:validate": "wireit",
"ci:profile": "node ./projects/internals/ci/ci-profile.js",
"lighthouse": "wireit",
"clean": "git clean -dfX",
"format": "wireit",
Expand Down Expand Up @@ -102,6 +103,7 @@
"./projects/starters/vue:ci",
"./projects/styles:ci",
"./projects/themes:ci",
"./projects/internals/ci:ci",
"./projects/internals/design:ci",
"./projects/internals/metadata:ci",
"./projects/internals/patterns:ci",
Expand Down
Loading