Skip to content

COMP: Fix ccache hit rate on Azure DevOps to near-zero#6048

Merged
hjmjohnson merged 1 commit intoInsightSoftwareConsortium:mainfrom
hjmjohnson:comp-ccache-azure-fix
Apr 13, 2026
Merged

COMP: Fix ccache hit rate on Azure DevOps to near-zero#6048
hjmjohnson merged 1 commit intoInsightSoftwareConsortium:mainfrom
hjmjohnson:comp-ccache-azure-fix

Conversation

@hjmjohnson
Copy link
Copy Markdown
Member

@hjmjohnson hjmjohnson commented Apr 13, 2026

Fix ccache configuration on all 7 Azure DevOps pipelines — hit rate was 0.02%, now 99.95%.

Root cause and fix

Root cause: CCACHE_NODIRECT=1 (global variable) disabled ccache's fast "direct mode" which hashes source files directly. Instead, every file was preprocessed before hashing — a slower path that also changes hash inputs, causing near-total cache misses across runs.

Evidence: ARM CI (GitHub Actions) never had NODIRECT and achieves 98.5% hit rate. Azure DevOps with NODIRECT=1 was at 0.02% (1 hit out of 5160 cacheable calls).

Fixes applied to all 7 Azure pipelines:

  1. Remove CCACHE_NODIRECT=1 — enable direct mode
  2. Add CCACHE_SLOPPINESS=pch_defines,time_macros — prevent __DATE__/__TIME__ misses
  3. Add per-job name to cache key (ccache-v4 | OS | JobName | SHA) — prevents jobs with different build configs from poisoning each other's caches
  4. Remove ITK_USE_CCACHE:BOOL=ON — redundant with CMAKE_*_COMPILER_LAUNCHER=ccache
  5. Add ccache --show-stats + ccache --zero-stats steps for monitoring

Pipelines updated:

  • AzurePipelinesLinux.yml (3 jobs: Linux, LinuxLegacyRemoved, LinuxCxx20)
  • AzurePipelinesLinuxPython.yml
  • AzurePipelinesMacOS.yml
  • AzurePipelinesMacOSPython.yml
  • AzurePipelinesWindows.yml
  • AzurePipelinesWindowsPython.yml
  • AzurePipelinesBatch.yml
Test results from PR #6047
Metric Before fix Run 1 (cold) Run 2 (warm)
Hit rate 0.02% 0% (seeding) 99.95%
Direct hits 1 0 2166/2167
Misses 5159 ~2167 1

@github-actions github-actions bot added type:Compiler Compiler support or related warnings type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct labels Apr 13, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 13, 2026

Greptile Summary

This PR fixes a near-zero ccache hit rate (0.02%) on the Azure DevOps Linux pipeline by removing CCACHE_NODIRECT=1, which was forcing ccache into preprocessor mode and causing near-total cache misses across builds. Additional improvements include per-job cache key isolation, CCACHE_SLOPPINESS=pch_defines,time_macros to suppress __DATE__/__TIME__ invalidation, and ccache stats steps for ongoing monitoring.

Confidence Score: 5/5

Safe to merge — CI-only configuration change with well-evidenced improvement and no P0/P1 findings.

All changes are confined to CI YAML configuration. The root cause analysis is thorough and supported by measured data (0.02% → 99.95% hit rate). Each change is individually justified: removing CCACHE_NODIRECT restores direct mode, per-job cache keys prevent cross-configuration pollution, and the stats steps add observability. The only note is the deliberate time_macros sloppiness trade-off, which is P2 and standard practice for CI caching.

No files require special attention.

Important Files Changed

Filename Overview
Testing/ContinuousIntegration/AzurePipelinesLinux.yml Replaces CCACHE_NODIRECT=1 with CCACHE_SLOPPINESS=pch_defines,time_macros, adds per-job cache keys (ccache-v4

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Pipeline starts per job] --> B[Restore ccache per-job key]
    B --> C[zero-stats, evict old entries, show-config]
    C --> D[ctest build with ccache direct mode]
    D --> E{Cache hit?}
    E -->|Yes ~99.95pct| F[Reuse cached object]
    E -->|No| G[Compile and store]
    D --> H[show-stats always runs]
    H --> I[report diagnostics and publish]
Loading

Reviews (1): Last reviewed commit: "COMP: Fix near-zero ccache hit rate on A..." | Re-trigger Greptile

Comment thread Testing/ContinuousIntegration/AzurePipelinesLinux.yml
Remove CCACHE_NODIRECT=1 which disabled ccache's fast direct mode,
forcing preprocessor-mode hashing on every compilation unit. The ARM
CI (which never had NODIRECT) achieves 98.5% hit rate; Azure DevOps
was at 0.02%.

Applied to all 7 Azure DevOps pipelines:
- AzurePipelinesLinux.yml (3 jobs: Linux, LinuxLegacyRemoved, LinuxCxx20)
- AzurePipelinesLinuxPython.yml
- AzurePipelinesMacOS.yml
- AzurePipelinesMacOSPython.yml
- AzurePipelinesWindows.yml
- AzurePipelinesWindowsPython.yml
- AzurePipelinesBatch.yml

Changes per pipeline:
- Remove CCACHE_NODIRECT=1 (enable direct mode)
- Add CCACHE_SLOPPINESS=pch_defines,time_macros
- Add per-job name to cache key (ccache-v4 | OS | JobName | SHA)
- Remove ITK_USE_CCACHE:BOOL=ON (redundant with CMAKE_*_LAUNCHER)
- Add ccache --zero-stats + --show-config maintenance step
- Add ccache --show-stats step after build for monitoring

Tested in PR InsightSoftwareConsortium#6047: Run 2 achieved 99.95% hit rate (2166/2167
direct hits) after these fixes on the Linux pipeline.
@hjmjohnson hjmjohnson force-pushed the comp-ccache-azure-fix branch from 285b0be to fb17c4c Compare April 13, 2026 17:37
@hjmjohnson hjmjohnson changed the title COMP: Fix near-zero ccache hit rate on Azure DevOps ITK.Linux COMP: Fix ccache hit rate on Azure DevOps to near-zero Apr 13, 2026
@hjmjohnson hjmjohnson merged commit 188d6fe into InsightSoftwareConsortium:main Apr 13, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:Compiler Compiler support or related warnings type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants