Build the search index one package at a time - #486
Merged
Conversation
The hourly search index regeneration (which also runs at startup) decoded every package in the database into memory at once before building the index. The decoded packages now need more memory than the server has, so the server was being OOM-killed several times an hour (over 1,600 times in the last 30 days), taking the site down for a couple of minutes each time. Instead, decode packages one at a time and retain only each package's index entries, filling in reverse dependency counts once all packages have been seen. Serving the full production data set now peaks around 1.1GB resident instead of 3.6GB+. Also: - Use both vCPUs and cap the heap below available RAM (-N2 -A64m -M3G) - Update deprecated GitHub Actions (cache/upload-artifact v2 no longer run) and pin the runner to ubuntu-24.04 to match the server OS
- A package file which cannot be read or decoded is now skipped with a logged error instead of silently aborting the entire index build (which left the search index empty until the next successful hourly run). Verified by corrupting a package file in a local run against production data: the package is skipped and the rest of the index builds normally. - Fully evaluate the new index on the regeneration thread before publishing it, so the first search request after a regen no longer pays for building the trie. - Pin stack to 3.9.3 in CI (what "latest" resolves to today) so release builds stay reproducible. Memory, measured over 21 consecutive index rebuilds against the full production data set with +RTS -s -N2 -A64m -M3G: 715MB maximum residency, 2.2GB peak RTS memory in use, no growth across cycles.
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pursuit has been getting OOM-killed several times an hour (over 1,600 times in the past 30 days per the systemd restart counter), which is why the site regularly goes down. The root cause is the search index regeneration in
startRegenThread, which runs at startup and hourly:getAllPackagesdecoded the latest version of every package in the database into memory at once (~1,400 packages, 1.7GB of JSON) before building the index. The decoded packages now require more memory than the 4GB droplet has, so the kernel kills the process mid-build — currently within ~2 minutes of startup, meaning the server never finishes building its index and crash-loops indefinitely (nginx's static page cache is the only reason the site mostly still works).Fix
Decode packages one at a time and retain only each package's (comparatively small, fully forced) index entries, filling in reverse dependency counts once all packages have been seen. The resulting index is identical.
Verified against a full clone of the production data set (pursuit-backups): the server now peaks around 1.1GB resident with a fully built index, versus 3.6GB+ and an OOM kill before.
Also in this PR:
pursuit.service: use both vCPUs and cap the heap below available RAM (-N2 -A64m -M3G), so the RTS runs out of heap before the kernel OOM killer steps in.actions/cache@v2andupload-artifact@v2have been disabled by GitHub and no longer run at all) and pin the runner toubuntu-24.04to match the server OS (being upgraded separately).