From 30abb026a8309aa46e294d00190568073b5dc91f Mon Sep 17 00:00:00 2001 From: Thomas Honeyman Date: Sun, 5 Jul 2026 13:19:38 -0400 Subject: [PATCH] Serialise decoding of large package files Rendering any documentation page decodes the entire docs JSON for that package version, which transiently needs tens of times the file's size in heap. A few generated packages (react-icons, elmish-html, deku, ...) have files of 10MB or more with enormous numbers of rarely-cached documentation pages, so a handful of concurrent crawler requests for those pages could exhaust the heap and restart the server (observed 15 times in the 24 hours after v0.9.10 fixed the search index build's memory use). Decodes of files over 5MB now run one at a time; smaller packages (nearly all of them) are unaffected. The file is only read once the lock is held, so queued requests do not pin file contents, and the queue is bounded: excess requests fail fast with a 503 rather than accumulating while clients time out. The hourly index regeneration is exempt from the bound so the index never silently omits a package. Package pages also no longer decode the same file twice when the requested version is the latest. Verified against the full production data set: a 30-way concurrent burst of requests for a 15MB package's version page kills the previous code (heap exhausted at -M3400m, 3.7GB RSS) and is served by this code as 16 200s and 14 fast 503s with RSS flat at 2.6GB. Also: - robots.txt: disallow SEO/bulk crawlers (Semrush, Ahrefs, DotBot, MJ12, Amazonbot, Bytespider, PetalBot), which generate most of the long-tail page traffic - Evict page-cache files not accessed in 90 days via a weekly cron job installed by the deploy, so the cache cannot fill the disk - Raise the heap limit to -M3400m now that the server has swap - Add deploy/SERVER.md documenting server-only and DO account state --- CHANGELOG.md | 29 +++++++++++++++ config/robots.txt | 25 +++++++++++++ deploy/SERVER.md | 57 ++++++++++++++++++++++++++++ deploy/cache-eviction.sh | 24 ++++++++++++ deploy/pursuit.service | 9 +++-- deploy/remote.sh | 3 ++ pursuit.cabal | 2 +- src/Application.hs | 2 + src/Foundation.hs | 6 +++ src/Handler/Database.hs | 80 ++++++++++++++++++++++++++++++++++++---- src/Handler/Packages.hs | 7 +++- 11 files changed, 231 insertions(+), 13 deletions(-) create mode 100644 deploy/SERVER.md create mode 100755 deploy/cache-eviction.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 85edde9..3570528 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,35 @@ the most up-to-date version of this file. ## Unreleased +## v0.9.11 + +- Serialise decoding of large package files (@thomashoneyman) + + Rendering any documentation page decodes the entire docs JSON for that + package version, which transiently needs tens of times the file's size in + heap. A few generated packages (react-icons, elmish-html, deku, ...) have + files of 10MB or more with enormous numbers of rarely-cached documentation + pages, so a handful of concurrent crawler requests for those pages could + exhaust the heap and restart the server. Decodes of files over 5MB now run + one at a time; smaller packages (nearly all of them) are unaffected. The + file is only read once the lock is held, so queued requests do not pin file + contents, and the queue is bounded (excess requests fail fast with a 503 + rather than accumulating while clients time out). The hourly index + regeneration is exempt from the bound so the index never silently omits a + package. + +- Do not decode a package version's file twice when rendering the package + page for the latest version (@thomashoneyman) +- Disallow SEO/bulk crawlers (Semrush, Ahrefs, DotBot, MJ12, Amazonbot, + Bytespider, PetalBot) in robots.txt (@thomashoneyman) +- Evict page-cache files not accessed in 90 days via a weekly cron job + installed by the deploy, so the cache cannot fill the disk + (@thomashoneyman) +- Raise the heap limit to `-M3400m` now that the server has swap to absorb + transient spikes above physical RAM (@thomashoneyman) +- Add `deploy/SERVER.md` documenting server-only and DigitalOcean account + state (@thomashoneyman) + ## v0.9.10 - Build the search index one package at a time (@thomashoneyman) diff --git a/config/robots.txt b/config/robots.txt index 7d329b1..6c5308a 100644 --- a/config/robots.txt +++ b/config/robots.txt @@ -1 +1,26 @@ User-agent: * +Disallow: + +# SEO and bulk crawlers provide no value to a documentation site and generate +# most of the traffic to the enormous long tail of uncached package pages, +# each of which is expensive to render (see Handler.Database.lookupPackage). +User-agent: SemrushBot +Disallow: / + +User-agent: AhrefsBot +Disallow: / + +User-agent: DotBot +Disallow: / + +User-agent: MJ12bot +Disallow: / + +User-agent: Amazonbot +Disallow: / + +User-agent: Bytespider +Disallow: / + +User-agent: PetalBot +Disallow: / diff --git a/deploy/SERVER.md b/deploy/SERVER.md new file mode 100644 index 0000000..9818465 --- /dev/null +++ b/deploy/SERVER.md @@ -0,0 +1,57 @@ +# Server notes for pursuit.purescript.org + +This file documents the state that lives only on the server or in the +DigitalOcean account, which no deploy touches and which would otherwise be +tribal knowledge. Last verified: 2026-07-05. + +## The droplet + +- DigitalOcean droplet 177127298 (`pursuit.purescript.org`), s-2vcpu-4gb, + nyc1, IP 206.189.189.151. +- Ubuntu 24.04 +- DNS for purescript.org is managed at Namecheap, not DigitalOcean. +- The CI runner in `.github/workflows/ci.yml` must build on the same Ubuntu + release the server runs, or the binary's glibc requirement won't be + satisfiable. Bump both together. + +## Server-only state (not managed by deploys) + +- **Swap**: 4GB `/swapfile`, registered in `/etc/fstab`, with + `vm.swappiness=10` set in `/etc/sysctl.d/99-swappiness.conf`. +- **Service enablement**: `pursuit` and `nginx` are `systemctl enable`d. +- **Backups**: root crontab runs `deploy/backup.sh` daily at 13:00 UTC, + copying `data/verified/` (the actual database) to `/var/www/pursuit-backups` + and pushing to the `purescript/pursuit-backups` GitHub repo. The page cache + (`data/cache/`) is *not* backed up; it is pure derived data. +- **nginx reload cron**: root crontab reloads nginx daily at 12:00 UTC to pick + up renewed TLS certificates. +- **TLS**: certbot (apt package) with the systemd `certbot.timer`; webroot + under `/var/www/letsencrypt-webroot`. +- **Cache eviction**: `/etc/cron.weekly/pursuit-cache-eviction` (installed by + `remote.sh` from `deploy/cache-eviction.sh`) deletes cache files not + accessed in 90 days so the cache cannot fill the disk. + +## DigitalOcean account state + +- **Uptime check** `pursuit-search-backend` targets + `https://pursuit.purescript.org/search?q=maybe` from us_east and eu_west. + The target is deliberately a search URL: search always exercises the + backend, whereas package pages are served by nginx from the page cache + even when the backend is down (which is how the 2025-26 OOM crash loop + went unnoticed for months). One alert: down for 2+ minutes. +- **Alert policy**: droplet memory utilization above 90% sustained for one + hour. This is the early warning that the package database is outgrowing + the droplet again. +- Alerts email *****@thomashoneyman.com. + +## Memory background + +The whole-package docs JSON is the unit of storage; rendering any single +documentation page decodes the entire file for that package version, which +transiently needs tens of times the file size in heap. A few generated +packages (react-icons, elmish-html, deku, google-apps, ...) have 10-25MB +files, so concurrent uncached requests for their pages can exhaust the heap. +`Handler.Database.lookupPackage` therefore serialises decodes of files over +5MB, and the search index is built one package at a time +(`createSearchIndexFromDatabase`). If heap-exhaustion restarts reappear in +the journal (`journalctl -u pursuit | grep "Heap exhausted"`), start there. diff --git a/deploy/cache-eviction.sh b/deploy/cache-eviction.sh new file mode 100755 index 0000000..dc9bc89 --- /dev/null +++ b/deploy/cache-eviction.sh @@ -0,0 +1,24 @@ +#!/bin/sh + +# Evict pursuit page-cache files that nobody has accessed in 90 days, so that +# the cache (which otherwise grows without bound as crawlers walk the long +# tail of package documentation pages) cannot fill the disk. +# +# This is safe: the cache is pure derived data. Everything in it is rendered +# from /var/www/pursuit/data/verified, nginx treats a missing cache file as +# "proxy to the backend", and an evicted page is simply re-rendered and +# re-cached on the next request for it. Backups do not include the cache. +# +# The finds tolerate errors: package uploads clear cache directories +# concurrently (Handler.Caching.clearCache), so files can vanish mid-walk. +# +# Installed to /etc/cron.weekly by deploy/remote.sh. + +CACHE_DIR=/var/www/pursuit/data/cache + +[ -d "$CACHE_DIR" ] || exit 0 + +find "$CACHE_DIR" -type f -atime +90 -delete 2>/dev/null || true +find "$CACHE_DIR" -mindepth 1 -type d -empty -delete 2>/dev/null || true + +exit 0 diff --git a/deploy/pursuit.service b/deploy/pursuit.service index 8963ca1..1570ae8 100644 --- a/deploy/pursuit.service +++ b/deploy/pursuit.service @@ -4,10 +4,11 @@ Description=Web service for hosting of PureScript API documentation [Service] Type=simple User=www-data -# The server has 2 vCPUs and 4GB of RAM. The heap limit needs to leave room -# for nginx and the rest of the system, so that the RTS runs out of heap -# before the kernel's OOM killer steps in. -ExecStart=/usr/local/bin/pursuit +RTS -N2 -A64m -M3G -RTS +# The server has 2 vCPUs, 4GB of RAM and 4GB of swap. The heap limit needs to +# leave room for nginx and the rest of the system; spikes above physical RAM +# are absorbed by swap, and the RTS runs out of heap before the kernel's OOM +# killer would step in. +ExecStart=/usr/local/bin/pursuit +RTS -N2 -A64m -M3400m -RTS Restart=always RestartSec=5s Environment="PURSUIT_APPROOT=https://pursuit.purescript.org" diff --git a/deploy/remote.sh b/deploy/remote.sh index 4b549f3..93c0550 100755 --- a/deploy/remote.sh +++ b/deploy/remote.sh @@ -56,6 +56,9 @@ rm -r "$tmpdir" cp /var/www/pursuit/deploy/nginx.conf /etc/nginx/sites-enabled/pursuit.conf systemctl reload nginx +# install cache eviction cron job +install -m 755 /var/www/pursuit/deploy/cache-eviction.sh /etc/cron.weekly/pursuit-cache-eviction + # install systemd service confing cp /var/www/pursuit/deploy/pursuit.service /etc/systemd/system/pursuit.service systemctl daemon-reload diff --git a/pursuit.cabal b/pursuit.cabal index a33cebb..4a67c9f 100644 --- a/pursuit.cabal +++ b/pursuit.cabal @@ -1,5 +1,5 @@ name: pursuit -version: 0.9.10 +version: 0.9.11 cabal-version: >= 1.8 build-type: Simple license: MIT diff --git a/src/Application.hs b/src/Application.hs index 562a872..02ce7fc 100644 --- a/src/Application.hs +++ b/src/Application.hs @@ -59,6 +59,8 @@ makeFoundation appSettings = do appHttpManager <- newManager appLogger <- newStdoutLoggerSet defaultBufSize >>= makeYesodLogger appSearchIndex <- newTVarIO emptySearchIndex + appLargeDecodeLock <- newMVar () + appLargeDecodeWaiters <- newTVarIO 0 let foundation = App{..} void (startRegenThread foundation) return foundation diff --git a/src/Foundation.hs b/src/Foundation.hs index a681a4c..25f6d9d 100644 --- a/src/Foundation.hs +++ b/src/Foundation.hs @@ -68,6 +68,12 @@ data App = App , appHttpManager :: Manager , appLogger :: Logger , appSearchIndex :: TVar SearchIndex + , appLargeDecodeLock :: MVar () + -- ^ Held while decoding a large package's JSON; see + -- 'Handler.Database.lookupPackage'. + , appLargeDecodeWaiters :: TVar Int + -- ^ The number of threads holding or waiting for 'appLargeDecodeLock', + -- used to bound the queue. } instance HasHttpManager App where diff --git a/src/Handler/Database.hs b/src/Handler/Database.hs index 4c2306a..c468871 100644 --- a/src/Handler/Database.hs +++ b/src/Handler/Database.hs @@ -19,7 +19,7 @@ import qualified Data.NonNull as NN import qualified Data.Text as T import Data.Version (Version, showVersion) import System.Directory - (getDirectoryContents, getModificationTime, doesDirectoryExist) + (getDirectoryContents, getModificationTime, doesDirectoryExist, getFileSize) import Web.Bower.PackageMeta (PackageName, mkPackageName, runPackageName) import qualified Language.PureScript.Docs as D @@ -68,7 +68,7 @@ createSearchIndexFromDatabase = do case mversion of Nothing -> return Nothing Just version -> do - mpkg <- hush <$> lookupPackage name version + mpkg <- hush <$> lookupPackageWithPolicy WaitWhenBusy name version case mpkg of Nothing -> return Nothing Just pkg -> do @@ -89,17 +89,83 @@ data SomethingMissing deriving (Show, Eq, Ord) lookupPackage :: PackageName -> Version -> Handler (Either SomethingMissing D.VerifiedPackage) -lookupPackage pkgName version = do +lookupPackage = lookupPackageWithPolicy FailWhenBusy + +data LargeDecodePolicy = FailWhenBusy | WaitWhenBusy + +lookupPackageWithPolicy :: LargeDecodePolicy -> PackageName -> Version -> Handler (Either SomethingMissing D.VerifiedPackage) +lookupPackageWithPolicy policy pkgName version = do file <- packageVersionFileFor pkgName version - mcontents <- liftIO (readFileMay file) - case mcontents of - Just contents -> - Right <$> decodeVerifiedPackageFile file contents + msize <- liftIO (catchDoesNotExist (getFileSize file)) + mpkg <- case msize of + Nothing -> return Nothing + Just size + | size >= largeDecodeThreshold -> + withLargeDecodeLock policy (decodeFrom file) + | otherwise -> + decodeFrom file + case mpkg of + Just pkg -> return (Right pkg) Nothing -> do -- Work out whether there's no such package or just no such version dir <- packageDirFor pkgName dirExists <- liftIO $ doesDirectoryExist dir return $ Left $ if dirExists then NoSuchPackageVersion else NoSuchPackage + where + -- The file is read inside the lock (rather than before queueing for it) so + -- that threads waiting their turn do not each pin a copy of a large file's + -- bytes. The decoded package is forced before the lock is released; the + -- decode is eager in practice, but we make certain the allocation spike + -- stays inside the lock. + decodeFrom :: String -> Handler (Maybe D.VerifiedPackage) + decodeFrom file = do + mcontents <- liftIO (readFileMay file) + case mcontents of + Nothing -> return Nothing + Just contents -> do + pkg <- decodeVerifiedPackageFile file contents + pkg `deepseq` return (Just pkg) + +-- | Decoding a package's docs JSON transiently needs tens of times the file's +-- size in memory, and a few generated packages (react-icons, elmish-html, +-- deku, ...) have files of 10MB or more, so a handful of concurrent requests +-- for their (rarely cached, because there are so many of them) documentation +-- pages can exhaust the heap. Decodes of large files therefore run one at a +-- time; packages below the threshold - nearly all of them - are unaffected. +-- +-- The queue for the lock is bounded: once 'maxLargeDecodeWaiters' threads +-- hold or await the lock, further requests fail fast with a 503 rather than +-- accumulating without limit while clients time out. The hourly search index +-- regeneration ('WaitWhenBusy') is exempt from the bound - it must not +-- silently omit a package from the index just because the server is busy, and +-- as a single sequential thread it adds at most one waiter. +withLargeDecodeLock :: LargeDecodePolicy -> Handler a -> Handler a +withLargeDecodeLock policy action = do + app <- getYesod + let counter = appLargeDecodeWaiters app + admitted <- case policy of + WaitWhenBusy -> do + atomically (modifyTVar' counter (+ 1)) + return True + FailWhenBusy -> atomically $ do + n <- readTVar counter + if n >= maxLargeDecodeWaiters + then return False + else do + writeTVar counter (n + 1) + return True + if admitted + then + withMVar (appLargeDecodeLock app) (const action) + `finally` atomically (modifyTVar' counter (subtract 1)) + else + sendResponseStatus serviceUnavailable503 + ("Too many concurrent requests for large packages; try again shortly." :: Text) + where + maxLargeDecodeWaiters = 16 :: Int + +largeDecodeThreshold :: Integer +largeDecodeThreshold = 5 * 1024 * 1024 availableVersionsFor :: PackageName -> Handler [Version] availableVersionsFor pkgName = do diff --git a/src/Handler/Packages.hs b/src/Handler/Packages.hs index e26c967..b450a46 100644 --- a/src/Handler/Packages.hs +++ b/src/Handler/Packages.hs @@ -219,7 +219,12 @@ findPackageWithLatest :: findPackageWithLatest pkgName version cont = do findPackage pkgName version $ \pkg -> do latestVersion <- fromMaybe version <$> getLatestVersionFor pkgName - latestPkg <- fromMaybe pkg . hush <$> lookupPackage pkgName latestVersion + -- Avoid decoding the same package file twice when the requested version + -- is the latest one; for large packages a decode is expensive. + latestPkg <- + if latestVersion == version + then return pkg + else fromMaybe pkg . hush <$> lookupPackage pkgName latestVersion cont pkg latestPkg packageNotFound :: PackageName -> Handler a