crowdsecurity · sabban · Jul 17, 2026
diff --git a/skills/crowdsec/SKILL.md b/skills/crowdsec/SKILL.md
@@ -116,6 +116,7 @@ Hub. If it's far behind `version.crowdsec.net/latest`, pull a newer tag
 | "caddy bouncer", "caddy module / xcaddy" | [references/configure/bouncers/web-servers.md](./references/configure/bouncers/web-servers.md) § Caddy |
 | "wrong source IP", "real client IP", "behind Cloudflare / reverse proxy / NPM", "X-Forwarded-For", "everyone shows as the proxy IP" | [references/configure/bouncers/web-servers.md](./references/configure/bouncers/web-servers.md) — per-bouncer real-IP/trusted-proxy sections |
 | "AppSec", "WAF", "virtual patching", "block by request shape" | [references/appsec/](./references/appsec/) — overview, deploy, configure, troubleshoot |
+| "bot detection", "challenge mode", "SendChallenge", "MatchKnownBot", "proof of work challenge", "appsec-bot-challenge" | [references/appsec/configure.md](./references/appsec/configure.md) § Bot-detection / challenge mode (config) · [references/appsec/deploy.md](./references/appsec/deploy.md) § Bot-detection / challenge mode (install + bouncer wiring) — *early feature, not in a numbered release* |
 | "Console", "enroll", "share signals" | [references/install/console.md](./references/install/console.md) |
 | "upgrade", "back up", "roll back", "new version", "tainted items after upgrade" | [references/operate/upgrades.md](./references/operate/upgrades.md) |
 | "old/outdated version", "`cscli` command or flag missing", "hub item won't install", "behavior doesn't match the docs", "installed from the distro package" | [references/operate/upgrades.md](./references/operate/upgrades.md) § Detect & fix an outdated / distro-packaged install (see **Step 1.5** above) |

diff --git a/skills/crowdsec/references/appsec/configure.md b/skills/crowdsec/references/appsec/configure.md
@@ -4,6 +4,10 @@ verified:
     version: "1.7.8"
     env: systemd
     notes: "appsec-configs/rules list+inspect, metrics rules table; fixed eval-time claim"
+  - date: 2026-07-17
+    version: "1.7.5-174-g66ab61fc-dirty"
+    env: docker
+    notes: "bot-detection/challenge config: challenge: block, SendChallenge/MatchKnownBot/ExemptFromChallenge/RejectSubmission hooks, on_challenge_submit phase, metrics"
 ---
 
 # AppSec — Configure
@@ -97,17 +101,131 @@ Disabling a rule the appsec-config still references will trip `unable to load in
 
 ## Hooks
 
-Hooks let you mutate the request, add context, or short-circuit evaluation. They fire at three phases:
+Hooks let you mutate the request, add context, or short-circuit evaluation. They fire at up to six phases (the last two only exist for bot-detection/challenge mode, see below):
 
 | Phase | When | Typical use |
 |---|---|---|
 | `on_load` | Once at startup. | Hydrate variables, compile regex caches. |
 | `pre_eval` | Before any rule runs against a request. | Inject custom variables from request headers, classify the request, decide if rules should evaluate at all. |
 | `on_match` | After a rule has matched but before the verdict is returned. | Change the action (`ban` → `captcha`), set a custom HTTP response, append context for scenarios. |
-| `post_eval` | After all rules have evaluated. | Log enrichment; rarely modifies the verdict. |
+| `post_eval` | After all rules have evaluated. | Log enrichment; rarely modifies the verdict. Also where the bot-detection challenge gate lives — § Bot-detection / challenge mode below. |
+| `on_challenge` | In-band only. A request carries an already-valid challenge cookie. | Branch on the decoded `fingerprint` object (e.g. force a re-challenge on a mismatch). |
+| `on_challenge_submit` | In-band only. A client POSTs to `/crowdsec-internal/challenge/submit`, after crypto validation. | Reject a cryptographically-valid but suspicious submission (`fingerprint.IsBot()` → `RejectSubmission(...)`). |
 
 Hooks are written in the `expr` language. They are deterministic and must not perform I/O. Errors in hooks bubble to the agent log and (for `pre_eval` / `on_match`) can drop or duplicate a request — test thoroughly with `cscli explain` (where supported) before enabling in production.
 
+## Bot-detection / challenge mode (early feature)
+
+**Not in a numbered release yet.** Engine support merged to crowdsec `master` after v1.7.8 (no
+published canonical docs page as of this writing); the hub collection
+(`crowdsecurity/appsec-bot-challenge`) is still an upstream `[do-not-merge]` PR
+(`crowdsecurity/hub#1826`, branch `test-waf-challenge-mode-scenarios`) — install it via the
+`hub_branch` override, see [../configure/hub.md](../configure/hub.md) § Pinning to a hub
+branch. Only bouncers that understand the structured JSON challenge envelope can render it —
+see [deploy.md](./deploy.md) § Bot-detection / challenge mode for the bouncer-side
+requirement.
+
+This serves visitors a lightweight proof-of-work + browser-fingerprint challenge instead of a
+hard block, then lets solved, non-bot clients through. Configuration lives in a top-level
+`challenge:` block on an appsec-config — combines field-by-field with other loaded configs,
+same as any other appsec-config field:
+
+```yaml
+challenge:
+  master_secret: "<64-char hex, or ≥32-byte passphrase>"   # unset = random, single-instance only
+  key_rotation_interval: 5m    # min 30s; must match across instances in HA
+  max_live_epochs: 3           # past epochs still accepted (slow clients)
+  cookie_ttl: 12h              # independent of key rotation
+```
+
+Leaving `master_secret` unset is fine on a single instance (a random secret is generated at
+startup, invalidating outstanding cookies on every restart). Running more than one AppSec
+instance requires setting `master_secret` and `key_rotation_interval` identically on all of
+them, or cookies minted by one are rejected by the others.
+
+### The core gate
+
+The shipped `crowdsecurity/appsec-bot-challenge-simple` config challenges every in-band
+request unconditionally — known-bot recognition is handled upstream by separate exemption
+configs (below), not by this gate:
+
+```yaml
+inband:
+  post_eval:
+    - filter: "true"
+      apply:
+        - SendChallenge()
+  on_challenge_submit:
+    - filter: "fingerprint.IsBot()"
+      apply:
+        - RejectSubmission("known bot (fast bot detection)")
+```
+
+### Known-bot exemption
+
+Nine opt-in `crowdsecurity/appsec-bot-challenge-exclude-*` configs (search engines, AI
+crawlers, social, monitoring, plus path-based ones for crawler files/feeds/webhooks/static/API
+routes) ship with the collection. Each runs a `pre_eval` hook matching a verified bot against a
+downloaded datafile, then flags the request so the core gate's `SendChallenge()` becomes a
+no-op for it — no cookie minted, re-evaluated every request:
+
+```yaml
+inband:
+  pre_eval:
+    - filter: 'MatchKnownBot(req.RemoteAddr, req.UserAgent(), req.URL.Path, "legit_bots/googlebot.json")'
+      apply:
+        - ExemptFromChallenge("googlebot")
+```
+
+`MatchKnownBot` requires network verification (exact IP, CIDR range, or forward-confirmed
+reverse DNS) in addition to the User-Agent match — a spoofed UA alone never exempts a request.
+To recognize your own bot, write a datafile (newline-delimited JSON, one entry per line) and
+declare it under the config's `data:` block:
+
+```yaml
+data:
+  - dest_file: my-bots.json
+    type: bots
+    source_url: https://example.test/my-bots.json   # cwhub-downloaded; for a local-only file,
+                                                       # bind-mount it into place instead and any
+                                                       # source_url value is fine — it's never
+                                                       # fetched if the file already exists
+```
+
+```json
+{"name":"my-internal-probe","user_agent":"MyProbe/1\.0","ranges":["10.0.0.0/8"]}
+```
+
+At least one of `ips`/`ranges`/`rdns` is required per entry — a User-Agent-only definition is
+rejected at load time (trivially spoofable).
+
+### Challenge-related hook functions
+
+| Function | Phase(s) | Effect |
+|---|---|---|
+| `SendChallenge()` | `post_eval`, `on_challenge` only | Serves the challenge for this request. No-op if the request already carries a valid cookie or was flagged exempt. |
+| `MatchKnownBot(ip, ua, path, datafile...)` | any | `true` if the request matches a bot datafile entry (UA + path preconditions, then IP/range/rDNS verification). |
+| `ExemptFromChallenge(reason)` | any | Flags the request exempt for `SendChallenge()`. Re-evaluated every request — mints no cookie. |
+| `GrantChallengeCookie(reason, ttl?)` | any | Mints a real challenge cookie without a solve — persists across requests until `ttl` (default `cookie_ttl`) expires. |
+| `RejectSubmission(reason)` | `on_challenge_submit` only | Rejects an otherwise crypto-valid submission. |
+| `DumpFingerprint(label)` | any (fingerprint must be populated) | Writes the decoded fingerprint as JSONL to `<datadir>/fingerprint_dumps/crowdsec_fp_dump_<label>.jsonl` — for offline tuning of your own bot rules. |
+
+`fingerprint.IsBot()` (bool) and `fingerprint.BotSignalCount()` (int) are the two most useful
+fields on the `fingerprint` object exposed to `on_challenge`/`on_challenge_submit`.
+
+### Verify
+
+```bash
+cscli metrics show appsec   # "Bot Detection Metrics" table: Requested/Submitted/Solved/Granted/
+                             # Exempt/Protocol Failures/Submissions Rejected/Cookies Invalid,
+                             # plus a per-reason "Bot Detection — Exempted" breakdown
+```
+
+A plain request to a challenged route returns `200` with an HTML challenge page and
+`user_cookies`/`user_headers` in AppSec's JSON envelope — **not** a bare `403` — so test through
+a bouncer that understands the envelope (see deploy.md), not with raw curl against `:7422`
+expecting a block/allow status code.
+
 ## Alerts and scenarios from AppSec
 
 Out-of-band rules emit events with `labels.type: appsec` and matching scenario fields. The hub ships scenarios that consume these — for instance, `crowdsecurity/appsec-virtual-patching` and `crowdsecurity/appsec-crs` aggregate matches into bucketed alerts. Confirm they're installed:

diff --git a/skills/crowdsec/references/appsec/deploy.md b/skills/crowdsec/references/appsec/deploy.md
@@ -8,6 +8,10 @@ verified:
     version: "1.7.8"
     env: k8s
     notes: "k8s with traefik + AppSec"
+  - date: 2026-07-17
+    version: "1.7.5-174-g66ab61fc-dirty"
+    env: docker
+    notes: "bot-detection/challenge collection install via hub_branch; Traefik local-plugin wiring for the structured challenge envelope (maxlerebourg/crowdsec-bouncer-traefik-plugin#343)"
 ---
 
 # AppSec — Deploy
@@ -124,7 +128,7 @@ The smoke test above proves the WAF works. For production you point a real bounc
 | Bouncer | Where to set the AppSec endpoint |
 |---|---|
 | `crowdsec-nginx-bouncer` (lua module) | `APPSEC_URL=http://127.0.0.1:7422` in `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf` (shell-style `KEY=VALUE`, empty by default = WAF off). The self-registered `API_KEY` already serves AppSec — reuse it. |
-| Traefik (`maxlerebourg/crowdsec-bouncer-traefik-plugin`) | Flat plugin options: `crowdsecAppsecEnabled: true` (default false), `crowdsecAppsecHost: <host>:<port>` (no scheme — `crowdsec:7422` in Docker Compose where the container is named `crowdsec`; `<release>-appsec-service.<namespace>.svc.cluster.local:7422` in Kubernetes), and the bouncer key in `crowdsecLapiKey`. In Kubernetes the Middleware `spec.plugin.<key>` must match the key registered in Traefik's `experimental.plugins.<key>` — NOT the module name `crowdsec-bouncer-traefik-plugin`. Full recipe in [../configure/bouncers/web-servers.md](../configure/bouncers/web-servers.md) § Traefik. |
+| Traefik (`maxlerebourg/crowdsec-bouncer-traefik-plugin`) | Flat plugin options: `crowdsecAppsecEnabled: true` (default false), `crowdsecAppsecHost: <host>:<port>` (no scheme — `crowdsec:7422` in Docker Compose where the container is named `crowdsec`; `<release>-appsec-service.<namespace>.svc.cluster.local:7422` in Kubernetes), and the bouncer key in `crowdsecLapiKey`. In Kubernetes the Middleware `spec.plugin.<key>` must match the key registered in Traefik's `experimental.plugins.<key>` — NOT the module name `crowdsec-bouncer-traefik-plugin`. Full recipe in [../configure/bouncers/web-servers.md](../configure/bouncers/web-servers.md) § Traefik. For bot-detection/challenge mode specifically, see § Bot-detection / challenge mode below — the released plugin doesn't parse the structured challenge envelope yet. |
 | `github.com/hslatman/caddy-crowdsec-bouncer` (Caddy module) | Two handlers required in the Caddy route — **`appsec` AND `crowdsec`** (see critical note below). The `appsec_url` field goes in the top-level `crowdsec` app config block. |
 | Any other AppSec-aware bouncer | Look for an `appsec_url` / `appsec.url` field; auth is always the bouncer's existing API key. |
 
@@ -135,6 +139,80 @@ After wiring: send a request through the real web server (not directly to 7422)
 
 See also: [../configure/bouncers/web-servers.md](../configure/bouncers/web-servers.md) for installing the bouncer in the first place.
 
+## Bot-detection / challenge mode (early feature)
+
+**Not in a numbered release yet** — see [configure.md](./configure.md) § Bot-detection /
+challenge mode for the config-side details (the `challenge:` block, hooks, known-bot
+exemption). This section covers installing the collection and the bouncer-side requirement.
+
+### Install
+
+The collection isn't on the released hub index. Point `cscli.hub_branch` at the upstream
+`[do-not-merge]` PR branch first (verified against `crowdsecurity/hub#1826`,
+`test-waf-challenge-mode-scenarios` — see [../configure/hub.md](../configure/hub.md) §
+Pinning to a hub branch for the full command sequence and the revert gotchas), then:
+
+```bash
+sudo cscli hub update
+sudo cscli collections install crowdsecurity/appsec-bot-challenge
+```
+
+This pulls `appsec-bot-challenge-simple` (the config that serves the challenge), all nine
+`appsec-bot-challenge-exclude-*` configs (each downloads its own verified-bot datafiles from
+`hub-data.crowdsec.net` at install time), four scenarios (rate-limiting on
+requested-but-never-solved / too-many-submissions / no-submission patterns, plus a
+bot-detected alert), a parser, and a context. Add the configs you want to your acquisition's
+`appsec_configs:` list alongside your base config (they combine field-by-field — see
+[configure.md](./configure.md) § Acquisition file) and reload.
+
+### Bouncer requirement — structured challenge envelope
+
+Unlike a normal AppSec block, a challenge response is a `403` carrying a JSON envelope
+(`action: "challenge"`, `http_status`, `user_body_content`, `user_headers`, `user_cookies`) that
+the bouncer must parse and relay to the browser verbatim — not just treat as a block. A
+compatible bouncer needs both of:
+
+- **Route the challenge protocol paths through the same AppSec-enabled middleware/route as the
+  protected application** — `/crowdsec-internal/challenge/*` (the challenge page, the
+  proof-of-work JS worker, and the submit endpoint). The bouncer answers these directly from
+  AppSec's envelope; they must never reach the real backend.
+- **Parse the envelope on any `403`** and write `http_status` (default `200` if absent),
+  `user_headers`, and each `user_cookies` entry as its own `Set-Cookie` header (never
+  comma-joined) to the client. An empty-body or non-JSON `403` still falls back to a normal
+  block — this is how older/incompatible bouncers degrade safely.
+
+As of this writing, this parsing isn't in the *released* `maxlerebourg/crowdsec-bouncer-traefik-plugin` —
+only in open PR `#343`. To test it, build the plugin as a Traefik **local plugin** from that
+branch rather than installing the registered catalog entry:
+
+```yaml
+# docker-compose.yml (Traefik service)
+command:
+  - "--experimental.localplugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
+volumes:
+  - ./plugin:/plugins-local/src/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin:ro   # git clone of the PR branch
+```
+
+```yaml
+# protected service labels
+- "traefik.http.routers.router-challenge.rule=PathPrefix(`/crowdsec-internal/challenge`)"
+- "traefik.http.routers.router-challenge.middlewares=crowdsec@docker"
+- "traefik.http.routers.router-challenge.service=<same service as the protected route>"
+```
+
+### Verify
+
+```bash
+curl -sS -D - http://<host>/<protected-path>
+# expect: 200, Content-Type: text/html, X-Remediation: challenge (or whatever
+# remediationheaderscustomname is set to), body is the challenge HTML — NOT a bare 403
+```
+
+A real browser will additionally solve the proof-of-work and reach the backend on the next
+request; curl alone only proves the envelope is being relayed, not a full solve — check
+`cscli metrics show appsec` (see [configure.md](./configure.md) § Verify) for `Ch. Requested`
+incrementing, and `Ch. Accepted` once something has actually solved it.
+
 ## Per-environment notes
 
 | Env | What changes |