feat(pokestop): consume Golbat /api/pokestop/available with endpoint-authoritative fallback (draft)

[jfberry]

Summary

Adds a ReactMap consumer for Golbat's new GET /api/pokestop/available (Golbat draft PR UnownHash/Golbat#383). A scanner source that has a Golbat endpoint now fetches the pokestop filter/available list from Golbat instead of running ~30 DISTINCT/GROUP BY SQL queries every 15 minutes — mirroring how Pokemon.getAvailable already consumes /api/pokemon/available. Everything else about pokestops (map markers, popups, search, submissions) is unchanged.

A source can now be dual (endpoint + DB creds): the migrated query (getAvailable) uses the endpoint, while un-migrated queries fall back to this.query() on the bound DB — so a Golbat pokestop endpoint is safe to enable without touching those features.

How it works

• server/src/models/pokestopAvailableMapper.js (new) — a pure mapper reproducing the SQL path's filter-key formulas exactly (quest rewards q/d/p/m/c/x/u + pokémon, invasions i/b, confirmed rocket a, lures l, showcases f/h, plus per-reward conditions). Its process() helper is byte-identical to the SQL path's.
• Pokestop.getAvailable wiring — an if (mem) endpoint branch that fetches, validates the response shape, and maps it. On failure (503 / fort_in_memory off, network error) it falls through to the SQL block: a dual source runs the SQL fallback on its bound knex; a pure-endpoint source has no knex, so it's dropped by the caller's Promise.allSettled.
• Dual endpoint+DB sources (DbManager) — a schema carrying both endpoint and DB creds registers the endpoint and builds a knex connection; the endpoint context (mem/secret/httpAuth) is overlaid onto the schema-checked DB context. Migrated methods read mem; un-migrated ones use the bound DB. Pure-endpoint and pure-DB schemas behave exactly as before.
• applyRocketPokemonFallback (shared helper) — the config-derived rocket-pokémon a-keys (from state.event.invasions[*].encounters, gated by fallbackRocketPokemonFiltering, default on) are emitted from one helper used by both the SQL and endpoint paths, so those keys are identical by construction.
• Types — AvailablePokestops (+ members); ApiEndpoint.httpAuth typed; DbConnection gains optional endpoint/secret/httpAuth for the dual shape.

Configuring & testing

Requires the Golbat side up: [general] fort_in_memory = true and the API secret matching ReactMap's. Sanity-check Golbat directly:

curl -H "X-Golbat-Secret: <secret>" http://<golbat-host>:<port>/api/pokestop/available   # 200 + {quests,invasions,lures,showcases}

Most instances already run Golbat for pokémon: a scanner DB source serving ["gym","pokestop","spawnpoint",...] plus a separate pure-endpoint golbat source for ["pokemon","device"]. To route the pokestop available list through Golbat while keeping pokestop markers/popups/search on the DB, just add the two endpoint fields to your existing scanner DB source (the one whose useFor includes pokestop):

// config/local.json → database.schemas[]  (your existing scanner DB source)
{
  "host": "127.0.0.1", "port": 3306,
  "username": "...", "password": "...", "database": "golbat_db",
  "endpoint": "http://{golbat_address}:{golbat_port}",   // ← add (same as your pokemon golbat source)
  "secret": "<golbat api secret>",                       // ← add
  "useFor": ["gym", "pokestop", "spawnpoint", "weather", "station", "..."]
}

That's the whole change — the source becomes dual. Only pokestop consumes the endpoint (getAvailable); gym/station/etc. don't check mem, so they keep using the DB; and pokestop markers/popups/search/submissions fall back to this.query() on the same DB. Your pure-endpoint pokémon source is untouched. (If you'd rather scope it tighter, split a useFor: ["pokestop"]-only copy of the DB source with the endpoint added and remove pokestop from the original — but it isn't necessary.)

Behavior:

• pokestop filter options ← Golbat /api/pokestop/available (offloads the ~30-query, 15-min SQL)
• pokestop markers / popups / search / submissions ← the bound DB (unchanged)
• Golbat 503 → getAvailable falls back to SQL on the DB

Verify: filter options populate and markers/popups/search still work; cross-check the filter list against your pre-change list (the form_id 0 / type-20 golden check, on live data).

Confirming the endpoint is live (logs): on each ~15-min refresh, ReactMap logs a genuine source-aware line when the pokestop available list comes from Golbat:

ℹ [POKESTOPS] [POKESTOP] loaded available from Golbat endpoint http://<golbat>:<port>/api/pokestop/available — 245 filter keys (245 quests, 6 invasions, 11 lures, 3 showcases), 215 reward conditions

Note the pre-existing [DB] Querying available for Pokestop line is emitted by the DbManager fan-out before any source picks endpoint-vs-SQL, so it does not indicate which path ran — the [POKESTOPS] loaded … from Golbat endpoint line does. If the endpoint is down, you instead get a [POKESTOPS] [POKESTOP] /api/pokestop/available unavailable/error … warn and it falls back to SQL. (A pure-DB source stays silent, as before.)

Verification

Per maintainer preference, no test suite was added (this repo has none). Correctness was established by a line-by-line golden comparison of the mapper against the SQL key-building (all reward-type formulas, the -form rule, the u-set, i/b, showcases, lures, with_ar merge confirmed to match), throwaway sanity scripts, an end-to-end output-shape check (mapper output is structurally identical to the SQL return and consumed identically by DbManager.getAvailable + filters/builder/pokestop.js), and eslint/prettier clean (adding the dual-source types reduced the pre-existing tsc error count by one).

Residual golden check (before enabling in prod)

Diff mapAvailablePokestops(liveGolbatResponse) vs the SQL getAvailable on the same data: form_id === 0 → bare <id>; type-20 real data (m<id> vs u20); a-key slot 2/3 when fallbackRocketPokemonFiltering is off. If any diverge, the cleaner fix is Golbat-side (convey null form / a type-20 sentinel) — coordinate via #383.

Follow-ups (deferred, non-blocking)

• Extract a shared evalQuery util — Pokestop.evalQuery is a ~47-line copy of Pokemon.evalQuery (task-scoping artifact).
• Optionally migrate getOne (simple, endpoint exists) / getAll (complex, needs incidents added to Golbat's scan) to shave DB load — not needed for correctness thanks to the dual-source DB fallback.
• fetchJson logs the request payload (incl. base64 auth header) on every 503 — pre-existing, worth quieting.

Depends on

Golbat UnownHash/Golbat#383 (the /api/pokestop/available endpoint). Draft until that lands and the golden check is run.

🤖 Generated with Claude Code

[jfberry]

Status of this branch: does not operate in its current form as only the get available branch is ready; examining whether we should implement other api calls for pokestops here, or whether to allow a hybrid option

[jfberry]

cd ~/ReactMap
# sanity: confirm the file + a pokestops array (find it if the path differs)
jq '.pokestops | length' server/.cache/available.json    # or: find . -name available.json

# fresh endpoint snapshot (so both are close in time)
curl -sS -H "X-Golbat-Secret: whatever" http://localhost:9001/api/pokestop/available > /tmp/avail.json

# diff: SQL (available.json) vs endpoint (mapper)
node -e '
  const fs=require("fs");
  const {mapAvailablePokestops}=require("./server/src/models/pokestopAvailableMapper");
  const sql=new Set(JSON.parse(fs.readFileSync("server/.cache/available.json","utf8")).pokestops);
  const ep =new Set(mapAvailablePokestops(require("/tmp/avail.json"),{invasions:{}}).available);
  const noA=s=>new Set([...s].filter(k=>k[0]!=="a"));   // a-keys need real event config; skip here
  const S=noA(sql), E=noA(ep);
  console.log("SQL",S.size," endpoint",E.size);
  console.log("only in SQL   :",[...S].filter(k=>!E.has(k)).sort().join(" ")||"(none)");
  console.log("only in endpt :",[...E].filter(k=>!S.has(k)).sort().join(" ")||"(none)");
'

output:

SQL 245  endpoint 245
only in SQL   : (none)
only in endpt : (none)

[jfberry]

How to configure ReactMap to use this

The pokestop available/filter list is served by Golbat's GET /api/pokestop/available; everything else about pokestops (map markers, popups, search, submission cells) still runs on the DB. So the scanner source needs both a Golbat endpoint and DB creds — a "dual" source (supported by the DbManager change in this PR).

Golbat side

• [general] fort_in_memory = true (otherwise the endpoint returns 503)
• API secret matching what ReactMap sends — the same one you already use for /api/pokemon/*
• Sanity check:

  curl -H "X-Golbat-Secret: <secret>" http://<golbat-host>:<port>/api/pokestop/available
  # 200 + {"quests":[…],"invasions":[…],"lures":[…],"showcases":[…]}

ReactMap side

Most instances already run Golbat for pokémon: a scanner DB source serving ["gym","pokestop","spawnpoint",…] plus a separate pure-endpoint golbat source for ["pokemon","device"]. To route the pokestop available-list through Golbat, just add the two endpoint fields to your existing scanner DB source (the one whose useFor includes pokestop), in config/local.json → database.schemas[]:

{
  "host": "127.0.0.1", "port": 3306,
  "username": "…", "password": "…", "database": "golbat_db",
  "endpoint": "http://{golbat_address}:{golbat_port}",   // ← add (same URL as your pokemon golbat source)
  "secret": "<golbat api secret>",                       // ← add
  "useFor": ["gym", "pokestop", "spawnpoint", "weather", "station", "…"]
}

That single change makes the source dual:

• pokestop filter options ← Golbat /api/pokestop/available (offloads the ~30-query, 15-min SQL)
• pokestop markers / popups / search / submissions ← the bound DB (unchanged)
• gym/station/spawnpoint/etc. ignore the endpoint (their queries don't check mem) → keep using the DB
• Golbat 503 (fort_in_memory off) → getAvailable falls back to SQL on the same DB

Your existing pure-endpoint pokémon source is untouched. Restart ReactMap after the config change.

(If you prefer tighter scoping, instead split off a useFor: ["pokestop"]-only copy of the DB source with the endpoint added and drop pokestop from the original — not necessary, since gym/station ignore mem.)

Verify (optional golden check)

Filter options should populate and markers/popups/search should still work. To confirm the endpoint-derived filter list matches the SQL one on the same forts:

cd ~/ReactMap
curl -sS -H "X-Golbat-Secret: <secret>" http://<golbat>:<port>/api/pokestop/available > /tmp/avail.json
# available.json only flushes on graceful shutdown — restart ReactMap first so it's a fresh SQL snapshot
node -e '
  const fs=require("fs");
  const {mapAvailablePokestops}=require("./server/src/models/pokestopAvailableMapper");
  const sql=new Set(JSON.parse(fs.readFileSync("server/.cache/available.json","utf8")).pokestops);
  const ep =new Set(mapAvailablePokestops(require("/tmp/avail.json"),{invasions:{}}).available);
  const noA=s=>new Set([...s].filter(k=>k[0]!=="a")); // a-keys need real event config; skip
  const S=noA(sql), E=noA(ep);
  console.log("SQL",S.size," endpoint",E.size);
  console.log("only in SQL   :",[...S].filter(k=>!E.has(k)).sort().join(" ")||"(none)");
  console.log("only in endpt :",[...E].filter(k=>!S.has(k)).sort().join(" ")||"(none)");
'

On live production data this matched byte-for-byte (245/245, no diffs). Any remaining diffs after a fresh restart are just time-sensitive i/l keys between the two pulls.

Requires Golbat UnownHash/Golbat#383 (the /api/pokestop/available endpoint) deployed with fort_in_memory on.

[jfberry]

This is working in my production environment

[jfberry]

This is now ready for review @Mygod - I'm building a second PR which does a DNF lookup of the gyms / pokestops / stations but this first PR solves a real problem with the 15-minute entire database scan so I think worth landing first if you are comfortable with this approach

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b07316de5b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".