v0.5.1009 regression: setInterval+async MySQL query loop pegs CPU and wedges Fastify HTTP

[proggeramlug]

Symptom

A 1Hz `setInterval` that ticks an async function reading from MySQL via `@perryts/mysql` wedges the perry runtime:

• CPU pins at 99% steady-state
• `/healthz` (and every other route) times out — Fastify accepts TCP connections but never writes a response
• No `fatal:` error, no crash, no log output during the wedge
• Profile sample (`sample 3`) shows the hot path is `madvise()` syscalls deep inside the perry runtime — GC thrashing

Same shop-admin code, same SQL queries, same `@perryts/mysql` ran cleanly on v0.5.1008 earlier today. v0.5.1009 wedges within 1–2 tick intervals.

Bisect

• Boot completes all 7 steps including `ready`
• HTTP responds for the first few requests within the warm window... actually no, `/healthz` times out from the first request after `ready`
• Commenting out `new JobLoop().start()` in shop-admin's `server/main.ts` → boot is clean, HTTP responds in <1s, CPU 0%
• Re-enabling `new JobLoop().start()` → CPU → 99% within 1s, HTTP wedges

The JobLoop is:

```ts
class JobLoop {
private running = false;
start(): void {
this.timer = setInterval(() => void this.tick(), 1000);
}
private async tick(): Promise {
if (this.running || this.stopping) return;
this.running = true;
try {
const jobs = await claimJobs(this.workerId, 10); // ← @perryts/mysql query w/ FOR UPDATE SKIP LOCKED
for (const job of jobs) await this.runOne(job);
} finally {
this.running = false;
}
}
}
```

`claimJobs` does a `SELECT … FOR UPDATE SKIP LOCKED` then `UPDATE` to atomically grab pending rows. On a fresh DB with no jobs, it returns an empty array.

Minimal repro (didn't reproduce)

```ts
import Fastify from "fastify";
const app = Fastify({ logger: false });
app.get("/healthz", async () => ({ ok: true }));
await app.listen({ host: "127.0.0.1", port: 18099 });

class Loop {
private running = false;
start(): void { setInterval(() => void this.tick(), 1000); }
private async tick(): Promise {
if (this.running) return;
this.running = true;
try { await new Promise((r) => setTimeout(r, 10)); }
finally { this.running = false; }
}
}
new Loop().start();
```

This does not reproduce — CPU 0%, healthz 200. So the trigger is something more specific:

• The MySQL query in the tick (`@perryts/mysql` is a pure-TS driver — compiled by perry, not a native binding)
• Or the volume of code in the surrounding compile unit (shop-admin has ~68 server TS files)
• Or a specific HIR/codegen change in v0.5.1009 that surfaces only at scale

Adding `@perryts/mysql` to my repro might catch it; haven't traced further yet because the bisect was clear and the user's first-Shopware-connect deadline is hot.

v0.5.1008 → v0.5.1009 diff highlights

(from `git log origin/main`)

```
b6b790c fix(hir): #1070 — setErrorHandler err param mis-tagged as fastify Request (#1077)
92f28e2 feat(crypto): wire createCipheriv/createDecipheriv for AES-256-CBC + AES-256-GCM (#1075)
35cf1b4 fix(codegen): refs #420 #618 — narrow v0.5.758 walk-stop to leaf-only with fields
0ee7f07 fix(hir): closes #542 #543 followup — m.keys()/m.values() on Map|undefined narrowed param
bd9e897 llvm-backend: union narrowing fixes — null/undef compare, toString dispatch, strict string concat
430ade1 fix: narrow IndexGet string key detection to avoid regression
```

The HIR narrowing / union narrowing changes are the most likely suspects given the GC-thrash signature.

Sample (perry v0.5.1009 wedged server)

```
2140 thread_start (in libsystem_pthread.dylib)
2140 Thread_: tokio-rt-worker (×4)
2140 Thread_: com.apple.main-thread
… ??? (in server) … ??? (in server) … madvise (in libsystem_kernel.dylib)
```

The deepest leaf is consistently `madvise`. With a stripped binary I can't trace which perry-runtime function is calling `madvise` in a tight loop, but the pattern fits the perry runtime's free-list returning pages to the OS over and over.

Workaround used

Commented out `new JobLoop().start()` in shop-admin's main.ts. Cron schedules are fine to keep registered (they don't fire on the wall clock during a typical test session). This unblocks the first Shopware-6 connect e2e while the regression gets sorted.

Pointer

Either a GC trigger in the HIR-narrowed Map/iter code path, or a tight allocation loop in the new union-narrowing code path. `bd9e8977` and `35cf1b43` are the most likely culprits to revert-test.

Local checkout: perry 0.5.1009, clean main, no edits.

[proggeramlug]

Re-tested against v0.5.1014 with the fix commit e538caa applied. Boot is now clean (the #1113 half of the fix works — app.server.on("upgrade", ...) no longer throws), but the JobLoop CPU-wedge half still reproduces verbatim:

[5/7] attach realtime ws broker      ← no more fatal here ✓ (#1113 fix)
[6/7] start job loop
[7/7] schedule cron jobs
ready

(5s later — same shop-admin JobLoop, same @perryts/mysql claimJobs() tick)
cpu: 98.6%
curl http://127.0.0.1:18080/healthz → timeout after 5s

shop-admin's JobLoop body unchanged from the original report:

class JobLoop {
  private running = false;
  start(): void { setInterval(() => void this.tick(), 1000); }
  private async tick(): Promise<void> {
    if (this.running || this.stopping) return;
    this.running = true;
    try { const jobs = await claimJobs(this.workerId, 10); ... }
    finally { this.running = false; }
  }
}

claimJobs does SELECT ... FOR UPDATE SKIP LOCKED via @perryts/mysql. On an empty jobs table it returns [] after a few ms. With JobLoop disabled, shop-admin runs cleanly on v0.5.1014. Re-enabling it instantly pins CPU.

Whatever e538caa fixed for #1114 didn't address shop-admin's case. The earlier minimal repro I tried (setInterval + setTimeout instead of MySQL) also didn't reproduce — so the trigger almost certainly involves the @perryts/mysql query path inside the tick, not setInterval alone.

Will leave my JobLoop disabled in shop-admin and re-test once a fix lands.

[proggeramlug]

Re-tested against perry v0.5.1016 with commit b799e647 (#1164) applied. Still wedges — the per-tick scratch-Vec fix didn't help shop-admin's @perryts/mysql-backed JobLoop.

=== boot ===                        ← clean, no fatal
[1/6] init MySQL pool
[2/6] run migrations
[3/6] ensure active crypto KID
[4/7] start fastify
[5/7] attach realtime ws broker     ← #1113 half still good
[6/7] start job loop
[7/7] schedule cron jobs
ready

=== CPU sampled every 1s while JobLoop ticks ===
  t=1s  cpu=0.6%       ← idle right after [7/7]
  t=2s  cpu=43.5%      ← first setInterval fire (1s mark)
  t=3s  cpu=97.0%
  t=4s  cpu=99.2%
  t=5s  cpu=98.7%      ← pinned

=== healthz after 5s ===
curl: (28) Operation timed out after 5006 milliseconds with 0 bytes received

So the wedge starts on the very first JobLoop tick — t=1s the first setInterval fires, calls await claimJobs(workerId, 10) against an empty jobs table, that resolves to [] after a few ms, the tick finishes... and then CPU climbs into the 90s and never comes down. healthz hangs from there on.

Two possibly-relevant details for whoever picks this up:

1. The DB is fresh — jobs table empty, no rows to claim. So whatever the wedge is, it's triggered by the empty-result path of claimJobs, not by handler dispatch / row processing.
2. The wedge isn't pre-existing — it's instantly triggered by the first setInterval fire. Disabling jobs.start() keeps CPU at 0% with the same source compile.

shop-admin's claimJobs is in server/persistence/jobs.ts:

export async function claimJobs(workerId: string, limit: number): Promise<Job[]> {
  return tx(async (conn) => {
    const rows = await conn.query<JobRow>(
      `SELECT id, kind, payload, attempts, maxAttempts
         FROM jobs
        WHERE status = 'pending' AND runAfter <= NOW(3)
        ORDER BY runAfter ASC
        LIMIT ? FOR UPDATE SKIP LOCKED`,
      [limit],
    );
    if (rows.length === 0) return [];
    const ids = rows.map((r) => r.id);
    // ... UPDATE the claimed rows
    return rows.map(toJob);
  });
}

(via @perryts/mysql tx() — opens a connection, BEGIN, runs the inner fn, COMMIT/ROLLBACK, releases.)

So the trigger is: open a fresh @perryts/mysql pool connection in a setInterval tick + run a SELECT ... FOR UPDATE SKIP LOCKED that returns no rows + return out of the tx. Repeat at 1Hz. That's apparently enough to drive perry's GC into a tight allocation loop.

If you'd like, I can write a smaller standalone repro (just @perryts/mysql + a setInterval tick that does a single SELECT 1 in a transaction) and post it as a separate gap-suite test — should narrow whether the trigger is the tx() transaction wrapper, the FOR UPDATE SKIP LOCKED clause, or just any @perryts/mysql query inside a setInterval.

Leaving JobLoop disabled in shop-admin until this lands.