fix: add NOT EXISTS guard to stop notification_id_seq burn on repeated job runs

[dylanjeffers]

Problem

The notification_id_seq column hit INT max (2³¹−1) on July 29 because all four sub-steps in RemixContestNotificationsJob re-scan the same events on every 30-second tick and attempt duplicate INSERTs.

Postgres allocates a sequence ID before checking ON CONFLICT DO NOTHING, so every wasted attempt permanently burns an ID:

Step	Window	Re-scans/event/day	Burn multiplier
fan_remix_contest_ending_soon	72 h	8,640	× N fan recipients
artist_remix_contest_ending_soon	48 h	5,760	× 1 (host only)
fan_remix_contest_ended	24 h	2,880	× N fan recipients
artist_remix_contest_ended	24 h	2,880	× 1 (host only)

July 29 spike: The Summer Cypher contest (1.97 M followers) entered the 72-hour ending-soon window. Each 30-second tick attempted 1.97 M INSERTs. After ~543 ticks (~4.5 h), ~1.07 B sequence IDs were burned — exhausting INT range and taking down notifications for all users.

Chronic baseline (~18 M IDs/day burned): Smaller active contests (combined fan audiences on the order of a few thousand) still burned millions of IDs daily through the same mechanism.

Fix

Add an event-level NOT EXISTS guard to each of the four SQL blocks:

AND NOT EXISTS (
    SELECT 1 FROM notification
    WHERE group_id = 'fan_remix_contest_ending_soon:' || e.event_id::text
)

Once any notification row exists for an event (i.e., after the first successful fan-out pass), subsequent job runs skip that event entirely. The per-recipient ON CONFLICT DO NOTHING is kept as a safety net for partial-failure retries.

Effect: Each contest triggers at most one fan-out pass instead of thousands. Sequence consumption drops from O(recipients × job_runs_in_window) to O(recipients × 1) per event.

What this does NOT change

• No user-visible notification is dropped. The guard only prevents re-attempting INSERTs that would conflict anyway.
• The bigint migration already deployed by Ray stays in place as the unblock. This PR stops the source of future exhaustion.

Testing

Existing TestRemixContest_Ended already asserts idempotency (runs the job twice and verifies the count stays the same). fanEndingSoon idempotency is implicitly covered: after the first run the NOT EXISTS check short-circuits, so no inserts are attempted on re-runs.

[dylanjeffers]

Updated approach: CronJob as primary fix

This PR has been updated. The primary fix is now converting RemixContestNotificationsJob from a 30-second goroutine to a Kubernetes CronJob running every 20 minutes. The NOT EXISTS guard stays as belt-and-suspenders.

Why CronJob instead of just NOT EXISTS?

The NOT EXISTS guard fixes burn-per-run, but the job still polled every 30 seconds against the same 24-72h windows — 2,880 no-op queries/day per active contest. The CronJob cuts this to 72/day (40x reduction).

What changed in this update:

• jobs/create_remix_contest_notifications.go — added RunE(ctx) error for the binary
• indexer/indexer.go — removed RemixContestNotificationsJob.ScheduleEvery
• cmd/remix_contest_notifications/main.go — new binary: connects to DB, runs job once, exits non-zero on error
• Dockerfile — builds /bin/remix-contest-notifications alongside /bin/bridge
• k8s/remix-contest-notifications-cronjob.yaml — CronJob spec (20-min schedule, concurrencyPolicy: Forbid)

Deploy note: Update namespace, image tag, and secretRef in the k8s manifest to match your cluster before applying.