Skip to content

fix(etl): playlist_seen PK, ErrNoRows on soft-deletes, attribute dispatch errors - #310

Merged
raymondjacobson merged 1 commit into
mainfrom
etl/playlist-seen-and-no-rows-followup
May 22, 2026
Merged

fix(etl): playlist_seen PK, ErrNoRows on soft-deletes, attribute dispatch errors#310
raymondjacobson merged 1 commit into
mainfrom
etl/playlist-seen-and-no-rows-followup

Conversation

@raymondjacobson

Copy link
Copy Markdown
Contributor

Summary

Follow-ups from the overnight prod-clone validation run (38,834 chain blocks, 69 deep diff cycles against the prod read replica). PR #308's fixes held up cleanly — these are the remaining two real findings + one observability gap, all surgical.

Changes

1. playlist_seen primary key mismatch with prod

Our 0005_notification_tables.up.sql declared the PK as (is_current, user_id, playlist_id, seen_at). Prod's schema has (user_id, playlist_id, seen_at) — no is_current. This caused an asymmetric bug:

  • The handler's ON CONFLICT (is_current, user_id, playlist_id, seen_at) matched our in-house PK but silently mismatched prod.
  • The fixed version ON CONFLICT (user_id, playlist_id, seen_at) matches prod but breaks against our in-house PK with 42P10 no unique or exclusion constraint matching.

Observed firing 7 times in the 11h validation run. Fix:

  • 0005_notification_tables.up.sql: declare the prod-correct PK on fresh databases.
  • 0027_fix_playlist_seen_pkey.up.sql (new): DROP+ADD the PK on databases that already ran the broken 0005.
  • notification.go:100: drop is_current from the ON CONFLICT target.

2. loadCurrentTrackRow / loadCurrentPlaylistRow propagated raw pgx.ErrNoRows on soft-deleted rows

validateTrackUpdate / validatePlaylistUpdate use trackExists / playlistExists which accept is_delete = true rows. The row-loaders loadCurrentTrackRow and loadCurrentPlaylistRow filter is_delete = false — so a track that was soft-deleted between validation and update returns raw pgx.ErrNoRows, which the dispatcher then logs at ERROR level as no rows in result set.

Observed firing 14 times during the 11h run, clustered with Track/Repost "already exists" rejections during follow surges — strongly correlated with high-activity windows where soft-deletes and updates race.

Wrap ErrNoRows in both row-loaders as a NewValidationError("X N is deleted or does not exist") so the dispatcher logs at WARN, not ERROR.

3. getUserHandle race-window ErrNoRows

Same shape as #2validateUserUpdate checks userExists, but if the user disappears in the narrow window before getUserHandle runs, the raw ErrNoRows leaks out. Fix: treat ErrNoRows as empty handle (callers compare against the new handle).

4. Attribute dispatch errors to a handler — observability fix

The dispatch-error log line at indexer.go:425 had no context fields, so when one fired we could see no rows in result set but had no clue which handler emitted it. Made #2 hard to attribute — only discovered via correlation analysis after multi-hour runs.

Add entity_type / action / entity_id / user_id / hash to the log so future occurrences are self-attributing. Mirrors the fields already on the WARN line for ValidationErrors.

Test plan

  • go build ./... + go vet ./... clean
  • Full go test ./pkg/etl/... against fresh Postgres: all green (including the previously-affected TestPlaylistSeenView_Success)
  • TestPlaylistSeenView_Success now exercises the new PK on the test side and passes
  • TestPlaylistUpdate_Success (which exercises loadCurrentPlaylistRow) still passes
  • No new test failures introduced

Open follow-ups (out of scope, called out in run findings)

  • Shares semantic mismatch with prod — we write ~5× more share rows than prod's python indexer. Product call (not an indexer bug).
  • Album vs playlist disambiguation — PR fix(etl): four prod-clone-run bugs + #307 test fixes folded in #308's is_album check produces save_type='album' where prod writes 'playlist'. Intentional, but downstream consumers should know.

🤖 Generated with Claude Code

… dispatch errors

Follow-ups from the overnight prod-clone validation run (38,834 chain
blocks, 69 deep diff cycles against the read replica).

1. playlist_seen PK was wrong. Our 0005 migration declared
   `PRIMARY KEY (is_current, user_id, playlist_id, seen_at)` but prod's
   schema has `(user_id, playlist_id, seen_at)`. The handler's ON CONFLICT
   target that names the prod columns gets rejected with 42P10 against
   the old in-house PK; the version that names is_current too (which
   the previous PR had) silently mismatches prod. Fix both sides:
     - Update 0005 to declare the prod-correct PK on fresh databases.
     - Add migration 0027 to DROP+ADD the PK on databases that already
       ran the broken 0005.
     - Drop is_current from the ON CONFLICT spec in notification.go.

2. loadCurrentTrackRow / loadCurrentPlaylistRow propagated raw
   pgx.ErrNoRows when the target was soft-deleted (is_delete = true)
   between validation and update. The validation (trackExists /
   playlistExists) accepts is_delete=true rows; these row-loaders filter
   them out. Race manifested as bursts of "no rows in result set" at
   indexer.go:425 during high-Repost / high-Follow surge windows. Wrap
   ErrNoRows as ValidationError so the dispatcher logs at WARN, not ERROR.

3. getUserHandle in user_update.go had the same shape — propagated raw
   ErrNoRows in the rare race window between userExists and the handle
   lookup. Treat ErrNoRows as empty handle (callers compare against the
   new handle).

4. The dispatch-error log line at indexer.go:425 had no context fields,
   so when one fired we could see "no rows in result set" but had no clue
   which handler emitted it. Add entity_type / action / entity_id /
   user_id / hash to the log so future occurrences are self-attributing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@raymondjacobson
raymondjacobson merged commit 77cfaee into main May 22, 2026
4 of 5 checks passed
@raymondjacobson
raymondjacobson deleted the etl/playlist-seen-and-no-rows-followup branch May 22, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant