Skip to content

fix: remove expand_fields from conference list and add server-side pagination#19

Merged
agonza1 merged 1 commit into
peermetrics:masterfrom
Boanerges1996:fix/pagination-and-expand-fields
Apr 8, 2026
Merged

fix: remove expand_fields from conference list and add server-side pagination#19
agonza1 merged 1 commit into
peermetrics:masterfrom
Boanerges1996:fix/pagination-and-expand-fields

Conversation

@Boanerges1996
Copy link
Copy Markdown
Contributor

Closes #17

Summary

  • Conference list was taking 56 seconds due to N+1 queries (expand_fields) and no pagination
  • Three changes that work together to fix this:

A) Remove expand_fields from conference list

  • List view (GET /conferences?appId=...) no longer inlines participants and issues
  • Detail view (GET /conferences/<id>) still returns them
  • The dashboard already fetches sessions/issues via separate API calls, so the inlined UUIDs were redundant

B) Server-side pagination (opt-in, backward compatible)

  • All list endpoints now accept ?limit= and ?offset= params
  • When present: returns { "results": [...], "count": 1234 }
  • When absent: returns original { "data": [...] } — no breaking change
  • Default limit=50, max=200, invalid values return 400
  • Applied to: conferences, sessions, participants, events

C) Fix BaseModel.filter() eager materialization

  • Removed .exists() call (result was discarded — wasted query)
  • Removed for obj in filtered: obj.prepare() loop that loaded ALL rows into memory
  • filter() now returns a lazy QuerySet so LIMIT/OFFSET translates to SQL
  • prepare() moved into serialize() loop to preserve App.prepare() behavior

Test results (local, 2079 conferences)

Test Result
No pagination params → { data: [...] } ✅ Backward compatible
?limit=3&offset=0{ results, count } ✅ 3 results, count=2079
?limit=3&offset=3 → page 2 ✅ Different results
Detail view → has participants/issues
List view → no participants/issues ✅ N+1 eliminated
limit=0 / limit=-1 → clamped to 1
limit=abc → 400 error
limit=500 → clamped to 200
Sessions, participants, events with pagination

Files changed

File Change
app/models/basemodel.py Lazy queryset in filter()
app/utils.py prepare() in serialize loop + paginate_and_serialize() helper
app/views/conference_view.py Remove expand_fields from list, add pagination
app/views/session_view.py Add pagination
app/views/participant_view.py Add pagination
app/views/event_view.py Pass request to retrieve_events, add pagination
app/views/conference_events_view.py Pass request to retrieve_events

…gination

Part A — Remove expand_fields from conference list:
- List view no longer inlines participants/issues (N+1 eliminated)
- Detail view (single conference) still returns expanded fields

Part B — Add opt-in server-side pagination:
- ?limit= and ?offset= params on conferences, sessions, participants, events
- Returns { results: [...], count: N } when pagination params are present
- Returns original { data: [...] } format when no params (backward compatible)
- Default limit=50, max=200, with input validation

Part C — Fix BaseModel.filter() eager materialization:
- Removed .exists() call (result was discarded)
- Removed eager for-loop that called prepare() on every row
- filter() now returns a lazy QuerySet so LIMIT/OFFSET work at DB level
- prepare() moved into serialize() to preserve App.prepare() behavior

Closes peermetrics#17
@agonza1 agonza1 merged commit b5ad74e into peermetrics:master Apr 8, 2026
Boanerges1996 added a commit to Boanerges1996/api that referenced this pull request Apr 9, 2026
…ants_count

After expand_fields was removed (PR peermetrics#19), the conference list no longer
includes participants or issues arrays. This broke the dashboard charts
for error/warning icons and participant counts.

Adds lightweight annotations via Exists() subqueries and Count() so the
list includes:
- has_errors (boolean)
- has_warnings (boolean)
- participants_count (integer)

No N+1 queries — all computed in the same SQL query.
agonza1 pushed a commit that referenced this pull request Apr 10, 2026
…ants_count (#22)

* fix: annotate conference list with has_errors, has_warnings, participants_count

After expand_fields was removed (PR #19), the conference list no longer
includes participants or issues arrays. This broke the dashboard charts
for error/warning icons and participant counts.

Adds lightweight annotations via Exists() subqueries and Count() so the
list includes:
- has_errors (boolean)
- has_warnings (boolean)
- participants_count (integer)

No N+1 queries — all computed in the same SQL query.

* fix: use Subquery for participants_count to avoid filtered join bug

Count('participants') reuses the filtered JOIN when participantId is in
the query string, returning 1 instead of the real total. Switched to a
Subquery that counts independently of the request's participant filter.

Addresses review feedback from Codex and Alberto.

* fix: exclude soft-deleted participants from participants_count

The Subquery used Participant.objects.filter() which bypasses the
is_active=True check, overcounting if any participants were soft-deleted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove expand_fields from conference list and add server-side pagination

2 participants