feat(apify): optional posts depth on social scrape endpoints

[sweetmantech]

Summary

Implements recoupable/chat#1836. Contract: recoupable/docs#258 — merge order: docs → api (this PR).

Adds an optional posts parameter (integer, 1–100) to both scrape endpoints and threads it into the platform Apify actor inputs:

Layer	Change
POST /api/socials/{id}/scrape	?posts=N query param, validated in validatePostSocialScrapeRequest (400 on 0/101/non-integer, parsed before auth)
POST /api/artist/socials/scrape	optional posts body field on artistSocialsScrapeBodySchema
scrapeProfileUrl / scrapeProfileUrlBatch	new optional posts arg forwarded to every platform ScrapeRunner
Twitter (apidojo/twitter-scraper-lite)	maxItems: posts ?? 1 — last N tweets with engagement metrics incl. viewCount (impressions)
YouTube (streamers/youtube-scraper)	maxResults: posts ?? 1, maxResultsShorts: posts ?? 0 — a requested depth includes Shorts, which the legacy snapshot excluded
Other platforms	accept and ignore the param (Instagram profile scrapes already return latestPosts)

Backwards compatible / cost-neutral by default: omitting posts produces the exact same actor inputs as before (unit-tested), so existing profile-snapshot callers and their Apify billing are unchanged.

Tests (TDD, red → green)

All new behavior was written test-first and confirmed failing (20 failures across 8 files) before implementation:

• startTwitterProfileScraping: default input unchanged; posts=20 → maxItems: 20
• startYoutubeProfileScraping: default input unchanged (Shorts excluded); posts=20 → maxResults/maxResultsShorts: 20
• scrapeProfileUrl + scrapeProfileUrlBatch: posts forwarded (and undefined when omitted)
• validatePostSocialScrapeRequest: posts=20 parses; 0/101/abc/1.5 → 400 before auth
• validateArtistSocialsScrapeBody + postArtistSocialsScrapeHandler: new suites (missing id → 400, posts pass-through, no-socials → [])

Full suite: 3745/3745 pass. Affected domains (lib/apify, lib/socials, lib/artist): 234/234. tsc --noEmit introduces no new errors (200 with this diff vs 203 pre-existing on base). ESLint clean on touched dirs.

Preview verification against the live deployment to follow as a PR comment.

🤖 Generated with Claude Code

---

Summary by cubic

Add optional posts depth to social scrape endpoints so callers can request up to N posts while keeping legacy single-item behavior by default (cost-neutral).

• New Features

  • POST /api/socials/{id}/scrape: accepts ?posts=1..100; invalid values return 400 before auth.
  • POST /api/artist/socials/scrape: accepts optional posts body field (1–100).
  • Threads posts through scrapeProfileUrl and scrapeProfileUrlBatch into platform scrapers.
  • Twitter (apidojo/twitter-scraper-lite): maxItems = posts ?? 1.
  • YouTube (streamers/youtube-scraper): maxResults = posts ?? 1, maxResultsShorts = posts ?? 0; when provided, Shorts are included.
  • Other platforms accept and ignore posts (no behavior change).

• Migration

  • No changes required for existing callers.
  • To request depth: use ?posts=N on /api/socials/{id}/scrape or { posts: N } on /api/artist/socials/scrape (1–100).

^{Written for commit c868d45. Summary will update on new commits.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
api	Ready	Preview	Jul 2, 2026 2:58pm

[coderabbitai]

Warning

Review limit reached

@sweetmantech, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 57 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c57fd0db-5946-47b1-8e4f-22ab7573b9d5

📥 Commits

Reviewing files that changed from the base of the PR and between e734c61 and c868d45.

⛔ Files ignored due to path filters (8)

• lib/apify/__tests__/scrapeProfileUrl.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
• lib/apify/__tests__/scrapeProfileUrlBatch.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
• lib/apify/twitter/__tests__/startTwitterProfileScraping.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
• lib/apify/youtube/__tests__/startYoutubeProfileScraping.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
• lib/artist/__tests__/postArtistSocialsScrapeHandler.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
• lib/artist/__tests__/validateArtistSocialsScrapeBody.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
• lib/socials/__tests__/postSocialScrapeHandler.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
• lib/socials/__tests__/validatePostSocialScrapeRequest.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**

📒 Files selected for processing (8)

• lib/apify/scrapeProfileUrl.ts
• lib/apify/scrapeProfileUrlBatch.ts
• lib/apify/twitter/startTwitterProfileScraping.ts
• lib/apify/youtube/startYoutubeProfileScraping.ts
• lib/artist/postArtistSocialsScrapeHandler.ts
• lib/artist/validateArtistSocialsScrapeBody.ts
• lib/socials/postSocialScrapeHandler.ts
• lib/socials/validatePostSocialScrapeRequest.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/social-scrape-posts-depth

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sweetmantech]

Preview verification — interim (2026-07-02)

Preview: api-5tofuvv0w-recoup.vercel.app (deployment resolved via repos/recoupable/api/deployments?sha=<PR head>, so it is built from this PR's commit).

Check (Done-when)	Documented	Actual	Result
?posts=0	400	400 (zod: posts must be between 1 and 100)	✅
?posts=abc	400	400	✅
?posts=101 equivalent (bulk body posts: 0)	400	400 {"missing_fields":["posts"],"error":"posts must be between 1 and 100"}	✅
No auth on /api/socials/{id}/scrape	401	401	✅
Validation ordering	param errors before auth	400 returned with an expired bearer token — validation short-circuits before auth, matching the unit test	✅
X posts=20 → ≥4 tweets w/ likeCount+viewCount	200 + dataset	⏳ pending — auth token expired mid-verification; re-running with a fresh token	⏳
YouTube posts=20 → 4 known Shorts w/ viewCount	200 + dataset	⏳ pending — same	⏳
Default (no posts) input byte-identical to today	unchanged	✅ unit-tested (start-input equality for both actors); legacy behavior also demonstrated by today's prod runs lbm3U2eqNzLpfCv8q (X, 1 item) / 3OTYCnusYtUeUlrGU (YT, 1 item, no Shorts)	✅

Note (observed live, pre-existing): POST /api/artist/socials/scrape accepts requests without any auth — logged with the credits audit on recoupable/chat#1836 as a follow-up; unchanged by this PR.

Will follow up with the two live posts=20 dataset results.

[cubic-dev-ai]

No issues found across 16 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Client as HTTP Client
    participant API as Next.js API Route
    participant PSR as validatePostSocialScrapeRequest
    participant ASB as validateArtistSocialsScrapeBody
    participant PSS as postSocialScrapeHandler
    participant PAS as postArtistSocialsScrapeHandler
    participant SA as selectAccountSocials
    participant SPU as scrapeProfileUrl
    participant SPB as scrapeProfileUrlBatch
    participant TA as Twitter Scraper
    participant YA as YouTube Scraper
    participant OA as Other Platform Scrapers
    participant AP as Apify Client

    Note over Client,AP: NEW: Optional "posts" depth parameter flow

    par POST /api/socials/{id}/scrape with query param
        Client->>API: POST /api/socials/{id}/scrape?posts=20
        API->>PSR: validatePostSocialScrapeRequest(request, id)
        PSR->>PSR: Parse query param with zod coerce
        alt posts invalid (0, 101, abc, 1.5)
            PSR-->>API: 400 response (before auth)
        else posts valid or omitted
            PSR->>SA: selectAccountSocials()
            SA-->>PSR: social record
            PSR-->>API: { social_id, posts }
        end
        API->>PSS: postSocialScrapeHandler()
        PSS->>SPU: scrapeProfileUrl(profileUrl, username, posts)
        SPU-->>PSS: scrape result with posts depth
        PSS-->>API: response
        API-->>Client: 200 with scrape data
    and POST /api/artist/socials/scrape with body field
        Client->>API: POST /api/artist/socials/scrape { posts: 20 }
        API->>ASB: validateArtistSocialsScrapeBody(body)
        alt posts invalid (0, 101, 1.5, "20")
            ASB-->>API: 400 response
        else posts valid or omitted
            ASB-->>API: validated body with posts
        end
        API->>PAS: postArtistSocialsScrapeHandler()
        PAS->>SA: selectAccountSocials()
        SA-->>PAS: artist socials list
        alt no socials found
            PAS-->>API: 200 with empty array
        else socials exist
            PAS->>SPB: scrapeProfileUrlBatch(inputs, posts)
            SPB->>SPU: scrapeProfileUrl(url, username, posts) per social
        end
        PAS-->>API: scrape results array
        API-->>Client: 200 with results
    end

    Note over SPU,AP: Platform-specific scraping behavior

    SPU->>SPU: Match platform from URL
    alt Twitter platform
        SPU->>TA: startTwitterProfileScraping(handle, posts)
        TA->>TA: maxItems = posts ?? 1
        TA->>AP: actor("apidojo/twitter-scraper-lite").start()
        AP-->>TA: run result
        TA-->>SPU: { runId, datasetId }
    else YouTube platform
        SPU->>YA: startYoutubeProfileScraping(handle, posts)
        YA->>YA: maxResults = posts ?? 1, maxResultsShorts = posts ?? 0
        Note over YA: posts includes Shorts, legacy snapshot excludes them
        YA->>AP: actor("streamers/youtube-scraper").start()
        AP-->>YA: run result
        YA-->>SPU: { runId, datasetId }
    else Other platforms (LinkedIn, Facebook, etc.)
        SPU->>OA: platform scraper(handle, posts)
        OA->>AP: start actor
        Note over OA: posts parameter accepted but ignored
        AP-->>OA: run result
        OA-->>SPU: { runId, datasetId }
    end
    SPU-->>PSS: scrape result

Loading

<sub>Auto-approved: Adds optional posts depth to social scrape endpoints with validation; forwards to Twitter and YouTube scrapers. Backward compatible, well-tested, and no risk to core logic or data integrity.

Re-trigger cubic</sub>

[sweetmantech]

Preview verification — final (2026-07-02) ✅

Completed the two pending live checks against preview api-5tofuvv0w-recoup.vercel.app via POST /api/artist/socials/scrape with {"posts": 20} (4 runs launched, all SUCCEEDED; datasets read back via prod GET /api/apify/runs/{runId}).

Check (Done-when)	Documented	Actual	Result
X posts=20 → tweets with likeCount + viewCount	200 + dataset	Run HY4mHujVAZMovCaiq: 20 tweets (10 own + 10 RTs). Pinned id 2072374207686607349 present: likes=2, replies=2, bookmarks=0, views=186 — impressions, which the connector path can never return (403 client-not-enrolled)	✅
YouTube posts=20 → the 4 known Shorts with stats	200 + dataset	Run jEgvw6xyaWinMv3TO: 40 items (20 videos + 20 Shorts). All four pinned Shorts present: 9C5XApWuT6E (26 views), 67KVW8jnDrU (56 views · 1 like), iPQwBlRvAF8 (31 views), AWPbUnQdN6U (36 views) — exactly matching the same-day YOUTUBE_GET_VIDEO_DETAILS_BATCH connector pull (26/56/31/36)	✅
Instagram / LinkedIn unaffected by posts	ignore param	1 profile item each (IG incl. latestPosts), same as legacy	✅

Two live-behavior nuances found and reconciled into the docs PR (recoupable/docs#258, second commit):

1. YouTube depth is per content type — posts=20 yields up to 20 videos + 20 Shorts (40 items), not 20 total.
2. X depth counts timeline items, not own posts — RTs/replies included, so an active account's older posts need a larger posts to reach (the Jun 24 post was beyond the latest-20 window here).

Note: the pinned-id check for 2069903614642913645 (Jun 24) was not reachable at depth 20 for the reason in (2) — the parameter behaved exactly as the actor defines it; the newest pinned id verified with full metrics. Docs ↔ API ↔ live responses now agree.

All Done-when criteria from recoupable/chat#1836 are verified. Ready for review; merge order docs#258 → this.

[sweetmantech]

Preview verification — re-run on current head c868d45 (2026-07-02) ✅

Head moved since the last verification (merge of test bringing in unrelated #742 — no scrape files changed), so the full suite was re-run against the fresh preview api-gm0jrbrbh-recoup.vercel.app (deployment resolved by sha). This round used a preview-environment Privy token, so the fully authenticated POST /api/socials/{id}/scrape?posts=N path was exercised directly (previously only reachable via the bulk endpoint).

Check (Done-when)	Documented	Actual	Result
No auth	401	401	✅
?posts=0 / ?posts=101 / ?posts=abc	400	400 / 400 / 400 (posts must be between 1 and 100)	✅
Invalid social UUID	400	400	✅
Authed X ?posts=20	200 + tweets w/ metrics	Run D9o7RLtI1Cehkuq5f: 20 tweets (10 own + 10 RTs); pinned 2072374207686607349: likes=2, replies=2, views=188 (impressions — up from 186 at the last pull, i.e. live data)	✅
Authed YouTube ?posts=20	200 + videos & Shorts	Run ojfSLqIUg81s8tGXA: 40 items (20 video + 20 shorts); all 4 pinned Shorts present — 9C5XApWuT6E 29 views, 67KVW8jnDrU 56 views · 1 like, iPQwBlRvAF8 31 views, AWPbUnQdN6U 36 views	✅

Environment note for future preview testing: previews verify tokens against a different Privy app than prod (prod tokens 401 on previews and vice versa) — worth documenting in the api CLAUDE.md at some point.

Docs contract is merged (recoupable/docs#258, 615cf5d). All Done-when criteria verified on the current head; ready to merge into test.