feat(apify): persist scrape results for the remaining 5 platforms#740
Conversation
Completes the persistence work started in #738 (registry + LinkedIn): TikTok, X/Twitter, YouTube, Threads, and Facebook scrapes now write back to socials instead of being poll-only (recoupable/chat#1833). Per platform (mirroring lib/apify/linkedin/): - start module attaches getApifyWebhooks() to the run (the root cause of "nothing persists" — only IG/LinkedIn runs ever called /api/apify). - handle<Platform>ProfileScraperResults upserts socials (keyed profile_url), with the field mapping taken from a REAL actor run of each platform: tiktok authorMeta (run G4YRI0e…), twitter author (ALVMZYX…), youtube channel fields (H0ZrIAs…), threads profile (9iiG1sD…), facebook page (ICZxdJB…). - registry entry under the actor id resolved from the Apify API. Round-trip hazards handled explicitly: twitter URLs are lowercased (actor echoes display casing, stored rows are lowercase — upsert would duplicate); youtube keys on inputChannelUrl (the URL we passed) because the actor's own channelUrl is the /channel/UC… form which never matches the stored @handle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 1 minute Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (7)
📒 Files selected for processing (11)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found across 18 files
Confidence score: 4/5
- In
lib/apify/twitter/handleTwitterProfileScraperResults.ts, mappingauthor.location || nullandauthor.description || nullcan prevent cleared profile fields from being persisted becauseupsertSocialsdrops nullish values, leaving stalesocials.region/socials.bioin place after users remove them on X — switch these assignments to preserve empty strings (or otherwise explicitly clear stored values) before merging. lib/apify/tiktok/__tests__/handleTiktokProfileScraperResults.test.tsclaims coverage for both empty datasets and missingauthorMeta, but only validates the empty-dataset path; a missing-authorMetaregression could slip through untested — add theitems[0]withoutauthorMetacase to de-risk future parser changes.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/apify/tiktok/__tests__/handleTiktokProfileScraperResults.test.ts">
<violation number="1" location="lib/apify/tiktok/__tests__/handleTiktokProfileScraperResults.test.ts:46">
P3: Test description claims to cover two scenarios (empty dataset OR no authorMeta) but only tests the empty-dataset path. The 'no authorMeta' case — where `items[0]` exists but lacks `authorMeta` — is untested. Since the handler guards on `!author?.profileUrl`, this path is exercised differently and should either be tested or removed from the description to avoid misleading future readers.</violation>
</file>
<file name="lib/apify/twitter/handleTwitterProfileScraperResults.ts">
<violation number="1" location="lib/apify/twitter/handleTwitterProfileScraperResults.ts:35">
P2: X profiles that remove their bio will keep stale `socials.bio`, because `author.description || null` converts the actor's empty string to `null` and `upsertSocials` drops nullish fields before upsert. Using nullish coalescing preserves an observed empty string so the scrape can clear stale text.</violation>
<violation number="2" location="lib/apify/twitter/handleTwitterProfileScraperResults.ts:38">
P2: X profiles that clear their location will keep stale `socials.region`, because `author.location || null` turns the empty string into a dropped nullish field. Preserving empty strings lets the upsert reflect the latest scrape result.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Client as External Scheduler
participant API as API Route (/api/apify)
participant Handler as ApifyWebhookHandler
participant Registry as getApifyResultHandler
participant TikTok as handleTiktokProfileScraperResults
participant Twitter as handleTwitterProfileScraperResults
participant YouTube as handleYoutubeProfileScraperResults
participant Threads as handleThreadsProfileScraperResults
participant Facebook as handleFacebookProfileScraperResults
participant ApifyClient as Apify Client (apifyClient)
participant DB as Database (socials)
Note over Client,DB: NEW: Webhook persistence flow for all 5 platforms
Client->>API: POST /api/apify (webhook payload)
API->>Handler: parse & validate request
Handler->>Registry: getApifyResultHandler(actorId)
alt TikTok actor (GdWCkxBtKWOsKjdch)
Registry-->>Handler: handleTiktokProfileScraperResults
Handler->>TikTok: handle(payload)
TikTok->>ApifyClient: listItems(datasetId)
ApifyClient-->>TikTok: items (post items)
TikTok->>TikTok: extract authorMeta (name, profileUrl, avatar, fans)
TikTok->>DB: upsertSocials([{ profile_url, username, avatar, followerCount }])
else Twitter actor (nfp1fpt5gUlBwPcor)
Registry-->>Handler: handleTwitterProfileScraperResults
Handler->>Twitter: handle(payload)
Twitter->>ApifyClient: listItems(datasetId)
ApifyClient-->>Twitter: items (tweet items)
Twitter->>Twitter: extract author (userName, url, followers)
Note over Twitter: NEW: lowercase profile_url to match stored rows (x.com/theasf)
Twitter->>DB: upsertSocials([{ profile_url, username, avatar, followerCount, region }])
else YouTube actor (h7sDV53CddomktSi5)
Registry-->>Handler: handleYoutubeProfileScraperResults
Handler->>YouTube: handle(payload)
YouTube->>ApifyClient: listItems(datasetId)
ApifyClient-->>YouTube: items (video items)
YouTube->>YouTube: extract inputChannelUrl, channelUsername, subscribers
Note over YouTube: NEW: key on inputChannelUrl, NOT channelUrl (/channel/UC…)
YouTube->>DB: upsertSocials([{ profile_url, username, avatar, followerCount, region }])
else Threads actor (kJdK90pa2hhYYrCK5)
Registry-->>Handler: handleThreadsProfileScraperResults
Handler->>Threads: handle(payload)
Threads->>ApifyClient: listItems(datasetId)
ApifyClient-->>Threads: items (profile items)
Threads->>Threads: extract username, url, follower_count
Threads->>DB: upsertSocials([{ profile_url, username, avatar, followerCount, bio }])
else Facebook actor (4Hv5RhChiaDk6iwad)
Registry-->>Handler: handleFacebookProfileScraperResults
Handler->>Facebook: handle(payload)
Facebook->>ApifyClient: listItems(datasetId)
ApifyClient-->>Facebook: items (page items)
Facebook->>Facebook: extract pageUrl, pageName, followers
Facebook->>DB: upsertSocials([{ profile_url, username, avatar, followerCount }])
else Unknown actor
Registry-->>Handler: undefined
Note over Handler: No-op, return early
end
Note over Client,DB: NEW: Each start module now attaches webhooks
Note over Client,DB: startTiktokProfileScraping -> actor.start(input, { webhooks: getApifyWebhooks() })
Note over Client,DB: startTwitterProfileScraping -> same pattern
Note over Client,DB: startYoutubeProfileScraping -> same pattern
Note over Client,DB: startThreadsProfileScraping -> same pattern
Note over Client,DB: startFacebookProfileScraping -> same pattern
DB-->>Handler: upsert result
Handler-->>API: { ok: true }
API-->>Client: 200 OK
Note over TikTok,Facebook: empty dataset → no-op, return { social: null }
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| bio: author.description || null, | ||
| followerCount: author.followers ?? null, | ||
| followingCount: author.following ?? null, | ||
| region: author.location || null, |
There was a problem hiding this comment.
P2: X profiles that clear their location will keep stale socials.region, because author.location || null turns the empty string into a dropped nullish field. Preserving empty strings lets the upsert reflect the latest scrape result.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/twitter/handleTwitterProfileScraperResults.ts, line 38:
<comment>X profiles that clear their location will keep stale `socials.region`, because `author.location || null` turns the empty string into a dropped nullish field. Preserving empty strings lets the upsert reflect the latest scrape result.</comment>
<file context>
@@ -0,0 +1,42 @@
+ bio: author.description || null,
+ followerCount: author.followers ?? null,
+ followingCount: author.following ?? null,
+ region: author.location || null,
+ };
+ await upsertSocials([social]);
</file context>
| region: author.location || null, | |
| region: author.location ?? null, |
| profile_url: normalizeProfileUrl(author.url).toLowerCase(), | ||
| username: author.userName, | ||
| avatar: author.profilePicture ?? null, | ||
| bio: author.description || null, |
There was a problem hiding this comment.
P2: X profiles that remove their bio will keep stale socials.bio, because author.description || null converts the actor's empty string to null and upsertSocials drops nullish fields before upsert. Using nullish coalescing preserves an observed empty string so the scrape can clear stale text.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/twitter/handleTwitterProfileScraperResults.ts, line 35:
<comment>X profiles that remove their bio will keep stale `socials.bio`, because `author.description || null` converts the actor's empty string to `null` and `upsertSocials` drops nullish fields before upsert. Using nullish coalescing preserves an observed empty string so the scrape can clear stale text.</comment>
<file context>
@@ -0,0 +1,42 @@
+ profile_url: normalizeProfileUrl(author.url).toLowerCase(),
+ username: author.userName,
+ avatar: author.profilePicture ?? null,
+ bio: author.description || null,
+ followerCount: author.followers ?? null,
+ followingCount: author.following ?? null,
</file context>
| bio: author.description || null, | |
| bio: author.description ?? null, |
| }, | ||
| ]); | ||
| }); | ||
| it("no-ops when the dataset is empty or has no authorMeta", async () => { |
There was a problem hiding this comment.
P3: Test description claims to cover two scenarios (empty dataset OR no authorMeta) but only tests the empty-dataset path. The 'no authorMeta' case — where items[0] exists but lacks authorMeta — is untested. Since the handler guards on !author?.profileUrl, this path is exercised differently and should either be tested or removed from the description to avoid misleading future readers.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/tiktok/__tests__/handleTiktokProfileScraperResults.test.ts, line 46:
<comment>Test description claims to cover two scenarios (empty dataset OR no authorMeta) but only tests the empty-dataset path. The 'no authorMeta' case — where `items[0]` exists but lacks `authorMeta` — is untested. Since the handler guards on `!author?.profileUrl`, this path is exercised differently and should either be tested or removed from the description to avoid misleading future readers.</comment>
<file context>
@@ -0,0 +1,51 @@
+ },
+ ]);
+ });
+ it("no-ops when the dataset is empty or has no authorMeta", async () => {
+ listItems.mockResolvedValue({ items: [] });
+ expect(await handleTiktokProfileScraperResults(payload)).toEqual({ social: null });
</file context>
Preview verification — 3 of 5 platforms persisted live (webhook → DB), zero duplicate rowsTested on this PR's preview (
The round-trip hazards — verified handled
Threads + FacebookNo product Net: with this PR, all 7 supported platforms (IG, LinkedIn, TikTok, X, YouTube, Threads, Facebook) persist scrape results to |
1 participant
Part of recoupable/chat#1833 — completes the persistence work from api#738 (registry + LinkedIn). Base:
main.What
TikTok, X/Twitter, YouTube, Threads, and Facebook scrapes now persist to
socialsinstead of being poll-only. Per platform, mirroringlib/apify/linkedin/:start…module passes{ webhooks: getApifyWebhooks() }(root cause of "nothing persists": only IG/LinkedIn runs ever called/api/apify).handle<Platform>ProfileScraperResultsupsertssocials(keyedprofile_url), mapping only fields observed in a real actor run of each platform (run ids in the code comments: TikTokG4YRI0e…, XALVMZYX…, YouTubeH0ZrIAs…, Threads9iiG1sD…, FacebookICZxdJB…).GdWCkxBtKWOsKjdch(tiktok),nfp1fpt5gUlBwPcor(x),h7sDV53CddomktSi5(youtube),kJdK90pa2hhYYrCK5(threads),4Hv5RhChiaDk6iwad(facebook).Round-trip hazards handled explicitly
upsertSocialskeys on exactprofile_url, so each handler must emit a URL that matches the stored row:x.com/TheASF) while stored rows are lowercase — the handler lowercases (X handles are case-insensitive). Without this, every scrape would create a duplicate row.channelUrlis the/channel/UC…form, which never matches the stored@handlerow — the handler keys oninputChannelUrl(the exact URL this service passed to the actor).Tests (TDD)
lib/apifysuite green (65 tests / 20 files); tsc + eslint clean.Preview verification (live scrape → webhook → DB diff on real rows) to follow as a comment.
🤖 Generated with Claude Code
Summary by cubic
Persist TikTok, X/Twitter, YouTube, Threads, and Facebook scrape results to
socialsvia Apify webhooks, completing the remaining platform persistence. This satisfies recoupable/chat#1833 and removes the poll-only behavior.New Features
handle<Platform>ProfileScraperResultsfor all 5 platforms to upsert intosocialskeyed byprofile_url.getApifyWebhooks()to allstart…modules so runs hit/api/apify.getApifyResultHandler.Bug Fixes
profile_urlto avoid duplicate rows caused by display casing.inputChannelUrl(matches stored@handle), not/channel/UC…URLs.Written for commit 31cd246. Summary will update on new commits.