feat(apify): registry webhook dispatch + LinkedIn persistence end-to-end#738
Conversation
…IG persistence) apifyWebhookHandler routed only the two Instagram actors via a hardcoded switch; every other platform (TikTok/X/YouTube/Threads/Facebook, and the new LinkedIn) hits "unhandled actorId" and persists NOTHING — those scrapes return data on the poll endpoint but never write back to socials (recoupable/chat#1833). Extract getApifyResultHandler (actorId → handler registry, SRP) and dispatch through it, so adding a platform is a single registry entry instead of another switch arm. Behavior-preserving — all existing webhook tests pass unchanged. The per-platform handlers (mirror handleInstagramProfileScraperResults) are the documented follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 23 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (4)
📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 3 files
Confidence score: 5/5
- In
lib/apify/apifyWebhookHandler.ts, the registry lookup is only checked for truthiness, so special actor IDs like__proto__can be treated as handlers and throw an internal exception instead of returning the expected unhandled-actor path; this is a narrow edge case but can produce incorrect error behavior if merged as-is — add a callable/type guard (e.g., verify the resolved value is a function) before invoking it.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/apify/apifyWebhookHandler.ts">
<violation number="1" location="lib/apify/apifyWebhookHandler.ts:20">
P3: Guard that the registry result is actually callable, not just truthy. Otherwise special actor ids such as `__proto__` produce an internal handler exception instead of the intended unhandled-actor response.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Apify as Apify Platform
participant Webhook as apifyWebhookHandler (POST /api/apify)
participant Validator as validateApifyWebhookRequest
participant Registry as getApifyResultHandler
participant IGProfile as handleInstagramProfileScraperResults
participant IGComments as handleInstagramCommentsScraper
participant DB as Database (socials / posts)
Note over Apify,DB: NEW: dispatch via actorId registry instead of switch
Apify->>Webhook: POST webhook payload (actorId, eventData)
Webhook->>Validator: validateApifyWebhookRequest(request)
alt Invalid payload
Validator-->>Webhook: error response
Webhook-->>Apify: 400 error (unchanged)
else Valid payload
Validator-->>Webhook: validated payload { actorId, ... }
Webhook->>Registry: getApifyResultHandler(actorId)
Registry-->>Webhook: ApifyResultHandler | undefined
alt Known actorId (Instagram profile/comments)
Note over Registry,IGProfile: e.g. actorId = "dSCLg0C3YEZ83HzYX"
Webhook->>IGProfile: handler(validated payload)
IGProfile->>DB: upsert socials + posts
DB-->>IGProfile: success
IGProfile-->>Webhook: result object
Webhook-->>Apify: 200 { status: "ok", ... }
else Known actorId (Instagram comments)
Note over Registry,IGComments: actorId = "SbK00X0JYCPblD2wp"
Webhook->>IGComments: handler(validated payload)
IGComments->>DB: upsert comments data
DB-->>IGComments: success
IGComments-->>Webhook: result object
Webhook-->>Apify: 200 { status: "ok", ... }
else Unknown actorId (unhandled)
Note over Registry,Webhook: any other actorId → undefined handler
Webhook->>Webhook: log warning, return "Unhandled actorId"
Webhook-->>Apify: 200 { status: "error", error: "Unhandled actorId: ..." }
end
end
Note over Webhook: Always returns 200 to Apify (prevents retries)
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| const { actorId } = validated.eventData; | ||
|
|
||
| const handler = getApifyResultHandler(actorId); | ||
| if (!handler) { |
There was a problem hiding this comment.
P3: Guard that the registry result is actually callable, not just truthy. Otherwise special actor ids such as __proto__ produce an internal handler exception instead of the intended unhandled-actor response.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/apifyWebhookHandler.ts, line 20:
<comment>Guard that the registry result is actually callable, not just truthy. Otherwise special actor ids such as `__proto__` produce an internal handler exception instead of the intended unhandled-actor response.</comment>
<file context>
@@ -20,23 +16,18 @@ export async function apifyWebhookHandler(request: NextRequest): Promise<NextRes
const { actorId } = validated.eventData;
+ const handler = getApifyResultHandler(actorId);
+ if (!handler) {
+ console.warn(`[WARN] apifyWebhookHandler: unhandled actorId ${actorId}`);
+ return NextResponse.json(
</file context>
| if (!handler) { | |
| if (typeof handler !== "function") { |
…-IG platform) Per review, the registry alone wasn't worth shipping — wire one new platform through the full persistence path (recoupable/chat#1833): - startLinkedinProfileScraping now attaches getApifyWebhooks() to the run — the root cause of "nothing but IG persists" is that only the Instagram start modules registered the per-run webhook; every other platform's runs never called /api/apify at all. - New handleLinkedinProfileScraperResults: upserts the scraped profile into socials (keyed on profile_url) — username/avatar/bio/followerCount/region. Field mapping taken from a REAL harvestapi run (TNwgXpQmhvoCHsKg8), including the {element,error,status} wrapper shape for missing profiles (skipped). - Registry: LpVuK3Zozwuipa5bp (harvestapi/linkedin-profile-scraper, id resolved from the Apify API) → the new handler. TikTok/X/YouTube/Threads/Facebook remain poll-only — each needs the same webhook attachment + a handler with its real output shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sweetmantech
changed the title
There was a problem hiding this comment.
1 issue found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/apify/linkedin/handleLinkedinProfileScraperResults.ts">
<violation number="1" location="lib/apify/linkedin/handleLinkedinProfileScraperResults.ts:43">
P2: LinkedIn webhook persistence can fail for valid-looking items that include `linkedinUrl` but omit `publicIdentifier`, because the inserted `socials` row has no required `username`. Falling back to the handle parsed from `linkedinUrl` would keep the upsert valid and consistent with scrape startup.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
|
||
| const social = { | ||
| profile_url: normalizeProfileUrl(first.linkedinUrl), | ||
| username: first.publicIdentifier, |
There was a problem hiding this comment.
P2: LinkedIn webhook persistence can fail for valid-looking items that include linkedinUrl but omit publicIdentifier, because the inserted socials row has no required username. Falling back to the handle parsed from linkedinUrl would keep the upsert valid and consistent with scrape startup.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/linkedin/handleLinkedinProfileScraperResults.ts, line 43:
<comment>LinkedIn webhook persistence can fail for valid-looking items that include `linkedinUrl` but omit `publicIdentifier`, because the inserted `socials` row has no required `username`. Falling back to the handle parsed from `linkedinUrl` would keep the upsert valid and consistent with scrape startup.</comment>
<file context>
@@ -0,0 +1,52 @@
+
+ const social = {
+ profile_url: normalizeProfileUrl(first.linkedinUrl),
+ username: first.publicIdentifier,
+ avatar: first.photo ?? null,
+ bio: first.about ?? first.headline ?? null,
</file context>
Preview verification — LinkedIn persistence works end-to-end (webhook → DB write)Tested on this PR's preview ( The definitive check — the webhook wrote the DB row, not a humanCaptured the
Full chain, verified live: Why nothing but IG ever persisted (root cause, confirmed)Webhooks are attached per-run, and only the Instagram start modules passed Also verified
Remaining follow-ups (documented in the registry doc comment + chat#1833)TikTok/X/YouTube/Threads/Facebook stay poll-only until each gets the same two-part treatment (webhook attachment + handler from its real output shape — captured samples for TikTok/X/YouTube are in chat#1833). |
Completes the persistence work started in #738 (registry + LinkedIn): TikTok, X/Twitter, YouTube, Threads, and Facebook scrapes now write back to socials instead of being poll-only (recoupable/chat#1833). Per platform (mirroring lib/apify/linkedin/): - start module attaches getApifyWebhooks() to the run (the root cause of "nothing persists" — only IG/LinkedIn runs ever called /api/apify). - handle<Platform>ProfileScraperResults upserts socials (keyed profile_url), with the field mapping taken from a REAL actor run of each platform: tiktok authorMeta (run G4YRI0e…), twitter author (ALVMZYX…), youtube channel fields (H0ZrIAs…), threads profile (9iiG1sD…), facebook page (ICZxdJB…). - registry entry under the actor id resolved from the Apify API. Round-trip hazards handled explicitly: twitter URLs are lowercased (actor echoes display casing, stored rows are lowercase — upsert would duplicate); youtube keys on inputChannelUrl (the URL we passed) because the actor's own channelUrl is the /channel/UC… form which never matches the stored @handle. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Part of recoupable/chat#1833 (data freshness). Base:
main.Problem
Scrape results for every non-Instagram platform never persist to
socials— they're only retrievable via the poll endpoint. Two root causes (both verified):apifyWebhookHandlerrouted only the two Instagram actor ids via a hardcodedswitch.getApifyWebhooks()— other platforms' runs never call/api/apifyat all.This PR
Registry foundation:
getApifyResultHandler(actorId)— actorId → handler registry (SRP, own file + tests), replacing theswitch. Behavior-preserving for IG.LinkedIn persisted end-to-end (first non-IG platform, per review — structure alone wasn't worth shipping):
startLinkedinProfileScrapingattachesgetApifyWebhooks()to the run.handleLinkedinProfileScraperResults: upserts the scraped profile intosocials(keyedprofile_url) —username/avatar/bio/followerCount/region. Field mapping from a real harvestapi run (TNwgXpQmhvoCHsKg8, 2026-07-01), including the{element, error, status}wrapper for missing profiles (skipped, no write).LpVuK3Zozwuipa5bp(harvestapi/linkedin-profile-scraper— id resolved from the Apify API) → handler.Tests (TDD)
RED→GREEN: handler (real profile shape → exact upsert; 404-wrapper → no write; empty dataset → no-op), registry mapping, webhook attachment on start. Full
lib/apifysuite green (54); tsc + eslint clean.Remaining follow-ups (documented in the registry's doc comment + chat#1833)
TikTok/X/YouTube/Threads/Facebook stay poll-only until each gets the same two-line webhook attachment + a handler built from its real output shape (captured samples for TikTok/X/YouTube exist in chat#1833).
🤖 Generated with Claude Code