feat(apify): add LinkedIn profile scraping (harvestapi)#737
Conversation
LinkedIn was the only one of {YouTube, TikTok, LinkedIn, X, Instagram} unsupported
by POST /api/socials/{social_id}/scrape — a linkedin.com profile returned 400
"Unsupported social media platform" (recoupable/chat#1833). Add:
- lib/apify/linkedin/startLinkedinProfileScraping.ts → harvestapi/linkedin-profile-scraper
(mirrors the tiktok/youtube modules).
- scrapeProfileUrl: dispatch linkedin.com → the new scraper.
- getSocialPlatformByLink: linkedin.com → LINKEDIN (was NONE).
Note: harvestapi input shape (profiles: [url]) should be confirmed against the
actor on the preview; result persistence is handled in the companion webhook PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 36 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds a new LinkedIn profile scraper module that invokes an Apify actor to scrape LinkedIn profiles, returning run/dataset identifiers. Wires this scraper into the existing platform-based scraper dispatch table and extends platform link detection to recognize LinkedIn URLs. ChangesLinkedIn Scraping Integration
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant scrapeProfileUrl
participant startLinkedinProfileScraping
participant ApifyActor
Caller->>scrapeProfileUrl: profile URL
scrapeProfileUrl->>scrapeProfileUrl: match linkedin.com in PLATFORM_SCRAPERS
scrapeProfileUrl->>startLinkedinProfileScraping: handle/URL
startLinkedinProfileScraping->>ApifyActor: run harvestapi/linkedin-profile-scraper
ApifyActor-->>startLinkedinProfileScraping: run status, id, defaultDatasetId
startLinkedinProfileScraping-->>scrapeProfileUrl: {runId, datasetId} or null/error
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found across 5 files
Confidence score: 2/5
- In
lib/apify/linkedin/startLinkedinProfileScraping.ts, the actor payload usesprofilesinstead of the expectedurls, so LinkedIn runs may ignore the requested profile and return incorrect/no data — switch the payload key tourlsand verify with a real profile URL before merging. - In
lib/apify/scrapeProfileUrl.ts, the/in/<slug>fallback can resolve toinwhen username is empty, which can scrape the wrong LinkedIn profile and propagate bad identity data — add LinkedIn-specific URL parsing (or guardrails for empty usernames) and cover it with a targeted test before merging. - In
lib/artists/getSocialPlatformByLink.ts, rawlinkedin.comsubstring matching can misclassify domains likenotlinkedin.com, leading to incorrect platform routing downstream — tighten matching to hostname boundaries (or URL host parsing) to de-risk false positives.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/artists/getSocialPlatformByLink.ts">
<violation number="1" location="lib/artists/getSocialPlatformByLink.ts:13">
P2: The LinkedIn detection matches any string containing `linkedin.com`, so unrelated URLs like `notlinkedin.com` can be misclassified as LinkedIn. Match the hostname boundary instead of using a raw substring check.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Client
participant API as POST /api/socials/{id}/scrape
participant PlatformDetector as getSocialPlatformByLink
participant ScraperDispatcher as scrapeProfileUrl
participant LinkedInScraper as startLinkedinProfileScraping
participant Apify as Apify Client (Actor API)
participant HarvestAPI as harvestapi/linkedin-profile-scraper
Note over Client,HarvestAPI: NEW: LinkedIn profile scraping flow
Client->>API: POST with linkedin.com URL
API->>PlatformDetector: getSocialPlatformByLink(url)
PlatformDetector-->>API: "LINKEDIN" (NEW: linkedin.com detection)
API->>ScraperDispatcher: scrapeProfileUrl(url, username)
ScraperDispatcher->>ScraperDispatcher: Match linkedin.com → startLinkedinProfileScraping
ScraperDispatcher->>LinkedInScraper: startLinkedInProfileScraping(handle)
LinkedInScraper->>LinkedInScraper: Build input { profiles: [targetUrl] }
LinkedInScraper->>Apify: actor("harvestapi/linkedin-profile-scraper").start(input)
Apify->>HarvestAPI: Start LinkedIn scrape job (external API)
HarvestAPI-->>Apify: Run object (id, defaultDatasetId, status)
alt Run started successfully
Apify-->>LinkedInScraper: run
LinkedInScraper-->>ScraperDispatcher: { runId, datasetId }
ScraperDispatcher-->>API: { supported: true, runId, datasetId }
API-->>Client: 200 OK
else Run failed or aborted
Apify-->>LinkedInScraper: run with status FAILED/ABORTED
LinkedInScraper->>LinkedInScraper: Throw OUTSTANDING_ERROR
LinkedInScraper-->>ScraperDispatcher: error
ScraperDispatcher-->>API: null / error
API-->>Client: 400 / 500 error
end
Note over API,Client: Persistence of results not implemented in this PR (follow-up)
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| if (link.includes("instagram.com")) return "INSTAGRAM"; | ||
| if (link.includes("spotify.com")) return "SPOTIFY"; | ||
| if (link.includes("tiktok.com")) return "TIKTOK"; | ||
| if (link.includes("linkedin.com")) return "LINKEDIN"; |
There was a problem hiding this comment.
P2: The LinkedIn detection matches any string containing linkedin.com, so unrelated URLs like notlinkedin.com can be misclassified as LinkedIn. Match the hostname boundary instead of using a raw substring check.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/artists/getSocialPlatformByLink.ts, line 13:
<comment>The LinkedIn detection matches any string containing `linkedin.com`, so unrelated URLs like `notlinkedin.com` can be misclassified as LinkedIn. Match the hostname boundary instead of using a raw substring check.</comment>
<file context>
@@ -10,6 +10,7 @@ export function getSocialPlatformByLink(link: string): string {
if (link.includes("instagram.com")) return "INSTAGRAM";
if (link.includes("spotify.com")) return "SPOTIFY";
if (link.includes("tiktok.com")) return "TIKTOK";
+ if (link.includes("linkedin.com")) return "LINKEDIN";
if (link.includes("apple.com")) return "APPLE";
if (link.includes("youtube.")) return "YOUTUBE";
</file context>
| if (link.includes("linkedin.com")) return "LINKEDIN"; | |
| if (/(^|\/\/|\.)linkedin\.com(\/|$)/.test(link)) return "LINKEDIN"; |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
lib/apify/linkedin/startLinkedinProfileScraping.ts (1)
1-45: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffExtract shared Apify start/validation logic. Several profile scrapers repeat the same
run.id/defaultDatasetIdguard andFAILED/ABORTEDhandling. A small helper aroundapifyClient.actor(...).start(...)would keep the modules aligned; leave the platform-specific input shaping in each scraper.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/apify/linkedin/startLinkedinProfileScraping.ts` around lines 1 - 45, Extract the repeated Apify run validation from startLinkedinProfileScraping into a shared helper used by the other profile scraper modules. Keep the LinkedIn-specific input shaping in startLinkedinProfileScraping, but move the common run.id/defaultDatasetId guard and FAILED/ABORTED handling into a reusable function around apifyClient.actor(...).start(...). Refer to startLinkedinProfileScraping, apifyClient.actor, and OUTSTANDING_ERROR when wiring the helper so the scraper modules stay consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/apify/linkedin/startLinkedinProfileScraping.ts`:
- Around line 33-35: The terminal-status check in startLinkedinProfileScraping’s
.start() path is misplaced because .start() returns before the run completes, so
FAILED and ABORTED will not be seen here. Update this flow by either switching
the LinkedIn run invocation to .call() if this function must wait for
completion, or remove the run.status check from startLinkedinProfileScraping and
move terminal-state handling into the polling/completion path that observes the
finished run.
- Line 26: The Apify actor start call in startLinkedinProfileScraping is missing
an abort/timeout guard, so it can hang indefinitely on the request path. Update
the scrape flow around
apifyClient.actor("harvestapi/linkedin-profile-scraper").start(input) to use an
AbortController or equivalent timeout mechanism, pass the signal into the
request if supported, and ensure the timeout is handled cleanly by surfacing a
controlled error instead of blocking the request.
---
Nitpick comments:
In `@lib/apify/linkedin/startLinkedinProfileScraping.ts`:
- Around line 1-45: Extract the repeated Apify run validation from
startLinkedinProfileScraping into a shared helper used by the other profile
scraper modules. Keep the LinkedIn-specific input shaping in
startLinkedinProfileScraping, but move the common run.id/defaultDatasetId guard
and FAILED/ABORTED handling into a reusable function around
apifyClient.actor(...).start(...). Refer to startLinkedinProfileScraping,
apifyClient.actor, and OUTSTANDING_ERROR when wiring the helper so the scraper
modules stay consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 250f01b8-f138-4411-81c9-e0f8d385bc3c
⛔ Files ignored due to path filters (2)
lib/apify/__tests__/scrapeProfileUrl.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/artists/__tests__/getSocialPlatformByLink.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**
📒 Files selected for processing (3)
lib/apify/linkedin/startLinkedinProfileScraping.tslib/apify/scrapeProfileUrl.tslib/artists/getSocialPlatformByLink.ts
| const input = { profiles: [targetUrl] }; | ||
|
|
||
| try { | ||
| const run = await apifyClient.actor("harvestapi/linkedin-profile-scraper").start(input); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
No timeout on external Apify call.
apifyClient.actor(...).start(input) has no timeout/abort mechanism. If the Apify API hangs, this call could block the request indefinitely, since it appears to run on a request-handling path via scrapeProfileUrl.
🕐 Suggested guard using AbortController
- const run = await apifyClient.actor("harvestapi/linkedin-profile-scraper").start(input);
+ const controller = new AbortController();
+ const timeout = setTimeout(() => controller.abort(), 30_000);
+ let run;
+ try {
+ run = await apifyClient
+ .actor("harvestapi/linkedin-profile-scraper")
+ .start(input, { signal: controller.signal });
+ } finally {
+ clearTimeout(timeout);
+ }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/linkedin/startLinkedinProfileScraping.ts` at line 26, The Apify
actor start call in startLinkedinProfileScraping is missing an abort/timeout
guard, so it can hang indefinitely on the request path. Update the scrape flow
around apifyClient.actor("harvestapi/linkedin-profile-scraper").start(input) to
use an AbortController or equivalent timeout mechanism, pass the signal into the
request if supported, and ensure the timeout is handled cleanly by surfacing a
controlled error instead of blocking the request.
| if (run.status === "FAILED" || run.status === "ABORTED") { | ||
| throw new Error(OUTSTANDING_ERROR); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does Apify JS client actor().start() return a run with terminal status like FAILED or does it only return the initial queued status?
💡 Result:
The Apify JS client's actor.start method starts an actor and immediately returns the run object, which reflects the initial state of the run, such as "READY" or "RUNNING" [1][2][3]. It does not wait for the actor to complete, so it will not return a terminal status like "SUCCEEDED" or "FAILED" at the time of resolution [1][2][3]. If you need to wait for the actor to finish, you should use the actor.call method instead, which polls the run status and returns the run object only after the actor has completed [2][4]. Alternatively, you can use the start method with a waitForFinish option (in some versions/configurations) or manage the wait manually using the returned run ID [5][2].
Citations:
- 1: https://docs.apify.com/api/client/js/reference/class/ActorClient
- 2: https://docs.apify.com/api/client/js/reference/next/class/ActorClient
- 3: https://docs.apify.com/api/client/js/reference/class/ActorClient.md
- 4: https://docs.apify.com/api/client/js/reference/class/ApifyClient
- 5: https://docs.apify.com/api/client/js/reference/interface/ActorStartOptions.md
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file with line numbers.
sed -n '1,220p' lib/apify/linkedin/startLinkedinProfileScraping.ts | cat -n
# Find nearby Apify start/call usage for context.
rg -n "actor\\.(start|call)|run\\.status|FAILED|ABORTED" lib/apify -SRepository: recoupable/api
Length of output: 3192
Move terminal-status handling out of this start path .start() resolves before the run finishes, so FAILED/ABORTED won’t be observed here. If this step should wait for completion, use .call(); otherwise, handle terminal states in the polling path instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/linkedin/startLinkedinProfileScraping.ts` around lines 33 - 35, The
terminal-status check in startLinkedinProfileScraping’s .start() path is
misplaced because .start() returns before the run completes, so FAILED and
ABORTED will not be seen here. Update this flow by either switching the LinkedIn
run invocation to .call() if this function must wait for completion, or remove
the run.status check from startLinkedinProfileScraping and move terminal-state
handling into the polling/completion path that observes the finished run.
…dles Two bugs caught verifying against the live harvestapi actor + a real LinkedIn social (recoupable/chat#1833): - The actor's input schema takes `urls` ("Profile URLs") / publicIdentifiers / queries — not `profiles`. Sending `profiles` would start a run with no targets. - getUsernameFromProfileUrl returned the LinkedIn path prefix ("in") instead of the handle, so a stored username of "in" would make the scraper build linkedin.com/in/in. Teach it linkedin.com/{in,company,school}/<handle>; this also fixes the username updateArtistSocials stores for LinkedIn adds. Verified: harvestapi input schema pulled from the actor's latest build (uZgrDy65OCHs0Chum); RED→GREEN tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…the scraper
Live preview test: the run queried linkedin.com/in/in → 404 "Profile not found" —
the dispatcher passed the DB row's stored username ("in", written by the old
path-prefix helper bug), which the helper fix can't repair for existing rows.
Add a per-entry input selector to PLATFORM_SCRAPERS; LinkedIn takes the full
profile_url (the field the platform was matched on), making stored-username
quality irrelevant for LinkedIn scrapes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rd in the module
Per review: don't modify the shared PLATFORM_SCRAPERS config for one platform.
Revert the input-selector change; instead startLinkedinProfileScraping rejects
LinkedIn path prefixes ("in"/"company"/"school") as handles — legacy rows stored
the prefix as username, and scraping linkedin.com/in/in would hit a real, wrong
profile. Correct usernames come from the fixed getUsernameFromProfileUrl going
forward; bad legacy rows now fail loudly instead of scraping the wrong person.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Preview verification — LinkedIn scrape works end-to-end (and the input-shape caveat was real)Verified against a real LinkedIn social ( Before / after
The PR-body caveat was warranted — 3 real bugs found & fixed during verification
All RED→GREEN TDD'd; Notes
Net: |
1 participant
Part of recoupable/chat#1833 (platform coverage). Base:
main.Why
LinkedIn was the only one of {YouTube, TikTok, LinkedIn, X, Instagram} unsupported by
POST /api/socials/{social_id}/scrape— alinkedin.comprofile returned400 "Unsupported social media platform"(verified live).getSocialPlatformByLinkalso returnedNONEfor LinkedIn.Fix
lib/apify/linkedin/startLinkedinProfileScraping.ts→harvestapi/linkedin-profile-scraper(mirrors the tiktok/youtube modules). Ref: https://apify.com/harvestapi/linkedin-profile-scraper (+harvestapi/linkedin-profile-postsfor posts, follow-up).scrapeProfileUrl.ts: dispatchlinkedin.com→ the new scraper.getSocialPlatformByLink.ts:linkedin.com→LINKEDIN.Tests (TDD)
getSocialPlatformByLink.test.ts(LINKEDIN) +scrapeProfileUrl.test.ts(dispatches linkedin.com; unsupported → null). tsc + eslint clean.Follow-ups (noted, not in this PR)
profiles: [url]) against the actor on the preview.🤖 Generated with Claude Code
Summary by cubic
Adds LinkedIn profile scraping via
harvestapi/linkedin-profile-scraper. The scrape API now supports LinkedIn; the scraper builds a profile URL from a handle or accepts a full URL and rejects invalid legacy handles.New Features
startLinkedinProfileScrapingusingharvestapi/linkedin-profile-scraper.linkedin.comto the LinkedIn scraper and mapped it toLINKEDINingetSocialPlatformByLink.Bug Fixes
in,company, orschool; accept full profile URLs, trim spaces, and ignore a leading@.urlswhen starting runs.getUsernameFromProfileUrlnow extracts LinkedIn handles from/in,/company, and/school.Written for commit b65c1d6. Summary will update on new commits.
Summary by CodeRabbit
@in LinkedIn handles.