|
| 1 | +--- |
| 2 | +title: 'Scaling SecretAgent Scrapes with Handlers' |
| 3 | +path: /handling-scale |
| 4 | +date: 2020-12-29 |
| 5 | +summary: "We needed a simpler approach to scaling out to multiple machines running SecretAgent and 1000s of waiting actions. So we added a new concept called Handlers." |
| 6 | +--- |
| 7 | + |
| 8 | +When you start using SecretAgent, you often copy and paste the default examples. As we started to use SecretAgent on larger extraction efforts, it became clear that we didn't have a clear story for "how" you go from that starting example to running 2, or even 1000 scrapes. |
| 9 | + |
| 10 | +As you start to think about structuring a bigger effort, a bunch of questions come up: |
| 11 | + |
| 12 | +- Do you create a new SecretAgent instance every time? Or do you simply add tabs? |
| 13 | +- How expensive is it to create many instances? |
| 14 | +- How should I make sure not to overload the host machine with the number of scrapes running at the same time? |
| 15 | +- How do I add new machines when I max out the current one? |
| 16 | + |
| 17 | +As we explored simplifying this story, we wanted to make the progression of "examples" through to full-scrapes a smooth process. Something like this: |
| 18 | + |
| 19 | +#### Step 1: Try Out an Example |
| 20 | + |
| 21 | +Trying out examples should require as little setup as possible, so we added a new `default export` that's a ready-to-go client for SecretAgent. |
| 22 | + |
| 23 | +```js |
| 24 | +import agent from 'secret-agent'; |
| 25 | + |
| 26 | +(async () => { |
| 27 | + // no initilization required! |
| 28 | + await agent.goto('https://ulixee.org'); |
| 29 | + const datasetLinks = await agent.document.querySelectorAll('a.DatasetSummary'); |
| 30 | + for (const link of datasetLinks) { |
| 31 | + const name = await link.querySelector('.title').textContent; |
| 32 | + const href = await link.getAttribute('href'); |
| 33 | + const dataset = { name, href }; |
| 34 | + console.log('Ulixee Dataset', dataset); |
| 35 | + } |
| 36 | + |
| 37 | + await agent.close(); |
| 38 | +})(); |
| 39 | +``` |
| 40 | + |
| 41 | +#### Step 2: Run Multiple Scrapes |
| 42 | + |
| 43 | +Agent instances are lightweight, but what do you do when you need to queue up thousands of them to run. Until now, you've been on your own to use libraries like `p-queue`, keeping track of promises, or simply waiting and looping. |
| 44 | + |
| 45 | +We introduced a new idea into SecretAgent called a [`Handler`](/docs/basic-interfaces/handler) to help run multiple scrapes in one session. Handlers manage the concurrency of multiple scrapes to ensure your machine doesn't get overloaded and hang. We designed it so your code should require almost no changes to transition to many scrapes. |
| 46 | + |
| 47 | +```js |
| 48 | +import { Handler } from 'secret-agent'; |
| 49 | + |
| 50 | +(async () => { |
| 51 | + const handler = new Handler({ maxConcurrency: 5 }); |
| 52 | + |
| 53 | + handler.dispatchAgent(async agent => { |
| 54 | + // agent is automatically created for us |
| 55 | + await agent.goto('https://ulixee.org'); |
| 56 | + const datasetLinks = await agent.document.querySelectorAll('a.DatasetSummary'); |
| 57 | + for (const link of datasetLinks) { |
| 58 | + const name = await link.querySelector('.title').textContent; |
| 59 | + const href = await link.getAttribute('href'); |
| 60 | + const dataset = { name, href }; |
| 61 | + |
| 62 | + // add a name to each agent so we can find each scrape on Replay |
| 63 | + const agentOptions = { name }; |
| 64 | + handler.dispatchAgent(getDatasetCost, link, agentOptions); |
| 65 | + } |
| 66 | + }); |
| 67 | + |
| 68 | + // only 5 agents will be active at a given time until all are done |
| 69 | + await handler.waitForAllDispatches(); |
| 70 | + await handler.close(); |
| 71 | +})(); |
| 72 | + |
| 73 | +// my data gets passed in once an agent is available |
| 74 | +async function getDatasetCost(agent, dataset) { |
| 75 | + const { name, href } = dataset; |
| 76 | + if (!href.startsWith('http')) href = `https://ulixee.org${href}`; |
| 77 | + console.log(href); |
| 78 | + await agent.goto(href); |
| 79 | + await agent.waitForAllContentLoaded(); |
| 80 | + const cost = await agent.document.querySelector('.cost .large-text').textContent; |
| 81 | + console.log('Cost of %s is %s', dataset.name, cost); |
| 82 | +} |
| 83 | +``` |
| 84 | + |
| 85 | +#### Step 3: Add Scraping Machines |
| 86 | + |
| 87 | +You might find that you need to increase the speed of your scrapes. So the next transition you'll likely want to make is to add remote machines. Handlers are built to round-robin between multiple [`CoreConnections`](/docs/advanced/core-connection). |
| 88 | + |
| 89 | +```js |
| 90 | +import { Handler } from 'secret-agent'; |
| 91 | + |
| 92 | +(async () => { |
| 93 | + const handler = new Handler( |
| 94 | + { |
| 95 | + maxConcurrency: 5, |
| 96 | + host: '192.168.1.1:2300', // fictional remote secret-agent #1 |
| 97 | + }, |
| 98 | + { |
| 99 | + maxConcurrency: 5, |
| 100 | + host: '192.168.1.2:2300', // fictional remote secret-agent #2 |
| 101 | + }, |
| 102 | + ); |
| 103 | + |
| 104 | +// ... everything else is the same! |
| 105 | + |
| 106 | + handler.dispatchAgent(async agent => { |
| 107 | + // agent is automatically created for us |
| 108 | + await agent.goto('https://ulixee.org'); |
| 109 | + ... |
| 110 | +``` |
| 111 | +
|
| 112 | +
|
| 113 | +#### Default Exports |
| 114 | +
|
| 115 | +To get to this setup, you'll notice some changes in the default exports when you install SecretAgent 1.3.0-alpha.1. The default exports that come out of the `secret-agent` and `@secret-agent/client` packages is now a pre-initialized instance of the `Agent` class (`SecretAgent` was renamed to `Agent`). |
| 116 | +
|
| 117 | +[`Handler`](/docs/basic-interfaces/handler) and [`Agent`](/docs/basic-interfaces/agent) are available as exports from both the `secret-agent` and `@secret-agent/client` if you'd like to continue to use those. To customize a "Remote" `SecretAgent` for an [`Agent`](/docs/basic-interfaces/agent), you can create a new instance with a [`coreConnection`](/docs/basic-interfaces/agent#constructor) parameter, or use the [`.configure()`](/docs/basic-interfaces/agent#configure) function. |
| 118 | +
|
| 119 | +
|
| 120 | +#### That's it! |
| 121 | +
|
| 122 | +That's our change. We hope it leads to a very simple model to understand how to scale up your SecretAgent instances. Feedback is welcome as always on any of our channels (listed in header)! |
0 commit comments