Skip to content

Latest commit

 

History

History
19 lines (16 loc) · 615 Bytes

README.md

File metadata and controls

19 lines (16 loc) · 615 Bytes

Marketplace Scraper Playwright

A scraper for Marketplace listing using NodeJS and Playwright

Current Architecture

The scraper-js app continuously scraps the same job params and checks for duplicates against Redis keys. It then pushes new items to a Redis queue.

Config

  • See scraper-js/config.js

Running

node index.js

Tracing

playwright show-trace trace.zip

Docker Compose

There is a development Kafka and Redis set up in /docker-compose. In order to run the apps you need to bring this up with docker compose up in that directory