-
Have python and pip working
-
Install virtualenv ( https://virtualenv.pypa.io/en/stable/installation/ )
-
make a directory where your project will run
mkdir p3-gpi-export -
Tell it to run in a vritualenv
virtualenv p3-gpi-export -
Go in the directory
cd p3-gpi-export -
Start the virtualenv
source bin/activate -
Install scrapy
pip install Scrapy -
Install dateparser
pip install dateparser -
Initiate a scrapy project
scrapy startproject p3_gpi_export -
Download the crawler
git clone https://github.com/greenpeace/planet4-gpi-export -
Copy the crawler file in the scrapy project crawlers
cp planet4-gpi-export/all_spider.py p3_gpi_export/p3_gpi_export/spiders/all_spider.py -
Go in the directory of the scrapy project
cd p3_gpi_export/ -
Run the crawler
scrapy crawl all
greenpeace/planet4-gpi-export
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|