pywebcrawl

Crawls the web looking for provided content. Assigns pages thresholds for recording or throwing away the page.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
crawler.py		crawler.py
main.py		main.py
runlist.txt		runlist.txt
site.txt		site.txt

Provide feedback