Add a general delay between crawl requests

[navarone-feekery]

Problem Description

Currently users cannot configure a delay between crawl requests. A simple x seconds delay configurable from the yaml file would allow users to slow down crawls.

This is similar to #46 (abiding by Crawl-delay in robots.txt). However, a general delay would be useful for users who can't or don't want to change the robots.txt.

When both are implemented, the highest value of the two should be used for the crawl delay.

Proposed Solution

Add a configurable crawl delay in yaml file.

Additional Context

Link to discuss forum: https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/4