Skip to content

Fix for #133 use requests Session object with Retry capabilities - #319

Merged
justb4 merged 3 commits into
masterfrom
133-retry-on-fail
Apr 26, 2020
Merged

Fix for #133 use requests Session object with Retry capabilities#319
justb4 merged 3 commits into
masterfrom
133-retry-on-fail

Conversation

@justb4

@justb4 justb4 commented Apr 25, 2020

Copy link
Copy Markdown
Member

This fix should improve HTTP connection handling by GHC Probes significantly:

  1. use requests Session objects to reuse connections (works via urllib3)
  2. implement the golden tip from: https://www.peterbe.com/plog/best-practice-with-retries-with-requests . Within the Probe object a Session with a retry/backoff schema is created.
  3. upgrade requests package to latest version 2.23.0

Especially in Probes where all N Layers of a WMS/Tiling endpoint are accessed/crawled this already gives a factor 3 performance improvement. In the old case a new connection was setup for each request.

The false negatives should be of the past. Will first observe for some time on the GHC demo server (where I get this stream of Failing/Fixed emails constantly...).

OPTION for later: may make this mechanism and its parameters configurable.

@justb4 justb4 added this to the Version 0.8.0 milestone Apr 25, 2020
@justb4
justb4 requested review from borrob and tomkralidis April 25, 2020 15:08
@justb4 justb4 self-assigned this Apr 25, 2020
@justb4
justb4 merged commit 6ff109b into master Apr 26, 2020
@justb4

justb4 commented Apr 26, 2020

Copy link
Copy Markdown
Member Author

@tomkralidis @borrob merging anyway. Session also mimics user-browser behavior (keeping connections) Fully tested locally with both Docker and regular. Running on demo.geohealthcheck.org will give a more realistic scenario. (and see if the email stream is reduced).

@tomkralidis
tomkralidis deleted the 133-retry-on-fail branch April 26, 2020 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant