Allow multiple retries when testing a resource

[alexandreleroux]

Feature request: allow multiple retries when testing a resource.

Example: when testing a WMS server, don't consider it down after the initial test, allow multiple retries (e.g. up to 3), done immediately after the initial one fails, to confirm that the resource is indeed unavailable.

[hannesaddec]

More considerations.. gap between re-tries should vary – certain WMS installations exist which go into sleep mode if no activity occurs for xx min (e.g. 30), and need some time to "wake" up. Retesting after 5 sec, 30 sec, 120 sec. If a server is not reachable after 2 minutes – consider it down. From: Alexandre Leroux [mailto:notifications@github.com] Sent: Tuesday, July 11, 2017 7:53 PM To: geopython/GeoHealthCheck Cc: Subscribed Subject: [geopython/GeoHealthCheck] Allow multiple retries when testing a resource (#133) Feature request: allow multiple retries when testing a resource. Example: when testing a WMS server, don't consider it down after the initial test, allow multiple retries (e.g. up to 3), done immediately after the initial one fails, to confirm that the resource is indeed unavailable. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#133>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMn2MVki3n2p2O3TNKUp_8irzzxWVxH_ks5sM7Z5gaJpZM4OUl6m>.

[justb4]

This is an interesting but tricky feature request that needs some discussion. I know of cases where a load-balancer dispatches requests to several backend OWS-servers (e.g. GeoServer). One or more may be down, such that e.g. with 3 servers, two down, one up, one in every three requests succeeds. Then the retry-mechanism would falsely assume the service is healthy. Also I think the Python requests package we use for HTTP requests, has a built-in retry mechanism and may even wait up to 5 mins before giving up (seen in practice).

But for the cases of servers that need a sort of "Wake-Up" we may develop a simpleHttpWakeupProbe that is executed first for a Resource. It can be parameterized for retries: number and interval. Would be simple to implement, based on the existing HttpGet Probe.

[justb4]

Adding a note: interesting implementation with discussion:
https://www.peterbe.com/plog/best-practice-with-retries-with-requests

[justb4]

This issue becomes relevant again. It looks that now and then "false negatives" occur: failure report, but in fact the service is ok. Happens especially in Probes that issue many requests in sequence, like going through all Layers of a WMS or Tiling service. The most common error is a Read Timeout.

One first addition that may help is using the Requests package Session within a Probe, this takes advantage of connection pooling in urllib3. See
https://2.python-requests.org/en/v3.0.0/user/advanced/

But also https://www.peterbe.com/plog/best-practice-with-retries-with-requests remains very relevant.

[justb4]

Think we implemented this using the requests Session object, via PR #319, thus closing.