IP Geocoder (freegeoip) errors when adding Resource

[justb4]

Seeing errors for freegeoip when requesting lat-lon location:

TLDR;

• implement freegeoip new API as described here
• implies requesting API key (NB max 10000 reqs a month!)
• optional: allow user-supplied API key
• once stable again implement Configurable geocoder #210 for a more flexible IP-geocoder solution

2018-10-08 11:13:40,509 - GeoHealthCheck.util - INFO - Geocoding geodata.nationaalgeoregister.nl with http://freegeoip.net/json/geodata.nationaalgeoregister.nl
2018-10-08 11:13:40,585 - GeoHealthCheck.util - ERROR - Could not derive coordinates: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/GeoHealthCheck/GeoHealthCheck/util.py", line 162, in geocode
    content = json.loads(urlopen(url).read())
  File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/local/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/local/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
2018-10-08 11:13:40,590 - GeoHealthCheck.models - ERROR - Could not derive coordinates: Could not derive coordinates: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/GeoHealthCheck/GeoHealthCheck/models.py", line 382, in __init__
    self.latitude, self.longitude = util.geocode(url)
  File "/GeoHealthCheck/GeoHealthCheck/util.py", line 167, in geocode
    raise ValueError(msg)
ValueError: Could not derive coordinates: HTTP Error 403: Forbidden
2

So we'll need a fix anyway, as freegeoip has changed API and conditions, http://freegeoip.net/shutdown:

{
--
  | "0": "#################################################################################################################################",
  | "1": "#                                                                                                                               #",
  | "2": "# IMPORTANT - PLEASE UPDATE YOUR API ENDPOINT                                                                                   #",
  | "3": "#                                                                                                                               #",
  | "4": "# This API endpoint is deprecated and has now been shut down. To keep using the freegeoip API, please update your integration   #",
  | "5": "# to use the new ipstack API endpoint, designed as a simple drop-in replacement.                                                #",
  | "6": "# You will be required to create an account at https://ipstack.com and obtain an API access key.                                #",
  | "7": "#                                                                                                                               #",
  | "8": "# For more information on how to upgrade please visit our Github Tutorial at: https://github.com/apilayer/freegeoip#readme      #",
  | "9": "#                                                                                                                               #",
  | "a": "#################################################################################################################################"
  | }

[tomkralidis]

We geocode only once when registering a resource (i.e not on every run).

Options:

1. have a GHC deployment register an API key
2. have GHC have a centrally managed API key
3. investigate similar alternative services

What do folks think about switching to IP-API ? Usage limits specified here

[justb4]

True, geocoding is only done once on Resource creation, in __init__(), which appears to not be called on other Resource object-creations, like on Run or edits. Possibly SQLAlchemy-specific. So GHC is mild on API-calls here.

How to proceed? I am in favor to use IP-API for now, is a two-liner fix, already seen working, close this issue, and then elaborate #210. It would not bee too hard to use our Plugin system with IP-API as the default Plugin. Other providers can then be configured at will. Most will just need a url, key and return latitude, longitude in JSON, so all those could use a single Plugin class. Within the config_site.py, we just configure and parameterize the Plugin:

GEOIP = {
  'plugin': 'GeoHealthCheck.plugins.geoip.generic',
  'parameters': {
    'url_template': 'https://mygeoipprovider.com/?key=<mykey>&ip=%s&format=json',
    'lat_field': 'latitude',
    'lon_field': 'longitude'
  }
}

# default IP-API in config_main.py

GEOIP = {
  'plugin': 'GeoHealthCheck.plugins.geoip.generic',
  'parameters': {
    'url_template': 'http://ip-api.com/json/%s',
    'lat_field': 'lat',
    'lon_field': 'lon'
  }
}

# Create at runtime
geoip_provider = Factory.create_obj(CONFIG['GEOIP']['plugin'])
geoip_provider.init(CONFIG['GEOIP']['parameters'])
lat, lon = geoip_provider.lookup(ip)
# etc

This will cover most of the Geo-IP providers I've seen, or APIs on self-hosted Geo-IP Databases, or maybe even a local DB lookup, as @cezio suggests in #210. Our Plugin system can instantiate and parameterize the Plugin. Possibly this could be an array of geocoders, if one fails.

[justb4]

Ok, switched to IP-API, for now. Existing Resources that still have lat,lon 0.0, 0.0 can be recoded simply by going to Resource Edit and then Save. This also caters for when the geocoding on create Resource failed for some reason. I propose to close and concentrate on #210.

[justb4]

Closing, now on with #210.