Make IGeocoder ops async using azure storage/functions for CSV Import

[mgmccarthy]

cc @tonysurma @MisterJames @dpaquette @stevejgordon

Right now, for the CSV import, we're invoking the IGeocoder once per Request. The problem/concern here is since we're not limiting how many requests we process in a CSV import:

• we could exceed the amount of requests supported in a given time frame to Google Maps (which is what IGeocoder is calling behind the scenes). Since IGeocoder does not have an "batch" API, we might need to see if there is a max request per time period allowed, and then throttle to calls to the API to not exceed that threshold
• b/c we're combining a transactional resource (the db) with a non-transactional resource (web service call via IGeocoder) in the same logical transaction (ImportRequestsCommandHandler.Handle), we could end up with the user waiting for the success message to come back to them, or in a worst case scenario, get a timeout caused by the web service.

After talking to @MisterJames and @dpaquette about it, we want to avoid using Hangfire for this (which was my initial approach) b/c this does not represent a scheduled operation, but purely, as async operation. Since azure storage/queues combined with webjobs can do the same thing, and it does not utilize web server resources like Hangfire, we want to use azure for this type of operation.

The high level items that would need to be accomplished for this to happen is outlined in Issue #1626. Once this code/webjob is in place and working, we can update the code in ImportRequestsCommandHandler to send a message on the azure queue for lat/long lookup

[mgmccarthy]

@dpaquette did you ever get a chance to check into how a shared EF 7 context would work (if it can work) across multiple projects in a single solution?

cc @stevejgordon

[mgmccarthy]

@dpaquette @stevejgordon @tonysurma @MisterJames

I have an asp.net core MVC solution running locally where the entities, the DbContext and the migrations are kept in a separate assembly (project) than the web project.

seems like running migrations like this works and I've successfully used the DbContext from the separate assembly.

I know there is code where Hangfire is being used to make async calls, when we really should be using azure storage/web jobs. By extracting the DbContxt and migrations from the Web/AllReady project, we allow the DbContext to be used by the .WebJob project, which will allow us to write azure functions which use DbContext.

That can move the project away from utilizing Hangfire for all but scheduled tasks (which is why is was introduced).

[stevejgordon]

@mgmccarthy Sounds like great progress. One thing to consider is that @MisterJames suggested we look might want to look at moving away from WebJobs to true Azure Functions - Serverless. I'm not sure if Azure Functions can/should consume entity framework for data access.

[mgmccarthy]

@stevejgordon I'll do some reading on azure functions and see if DI is possible If it's not, we could still use service locator (event though it's not a "best practice") and still achieve the same result

[mgmccarthy]

@stevejgordon looks like our current functions we have written in AllReady.NotificationsWebJob just directly new up the required dependencies. ProcessSmsQueueMessage creates a new TwilioRestClient and
ProcessEmailQueueMessage creates a new SendGridMessage.

That being said, the first step in moving towards allowing an azure function to work with DbContext would be to move AllReadyContext to a separate assembly. Once that is done, we can new up the DbContext directly in the function. If/when that works, we could always look to moving towards using service locator if we want. This blog posts points out how to do just that: https://blog.kloud.com.au/2016/11/21/managing-dependencies-in-azure-functions/

But it looks like none of our current azure functions are testable (and that's not necessarily a bad thing).