Spike submissions stats

[thomasiles]

Spike for showing total submission stats

Trello card: https://trello.com/c/usentmeF/2918-timebox-2-days-spike-to-add-submission-data-into-the-admin-reports

This spike explores creating a report with submissions for a set period, adding manually entered baseline to results.

There are two big issues which prevent this feature being used as the source of truth for our submission stats.

• Our stats aren't saved for longer than 15 months
• The stats include test submissions, which make them much less useful

To solve the issue with stats only going back 15 months, we could:

• Start storing the daily total submissions. We could use a batch job to fetch them from AWS and then calculate totals using the stored stats.
• Only show stats for fixed time periods, e.g. the last 6 months, avoiding the need to store them but not showing a total.
• Keep a baseline number to add to the rolling stats. We would need to periodically update the figure. This would allow us to show total stats but at the cost of needed to update the figure. This is the approach this spike took.

The second issue, test submissions included in our the stats is harder. I think the easiest solution for this is to add another dimension to our submission stats, organisation. We could then exclude a list of organisations which we know produce test stats. This would be the org our end-to-end tests use and the internal forms team.

This spike doesn't do that.

How we store submission counts

We store submission counts using AWS CloudWatch metrics. Every time a form is submitted we add a count for <formid, environment>.

These metrics are stored for a maximum of 15 months at a granularity of 1 day.
We replay these metrics to form creators on the "live" view of a form in Admin.

We don't store these metrics in our database.
Every time a submission total for a form is displayed we are issuing a query to AWS.
This keeps the code simple and ensures the data is always fresh.
But it stops us showing stats for anything beyond 14 months ago.
It also means the stats could go down as well as up.
This makes keeping an overall total based on these results harder.

Showing totals

To show a total submissions, we can query AWS to get a sum of the metrics for all forms in an environment for each day for the last 12 months (or anything less than 15 months).
This is slow but lets us calculate lots of statistics, like daily, weekly and monthly submissions as well as partial counts for the current day, week, and month.

There is a problem with this approach.
The query returns all submissions which includes the submissions for test forms.
Our end-to-end tests create and submit a large number of forms. Over 1000 a month on dev.

To calculate the current figures, Anne queries Splunk and uses set queries to remove the values based on the titles of the test forms.

Things to consider when reviewing

• Ensure that you consider the wider context.
• Does it work when run on your machine?
• Is it clear what the code is doing?
• Do the commit messages explain why the changes were made?
• Are there all the unit tests needed?
• Do the end to end tests need updating before these changes will pass?
• Has all relevant documentation been updated?

[github-actions]

🎉 A review copy of this PR has been deployed! You can reach it at: https://pr-2766.admin.review.forms.service.gov.uk/

It may take 5 minutes or so for the application to be fully deployed and working. If it still isn't ready
after 5 minutes, there may be something wrong with the ECS task. You will need to go to the integration AWS account
to debug, or otherwise ask an infrastructure person.

For the sign in details and more information, see the review apps wiki page.