feat: add SurveyReportUpload and add send report method

[Alec4r]

Description:

This PR adds a new model SurveyReportUpload to save the status and details of the send request.

SurveyReport model (#31183).

ISSUE:
openedx/axim-engineering#349

APPROVED DESIGN:
https://docs.google.com/document/d/1tm1O5BQsAwekphI5IN9pBn8behQDZigXePLGqCl5he8

DECISIONS:

• We decided add a new model to save all the information about each send request.
• We decided send the survey report data from json to a API endpoint, I hope that tCrill supply this endpoint.
• We decided that each report generated by cli will be send automatically, unless the user set the option --no-send in the command.

How to test

First, You should define the settings: SURVEY_REPORT_ENDPOINT with a valid url, you can use some Mock API Service like https://beeceptor.com/.

Then, with a dev environment running, you should run the command python manage.py lms generate_report or python manage.py lms generate_report --no-send.

[openedx-webhooks]

Thanks for the pull request, @Alec4r! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[Alec4r]

@jalondonot @felipemontoya @ormsbee @jmakowski1123 @mariajgrimaldi We would appreciated your comments in this PR.

[felipemontoya]

@ormsbee do you have something different in mind for sending this report?

We have explored things like having a zappier that formats and sends to google docs, but we are open to anything.

[ormsbee]

I think we should have a real URL enabled by default (if one exists yet), and a timeout on this request so it doesn't block for too long. But other than that, this seems fine to me.

[ormsbee]

Is there an endpoint to send to at this point?

[Alec4r]

@ormsbee No, we don't have an endpoint for it, @felipemontoya some approach for this?

[felipemontoya]

I can produce a Zappier webhook and we leave that as the default for now.
https://help.zapier.com/hc/en-us/articles/8496326446989

Webhooks are available only in paid plans. I can put it in the edunext account.

[Alec4r]

@felipemontoya @jmakowski1123 @e0d I added the site name in a new field and I added a setting to make the site name anonymous.

new survey report file: https://docs.google.com/spreadsheets/d/1PWhWZ0XN6tEo8xhDh3kTSuGMrEYAE6tDBQu2kdF8tZA/edit?usp=sharing

setting: ANONYMOUS_SURVEY_REPORT is False by default.

[e0d]

How is the hash computed? Ideally it would be a consistent value -- it wouldn't change if the secret key was updated for example.

[Alec4r]

@e0d I'm using a sha256 from the hashlib so the hash will be always the same.

[ormsbee]

At the risk of bikeshedding on this one, if we really want to have a fixed ID that each site carries for anonymization purposes, please make it so that the value is randomly generated and stored in the database. A simple hash of the base URL is too easy to figure out because there are relatively few Open edX sites out there, and we can run SHA256 against all of them. Salting doesn't help because either the salt is randomly generated per-site (in which case it might as well be the anonymous value), or it's linked against some secret that might change.

We've walked up and down this path a couple of times with anonymous user IDs.

[Alec4r]

@ormsbee I tough 2 difference options for this:

Setting: We can define a new setting to save the ID, the problem with this is that the ID could be changed by the user whenever he wants, so maybe is not the best idea.

Model: We can create a new model just to save the ID, this will help us to persist the ID without problems and the ID will never change, we can use a get_or_create in the send method.

What do you think?

[mphilbrick211]

Per Jorge, flagging this for @e0d and @jmakowski1123 to weigh in on Dave's feedback before proceeding. Thanks!

[mphilbrick211]

@e0d @mariajgrimaldi @jmakowski1123 @felipemontoya @ormsbee - per Jorge, this is ready for another round of reviews. Please take a look! Thanks :)

[felipemontoya]

@Alec4r going back to Dave's comments actually he already mentioned

please make it so that the value is randomly generated and stored in the database

This would be the option 2 where we store the value in a model. I think we should go that route and not the settings one. Writing the value in settings would require that we randomize the creation of that ID at the first deploy, which is out of our control from this work and also it would make it very easy to loose track of the generated ID and change it for cases where an install is being updated in it's technology. We have seen such cases when we are migrating old installs to something docker based.

[felipemontoya]

Thanks a lot @Alec4r. This would solve the last remaining issue with the format.

From my perspective we are ready with the code portion of this. +1. I'm leaving an approval here.

However the current zap in the production configuration belongs to edunext. I suppose we want to have a tCRIL zappier account for this right from the start.

@e0d, @jmakowski1123 is there something we can do to help with the creation of this zap? The instructions where provided by @Alec4r some time ago.

https://docs.google.com/document/d/1rLxVu5bC-XhEmKAGwF2FSyG6t8JVuVK26SS81kCgEmQ/edit

[mariajgrimaldi]

All my suggestions/questions were addressed so I'm good with this! Thank you!

[ormsbee]

@Alec4r: Is this the set of commits you want merged? We'd normally squash it to fewer commits, but the individual messages here seem pretty intentional.

@mariajgrimaldi, @felipemontoya: Anyhow, please merge (or squash+merge) at your discretion.

[e0d]

I've created the Zap. @felipemontoya has the URL for the endpoint. PM me in slack if you need it as well. The Zap should be published and live now.

[openedx-webhooks]

@Alec4r 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the production environment.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the production environment.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production.

[edx-pipeline-bot]

EdX Release Notice: This PR has been deployed to the production environment.