Skip to content

Add migration to populate site_values in SiteConfigurationHistory - #23236

Merged
pwnage101 merged 1 commit into
masterfrom
pwnage101/deng-18-rename_site_configuration_siteconfiguration_values_2.2
Mar 3, 2020
Merged

Add migration to populate site_values in SiteConfigurationHistory#23236
pwnage101 merged 1 commit into
masterfrom
pwnage101/deng-18-rename_site_configuration_siteconfiguration_values_2.2

Conversation

@pwnage101

@pwnage101 pwnage101 commented Feb 28, 2020

Copy link
Copy Markdown
Contributor

Right now the ORM is very unhappy about the JSONField site_values
in SiteConfigurationHistory containing non-JSON (empty strings). We
cannot even write a data migration using the ORM to populate the field
because that causes a JSONDeserializationError. Therefore, we must
bypass the ORM and populate the values with raw SQL.

DENG-18


Testing in devstack:

DB state before migration:

mysql> select site_values from site_configuration_siteconfigurationhistory;
+-------------+
| site_values |
+-------------+
|             |
|             |
+-------------+
2 rows in set (0.00 sec)

Running the migration:

root@lms:/edx/app/edxapp/edx-platform# python manage.py cms migrate
...
Operations to perform:
  Apply all migrations: admin, announcements, api_admin, assessment, auth, block_structure, bookmarks, catalog, celery_utils, completion, consent, content_libraries, content_type_gating, contentserver, contentstore, contenttypes, cornerstone, course_action_state, course_creators, course_date_signals, course_duration_limits, course_groups, course_modes, course_overviews, courseware, coursewarehistoryextended, crawlers, credit, dark_lang, database_fixups, degreed, discounts, django_comment_common, django_notify, djcelery, edx_oauth2_provider, edx_proctoring, edx_when, edx_zoom, edxval, embargo, enterprise, entitlements, experiments, integrated_channel, lx_pathway_plugin, milestones, oauth2, oauth2_provider, oauth_dispatch, organizations, problem_builder, redirects, sap_success_factors, schedules, self_paced, sessions, site_configuration, sites, static_replace, student, submissions, survey, system_wide_roles, tagging, theming, track, user_api, user_authn, user_tasks, verified_track_content, verify_student, video_config, video_pipeline, waffle, waffle_utils, wiki, workflow, xapi, xblock_config, xblock_django
Running migrations:
  Applying site_configuration.0005_populate_siteconfig_history_site_values... OK

DB state after migration:

mysql> select site_values from site_configuration_siteconfigurationhistory;
+-------------+
| site_values |
+-------------+
| {}          |
| {}          |
+-------------+
2 rows in set (0.00 sec)

Running the reverse migration:

root@lms:/edx/app/edxapp/edx-platform# python manage.py cms migrate site_configuration 0004_add_site_values_field
...
Operations to perform:
  Target specific migration: 0004_add_site_values_field, from site_configuration
Running migrations:
  Rendering model states... DONE
  Unapplying site_configuration.0005_populate_siteconfig_history_site_values... OK

DB state after reverse migration:

mysql> select site_values from site_configuration_siteconfigurationhistory;
+-------------+
| site_values |
+-------------+
|             |
|             |
+-------------+
2 rows in set (0.00 sec)

@pwnage101

Copy link
Copy Markdown
Contributor Author

I also just checked the PostgreSQL syntax and it should be valid, but I did not test: https://www.postgresql.org/docs/9.1/sql-update.html

@doctoryes

Copy link
Copy Markdown
Contributor

I also just checked the PostgreSQL syntax and it should be valid, but I did not test: https://www.postgresql.org/docs/9.1/sql-update.html

Hmmm - I'm curious why the raw SQL needs to be PostgreSQL-complaint. Are there installations which use it? Does edx.org use it?

@doctoryes doctoryes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's simple - 💟

@@ -0,0 +1,30 @@
# -*- coding: utf-8 -*-
# Written by Troy.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Maybe use your Github handle? Or edX email address here? Or just remove this line? Because git knows all...

@pwnage101

Copy link
Copy Markdown
Contributor Author

Hmmm - I'm curious why the raw SQL needs to be PostgreSQL-complaint. Are there installations which use it? Does edx.org use it?

edx.org does not, but I just figured since one reason we're doing this PR is for the open source community and who knows what they might be trying out.

Right now the ORM is very unhappy about the JSONField `site_values`
in SiteConfigurationHistory containing non-JSON (empty strings).  We
cannot even write a data migration using the ORM to populate the field
because that causes a JSONDeserializationError.  Therefore, we must
bypass the ORM and populate the values with raw SQL.

DENG-18
@pwnage101
pwnage101 force-pushed the pwnage101/deng-18-rename_site_configuration_siteconfiguration_values_2.2 branch from c3ab1b8 to 06b5470 Compare March 2, 2020 16:44
@pwnage101

Copy link
Copy Markdown
Contributor Author

jenkins run js

@pwnage101

Copy link
Copy Markdown
Contributor Author

jenkins run bokchoy

@edx-status-bot

Copy link
Copy Markdown

Your PR has finished running tests. There were no failures.

@pwnage101
pwnage101 merged commit bfd4598 into master Mar 3, 2020
@pwnage101
pwnage101 deleted the pwnage101/deng-18-rename_site_configuration_siteconfiguration_values_2.2 branch March 3, 2020 16:05
@edx-pipeline-bot

Copy link
Copy Markdown
Contributor

EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production.

@edx-pipeline-bot

Copy link
Copy Markdown
Contributor

EdX Release Notice: This PR may have caused e2e tests to fail on Stage. If you're a member of the edX org, please visit #e2e-troubleshooting on Slack to help diagnose the cause of these failures. Otherwise, it is the reviewer's responsibility. E2E tests have failed. https://gocd.tools.edx.org/go/tab/pipeline/history/deploy_to_stage

@edx-pipeline-bot

Copy link
Copy Markdown
Contributor

EdX Release Notice: This PR has been deployed to the production environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants