Introduce Answer.course_key and deprecate course_id. by mtyaka · Pull Request #131 · open-craft/problem-builder

mtyaka · 2016-11-23T08:25:02Z

This patch introduces a new 255-char course_key field to the Answer model. The old 50-char course_id field is still present, but is deprecated and will be removed in next release.

It removes the previous migration (which was never deployed to production) which extended the existing course_id field to 255 characters, which was found to not be the optimal approach on the edx.org production database due to the relevant table's size. We cannot simply extend the existing course_id field because we have some large problem_builder_answer tables in production and the migration to extend the column would lock the table causing issues in production.

The code is updated so that it uses the new course_key column, but falls back to course_id.
In next release, a data migration to copy course_id data to course_key will be provided and the course_id column dropped.

Below is the output of sqlmigrate for the new migration:

BEGIN;
ALTER TABLE `problem_builder_answer` ADD COLUMN `course_key` varchar(255) NULL;
ALTER TABLE `problem_builder_answer` ADD CONSTRAINT `problem_builder_answer_student_id_2ce682a818c95cbc_uniq` UNIQUE (`student_id`, `course_key`, `name`);
CREATE INDEX `problem_builder_answer_c8235886` ON `problem_builder_answer` (`course_key`);

COMMIT;

Next release will add a data migration to copy contents of course_id to the new course_key column. EdX will fake this migration and copy data in batches instead.
Next release will also drop the deprecated course_id column.

See https://github.com/edx/edx-platform/pull/14013#issuecomment-261635454 for more information.

Environments: edx.org and edge.edx.org

Merge deadline: ASAP - this is blocking a course on Edge

JIRA tickets: TNL-5932

Discussions: See https://github.com/edx/edx-platform/pull/14013 and JIRA ticket.

Sandbox URL:

Partner information: hosted on edX Edge (Davidson College)

Testing instructions:

Check that you can successfully add a Problem Builder "Long Answer" component to a course with a course key that is longer than 50 character and that answers answered on an older version of problem-builder continue to work correctly with this patch.

Install problem-builder v2.6.1
Create a course with a combined course key less than 50 characters.
Add a Problem Builder component, then add the Long Answer component.
Answered several questions.
Create a course with a combined course key more than 50 characters.
Add Unit with Problem Builder and a Long Answer element - watched it fail.
Installed problem-builder from this PR.
Run migrations.
Verify that the answers are still present in the first course.
Verify that the Long Answer element in the course with the very long course ID is working now, and you can submit answers to it.

Reviewers

@pomegranited

Migration locks the table, which is not desired since the production table on edx.org is quite large. The course_id field will be extended in a multiple steps instead. For more info see: - https://github.com/edx/edx-platform/pull/14013 - #131

pomegranited

👍 @mtyaka

I tested this on my devstack:
1. Installed problem-builder v2.6.1
2. Created a course, and imported the Problem Builder Demo Course.
3. Answered several questions.
4. Created a test course with a very long course ID
5. Added Unit with Problem Builder and a Long Answer element - watched it fail.
6. Installed problem-builder from this PR
7. Ran migrations
8. Verified that my answers are still present in the Demo course.
9. Verified that the Long Answer element in the course with the very long course ID is working now, and I can submit answers to it.
I tested this on the sandbox instance:
1. Created a course with a course ID > 50 chars
2. Added Unit with Problem Builder and a Long Answer element.
3. Verified that I could answer the question in the LMS.
I read through the code
~~I checked for accessibility issues~~
Includes documentation

pomegranited · 2016-11-24T00:43:27Z

+    # course_key is the new course_id replacement with extended max_length.
+    # We need to allow NULL values during the transition period,
+    # but will be set to null=False in next release.
+    course_key = models.CharField(max_length=255, db_index=True, null=True)


Will the uniqueness constraint ('student_id', 'course_key', 'name') get us into trouble if all the initial course_key values are null?

No, multiple NULL values are not considered to be identical to each other so they don't violate the uniqueness constraint.

This introduces a new 255-char course_key field to the Answer model. The old 50-char course_id field is still present, but is deprecated and will be removed in next release. We cannot simply extend the existing course_id field because we have some large problem_builder_answer tables in production and the migration to extend the column would lock the table causing issues in production. The code is updated so that it uses the new course_key column, but falls back to course_id. In next release, a data migration to copy course_id data to course_key will be provided and the course_id column dropped.

mtyaka · 2016-11-25T07:13:17Z

Rebased from f0d0ce5.

mtyaka · 2016-11-25T07:20:11Z

Thank you @pomegranited!

maxrothman · 2016-11-29T18:56:32Z

+                         WHERE name = %s
+                           AND student_id = %s
+                           AND (course_key = %s
+                                OR course_id = %s)'''


Why are you writing raw SQL here rather than using the ORM?

@maxrothman Only because I was not familiar with django Q objects and didn't know django ORM supports OR queries. Thanks for the hint, I opened #136 to convert the raw SQL query to Q objects - can you please review?

If it looks good, I'll update the version number at https://github.com/edx/edx-platform/pull/14044 and we can finally get that one merged.

It includes support for course keys longer than 50 characters. See: open-craft/problem-builder#131

mtyaka force-pushed the mtyaka/extended-course-key branch 6 times, most recently from e4fd2d5 to f0d0ce5 Compare November 24, 2016 08:20

mtyaka mentioned this pull request Nov 24, 2016

Update problem-builder dependency. edx/edx-solutions-edx-platform#757

Merged

2 tasks

mtyaka mentioned this pull request Nov 24, 2016

Remove migration that extends course_id column #133

Merged

1 task

pomegranited approved these changes Nov 24, 2016

View reviewed changes

mtyaka force-pushed the mtyaka/extended-course-key branch from f0d0ce5 to b855567 Compare November 25, 2016 07:13

Bump version to 2.6.4.

113d5d6

mtyaka merged commit 2f322a8 into master Nov 25, 2016

mtyaka deleted the mtyaka/extended-course-key branch November 25, 2016 07:37

mtyaka mentioned this pull request Nov 25, 2016

Bump problem-builder to v2.6.5 openedx/openedx-platform#14044

Merged

3 tasks

maxrothman reviewed Dec 6, 2016

View reviewed changes

mtyaka mentioned this pull request Dec 7, 2016

Use django Q objects instead of raw SQL query #136

Merged

2 tasks

mtyaka added a commit to open-craft/openedx-platform that referenced this pull request Dec 12, 2016

Bump problem-builder to v2.6.5.

86177cb

It includes support for course keys longer than 50 characters. See: open-craft/problem-builder#131

mtyaka added a commit to open-craft/openedx-platform that referenced this pull request Dec 13, 2016

Bump problem-builder to v2.6.5.

d97555c

It includes support for course keys longer than 50 characters. See: open-craft/problem-builder#131

mtyaka mentioned this pull request Dec 13, 2016

Bump problem-builder to v2.6.5. openedx/openedx-platform#14143

Closed

adampalay pushed a commit to openedx/openedx-platform that referenced this pull request Dec 14, 2016

Bump problem-builder to v2.6.5.

90707af

It includes support for course keys longer than 50 characters. See: open-craft/problem-builder#131

adampalay mentioned this pull request Dec 14, 2016

Bump problem-builder to v2.6.5. openedx/openedx-platform#14148

Merged

mtyaka mentioned this pull request Jan 13, 2017

Copy course_id into course_key and remove temporary transition code #138

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce Answer.course_key and deprecate course_id. - #131

Introduce Answer.course_key and deprecate course_id.#131
mtyaka merged 2 commits into
masterfrom
mtyaka/extended-course-key

mtyaka commented Nov 23, 2016 •

edited

Loading

Uh oh!

pomegranited left a comment

Uh oh!

pomegranited Nov 24, 2016

Uh oh!

mtyaka Nov 25, 2016

Uh oh!

mtyaka commented Nov 25, 2016

Uh oh!

mtyaka commented Nov 25, 2016

Uh oh!

maxrothman Nov 29, 2016

Uh oh!

mtyaka Dec 7, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mtyaka commented Nov 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pomegranited left a comment

Choose a reason for hiding this comment

Uh oh!

pomegranited Nov 24, 2016

Choose a reason for hiding this comment

Uh oh!

mtyaka Nov 25, 2016

Choose a reason for hiding this comment

Uh oh!

mtyaka commented Nov 25, 2016

Uh oh!

mtyaka commented Nov 25, 2016

Uh oh!

maxrothman Nov 29, 2016

Choose a reason for hiding this comment

Uh oh!

mtyaka Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mtyaka commented Nov 23, 2016 •

edited

Loading