Skip to content

doc: ADR for removing MongoDB usage. #29098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ormsbee merged 1 commit into from
Nov 9, 2021
Merged

doc: ADR for removing MongoDB usage. #29098

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions common/lib/xmodule/xmodule/docs/decisions/0002-remove-mongodb-dependency.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
Remove MongoDB as a Dependency
------------------------------

Context
=======

The edx-platform repo uses MongoDB for storing content (the ModuleStore and ContentStore interfaces). MongoDB was chosen early on in the Open edX project because:

* **Course content is very freeform**, and XBlock is a pluggable interface. So while the data stored for videos, problems, crystallography simulations, and circuit schematic editors might have some overlap, individual XBlock are permitted to extend that storage any way they like. At the time, this struck us as a more document-database-centric than SQL.
* **MongoDB’s GridFS** seemed liked a good solution for the static assets that need to be managed on a per-course basis (e.g. PDFs).
* **MySQL at the time did not support JSON fields.** This would have made certain search operations more cumbersome. (MySQL itself was chosen because PostgreSQL was not available on RDS at the time).

Since that time, several things have changed:

#. We've been trying to reduce the complexity of the stack needed to run Open edX.
#. We've switched over from the original ``DraftModuleStore`` to ``DraftVersioningModuleStore``. The latter drops most query patterns and mostly uses MongoDB as a simple key value store.
#. We've adopted django-storages for pluggable file/blob storage.

Decisions
=========

All usage of MongoDB in edx-platform will be removed or replaced in the following ways:

* Old Mongo (``DraftModuleStore``) will be removed as a storage backend altogether. This is already covered under DEPR-58, and affects only old style courses with course keys in the "Org/Course/Run" format (as opposed to the "course-v1:Org+Course+Run" format).
* ContentStore storage of course static assets will be moved to django-storages, allowing for pluggable backends (e.g. files, S3, GridFS).
* Split Modulestore (``DraftVersioningModuleStore``) will be moved to use the Django ORM for the active version lookup, and django-storages to replace key-value storage of what are structure and definition documents today.

@awaisdar001 awaisdar001 Nov 3, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Django ORM" => are you planning to use any existing library for versioning or formulating version look-up schema from scratch?

@bradenmacdonald bradenmacdonald Nov 3, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Versioning will use the same implementation as it does today (in MongoDB), but the data will just be stored in a MySQL row instead of a MongoDB document. You can see the details in https://github.com/edx/edx-platform/pull/29058

However, the move to the MySQL table does allow us to also use django_simplehistory, so we've turned that on for the active versions table, and it provides a historical record of the changes to each course.


Note that this does not include the use of MongoDB for the forums experience. While I still believe that removing MongoDB for that service is desirable, it's beyond the scope of this particular ADR.