Content libraries prototype

[bradenmacdonald]

Background: This is an early-stage prototype implementation of Content Libraries (SOL-11). Once this PR is ready, it is intended to be merged so that it can be used as a common basis for further work on the feature. It is not intended for use on production instances anytime soon, so is disabled by default using a feature flag. Backwards-incompatible/breaking changes may be introduced in the near future.

Discussion: Architecture discussed extensively on the wiki and in meetings, then the revised proposal was presented to the Arch Council on Oct. 21 and given thumbs up. Some open questions remain.

Partner information: 3rd party-hosted open edX instance, for an edX solutions client.

Merge deadline: TBD

Dependencies: Requires openedx/opaque-keys#46, which is already merged.

Status: Demonstrates the very basic features of libraries.

Future: The following will not be part of this PR:

• UI/UX improvements, including pagination (code for this is ready, PR coming after this one merged)
• Course-specific overrides (implementing this requires careful thought and code review so I'd like to do it separately in an upcoming pull request). Note that this means that at present, if you import content from a library into a course, then edit the imported blocks from the course view in studio, any such changes will be lost when synchronizing/updating with the latest content blocks from the library. (code for this is ready, PR coming after this one merged)
• filtering/tagging
• Support of libraries from non-split courses
• Performance tests (can we have 1,000 problems in the library? 10,000?) - very much depends on how course overrides are implemented, so this is deferred for now
• Permissions - currently uses the same permissions system as courses. (code for this is ready, PR coming after this one merged)

Testing Instructions:

1. Check out this branch and do paver install_python_prereqs
2. As 'edxapp' user, edit ~/cms.env.json and add "ENABLE_CONTENT_LIBRARIES": true to the FEATURES object.
3. Edit ~/cms.auth.json. Go to MODULESTORE > default > OPTIONS > stores - it should contain two or three entries. Re-order the entries so that the store with "NAME": "split", comes first in the stores list. (This makes split the default modulestore for any new courses.)
4. Start paver devstack studio and go to localhost:8001 in your browser.
5. Log in as staff, and create a new library (a new "New library" button should appear at top right of studio home page).
6. Add some content to the library.
7. Go to studio home and go into a course or create a new course.
8. Go to Settings > Advanced Settings and add "library_content" to advanced_modules.
9. Go to Content > Outline. Create a new section and subsection, then edit it. Click "Advanced" and add a new "library_content" block.
10. Once the block is added, click "Edit" to edit its settings. Most are explained on the settings editor. For the "Library" setting, you must click "Add", then paste the LibraryLocator of your library. This looks like library-v1:ProblemX+PR0B. (It's in the URL when you go to the library in studio). Hit Save.
11. Click "Update now with latest components from the library".
12. Optional: Click "View >" to preview all the blocks found in the library.
13. Hit "Publish", then start up the LMS and go to localhost:8000. Browse into the course, and you should see that the block is replaced by random blocks from the library. To test the randomization, click the lowest "STAFF DEBUG INFO" link underneath the module, then click "Delete Student State", then refresh the page.

[openedx-webhooks]

Thanks for the pull request, @bradenmacdonald! I've created OSPR-173 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams. Feel free to add information to the ticket which can help Product understand the context for the PR - supporting documentation, edx-code email threads, timeline information ('this must be merged by XX date', and why that is), partner information (this is for a course on edx.org, for example), etc.

All technical communication about the code itself will still be done via the Github pull request interface. As a reminder, our process documentation is here.

[sarina]

@bradenmacdonald I assume this doesn't need any review attention yet; let me know if/when that changes.

[coleshaw]

Trying to run this in my devstack, I have run into two issues so far. Not sure if others will see the same things since my devstack is pretty old, but wanted to document them.

1. Had to update node.js version, following the solutions from therefromhere's comment on this SO thread: https://stackoverflow.com/questions/12913141/installing-from-npm-fails/21715730#21715730
2. Had to apply the mongo auth patch, from the Google Groups thread, here: https://groups.google.com/forum/#!searchin/edx-code/pymongo/edx-code/esHKA2Zs2lY/CDbTMZNppQcJ

So far I have no "add library" button, but will dig into that further (stuck at step 5).

[antoviaque]

@cjshawMIT Thank you for helping to test this! For the issues 1&2, this seem because you are updating an older devstack image, right? Ie, did you had these issues to run the current master too, or just the current branch?

For your issue with the 'Add library' missing, I had a quick look at the code, I'll comment inline.

[antoviaque]

@bradenmacdonald Wouldn't that cause the "New library" button to only appear if there are already libraries in the DB? Cf https://github.com/edx/edx-platform/pull/5779/files#diff-968532c6bba75e1f61d1e1ddb51d7842R432

[bradenmacdonald]

@antoviaque Yep, good point. That explains @cjshawMIT's issue. Will fix.

[coleshaw]

@antoviaque , yes, I think it was johnnycake or pre-johnnycake, not sure. Am trying it again today on kifli. Thanks to both for looking into that!

[coleshaw]

Running on a new kifli devstack, I ran into this npm issue when trying to run paver devstack studio after checking out the PR. So I had to run sudo /edx/bin/update edx-platform master, check out the PR, then run paver devstack studio. May not apply to all systems, just noting it here in case someone else runs into the same issue.

[bradenmacdonald]

@cjshawMIT The "New Library" button should now show up, if you pull the fix I just pushed. Also, to clarify, the node.js, NPM, and mongo issues are all totally unrelated to libraries, and would apply if updating to the current master as well.

[bradenmacdonald]

@sarina (cc @dmitchell @antoviaque) I think this is ready for review now.

n.b. I have a separate PR that I will open soon that deals with Scope.settings fields and allows courses to override those on library blocks - it turned out to be a complex change so I want to make sure we can review that separately. See the notes under the "Future" heading in this PRs description.

[pdpinch]

Braden - We've got a WIP PR that touches on scope.settings overrides as well: https://github.com/edx/edx-platform/pull/5802 -- is there some common cause here?

When you submit your PR, could you tag @cpennington and @chrisrossi ?

Thanks.

On Nov 2, 2014, at 3:57 AM, Braden MacDonald notifications@github.com wrote:

n.b. I have a separate PR that I will open soon that deals with Scope.settings fields and allows courses to override those on library blocks - it turned out to be a complex change so I want to make sure we can review that separately. See the notes under the "Future" heading in this PRs description.

[coleshaw]

@bradenmacdonald , I may have done something stupid when trying to rebase this PR, but I now see the + New Library button -- yay! However, I get a missing plugin error when I try to create a new library ... did I miss an install step? This may also be a generic edX platform question, but it seems like the platform does not recognize the new Library entry points in the xmodule/setup.py. They are present in te file -- is there some trick to reloading / refreshing the entry points? Thanks! Error message below:

2014-10-31 16:24:50,131 ERROR 15169 [root] exceptions.py:9 - Uncaught exception from <class 'django.core.handlers.wsgi.WSGIHandler'>
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 109, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 20, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/views/library.py", line 77, in library_handler
    return _create_library(request)
  File "/edx/app/edxapp/edx-platform/common/djangoapps/util/json_request.py", line 24, in parse_json_into_request
    return view_function(request, *args, **kwargs)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/views/library.py", line 112, in _create_library
    fields={"display_name": display_name},
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py", line 89, in inner
    retval = func(field_decorator=strip_key_collection, *args, **kwargs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py", line 632, in create_library
    library = store.create_library(org, library, user_id, fields, **kwargs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py", line 1654, in create_library
    return self._create_courselike(locator, user_id, **kwargs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py", line 1549, in _create_courselike
    partitioned_fields = self.partition_fields_by_scope(root_category, fields)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/__init__.py", line 800, in partition_fields_by_scope
    cls = self.mixologist.mix(XBlock.load_class(category, select=prefer_xmodules))
  File "/edx/app/edxapp/venvs/edxapp/src/xblock/xblock/plugin.py", line 107, in load_class
    selected_entry_point = select(identifier, all_entry_points)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/__init__.py", line 1163, in prefer_xmodules
    return default_select(identifier, entry_points)
  File "/edx/app/edxapp/venvs/edxapp/src/xblock/xblock/plugin.py", line 37, in default_select
    raise PluginMissingError(identifier)
PluginMissingError: library
2014-10-31 16:24:50,143 ERROR 15169 [django.request] base.py:213 - Internal Server Error: /library/
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 109, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 20, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/views/library.py", line 77, in library_handler
    return _create_library(request)
  File "/edx/app/edxapp/edx-platform/common/djangoapps/util/json_request.py", line 24, in parse_json_into_request
    return view_function(request, *args, **kwargs)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/views/library.py", line 112, in _create_library
    fields={"display_name": display_name},
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py", line 89, in inner
    retval = func(field_decorator=strip_key_collection, *args, **kwargs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py", line 632, in create_library
    library = store.create_library(org, library, user_id, fields, **kwargs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py", line 1654, in create_library
    return self._create_courselike(locator, user_id, **kwargs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py", line 1549, in _create_courselike
    partitioned_fields = self.partition_fields_by_scope(root_category, fields)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/__init__.py", line 800, in partition_fields_by_scope
    cls = self.mixologist.mix(XBlock.load_class(category, select=prefer_xmodules))
  File "/edx/app/edxapp/venvs/edxapp/src/xblock/xblock/plugin.py", line 107, in load_class
    selected_entry_point = select(identifier, all_entry_points)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/__init__.py", line 1163, in prefer_xmodules
    return default_select(identifier, entry_points)
  File "/edx/app/edxapp/venvs/edxapp/src/xblock/xblock/plugin.py", line 37, in default_select
    raise PluginMissingError(identifier)
PluginMissingError: library
[31/Oct/2014 16:24:50] "POST /library/ HTTP/1.1" 500 38480

[dmitchell]

@cjshawMIT you'll need to reinstall opaque keys and run setup.py. paver should do it for you but may not. Ensure you run either paver install_prereqs or pip install -r github.txt (in the requirements dir). Then run python setup.py install in the opaque_keys directory.
@bradenmacdonald you'll need to ensure the standard paver commands do the right thing otherwise every server/person running this will hit the same problem. I think it should be right already, but someone should verify.

[coleshaw]

Thanks, Don, I will try it out as soon as I can get a fresh devstack. It is highly possible this is user error on my part and the paver files take care of everything properly. I am still trying to understand the edX stack and where / how to install various components. —Thanks!

[singingwolfboy]

Looks like this pull request needs to be rebased. Can you take care of that, @bradenmacdonald?

[bradenmacdonald]

@cjshawMIT @dmitchell I just confirmed, the normal paver process should set this up properly.

$ git checkout master
edxapp@precise64:~/edx-platform$ python
>>> import pkg_resources
>>> list(pkg_resources.iter_entry_points(group='course_key'))
[EntryPoint.parse('slashes = opaque_keys.edx.locator:CourseLocator'), EntryPoint.parse('course-v1 = opaque_keys.edx.locator:CourseLocator')] # Notice no library key support

$ git checkout content-libraries-prototype

edxapp@precise64:~/edx-platform$ paver devstack 
(installs python prereqs automatically)
(libraries now working)

@singingwolfboy No problem; rebased now. The conflict was extremely minor - just an adjacent line in requirements/github.txt

[bradenmacdonald]

@cjshawMIT You should not have run python setup.py install on common/lib/xmodule. Instead, to manually force setuptools to update it you should have just run python setup.py develop in that folder. To fix: Run pip uninstall xmodule then try paver install_python_prereqs or if that still does nothing, checkout master then run paver install_python_prereqs then checkout this branch again and paver install_python_prereqs.

New question answers:

• Nothing appears under Common Problem Types: that's really weird. Probably related to the XModule issues you were having. Check the normal course editing view - they two should offer the same options.
• First time you edit nothing happens: You're trying to use LibraryContentModule in a course that's stored on old mongo. Double-check that you followed step 3 of the instructions, and make sure you've created a new course. The course ID (in the studio URL) should look like "course-v1:something+something+something".

[coleshaw]

Hi Braden, thanks for your help -- you hit the nail on the head for both the issues. Apologies for my ignorance with the setup. Working great!

[bradenmacdonald]

@cjshawMIT You're welcome - thanks a lot for checking this branch out!

[bradenmacdonald]

@chrisndodge @marcotuts @antoviaque FYI, I just pushed a fix for the other issue seen where library_content was not available for use in studio.

[bradenmacdonald]

For those interested, implementations of pagination, Scope.settings overrides, and content library permissions are now linked in the "Future" section of this PR's description. I will open PRs for them once this one has been reviewed and merged, so that each one can be reviewed on its own.

[cahrens]

BTW, here is the current code coverage report for this PR:

https://jenkins.testeng.edx.org/job/edx-all-tests-auto-pr/18111/SHARD=1,TEST_SUITE=unit/artifact/reports/diff_coverage_combined.html

I wasn't sure how to find this since the build failed due to pep8/pylint errors. Here is how you can get to the report as long as the "unit" phase passed:
click on the passing unit build
then click "build artifacts"
then "reports"
then "diff_coverage_combined.html"

[bradenmacdonald]

Thanks @cahrens. I noticed however that the coverage report is wrong... for example, it's highlighting the new LibraryFactory code as not being used, although it clearly is used in at least four different test files. Also modulestore methods like get_library, create_library, and get_libraries are showing up as having no coverage but they are definitely used in the tests (e.g. the test test_list_libraries in test_libraries.py). Any ideas on why that is?

[cahrens]

I'd recommend you check in with test eng about diff cover issues.

I seem to recall that if your unit tests live in CMS but the code you are trying to count coverage for lives in common, the CMS unit tests will not count towards coverage. That may be an issue that you are hitting.

[sarina]

@bradenmacdonald I recommend you verify that all the methods are actually being called in the tests - perhaps by running tests individually and using print statements, or using a debugger and stepping through every function. Sometimes you think things are being executed when they actually aren't.

If diff-cover is actually mis-reporting things, I'll help you get in touch with test eng.

[bradenmacdonald]

Ah, thanks @cahrens, your comment explains it: all of my new tests live in CMS (since libraries are only used within the CMS).

I will increase the coverage of the CMS components. I think the changes in common are covered fairly well by the included tests, though they won't show up on coverage reports.

BTW, some aspects of this PR won't be covered by tests yet, like permissions and field overrides, since the implementations of those have already been completely changed in the next set of PRs that are waiting on this one.

(cc @sarina)

[bradenmacdonald]

I just added some more tests to increase the coverage of this PR (for cms anyways) and rebased it, as there have been a few conflicting changes on master in the meantime. All the tests and quality checks should be passing now, as this now includes the changes from #5814 (ignore TODO in pylint).

Edit: Spoke too soon. Apparently 5814 is not applied on the Jenkins server.

[cahrens]

Don't neglect Jasmine tests for JavaScript (and bok choy tests where appropriate)-- JavaScript code needs to have test coverage as well. You will find it much easier to write Jasmine tests if you use Backbone models and views. There are many examples in Studio.

[bradenmacdonald]

Hey @jzoldak, question for you: does code in common require its own tests in common or is it ok if the test coverage is provided by tests that live in cms? As we've been discussing in this PR, the coverage report shows the changed code in common has no coverage, since the testing has all been added to cms.

[dmitchell]

It's preferable to have the coverage local and targeted, but I'm not saying
it's a requirement.

On Mon, Nov 17, 2014 at 10:09 AM, Braden MacDonald <notifications@github.com

wrote:

Hey @jzoldak https://github.com/jzoldak, question for you: does code in
common require its own tests in common or is it ok if the test coverage
is provided by tests that live in cms? As we've been discussing in this
PR, the coverage report shows the changed code in common has no coverage,
since the testing has all been added to cms.

—
Reply to this email directly or view it on GitHub
https://github.com/edx/edx-platform/pull/5779#issuecomment-63317935.

[sarina]

I think it is better to add the tests next to the code. If the code lives
in common/ the tests should live in common/ so that it's clear to future
maintainers of the code where the tests are.

On Mon, Nov 17, 2014 at 10:14 AM, Don Mitchell notifications@github.com
wrote:

It's preferable to have the coverage local and targeted, but I'm not
saying
it's a requirement.

On Mon, Nov 17, 2014 at 10:09 AM, Braden MacDonald <
notifications@github.com

wrote:

Hey @jzoldak https://github.com/jzoldak, question for you: does code
in
common require its own tests in common or is it ok if the test coverage
is provided by tests that live in cms? As we've been discussing in this
PR, the coverage report shows the changed code in common has no
coverage,
since the testing has all been added to cms.

—
Reply to this email directly or view it on GitHub
https://github.com/edx/edx-platform/pull/5779#issuecomment-63317935.

—
Reply to this email directly or view it on GitHub
https://github.com/edx/edx-platform/pull/5779#issuecomment-63318798.

[bradenmacdonald]

This has been superseded by #6033 and #6046.