Content libraries analytics enhancements (SOL-121)

[bradenmacdonald]

Background: This PR implements three changes to support better analytics when content libraries are used:

• Adds a new field to the edit_info of blocks in split mongo's structures table, in order to track the original usage key of blocks copied from a library.
• Adds extra data to LMS events like problem_check such as the library usage ID of the problem (builds on Add module display name to module related server events #2696)
• Emits new assigned and removed events when students are randomly assigned problems from a library

Discussion: Has largely taken place on this Confluence wiki page in the comments and the section near the top entitled "Analytics Considerations".

Dependencies: Depends on the PRs already merged into the content-libraries feature branch.

Sandbox URL: TBD

Prior code reviews: open-craft#26

This implements support for analytics of content libraries, built to implement the ideas of Brian Wilson and Don Mitchell described on the Confluence wiki page in the section near the top entitled "Analytics Considerations".

New tracking events:

Two new events will appear in the tracking logs (see example below):

• edx.librarycontentblock.content.assigned - indicates that a student has been assigned their subset of blocks. Data includes:
  • location The BlockUsageLocator of the LibraryContentModule emitting the event
  • added: a list of data about the blocks added (usage keys in both the course and the library, the block's "update_version", and a flat list of the same information about any descendants the block has)
  • result: a list of the complete set of blocks now assigned to this student (will always equal added at first, but if max_count is increased, added may be different)
  • max_count: The new value of max_count. May be greater than len(result) if the library contains fewer matching blocks than max_count.
  • previous_count: The number of blocks that were previously assigned to this student (before this event)
• edx.librarycontentblock.content.removed - indicates that a previously-assigned block is no longer being shown to this student. This should be very rare. Data includes:
  • location The BlockUsageLocator of the LibraryContentModule emitting the event
  • removed: a list of the blocks that were removed (same format as "added", above but may not have the library usage key and version in some cases)
  • result: a list of the complete set of remaining blocks now assigned to this student
  • reason: Either overlimit (max_count was decreased by the course author) or invalid (Block was deleted from library or library setting was changed and no longer includes this block)
  • max_count: The new value of max_count. May be greater than len(result) if the library contains fewer matching blocks than max_count.
  • previous_count: The number of blocks that were previously assigned to this student (before this event)

To test: As the vagrant user you can run sudo tail -f /edx/var/log/tracking/tracking.log to watch the tracking log. If you sign in to the LMS as a student and view a course containing a LibraryContent block, the first time that you view the courseware should result in an "edx.librarycontentblock.content.assigned" entry appearing in the logs. This entry should appear as below.

Also the Django admin interface at http://localhost:8000/admin/courseware/studentmodule/ is helpful for resetting the student state for your test student user.

Example tracking log entries

First, I made a library with two main blocks - one without children and one with a hierarchy. This is what it looks like:

Next, if a student sees a library content block using this library that has recently been edited to go from showing 1 block to showing 2, they will generate this analytics "added" event:

{
    "username": "honor",
    "host": "precise64",
    "event_source": "server",
    "event_type": "edx.librarycontentblock.content.assigned",
    "context": {
        "course_user_tags": {},
        "user_id": 1,
        "org_id": "BradenX",
        "course_id": "course-v1:BradenX+TEST+1",
        "path": "/courses/course-v1:BradenX+TEST+1/courseware/f61bfa6228e948f59ac1107f6077e00c/1aac970a9128469ea78c8ef1c1d0a965/"
    },
    "time": "2015-01-06T05:47:09.501831+00:00",
    "ip": "10.0.2.2",
    "event": {
        "added": [
            {
                "usage_key": "block-v1:BradenX+TEST+1+type@vertical+block@2a983fb828ee8bf6e34b",
                "original_usage_key": "lib-block-v1:BradenX+OVRD+type@vertical+block@108c4492010d49c2b3844eebdd99626b",
                "original_usage_version": "54ab715f56c02c5a88bb5df9",
                "descendants": [
                    {
                        "original_usage_version": "54ab716d56c02c5a88bb5dfb",
                        "usage_key": "block-v1:BradenX+TEST+1+type@vertical+block@975fbb4d7cde4642e81e",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@vertical+block@d14de4dfc5fc4fc3a2b1e5c08de3c6e8"
                    },
                    {
                        "original_usage_version": "54ab717b56c02c5a88bb5dfd",
                        "usage_key": "block-v1:BradenX+TEST+1+type@html+block@af83d86bd23f3385cb8d",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@html+block@120657bdb6714a3096518e4fc020b1a9"
                    },
                    {
                        "original_usage_version": "54ab6fb356c02c591fefcdb9",
                        "usage_key": "block-v1:BradenX+TEST+1+type@html+block@2779f12e43582c7e0789",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@html+block@8d79a3991a2d467cb9095bd1fa365e71"
                    }
                ]
            }
        ],
        "location": "block-v1:BradenX+TEST+1+type@library_content+block@f3ebf2bfd1d047c284deebbff984e4e3",
        "previous_count": 1,
        "max_count": 2,
        "result": [
            {
                "usage_key": "block-v1:BradenX+TEST+1+type@problem+block@c279e825df7fb020cb2c",
                "original_usage_key": "lib-block-v1:BradenX+OVRD+type@problem+block@4b0043c672b54878b4600f7b1655435c",
                "original_usage_version": "54ab718c56c02c5a88bb5dff",
                "descendants": []
            },
            {
                "usage_key": "block-v1:BradenX+TEST+1+type@vertical+block@2a983fb828ee8bf6e34b",
                "original_usage_key": "lib-block-v1:BradenX+OVRD+type@vertical+block@108c4492010d49c2b3844eebdd99626b",
                "original_usage_version": "54ab715f56c02c5a88bb5df9",
                "descendants": [
                    {
                        "original_usage_version": "54ab716d56c02c5a88bb5dfb",
                        "usage_key": "block-v1:BradenX+TEST+1+type@vertical+block@975fbb4d7cde4642e81e",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@vertical+block@d14de4dfc5fc4fc3a2b1e5c08de3c6e8"
                    },
                    {
                        "original_usage_version": "54ab717b56c02c5a88bb5dfd",
                        "usage_key": "block-v1:BradenX+TEST+1+type@html+block@af83d86bd23f3385cb8d",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@html+block@120657bdb6714a3096518e4fc020b1a9"
                    },
                    {
                        "original_usage_version": "54ab6fb356c02c591fefcdb9",
                        "usage_key": "block-v1:BradenX+TEST+1+type@html+block@2779f12e43582c7e0789",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@html+block@8d79a3991a2d467cb9095bd1fa365e71"
                    }
                ]
            }
        ]
    },
    "agent": "Mozilla/5.0 ...",
    "page": "x_module"
}

And if I then edit the course and change the "count" back to 1, it will generate this analytics removed event when the student next views the course:

{
    "username": "honor",
    "host": "precise64",
    "event_source": "server",
    "event_type": "edx.librarycontentblock.content.removed",
    "context": {
        "course_user_tags": {},
        "user_id": 1,
        "org_id": "BradenX",
        "course_id": "course-v1:BradenX+TEST+1",
        "path": "/courses/course-v1:BradenX+TEST+1/courseware/f61bfa6228e948f59ac1107f6077e00c/1aac970a9128469ea78c8ef1c1d0a965/"
    },
    "time": "2015-01-06T05:36:15.601560+00:00",
    "ip": "10.0.2.2",
    "event": {
        "reason": "overlimit",
        "removed": [
            {
                "original_usage_version": "54ab718c56c02c5a88bb5dff",
                "descendants": [],
                "usage_key": "block-v1:BradenX+TEST+1+type@problem+block@c279e825df7fb020cb2c",
                "original_usage_key": "lib-block-v1:BradenX+OVRD+type@problem+block@4b0043c672b54878b4600f7b1655435c"
            }
        ],
        "location": "block-v1:BradenX+TEST+1+type@library_content+block@f3ebf2bfd1d047c284deebbff984e4e3",
        "previous_count": 2,
        "max_count": 1,
        "result": [
            {
                "usage_key": "block-v1:BradenX+TEST+1+type@vertical+block@2a983fb828ee8bf6e34b",
                "original_usage_key": "lib-block-v1:BradenX+OVRD+type@vertical+block@108c4492010d49c2b3844eebdd99626b",
                "original_usage_version": "54ab715f56c02c5a88bb5df9",
                "descendants": [
                    {
                        "original_usage_version": "54ab716d56c02c5a88bb5dfb",
                        "usage_key": "block-v1:BradenX+TEST+1+type@vertical+block@975fbb4d7cde4642e81e",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@vertical+block@d14de4dfc5fc4fc3a2b1e5c08de3c6e8"
                    },
                    {
                        "original_usage_version": "54ab717b56c02c5a88bb5dfd",
                        "usage_key": "block-v1:BradenX+TEST+1+type@html+block@af83d86bd23f3385cb8d",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@html+block@120657bdb6714a3096518e4fc020b1a9"
                    },
                    {
                        "original_usage_version": "54ab6fb356c02c591fefcdb9",
                        "usage_key": "block-v1:BradenX+TEST+1+type@html+block@2779f12e43582c7e0789",
                        "original_usage_key": "lib-block-v1:BradenX+OVRD+type@html+block@8d79a3991a2d467cb9095bd1fa365e71"
                    }
                ]
            }
        ]
    },
    "agent": "Mozilla/5.0 ...",
    "page": "x_module"
}

Information added to other events

This PR also adds tracking context data (usage_key, original_usage_key, and original_usage_version) to some normal events like problem_check:

{
    "username": "honor",
    "event_source": "server",
    "event_type": "problem_check",
    "context": {
        "course_user_tags": {},
        "user_id": 1,
        "org_id": "BradenX",
        "module": {
            "usage_key": "block-v1:BradenX+OVRD_TEST+1+type@problem+block@c279e825df7fb020cb2c",
            "original_usage_version": "54a2234656c02c0fdaa54f57",
            "display_name": "A Friendly Test Problem",
            "original_usage_key": "lib-block-v1:BradenX+OVRD+type@problem+block@4b0043c672b54878b4600f7b1655435c"
        },
        "course_id": "course-v1:BradenX+OVRD_TEST+1",
        "path": "/courses/course-v1:BradenX+OVRD_TEST+1/xblock/block-v1:BradenX+OVRD_TEST+1+type@problem+block@c279e825df7fb020cb2c/handler/xmodule_handler/problem_check"
    },
    "time": "2015-01-02T22:08:54.091389+00:00",
    "event": {
        "submission": {},
        "success": "incorrect"
    }
}

TODO:

• When viewing the course "Progress" page, spurious assigned events are emitted, because the XBlock assigns children but isn't able to save its state to the database - so in the future, children will be re-assigned. Addressing this issue appears to be too involved to achieve as part of this PR.

[openedx-webhooks]

Thanks for the pull request, @bradenmacdonald! It looks like you're a member of a company that does contract work for edX. If you're doing this work as part of a paid contract with edX, you should talk to edX about who will review this pull request. If this work is not part of a paid contract with edX, then you should ensure that there is an OSPR issue to track this work in JIRA, so that we don't lose track of your pull request.

To automatically create an OSPR issue for this pull request, just visit this link: http://openedx-webhooks.herokuapp.com/github/process_pr?number=6492&repo=edx%2Fedx-platform

[bradenmacdonald]

@antoviaque Can you please set up sandboxes and update the description?

@brianhw @lamagnifica @stroilova Can you please review?

@dmitchell Can you please review the change to split? (Storing the original usage key in edit_info when blocks are copied from a library - you may want to suggest a different name for the new fields)

[brianhw]

LGTM. 👍

[bradenmacdonald]

Test failure is unrelated (bok choy discussion pagination issue fixed on master via 1b7d5f3).

[stroilova]

@bradenmacdonald This looks wonderful. Great documentation + sample events also. Thank you for implementing problem_check annotations + descendants structure as well.

Minor questions:

1. If max_count is decreased, and some blocks are removed, is the max_count value output along with the "reason": "overlimit" flag?
2. If max_count is increased, is an add'l added event emitted, with no 'reason' for it?

That is, do you output max_count (i.e. the expected number of assigned pieces of content) along with each event? Then researchers would know when they have the 'complete set' of assigned pieces, all accounted for, or whether they should be scanning for more added/removed events.
(@brianhw what do you think here?)

In general, 👍

[bradenmacdonald]

@stroilova

1. No, but in general the new max_count is equal to the number of main entries in "result" - unless the library doesn't have enough blocks available.
2. An additional "assigned" event will be emitted for each student when they view the problem, yes. And yes, the "assigned" events don't have the "reason" field.

Let me know if you still feel I should add max_count to the events.

[bradenmacdonald]

Rebased and squashed. Note for reviewers: It's probably now easier to review this PR commit-by-commit rather than line-by-line of the combined diff. I've squashed the commits to tell a nice clean story :)

[brianhw]

@stroilova I'm wondering if what you really want is to have "initial_count" and "max_count" on each of these events. I don't see a consistent way otherwise to track which changes in assignments are due to changes in max_count. The only one that seems unambiguous in itself is an "overlimit" removal -- the length of results equals max_count. But if one has a decrease in max_count and a removal of an invalid block, the resulting event will only be "invalid". One can infer that there was a decrease in max_count only by the absence of a subsequent "assigned" event. Likewise, with an "assigned" event, additions may be caused by an initial assignment or replacing a removed block, in addition to being caused by increases in max_count. Outputting max_count by itself is useful when the length of results is less than max_count because there aren't enough valid blocks to select, as Braden points out. But it won't tell you by itself if the max_count changed. If you knew the initial_count, before any are removed or added, and you knew the max_count, you could identify when max_count changes, and also more easily identify those events representing initial adds. This is helpful when having multiple "initial adds" due to the lack of saving when progress pages are rendered. (One can more easily ignore the earlier "initial adds" and just keep the last one.)

What do you think?

[dmitchell]

Isn't deletion a change? What other change can cause invalid? Perhaps just emit "deleted"? Is the user selecting a subset from the library and thus if today they select [a, b, c] & tomorrow they select [a, c, d] this removes 'c`? (which is not a deletion but a user action).

[bradenmacdonald]

Yes deletion is a change. But it's also possible that the block can be configured to use library A [blocks a,b,c] and then changed to use library B [blocks d,e,f], causing all of the blocks in the course to need to be replaced. But that's not what I would call a "deletion".

Also yes, course author changes can result in the subset of questions used changing from [a,b,c] to [a,c,d] due to different filters being specified.

[dmitchell]

👍

[bradenmacdonald]

Thanks @dmitchell !

[stroilova]

@bradenmacdonald @brianhw I am fine with merging as-is because of time constraints. Perhaps we can add max_count later if needed. Braden, do you see any need to add it now?

I'm going to give a 👍 from my end.

[stroilova]

LGTM 👍