Architectural discovery - should tagging be built as a separate service?

[jmakowski1123]

From Braden:

We need to decide if tagging will be a separate service, or built into the LMS / Learning Core / etc.

Note that "taxonomy-connector", used by edx.org for skill tagging of videos and courses, is built as a library that's integrated into the discovery service: https://github.com/openedx/taxonomy-connector

Some prior tagging work was built as "tagstore" within blockstore but this approach was abandonded because the taxonomy models were not versioned in the same way that course content was.

[giovannicimolin]

References:

• https://discuss.openedx.org/t/when-to-make-a-new-backend-service/8267/7
• https://openedx.atlassian.net/wiki/spaces/AC/pages/1074397222/Architecture+Manifesto
• https://open-edx-proposals.readthedocs.io/en/latest/processes/oep-0057-proc-core-product.html
• https://open-edx-proposals.readthedocs.io/en/latest/processes/oep-0056-proc-architectural-advisory-process.html
• https://discuss.openedx.org/t/attacking-the-monolith-by-extracting-a-core/5383
• https://open-edx-proposals.readthedocs.io/en/latest/best-practices/oep-0049-django-app-patterns.html
• Learning Core Platform Arch Discovery platform-roadmap#67 (comment)

[giovannicimolin]

Implementation options

Pluggable Django App (edx-platform) alongside learning-core

Implement the new tagging service as a pluggable django app, and installed alongside learning-core in the edx-platform

Tagging content is central to enable content re-use, facilitate the implementation of flexible content structures different from the current implementation and allow adaptive learning.

Concerns:

1. If the tagging service ends up eventually being used for content search and marketing, it can scale disproportionally compared to the other apps. Is that a valid problem? Should we be concerned about this now?
   ....

2. My understanding is that tagging content is part of the core of the platform, but tagging things with marketing purposes is not. Because of this, along with the requirement to make the new tagging system generic enough to accommodate both, should we implement it as a separate service?

Possible solutions:

• Implement generic “namespaces” that support different taxonomies/tags for the same content. If the APIs are complete and generic enough - then don’t need to bake in marketing specific code into the tagging capabilities (we’ll need strong adherence to best practices for this to work).
• Implement a “tagging-core”, basically a generic tagging library that can be reused on the learning core and wherever tagging is required (this can cry that can be reused on the learning core and wherever tagging is required (this can cry that can be reused on the learning core and wherever tagging is required (this can cause UI and code fragmentation, but separates content and marketing concerns).

Implementing tagging as Discovery service plugin

Discovery is already used as the source of truth place for course/program/block metadata in the previous implementations, so keeping it here means keeping the platform metadata management "consistent" (specially since it needs to work with all kinds of contents - from atomic learning units all the way up to courses).

Concerns:

• The course discovery repository is not well documented and has "obscure" code that is only used by edX/2U.
• Some implementations are tied to specific closed-source services (example: openedx/taxonomy-connector uses a 3rd party service to tag video blocks automatically).
• As shown here, the data flow from the discovery service is somewhat confusing, and evolved as the needs for that repo changed, not sure we want to build the new tagging system following those footsteps.

Implement tagging as a new IDA

A new IDA dedicated for tagging would represent a break from the current codebases and force implementation using REST APIs.

Concerns:

• One more app to host: maintenance cost increases over time.
• There's a change that the same thing that happened to blockstore happens to this: ends up being transformed into a Django app.
• Extra response time between services when handling synchronous operations related to tagging (not sure if this is valid)/

Notes

No matter which option is picked: this new implementation needs to avoid the pitfalls mentioned in this post and we should implement it as with clear boundaries and APIs, following the principles outlined in OEP-49 and in the Architecture Manifesto.

---

ADR (following OEP-56) to follow once initial discussions here are done.

I'm leaning towards option #1 - building it as a pluggable app and installing it along learning-core. Being able to use Python APIs and working on a single codebase improve developer experience and onboarding speed.
IMO the separate IDA is a bit of overkill, and discovery is a messy repo to work on.

[giovannicimolin]

@bradenmacdonald @ormsbee @ChrisChV @jmakowski1123 This is ready for review.
I'm leaning towards option 1, but I'm curious to hear what you think. I also wrote a few concerns about each approach and on some, suggested solutions.

[ChrisChV]

@giovannicimolin Nice work!!

I'm leaning towards option #1 - building it as a pluggable app and installing it along learning-core. Being able to use Python APIs and working on a single codebase improve developer experience and onboarding speed.
IMO the separate IDA is a bit of overkill, and discovery is a messy repo to work on.

I agree with you, one of the objectives of the tagging app is that it works in the current version of the content libraries and then with learning-core. I feel that with this first option it will be easier to achieve this 👍

[bradenmacdonald]

I think option 1 makes sense, and I don't like option 2 because many people prefer/need to run the platform without the discovery service. Option 3 seems overkill without a compelling reason.

[ormsbee]

I'm in complete agreement with @bradenmacdonald. Let's roll with Option 1.

[giovannicimolin]

@ChrisChV @bradenmacdonald @ormsbee I created an ADR with the decision from this thread in the openedx-learning repo.

[ChrisChV]

@jmakowski1123 The ADR for this issue was merged. I think we can put this task on Done