[DEPR]: Course Discovery Service

[bradenmacdonald]

Proposal Date

23 September 2024

Ticket Acceptance Date

23 October 2024

Technology Removal Date

2025-2027

First Open edX Named Release Without This Functionality

TBD

Rationale

The Course Discovery Service provides a variety of optional functionality in Open edX, some of which is required for other services like the edx-enterprise suite. However, the fundamental architecture of the discovery service has proven to be problematic, and it is currently a limiting factor that makes it much more difficult than it should be to develop and integrate custom functionality for the Open edX platform such as ecommerce, enterprise integrations, subscriptions, and programs.

Here are some of the problems with its current architecture:

• Non-default: The discovery service is not installed by default on most instances of Open edX, which means that "core" functionality is never built using its APIs.

• Out of Sync Course Data: Fundamentally, the discovery service maintains at least two separate sets of models about courses (i.e. the Course model in its MySQL database, and the CourseRunDocument in Elasticsearch). Though there are various systems for keeping this data in sync with Studio (which holds the actual "source of truth"), they are imperfect and synchronization issues are a frequent source of bugs and errors.

• Bi-directional sync: Because the discovery service is the source of truth for some pieces of data, it actually syncs its data to Studio as well as from Studio. This has caused a variety of issues, including bugs with an infinite cycle of updates.

• Other Redundant/Conflicting Data Models: The discovery service has its own database tables for storing important data that is already stored by Studio/LMS (Discovery Organization vs. Core Organization, Mode vs. CourseMode, Person vs. UserProfile, CourseEditor vs. CourseAccessRole). Often, these models have either subtle or substantial differences from the Studio/LMS version, and there is no guarantee that they are in sync at all.

  • Part of the reason this problem has arisen is that there was historically no good mechanism to extend the models in the core platform (and their corresponding APIs) using plugins.

• Competing APIs: some operations like "list courses available to me" can be done with the core platform APIs, the discovery service APIs, or the edx-enterprise APIs. Needless to say, they have different behavior, functionality, caching, etc.

• Kitchen Sink / edX-Specific Functionality: The Course Discovery Service includes, in no particular order:

  • hard-coded edX-trademarked program types
  • hard-coded course run types like paid-bootcamp and bootcamp-2u (not to be confused with Tracks, Modes, or Seat Types which are all related)
  • Subject
  • Topic
  • Fact
  • CertificateInfo
  • SEO Metadata
  • "Taxi" (2U form library) lead capture data
  • random 2U "additional fields"
  • Collaborators
  • Product Value data (for Cost-per-click??)
  • product GeoLocation
  • Course location restriction
  • "CourseReview model to save Outcome Survey analytics coming from Snowflake"
  • Entitlements
  • Corporate Endorsements
  • Program Subscriptions
  • Degrees
  • Curriculum
  • Pathways
  • Person's Social Networks
  • an "API Client...to communicate with google spread sheets and drive images"
  • some Salesforce thing
  • various functionality built on Algolia
  • and many more things.

  It may be the case that the platform lacked ready-to-use extension points at the time these features were required, but they should have been built as extensions regardless.

• Problematic Programs implementation: If you want to group courses together into a "Program", you can do so using the discovery service programs feature. However: it is not well documented; it is limited in functionality; its only management UI is the Django admin; it is not flexible and only supports 2U's original use cases; and it depends on various other apps like a marketing site, ecommerce, etc.

Removal

The entire Course Discovery microservice and the corresponding code in edx-platform that integrates with it. However, see below - this is not going away anytime soon.

Replacement

To rectify this situation, this proposal calls for a series of improvements to the Open edX core (edx-platform and openedx-learning, etc.) that add a "catalog" feature and focus on improving platform extensibility, so that the kitchen sink of features listed above could be largely built using plugins. At the same time, the Discovery service would be officially deprecated.

New Flexible Catalog App

To replace one of the core features of the Discovery Service, a new "Catalog" application should be built into the core platform. The purpose of this app is to provide a unified API for answering the question of "What courses are available to me?". It should be extensible using plugins so that things like Subscriptions and Enterprise Licensing can be accounted for.

This catalog app may also have a flexible concept of "programs" which are essentially groups of related courses, but where each program can define requirements like "pass course A then B, then either C and/or D in order to complete the program." Significant product research and planning is required to define such requirements in more detail.

The catalog app should be developed with "micro-courses" in mind as well - courses significantly smaller than the average MOOC and more like a "Section" of today's typical courses.

The detailed design of the catalog app will happen in future ADRs (and/or an OEP) if this DEPR proposal is accepted. There is already a draft of one initial ADR to add an edx-platform representation of "Catalog Courses".

Extending Core Models

Many of the other "kitchen sink" features listed above were developed in the Discovery service because there wasn't a better place to put them at the time. Instead of finding the least bad place to put specialized functionality like this, we should modify the platform so that the core models can be extended by plugins.

For example, a plugin should be able to add custom fields to the UserProfile model, and have these fields (optionall) show up anywhere user profiles are seen or edited. Likewise the core Course model should be extensible, so a plugin could add (e.g.) a "Price" field to each course and choose to have it show up in the catalog API data, the default catalog UI (when user's browse for courses), and the default course "Advanced Settings" UI. The Open edX Hooks Extension Framework already provides some mechanisms for this, but others will need to be proposed in future ADRs/OEPs.

Roles & Permissions

Achieving a good implementation of a core catalog app and extensible core models will almost definitely depend on a major refactor of roles and permissions, the lack of which has been a long-standing pain point in Open edX.

Deprecation

Deprecating the Discovery Service now means three things:

First, proposing any new functionality for the Open edX Project whose implementation depends on the Discovery Service would be rejected.

Second, maintenance of the Discovery Service would become entirely the responsibility of those organizations that [still] use it (e.g. 2U).

Third, eventually the corresponding hooks and integration code in edx-platform (e.g. sync code, "programs cache, etc.) would be removed. This is likely many years away (see below). However, if this DEPR is accepted, we will make an effort to flag all of this code and corresponding APIs so that it's clear it will be eventually removed, and should not be used for implementing new functionality.

Migration

For any functionality that currently depends on the Discovery Service (e.g. "Programs"), there will be no effort to remove corresponding integration code from edx-platform until an alternative approach along the lines proposed has been developed, stablized, and released for enough time that organizations have a reasonable migration path. Details of that will depend on the implementation. Those who currently depend on the discovery service for functionality critical to their organization may continue to use it for foreseeable future, provided they are able to contribute to its ongoing maintenance.

The goal of this DEPR proposal is not to take the discovery service away as soon as possible, but to focus community resources on developing a better alternative, and to encourage stakeholders to avoid building any new features on top of the discovery service from this point forward.

[felipemontoya]

I'm all for the end-goal of this proposal.

I think every provider has met issues with course-discovery and would like to see a replacement that is actually part of the core, is organization agnostic and can be extended. I know as a provider I definitely want that. I even proposed a pattern of hooks to address this issue among others when I was looking at the possible roadmap for hooks for this year and beyond.

Implement Hook Pattern in Core Components of IDAs
Implement a standardized pattern of hooks in the platform's core components and other IDAs, which currently lack extensibility entry points such as course about page, course progress, search, homepage, login/registration, learner-dashboard, profile page, certificates, credentials, programs page, ecommerce APIs, discovery, catalog and enterprise views. This will involve adding filters at the beginning of each process and firing events at the end, allowing organizations to customize core logic to fit client needs.

Taken from this hooks roadmap document

Something that this proposal is not considering is how the information of the catalog will be displayed. I understand that 2U uses the catalog as an API to feed into their marketing site which is a whole other system also not in the core. Perhaps we could leverage a Course search MFE such as the one proposed here

Some specific comments:

• new catalog app: should this proposal take off, I believe we should put the core primitives like programs or catalog and write the most basic implementation possible. Then leave every bit of product development to plugins. I don't think that requirements for courses or anything so specific should be implemented in core. Mostly a models and calls to extension points should live there.

• extending core models: I love the idea. I however don't know how we could bring this into the hooks framework in a performant way. I'll be happy to collaborate on this until we reach a good solution.

• roles: this is a whole project on itself.

[bradenmacdonald]

Something that this proposal is not considering is how the information of the catalog will be displayed. I understand that 2U uses the catalog as an API to feed into their marketing site which is a whole other system also not in the core.

@felipemontoya My thinking is that a basic "Find Courses MFE" should be included with the platform (similar to how the LMS currently has a super basic page to browse available courses):

It would just use the new core catalog service to show courses to the current user (or to users who aren't yet logged in). I expect that it would be super basic. However, because it's using the new catalog service, it would correctly account for any plugins (subscriptions, enterprise, permissions) that are affecting the catalog - it just might not have any ability to show details of those plugins in the UI. So anyone who needs something more elaborate or custom can either (A) extend it using FPF, (B) replace it with their own custom MFE, or (C) turn it off completely and just integrate the REST API data into their existing external website or external LXP.

[felipemontoya]

Yes, this makes a lot of sense. In that case it would be even more important to lay the correct groundwork during the MFE conversion (platform-roadmap/issues/375) . CC @GlugovGrGlib

[tecoholic]

@bradenmacdonald A huge thanks for this proposal. As someone provision course-discovery and edx-enterprise suite, the recommendations for a new catalogue app and extension of the data models makes a lot of sense.

@felipemontoya

new catalog app: should this proposal take off, I believe we should put the core primitives like programs or catalog and write the most basic implementation possible. Then leave every bit of product development to plugins.

200% agree with this. If I boil down course-discovery and enterprise-catalog, it is simply a set of data and business rules made for a single business, that everyone else trying to use the software have to agree to or work around to satisfy their requirements. I am spending an unorthodox amount of hours just trying to get courses to show up for the learner in enterprise-learner-portal MFE. Business rules of which course gets displayed for which user should be left to the individual organisations to maintain as simple openedx-filter plugins. (side thought: maybe someone should build a low-code/GUI tool that will allow this to be done dynamically without having to put things down in Python config and requiring redeployment)

Some of the relevant work at OpenCraft

While Braden's proposal comes with the broader experience with the stack. I want to highlight a couple of things that are relevant to this proposal.

Note: I don't want this to be taken as an OpenCraft ad of sorts. Trust me, I am not authorised to do it. I merely want to highlight these things so, someone will tell me if there is a better way to do these things or at least provide an opportunity for the community to say, instead of doing this individually, let's do this together.

The catalog app should be developed with "micro-courses" in mind as well - courses significantly smaller than the average MOOC and more like a "Section" of today's typical courses.

There is a plugin that does this from existing courses. - https://github.com/open-craft/section-to-course. It takes a section of a course and publishes it as an independent course. I believe it also allows syncing the content if the parent course is ever updated (might be manual at the moment, but could be automatic).

This catalog app may also have a flexible concept of "programs" which are essentially groups of related courses, but where each program can define requirements like "pass course A then B, then either C and/or D in order to complete the program." Significant product research and planning is required to define such requirements in more detail.

https://github.com/open-craft/learning-paths-plugin/. This is a very early stage project that tried to implement "Programs" as a plugin. "Programs" is a merely a collection of a course, whereas with learning-path we are trying to implement a sequential -or- non-sequential collection with possible inclusion of micro-credentials like Badges.

[bradenmacdonald]

Here is a recent proposal from the community that gives an example of wanting to extend the core platform course model with custom fields: openedx/edx-platform#35593 - seems like a very relevant example.

[regisb]

Thanks a lot for putting this together @bradenmacdonald! I'm looking forward to not having to run a separate discovery service.

The great thing about this DEPR announcement is that it doesn't really require us to build anything new -- that is, until someone steps forward and tells us that Discovery is a required feature for them. Let's use this announcement to gauge community usage. We can then decide whether we want to backport some features in the core, and how.

Wouldn't you want to name a specific release as a deadline? Otherwise I'm afraid there won't be a sense of urgency in the community. I propose to target Teak.

[OmarIthawi]

I support this removal. Discovery, credentials, edx-enterprise and licensing-manager has been all 2U-specific.

It's more likely than not, that Discovery can be better implemented in a customer Marketing Site (e.g. Drupal or similar) than into an edx-platform tied service.

[kdmccormick]

Thanks Braden!

Could you clarify what happens if this is Accepted? Would we archive course-discovery, so that it is not part of Teak?

[bradenmacdonald]

@kdmccormick At a minimum, there will be a ban on developing any new functionality on top of the discovery service. I'd like to say that it won't be officially supported at all as of Teak (but perhaps an org like 2U may continue to maintain and use it if needed for their org's requirements). But I want to hear about who's using it, how they'd be affected, and how willing they are to help build replacement(s) for it and on what timeframe.

[feanil]

I think we should plan on dropping course-discovery by Teak unless there is a lot of community interest in reforming it. If 2U wants to continue to use it and maintain it, I think that can be done off of a fork like with some other services.

[scottrish]

Apologies for coming to this conversation late. While I generally agree with the proposal I think it's important that there is a migration path off the current Discovery service before it is deprecated. In particular I am not in favor of dropping official support until such time that the proposed replacement is available.