Introduce a dynamic addon imp registry to support Foreign Addon Imps

A Foreign Addon Imp is an addon implementation that is put outside of
gravyvalet codebase offered as a Django application. A collection of
Foreign Addon Imps is distributed as a Django project.

AddonRegistry class is the dynamic addon imp registry that holds the
addon imps in use. Even the built-in addon implementation needs to be
registered to be used, by adding the name and imp number pair to
`app.settings.ADDON_IMPS` configuration. `AddonImpNumbers` enum is
discarded. In addition, AddonImpRegistry class replaces the public
interface of `addon_service.common.known_imps` module.
Foreign Addon Imps needs to be registered to `ADDON_IMPS` too. Moreover,
they needs to be registered to `INSTALLED_APPS`.

Author: chiku

FOREIGN_ADDON_IMP_DEVELOPMENT.md

@@ -0,0 +1,167 @@
+# Foreign Addon Imp Development Guide
+
+This guide explains how to develop Foreign Addon Imps for gravyvalet, allowing you to create custom integrations that can be distributed as standalone Python packages.
+
+## Overview
+
+Foreign Addon Imps are Django apps that extend gravyvalet's functionality by implementing new external service integrations. They can be developed independently and distributed as regular Python packages.
+
+## Requirements
+
+- Python 3.9+
+- Django 4.2+
+- gravyvalet
+
+## Development Steps
+
+### Create Your Package Structure
+
+Create a standard Python package structure:
+
+```
+your_addon_imp_package/
+├── setup.py
+├── README.md
+├── your_addon_imp_app/
+│   ├── __init__.py
+│   ├── apps.py
+│   ├── your_imp.py # your AddonImp implementation
+│   └── static/
+│       └── {AppConfig.name}/    # typically, `your_addon_imp_package.your_addon_imp_app`
+│           └── icons/      # Place your addon's icon files here
+│               └── your_icon.svg
+```
+
+A foreign addon imp package can include multiple foreign addon imps. To do so,
+just include multiple Django apps that behave as foreign addon imps.
+
+### Implement Your Django App Config
+
+Create `apps.py` with a class that inherits from `ForeignAddonImpConfig`:
+
+```python
+from addon_toolkit.interfaces.foreign_addon_imp_config import ForeignAddonImpConfig
+from .your_imp import YourServiceImp
+
+class YourAddonImpConfig(ForeignAddonImpConfig):
+    name = "your_addon_imp_package.your_addon_imp_app"
+
+    @property
+    def imp(self):
+        """Return your AddonImp implementation class."""
+        return YourServiceImp
+
+    @property
+    def addon_imp_name(self):
+        """Return the unique name for your addon imp.
+
+        IMPORTANT: This name MUST be unique across all addon imps in
+        gravyvalet installation. Users will reference this name in their
+        ADDON_IMPS configuration.
+        """
+        return "YOUR_ADDON_IMP_APP_NAME"
+```
+
+### Implement Your AddonImp
+
+Create your AddonImp implementation based on the type of service:
+
+```python
+from addon_toolkit.imp import AddonImp
+from addon_toolkit.imp_subclasses.storage import StorageAddonImp
+
+class YourServiceImp(StorageAddonImp):
+    # Implement required methods and properties
+    # See addon_toolkit documentation for details
+    pass
+```
+
+The modules under the `addon_imps` package are good examples to
+implement this part.
+
+### Choose a Unique Addon Imp Name and Document the name
+
+**CRITICAL**: Your `addon_imp_name` must be unique to avoid conflicts.
+
+Before choosing a name, check built-in addon imp names in
+`addon_service.common.known_imps.KnownAddonImps` and avoid to use the
+names enumerated.
+
+Document the name clearly so users know exactly what to use. Since users
+can use the package name of the addon imp application instaed of
+`addon_imp_name` value, document the package name too is a recommended
+manner.
+
+### Adding Icons for Your Addon Imp
+
+Foreign addon imps can provide custom icons that will be available in
+the gravyvalet admin interface.
+
+#### Icon Directory Convention
+
+Place your icon files in the `static/{AppConfig.name}/icons/` directory
+within your addon imp app:
+
+```
+your_addon_imp_app/
+├── static/
+│   └── {AppConfig.name}    # typically, your_addon_imp_package.your_addon_imp_app/
+│       └── icons/
+│           ├── your_service.svg
+│           ├── your_service.png
+│           └── your_service_alt.jpg
+```
+
+#### Supported Formats
+
+- SVG (recommended for scalability)
+- PNG
+- JPG/JPEG
+
+### Package and Distribute
+
+Create a `setup.py` for your package:
+
+```python
+from setuptools import setup, find_packages
+
+setup(
+    name="your-addon-imp-package",
+    version="1.0.0",
+    packages=find_packages(),
+    include_package_data=True,  # Important for including static files
+    install_requires=[
+        "django>=4.2",
+    ],
+    package_data={
+        'your_addon_imp_app': [
+            'static/your_addon_imp_package.your_addon_imp_app/icons/*',  # Include icon files
+        ],
+    },
+)
+```
+
+## Installation and Usage
+
+Users can install and use your foreign addon imps by:
+
+1. Installing your package:
+```bash
+pip install your-addon-imp-package
+```
+
+2. Adding it to Django's `INSTALLED_APPS`:
+```python
+INSTALLED_APPS = [
+    # ... other apps
+    "your_addon_imp_package.your_addon_imp_app",
+]
+```
+
+3. Registering it in gravyvalet's `ADDON_IMPS`:
+```python
+ADDON_IMPS= {
+    # ... other addons
+    "YOUR_ADDON_IMP_APP_NAME": 5000,  # Use a unique (eg. ID >= 5000)
+}
+```

README.md

@@ -56,6 +56,56 @@ To configure OAuth addons:
 4. There fill your client id and client secret (instructions to obtain them are [here](./services_setup_doc/README.md))
 5. Now you should be able to connect these addons according to existing user flows (in ordinary osf app)
 
+## ...use foreign addon imps
+
+Foreign addon imps allow you to extend gravyvalet with additional integrations
+without modifying the core code.
+
+To use foreign addon imps:
+
+1. Install the foreign addon imp package(s):
+```bash
+pip install foreign-addon-imp-package-you-want
+```
+
+2. Add the foreign addon imp(s) to `INSTALLED_APPS` in your Django settings:
+```python
+INSTALLED_APPS = [
+    # ... existing apps ...
+    'foreign_addon_imp_package_you_want.app_name',
+    # ...
+    'addon_service',
+    # ...
+]
+```
+
+3. Register each foreign addon imp to `ADDON_IMPS` with a unique ID number:
+```python
+ADDON_IMPS = {
+    # ... other addons ...
+    "YOUR_ADDON_IMP_NAME": 5001,  # Use a unique number not used by other addons
+}
+```
+
+The name of each addon imp must be documented in the document of the foreign
+addon imp package. If 2 addon imp applications you want to use adopted identical
+names, use the package name instaed:
+
+```python
+ADDON_IMPS = {
+    # ... other addons ...
+    'foreign_addon_imp_package_you_want.app_name': 5001,
+}
+```
+
+The ID numbers must be:
+- Unique across all addon imps
+- Never changed once assigned (changing would break existing configurations)
+
+4. Restart gravyvalet to load the new foreign addon imps
+
+After these steps, the foreign addon imps will be available.
+
 ## ...configure a good environment
 see `app/env.py` for details on all environment variables used.
 

addon_service/addon_imp/models.py

@@ -3,7 +3,7 @@
 from functools import cached_property
 
 from addon_service.addon_operation.models import AddonOperationModel
-from addon_service.common import known_imps
+from addon_service.common.known_imps import AddonImpRegistry
 from addon_service.common.static_dataclass_model import StaticDataclassModel
 from addon_toolkit import AddonImp
 
@@ -19,12 +19,12 @@ class AddonImpModel(StaticDataclassModel):
 
     @classmethod
     def init_args_from_static_key(cls, static_key: str) -> tuple:
-        return (known_imps.get_imp_by_name(static_key),)
+        return (AddonImpRegistry.get_imp_by_name(static_key),)
 
     @classmethod
     def iter_all(cls) -> typing.Iterator[typing.Self]:
-        for _imp in known_imps.KnownAddonImps:
-            yield cls(_imp.value)
+        for _imp in AddonImpRegistry.get_all_addon_imps():
+            yield cls(_imp)
 
     @property
     def static_key(self) -> str:
@@ -35,7 +35,7 @@ def static_key(self) -> str:
 
     @cached_property
     def name(self) -> str:
-        return known_imps.get_imp_name(self.imp_cls)
+        return AddonImpRegistry.get_imp_name(self.imp_cls)
 
     @cached_property
     def imp_docstring(self) -> str:

addon_service/admin/__init__.py

@@ -1,11 +1,15 @@
 from django.contrib import admin
 
 from addon_service import models
-from addon_service.common import known_imps
 from addon_service.common.credentials_formats import CredentialsFormats
+from addon_service.common.known_imps import AddonImpRegistry
 from addon_service.common.service_types import ServiceTypes
 from addon_service.external_service.computing.models import ComputingSupportedFeatures
 from addon_service.external_service.storage.models import StorageSupportedFeatures
+from addon_toolkit.interfaces.citation import CitationAddonImp
+from addon_toolkit.interfaces.computing import ComputingAddonImp
+from addon_toolkit.interfaces.link import LinkAddonImp
+from addon_toolkit.interfaces.storage import StorageAddonImp
 
 from ..external_service.citation.models import CitationSupportedFeatures
 from ..external_service.link.models import (
@@ -26,13 +30,15 @@ class ExternalStorageServiceAdmin(GravyvaletModelAdmin):
     )
     raw_id_fields = ("oauth2_client_config", "oauth1_client_config")
     enum_choice_fields = {
-        "int_addon_imp": known_imps.StorageAddonImpNumbers,
         "int_credentials_format": CredentialsFormats,
         "int_service_type": ServiceTypes,
     }
     enum_multiple_choice_fields = {
         "int_supported_features": StorageSupportedFeatures,
     }
+    dynamic_choice_fields = {
+        "int_addon_imp": lambda: AddonImpRegistry.iter_by_type(StorageAddonImp),
+    }
 
 
 @admin.register(models.ExternalCitationService)
@@ -45,13 +51,15 @@ class ExternalCitationServiceAdmin(GravyvaletModelAdmin):
     )
     raw_id_fields = ("oauth2_client_config", "oauth1_client_config")
     enum_choice_fields = {
-        "int_addon_imp": known_imps.CitationAddonImpNumbers,
         "int_credentials_format": CredentialsFormats,
         "int_service_type": ServiceTypes,
     }
     enum_multiple_choice_fields = {
         "int_supported_features": CitationSupportedFeatures,
     }
+    dynamic_choice_fields = {
+        "int_addon_imp": lambda: AddonImpRegistry.iter_by_type(CitationAddonImp),
+    }
 
 
 @admin.register(models.ExternalLinkService)
@@ -64,14 +72,16 @@ class ExternalLinkServiceAdmin(GravyvaletModelAdmin):
     )
     raw_id_fields = ("oauth2_client_config", "oauth1_client_config")
     enum_choice_fields = {
-        "int_addon_imp": known_imps.LinkAddonImpNumbers,
         "int_credentials_format": CredentialsFormats,
         "int_service_type": ServiceTypes,
     }
     enum_multiple_choice_fields = {
         "int_supported_features": LinkSupportedFeatures,
         "int_supported_resource_types": SupportedResourceTypes,
     }
+    dynamic_choice_fields = {
+        "int_addon_imp": lambda: AddonImpRegistry.iter_by_type(LinkAddonImp),
+    }
 
 
 @admin.register(models.ExternalComputingService)
@@ -84,13 +94,15 @@ class ExternalComputingServiceAdmin(GravyvaletModelAdmin):
     )
     raw_id_fields = ("oauth2_client_config",)
     enum_choice_fields = {
-        "int_addon_imp": known_imps.ComputingAddonImpNumbers,
         "int_credentials_format": CredentialsFormats,
         "int_service_type": ServiceTypes,
     }
     enum_multiple_choice_fields = {
         "int_supported_features": ComputingSupportedFeatures,
     }
+    dynamic_choice_fields = {
+        "int_addon_imp": lambda: AddonImpRegistry.iter_by_type(ComputingAddonImp),
+    }
 
 
 @admin.register(models.OAuth2ClientConfig)