Draft: add rdmo.config app for Plugin model (plugin management)

[MyPyDavid]

Description

• adds a rdmo.config app
  • with a Plugin model
  • the Plugin behaves similar to any other element (Catalog, Taks, View)
• added a new setting PLUGINS = [ .. ] which replaces the legacy settings (e.g. PROJECT_IMPORT).

Deprecations and legacy

• removed rdmo.core.plugins
  • moved Plugin base class to rdmo.config.plugins and renamed to PluginBase

Related issue: #1413

[jochenklar]

Great! Lets discuss in the meeting.

[jochenklar]

I think we can do without this, the Plugin(-class) should know this.

[MyPyDavid]

I have now something like this as a utils.py function, to detect the plugin type from the class inheritance.

PLUGIN_BASES = {
    'optionset_provider': OptionsetProvider,
    'export': Export,
    'import': Import,
    'issue_provider': IssueProvider,
}

def detect_plugin_type(plugin_class):
    if not issubclass(plugin_class, Plugin):
        return "not_an_rdmo_plugin"

    for type_name, base_cls in PLUGIN_BASES.items():
        if issubclass(plugin_class, base_cls):
            return type_name
    return 'rdmo_plugin_unknown_type'

I kept it as a property in the Plugin model:

    @property
    def plugin_type(self) -> str:
        try:
            plugin_class = self.get_class()
        except Exception as e:
            return e.__class__.__qualname__.lower()
        return detect_plugin_type(plugin_class)

[MyPyDavid]

I'm getting some strange DB errors for mysql in ci(and sqlite in local testing).
In my local pytest with sqlite:

django.db.utils.OperationalError: foreign key mismatch - "questions_question" referencing "domain_attribute"

From the ci logs:

# at
query = b'ALTER TABLE `domain_attribute` DROP COLUMN `attributeentity_ptr_id`'

 django.db.utils.OperationalError:
  (1829, "Cannot drop column 'attributeentity_ptr_id': 
 needed in a foreign key constraint 'questions_questionse_attribute_id_30ee3ea9_fk_domain_at' of table 'questions_questionset'")

The problem can be resolved by removing the app 'rdmo.config' from the installed apps again..
Trying to see it is a bug somewhere in the migrations (0037_remove_attribute.py) but can't really find it so far..

[MyPyDavid]

I'm getting some strange DB errors for mysql in ci(and sqlite in local testing). In my local pytest with sqlite:

django.db.utils.OperationalError: foreign key mismatch - "questions_question" referencing "domain_attribute"

Think that I found the bug with help of GPT and could fix it by adding a single dependency to the ('domain', '0038_rename_attributeentity_to_attribute') in the questions.migrations.0032_meta.py file.

GPT:

You’ve got a classic multi-table-inheritance gotcha: the questions app points a FK at the child table’s primary key just before the domain app drops that child PK. SQLite complains with a vague “foreign key mismatch,” while MySQL is blunt: you can’t drop attributeentity_ptr_id on domain_attribute because it’s referenced by questions_questionset / questions_question.

Why this happens (in your tree)

• In domain 0037, the child model Attribute (multi-table inheritance) is dismantled: it first adds a dummy column, then drops the attributeentity_ptr field (which is the PK of the child table!), and finally deletes the model.
• Immediately after, domain 0038 renames the base class AttributeEntity → Attribute, so from here on “Attribute” refers to the base table (with a normal id PK), not the child.
• In the questions app, migration 0031 renames the field attribute_entity → attribute, and 0032 makes that FK point to to='domain.Attribute'. At the time 0032 runs (without extra dependencies), “domain.Attribute” still means the child table (pre-rename), so the FK targets the child table’s PK (attributeentity_ptr_id).
• Then domain 0037 tries to drop attributeentity_ptr on domain_attribute, but the FK from questions still points to that column → MySQL error 1829; SQLite “FK mismatch”.
  You also noticed removing rdmo.config “fixes” it. That’s consistent: the extra app changes the topological order so domain 0037 sneaks in before questions flips its FKs to the renamed/base model, exposing the ordering bug.

[jochenklar]

Ok got it! Good explanation by ChatGPT. This is why you don't do multi-table inheritance and why past-Jochen dismantled it.

[MyPyDavid]

• need to rebase to 2.4.0 now
  PS it ran without any conflicts

[MyPyDavid]

Could implement this already in #1488, see #1488 (comment)

[jochenklar]

Maybe create instance of plugin here an read plugin_type from the class. Handle raw.

[MyPyDavid]

are you sure that the raw case needs to be handled? I could not find any other model that does it on the save method, only the signal handlers are using it so far..
Otherwise it could be simply

    if raw:
        return super().save(*args, **kwargs)

right?

[jochenklar]

Yes you are right, no raw needed. Fixtures use a bulk insert.

[jochenklar]

Thanks for all the work. I added a lot of comments, and fixed some things already in #1574.

[jochenklar]

I think this is always empty, why should Plugin have any rows when running the migration?

[jochenklar]

Here we need to do something about existing integrations in the database, which do not have a configured plugin (anymore).

[jochenklar]

This should be restricted to PLUGIN_TYPES, right? I see that's in the serializer and in admin, but this could be a regular choices arg.

[jochenklar]

We should refactor this after the rebase and use filter_tasks_or_views_for_project since it is basically the same.

[jochenklar]

The plugin specific part can be a filter afterwards.

[jochenklar]

The order needs also be applied, since it matters when two plugins accept the same files for upload.

[jochenklar]

filter_element_for_project?

[jochenklar]

This needs to be .filter(models.Q(catalogs=None) | models.Q(catalogs=project.catalog)).

[jochenklar]

PLUGINS is a list, if not it should crash.

[jochenklar]

This is not an error. I don't know if there is a CommandInfo. Maybe just a self.stdout.write and return?

[jochenklar]

Maybe the script should not merge, but first insert the regular plugins and then read and insert the legacy plugins and skipping the ones that are already in the database (by python path).

[jochenklar]

This file is still to complex, please remove all unnecessary code and comments. Remove all typing. The management class should be at the top of file for readability. All methods (if they are needed) should be in Command.

Also I don't get the part about the database transactions.

[jochenklar]

Maybe the script should be split in two, one for the legacy settings, and one for the future regular RDMO setup. I guess that would make things easier.

[MyPyDavid]

thanks! The #1574 also looks good, I'll try to go through the comments ASAP :D