How would I use lazy loading with cloup?

[cbandera]

With click I am using the "Custom MultiCommand" approach documented here in Click 7.x or here in Click 8 in order to avoid loading all commands everytime my cli is started.

Since MultiCommand is not supported by cloup and the cloup.Group has different methods, I am unsure how to adapt the lazy loading approach with cloup.

Has anyone tried this out or can give me a hint on how to approach this?

[janluke]

This is an interesting question, thank you for asking! Cloup was definitely not built with this in mind. I'll look into this and come back with something.

to avoid loading all commands everytime my cli is started.

In my experience, unless you have a really huge number of commands, importing the commands has not been an issue in terms of loading time. The real issue has always been third-party dependencies with very slow import time (e.g. youtube-dl is one of them) and the solution was to import those packages only when needed.

Do you have a different experience?

[janluke]

Give a look to this example: https://github.com/janluke/cloup/tree/lazy-loading-example/examples/lazy_loading.

The idea is that rather than extending a MultiCommand, I define proxies (LazyCommand and LazyGroup) for lazy-loaded commands. Then I use those lazy commands as if they were normal commands.

The problem with this approach is that you need to be careful because any attribute/method access on the proxy which cannot be satisfied directly from the proxy itself will cause the command to be loaded. So if you go for this approach, I suggest to put logger.debug("loading module X") on each command module, similarly as I did in the example with print().

[cbandera]

Wow, I like the proxy pattern implementation with the __getattr__ trick a lot. I implemented it in a more complicated way... I'll give it a try after the holidays

[janluke]

Feel free to share the code or open a PR draft. This may be a feature candidate.

[cbandera]

Hi @janluke sorry for the long silence, some things crossed my plans for the timeline... 😄
My implementation looked like the following. I was basically creating two new subclasses and tried to manually identify those parts that actually really do need to load the module (i.e. 'get_short_help_str' and 'get_command'). This does come with it's own problems as well however. E.g. aliases are currently defined in the commands files, but in order to use them with a lazy loaded command they need to be additionally defined in the main cli.py file.... hence the sanity check in the help function:

from pathlib import Path
from typing import List

import cloup

commands_folder = # Path to your commands folder (could of course also be injected at runtime)


def _import_command(cmd_name: str) -> cloup.Command:
    """Load a command from disk."""
    cmd_file = commands_folder / f"cmd_{cmd_name}.py"
    code_obj = compile(cmd_file.read_text(), cmd_file, "exec")  # Compile as module
    # Read symbols into dictionary
    symbols = {}
    eval(code_obj, symbols, symbols)  # pylint: disable=W0123
    return symbols[cmd_name]  # Return the entry function for the command


class LazyCommand(cloup.Command):
    """Custom Command class to allow lazy importing."""

    def get_short_help_str(self, limit=45):
        """Lazy load implementation of parent class command. Additionally tests that the aliases are set correctly."""
        cmd = _import_command(self.name)
        if self.aliases != cmd.aliases:
            raise RuntimeError(
                f"Aliases of LazyCommand {self.name} ({self.aliases}) do not match actual cmd aliases ({cmd.aliases})"
            )
        return cmd.get_short_help_str(limit)


class LazyImportGroup(cloup.Group):
    """Custom Group class to allow lazy importing."""

    def get_command(self, ctx, cmd_name):
        return _import_command(cmd_name)

    def add_lazy_command(self, name: str, aliases: List[str], **kwargs):
        """Add a lazily imported command to this group.

        Lazy importing means that the respective code is only imported when necessary.

        Args:
            name (str): Name of the command.
                        This must match both a file 'cmd_{name}.py' in the 'commands' directory,
                        as well as the main entry point within this file.
            aliases (List[str]): A list of aliases for this command.
                                 Must match the aliases configured in 'cmd_{name}.py'!
            **kwargs: All additional args are forwarded to the parent class.
        """
        super().add_command(LazyCommand(name, aliases=aliases), **kwargs)