Split up Memacs into separate moldule repositories?

[novoid]

As mentioned in #69, there seems to be an issue with having so many dependencies that are installed when a user is interested in one Memacs module only:

I, too, am worried on adding all requirements for all modules. When somebody is only interested in one (simple) Memacs module, why should she/he install a lot of dependencies. Maybe we did a conceptual mistake in the early stage when we decided to use one single repository for all different Memacs modules.

How to deal with this? Split up Memacs into separate repositories/packages (one module = one Python pip package/repository)?

[sanzoghenzo]

Hi,
as stated in #80 (comment), you can use extra_requires to specify the dependencies, for example:

setup(
    name="memacs",
    ...
    extras_require={
        "lastfm":  ["pylast"],
        "twitter": ["twython"],
    }
)

so a user can pip install memacs[lastfm] if you only need lastfm functionality.

Another option is to split it in separate packages, but having them under the same namespace by using namespace packages.

[novoid]

Thanks @sanzoghenzo, this is really good input here!
Are you feeling confident to contribute a modified setup.py? If you could do me this favor, I'd be very glad since you seem to have some experiments with this option while I don't...

[sanzoghenzo]

Hi @novoid ,
honestly I moved to other stuff an I didn't have the time to look at this project...

If I have some spare time I'll look into that. Since I didn't write any of the code, it could take me quite some time and I can incur in errors; also, I don't know what granularity you wish to achieve (does every single module need to be installed separately?)

That said, I'll try to do it.

[sanzoghenzo]

Also note that, for the scripts to be installed as dependencies, and to ensure cross platform compatibility, I need to change #96 and move the script definition to the "console_script" of "entry_points" argument as I explained in #80 (comment).

[novoid]

With the latest contribution of @sanzoghenzo, at least specific module dependencies could be installed optionally. This definitively solves some issues we had.

However, the main question still applies for other reasons as well:

Wouldn't it be better to split the main framework and the modules to separate Git(Hub) repositories?

---

This would have several advantages:

• Third-party modules can be maintained by their original authors much better: clear authorship/owners, issue notifications, ...
• A cleaner separation between the Memacs framework and the modules.
  • Even in this repository, I added some "works-on-my-machine"/standalone modules in the tmp sub-hierarchy. Those should be replaced by proper modules but I don't feel too much pressure for now.
• more? (add comment)

Disadvantages:

• There could be Memacs modules that require older and specific Memacs framework versions.
  • This issue might also apply on a combined repository.
• more? (add comment)

---

In case the repo should be splitted, what are the main tasks to be done?

• Notify external authors
• Keep this repo as framework repo
• Move my own modules to separate repos
  • Losing all their history?
• How to inform the Memacs users of this change properly?

[novoid]

See #114 (comment) for an idea on splitting up within the same repo in order to install a module like pip install memacs[photos]

[sanzoghenzo]

See #114 (comment) for an idea on splitting up within the same repo in order to install a module like pip install memacs[photos]

Hi @novoid ,
the things I stated in that comment are exactly the same as we discussed here, there's no new information there (I didn't remember any of this 😅) - without splitting the project into multiple sub-packages, memacs will be installed in full and only the dependencies can be selectively installed.

The way to go is still to split the code:

setup.py
memacs/
    # No __init__.py here.
    photos/
        # Sub-packages have __init__.py.
        __init__.py
        photos.py

with setup.py containing:

from setuptools import setup, find_namespace_packages

setup(
    name='memacs-photos',
    ...
    packages=find_namespace_packages(include=['memacs.*'])
)

What is up to you is to decide if all these sub packages need to stay in the same repository (this one), or if you can/want create one repo per sub-package.

The latter could be a bit overkill (especially if a sub package consists of only one module) and you will loose the git history, but can be useful for transitioning (especially if you call the new metapackage something different from memacs, so they can live simultaneously on pypi).

[novoid]

Hi @novoid , the things I stated in that comment are exactly the same as we discussed here, there's no new information there (I didn't remember any of this sweat_smile) - without splitting the project into multiple sub-packages, memacs will be installed in full and only the dependencies can be selectively installed.

Thanks for the clarification so that even I was able to understand the implications of your suggested change. ;-)

In this case, I reopen #114 because it does not split up the Memacs modules into separate repositories, allowing for different authors to keep their authorship.

This issue here still follows the idea of separating the Memacs modules, #114 follows the idea of letting the user decide which dependencies she wants to set up.