Distributions, modules, compunits revisited?

[2colours]

See Raku/problem-solving#485

I don't intend to delve deep into this now - especially that I find it quite confusing and keep forgetting half of it.

Superficially, the situation is the following:

• "packages" are basically namespaces
• "modules" are namespaces with some extra metadata and behavior that is useful if you want to expose them (this level of abstraction doesn't exist in most languages - honestly, shame on them)
• "compilation units" (aka "compunits") are more like C#'s assemblies or source files corresponding to object files in the native world: this is the physical unit that you can execute or import in your code, and it may contain any number of modules (or packages for that matter)
• "distributions" (or "dists") are basically premade bundles of some Raku software that you can install and depend on, they can expose any number of modules and compunits - or they can just expose scripts (which are arguably also compunits but as things stand, not precompiled)

You can see the decoupling increase with each step: packages just organize code that is not API-worthy even internally, modules still exist within your own code but have API-related abilities, compilation units are files that could be shared so they reach out of code stricto sensu, and finally dists are in the scope of dependency managers.

Confer https://docs.raku.org/language/module-packages for the first two. By the way, I'm confused by the code snippet of the "Modules on disk" part - not sure why that name would automagically resolve a file!

There has been a lot of complaints about this system, some of them are related to the current problem:

• first, the correspondence between modules and compunits - and especially modules and dists - is unnecessarily weak
• second, (in)famously dependency resolution mainly (but not exclusively - to make matters more complicated) operates based on module names and metadata, not dist names and metadata, even though part of the metadata is only provided by the dist, and the whole system is a confusing leap of abstractions. We pretend there is no need to care about dists, even though they are the unit of publishing and receiving external code
• there is a plain path based fallback resolution strategy for linking modules(? compunits? whatever you write after use) to file paths, it's called CURFS (CompUnit Repository :: File System). However, CURFS sucks: it sabotages precompilation of any modules that might rely on it, needs to be allowed explicitly (either with the -I flag or use lib <path> in the code), and will eagerly build the given path into a repository which is unjustified and dog slow for large directories.

It's most notably the third complaint for this issue and it has also appeared to me a couple of times: private modules - in particular private local modules or in other words: internal modules - would be the main use case of CURFS. It's highly unlikely you want to version internal modules and anyway - since they are internal, you can solve the problem in any way you desire. This is actually a none-issue in pretty much all languages. But in Raku you cannot use CURFS if you ever want any of your modules precompiled. This means that even your internal module resolutions have to go through the very same path that an external user's calls would go through.

To make this somewhat worse, confer the second complaint: it's not only that all people who have installed this distribution will be able to use all these modules that you never meant to expose: in fact they can install the distribution by any of these 51 names, or depend on any of them which will in turn install this distribution.