Transpilation approach

[bakaq]

Continued from #25 (comment).

Currently we have one file that serves as both the package manager itself (receiving commands to manage projects and dependencies) and as the package loader (the thing that injects the actual code of packages inside. This is a low effort way to do it, but to do it properly we depend on the Prolog implementation having at least term expansion.

A more general way that we probably want to switch to eventually is to do transpilation: we would get all the relevant files (of the project and all it's dependencies) and turn it into something that the underlying Prolog implementation can actually understand before it actually runs it. That would make it possible, if we do it right, to do a package manager that works really well even for ISO Prolog systems that don't even have modules or term expansion. It also would mean that we don't need a loader. LogTalk is a very mature project that does basically that (that's why I've been calling this approach "reimplementing LogTalk"), but it's scope is much bigger (it's actually an entire object oriented language built on top of Prolog), and we could probably do some things way better.

So here's a guiding wishlist of features we would want Bakage to have when doing transpilation, including some that would help us both differentiate from and improve upon LogTalk:

1. It should work with purely ISO 13211-1 5.1e packages, and be able to run those in any ISO compliant Prolog implementation. (This is the bare minimum for an "ISO Prolog package manager")
2. It should be able to provide access to implementation specific features of systems. Things like dif/2 or library(clpz).
   • Crucially, the way this should be done is by feature detection, rather than by writing shims for specific systems (which is what I think LogTalk does). In that way, any new system that implements the required features is automatically supported.

Here are some things I think should be non-goals (at least in the middle term, but probably forever):

1. Providing "new language features" apart from the concept of packages (and maybe environments, more on this later). Things like objects, preprocessor, etc... LogTalk goes very bonkers with this but we really shouldn't.
2. Go out of our way to support ISO non-conforming behavior. This is both to reduce implementation scope and to put pressure on implementations that want to use this to take conformance more seriously.

Providing implementation defined and specific features to packages

The way I'm thinking about this is that a package should declare in it's manifest which features it needs. Something like this:

prolog_features([
  "quintus_modules_base",
  "use_module_library_clpz",
  "use_module_library_lists",
  "use_module_library_lists_contains_maplist/2"
]).

This would mean that this package uses some basic Quintus-style modules features (I think that would cover only the basics such as use_module/1 existing, and weirder behavior like how operators are handled would be a different feature), that it will use the builtin libraries library(clpz) and library(lists), and that library(lists) provides maplist/2 (it's in other places in some systems).

The package manager could then check if libraries have compatible features, enforce that the features of a package are a superset of the features of it's dependencies, do feature detection on a implementation to check if it's compatible with a package, etc...

But this is assuming that we just "pass-through" the implementation specific features, and that we would write code basically exactly as would be written for the Prolog implementation that will run the package in the end. We could also shim the features.

Environments (?)

This gets into something that may be a bad idea or out of scope, and it's definitely much more complicated, but we could also provide an unified environment that would shim the features that we want so that they appear in an unified way to package writers. For example, we could have this in the manifest:

prolog_environment([
    "quintus_modules"
    "library_lists",
    "library_dif"
]).

And the package code be like:

:- use_module(library(lists)).
:- use_module(library(dif)).

example(A, B, C) :-
    dif(A, 2),
    maplist(=, [1,2,3], [A, B, C]).

We would basically provide an unified and opinionated interface that on transpilation would rearrange things so that it would work on every system with compatible features. So here we have a library(lists) that provides maplist/3, but an implementation may provide that in library(apply) or even have it always be available instead, and the transpilation step would rearrange the code so that it also works in these cases. For that the package manager would do feature detection on the Prolog implementation to see if it supports the needed features in any way that it know how to deal with, and then transpile to that set of features as a target.

This is too much work and very close to actually reimplementing LogTalk, so I'm not sure we should tackle this in the middle-short term even if we agree it's a good idea.

Feature detection

Ok, so the thing that would make this all work: how do we actually do feature detection for implementation specific features in Prolog systems? ISO Prolog doesn't provide anything close to what we need here (there's the prolog flags, but they are nowhere close to as fine grained as would be necessary and implementations generally don't publicize many features there anyway). My idea here is to rely on the ISO Prolog IO predicates to detect features, and do really heavy end-to-end tests. I think this is fine, because we could cache them and so they would only need to be run 1 time for end users. We could even pre-populate the results for known systems, though I stress that it's very important that this could be done from 0 for both previously known systems and also any new ones that eventually appear.

So, for example, to test for Quintus-style modules we could:

1. Create a temporary directory, and two files mod.pl and prog.pl.
2. mod.pl would define a module as follows:

   :- module(mod, [example/0]).
   example :- write('Quintus-style modules are supported!'), nl.

3. prog.pl would use the module as follows:

   :- use_module(mod).
   :- initialization(main).
   
   main :- 
       catch(example, _, halt(1)),
       halt.

4. We would then use the Prolog implementation command that the user specified to run prog.pl, with a timeout if it hangs for some reason, and recording it's stdout, stderr and status code. If it matches what we expect (no timeout, nothing in stderr, "Quintus-style modules are supported!\n" in stdout, 0 status code) then we consider the feature supported for that system.

I think this generalizes quite well for basically any implementation specific feature that we would want to test. It would involve a lot of scripting, but we are already doing that here anyway.

As I said, this could be cached locally and then reused for later invocations. We could run the entire known feature tests on all publicly available Prolog implementations in CI periodically and then display them in a table in a site for documentation. This would help package writers know what tradeoffs they are making when choosing certain features. We could also ship the pre-compiled caches these CI runs would generate with the package manager so that you wouldn't need to run all the tests locally for known systems.

Linting?

I think we should in some way warn package writers if they are unknowingly using features that are not declared in the manifest, but this is probably really hard to do and maybe better as a different tool.

Conclusion

Sorry for the length of this, and how many steps ahead I'm thinking here, but this is something that's been on my mind for a very long time. I would love to have criticism here and maybe even some new ideas.