Transitive git dependencies

[bakaq]

How should we deal with transitive git dependencies? I'm thinking about the case where multiple packages in the dependency tree refer to one git package but pinned to different revisions. In the case of a parent package declaring the same git dependency of a child, I think it's clear that the parent's pin should override the child's (so pkg3 would be pinned to branch b in Example 1 below), but what about sibling dependencies (Example 2 below)?

flowchart LR
  subgraph ex1["Example 1"]
    direction TB
    pkg11["pkg1"]
    pkg21["pkg2"]
    pkg31["pkg3"]
  
    pkg11 --> pkg21
  
    pkg21 --"branch a"--> pkg31
    pkg11 --"branch b"--> pkg31
  end
  subgraph ex2["Example 2"]
    direction TB
    pkg12["pkg1"]
    pkg22["pkg2"]
    pkg32["pkg3"]
    pkg42["pkg4"]
  
    pkg12 --> pkg22
    pkg12 --> pkg32
  
    pkg22 --"branch a"--> pkg42
    pkg32 --"branch b"--> pkg42
  end
  ex1 ~~~ ex2

Loading

This wouldn't be a problem with actual versions, since we could then just do dependency resolution to find a suitable version, but we don't have the luxury of versions here.

[constraintAutomaton]

Can we make the resolution based on a lockfile? so our version would be the hash of the commit. If there is no lock we pick the latest.

[bakaq]

Huh, that's interesting, but how would that be different from just overriding the git hash in your root package in the tree? If I understood your idea correctly, both would "fix" the git hash that actually gets locked, overriding what the dependencies declare themselves.

Also, remember that git dependencies are useful even if the package does use "normal" versions, so this idea of yours may conflict with that. I use Scryer Prolog in some projects as a git dependency in Cargo for example, even though it also has normal versioning too. You may want to use this to get a new version that has a bugfix that isn't in a release yet, or other reasons. It's a pretty important escape hatch, and just happens to me easier to implement first which is why I started with it instead of normal versioning.

An idea that this just gave me is that we could error if the Example 2 of the OP happens, and in that case always force the user to declare an override git version in the root package. I think this may be the best way forward, as it doesn't fix a behavior and we could relax it on the future.

[constraintAutomaton]

Huh, that's interesting, but how would that be different from just overriding the git hash in your root package in the tree?
If I understood your idea correctly, both would "fix" the git hash that actually gets locked, overriding what the dependencies declare themselves.

I see it has a more flexible resolution approach, where the user can declare its policy in terms of compatibility between git hashes. We could declare dependencies as such:

dependency(Name, D, version(Major, Minor, Patch, R))

where R is a resolution method such as exact, any_patch, any_minor, any, etc.

Also, remember that git dependencies are useful even if the package uses "normal" versions, so this idea of yours may conflict with that.

You could set the version to any. For me "versioning", would be optional, because the default behavior should check for exact matching of git hash. I would sort of see it has we have the "physical" version provided by a dependency (for the moment we only have git and path) and we have a "logical" one with a version system on top. By default the "logical" one simply point to the "physical" one.

An idea this just gave me is that we could error if the case in Example 2 of the OP happens, and in that case always force the user to declare an override git version in the root package.

I agree, it makes sense to emphasize correctness by requiring an explicit resolution or rule in case of a violation, at least in the backend of how pkg.pl works , because we could enforce default behavior.

[constraintAutomaton]

both would "fix" the git hash that actually gets locked, overriding what the dependencies declare themselves.

I would see it has when it is locked it is locked, the new dependency have to be compatible with the locked dependency. If not we throw an error and the user should relax or change it's version.

[bakaq]

For me "versioning", would be optional, because the default behavior should check for exact matching of git hash.

You are getting into some really interesting stuff. Zig and Nix do this thing of "the version is the hash of the thing", with the versions they list being mostly a consistency check on top of it. The problem of that is that it's kinda impossible to do dependency resolution on that, so in that way it's a bit unfortunate, though it has some interesting properties (see this).

Also, it seems you are kinda reinventing a registry with this versions.pl thing. Would be users always required to write that for themselves? That seems kinda tiring. If not, who would host that and where? I was intending to do versions the Go way, just specially named tags in git repos, as that doesn't really need a registry (though it does benefit a lot from caches at scale) and therefore skips all the complexity that comes with them.

I would see it has when it is locked it is locked, the new dependency have to be compatible with the locked dependency. If not we throw an error and the user should relax or change it's version.

By "actually gets locked" I mean that all of this needs to be decided before even getting to the lockfile. For the module reasons I described earlier we can really only have one version of each package in the lockfile, the problem here is deciding what that would be in the case of git dependencies. Maybe the confusion here is that you think we need a lockfile for each dependency recursively? The lockfile is (at least in every implementation I know, like npm and Cargo) generated for the whole dependency tree at once.

[constraintAutomaton]

Also, it seems you are kinda reinventing a registry with this versions.pl thing. Would be users always required to write that for themselves? That seems kinda tiring. If not, who would host that and where?

I thought that next to your manifest file you would have a version file. The version file is like an event log. I imagined we could provide a predicate to write into this file, like:

V = increment_major, pkg_publish(V).

So, it’s like a local registry, because it only pertains to your package.

The problem of that is that it's kinda impossible to do dependency resolution on that, so in that way it's a bit unfortunate, though it has some interesting properties (see this).

It was really interesting! I really like the elegance/simplicity of the approach. But yeah, it seems to me it would only work in practice if packages are always ("correctly/actively") maintained. But, I feel like what I proposed is maybe too heavy...

By "actually gets locked" I mean that all of this needs to be decided before even getting to the lockfile [...] Maybe the confusion here is that you think we need a lockfile for each dependency recursively?

Hmm, I did not think we need a lockfile for each dependency. I thought that any new dependency has to be compatible with the currently locked dependencies if we have multiple dependency resolution policies.