RFC: Scheme for revising bounds to eliminate corners of the version cartesian products

[hvr]

Problem Statement

The basic abstracted problem can be demonstrated with two dependencies, foo and bar and each with two major API versions. For the general case, these two packages shall be considered independent of each other, i.e. neither depends upon the respective other.

Their exported API is

-- foo-1.1
module Foo where
f :: Char -> Int

-- foo-1.2
module Foo where
f :: Char -> Word

-- bar-0.1
module Bar where
g :: Int -> Bool

-- bar-0.2
module Bar where
g :: Word -> Bool

And now consider a consumer of bar and foo, i.e.

build-depends: foo >= 1.1 && < 1.3, bar >= 0.1 && < 1.3

and combines g and f like so

h :: Char -> Bool
h = Bar.g . Foo.f

Let's assume this definition is semantically sound when it typechecks.

However, only two out of 4 combinations of foo and bar are valid:

	foo-1.1	foo-1.2
bar-0.1	OK	FAIL
bar-0.2	FAIL	OK

This is just one example; another situation that's commonly encountered is something like

	foo-1.1	foo-1.2
doo-0.1	OK	FAIL
doo-0.2	OK	OK

or any rotation of these two. (NB: if only one out of four quadrants is "OK", it's trivial to resolve via revisions)

Solutions to the Problem

The real problem now is how to fix an package description which contains the inaccurate

build-depends: foo >= 1.1 && < 1.3, bar >= 0.1 && < 0.3

dependency specification without forbidding any of all the previously reachable sound quadrants.

A solution.

Indirect solution by introducing extra dummy packages

A possible solution would be to use the network-uri-flag hack, which involves creating a dummy package with an empty library like shown below to encode the valid combinations of foo and bar:

name: foo-bar-compatibility
version: 0

flag _choice
  manual: False

library
  if flag(_choice)
    build-depends: foo == 1.2.*, bar == 0.2.*
  else
    build-depends: foo == 1.1.*, bar == 0.1.*

and then this package foo-bar-compatibility needs to be added to the whitelisted packages allowed to be added retroactively via revisions into build-depends in the hackage-server revision validation logic. Consequently, the package consuming foo and bar would need to be revised into

build-depends: foo >= 1.1 && < 1.3, bar >= 0.1 && < 0.3, foo-bar-compatibility == 0

However, while this approach is possible, it comes at with a couple of obvious downsides:

• needs to create and maintain new extra packages for encoding inter-package relations
• management overhead of needing to maintain/grow an adhoc whitelist of these packages (currently this requires a redeploy of hackage-server)
• this is required for every combination of packages that exhibit the problem at hand
• the extra packages present a possible additional point of failure if mistakes are made

Direct solution

A more direct solution is to extend the hackage revision rules to allow the introduction of new automatic cabal flags via metadata revisions, with the following machine-checked limitations:

1. The ability to introduce cabal flags shall be limited to hackage trustees
2. Such cabal flags introduced via revisions must start with a reserved x-rev- prefix (or another prefix that doesn't yet occur in 01-index.tar at time of deployment)
3. cabal flags prefixed by x-rev- must be manual: False (Hackage will enforce this invariant during revisions)
4. Hackage will refuse uploads of 0th revisions with package descriptions containing x-rev- prefixed flag names; this is to ensure that non-trustees cannot create immutable x-rev- flags by accident.

Since these are automatic cabal flags which the cabal spec provides for the very purpose to express these kind of either/or dependency specifications involving multiple packages.

Direct solution for future cabal-versions

If, in a future cabal version, the cabal format provides a flag-less way to express, such as e.g.

    build-depends: foo >= 1.1 && < 1.3
    if lib-version(foo >= 1.2)
        build-depends: bar == 0.1.*
    else
        build-depends: bar == 0.2.*

the revision logic would be adapted to allow introducing if lib-version(...)-conditionals for cabal-version:s which support this predicate; while also disallowing the flag-introduction from the previous section for cabal-version:s which provide this more idiomatic construct.

[gbaz]

Just to be clear: can't you use nested conditionals in the example, at the cost of a potential explosion of expression size? I.e. (foo >= 1.2 && bar = 0.1.*) || (foo = 1.1.* && bar = 0.2.*) ?

Edit: oh wait, I guess that syntax isn't allowed, sigh.

[gbaz]

Ok, so for the "future" idea, I'd really like to have sugar to let conditionals be inlined into build-depends, so we could write: build-depends: foo >= 1.1 && < 1.3, if (lib-version(foo>=1.2) (bar == 0.1.*) (bar == 0.2.*), ... But that's orthogonal.

[cartazio]

Why isn’t || allowed?

[gbaz]

The general scheme is its a disjoint list of packages, each with a range. The range itself can use && and || both. But each package specifies its range independently. Herbert just explained to me offline one key thing -- this means that when each package version is fixed and when each flag is also fixed, there's no "degree of freedom" since expressions have a unique solution. Otherwise, overlapping or clauses could prevent this.

[gbaz]

Talking to herbert some more, I think that we can simplify his proposal and just allow anyone to add auto flags in revisions, full stop. That means that we don't need a whole bunch of special revision rules here, and instead we just increase slightly the revision surface area as a whole.

It does mean that those flags can now be altered to manual (which is a one-way op) even when we wouldn't want them to be -- but this is already the case with allowing other flags that were set as auto to be altered to manual, so we're no worse off.