Proposal for v1/package

[jrray]

Our spec for packages has for the most part mirrored the spec for recipes, with some relatively minor changes, like pinning values, when a recipe is build into a package. There is a lot of complexity in a recipe, particularly with respect to components and embedded packages, that end up also in the packages spec and puts a burden on the package solver to handle correctly. Bumping the package spec to v1/package creates an opportunity to rethink what a package needs to contain and shift some of the burden out of the solver and into the package builder.

For example, a built package contains both a build and install section, and both can contain var values that are required by the solver. More can be hiding in components and/or embedded packages. The Package trait doesn't provide a way to iterate over all the options/requirements of a package (though it could!), instead, a solver has to be aware of all the ways these options/requirements can exist in the package spec and handle them correctly.

The first part of the proposal is to remove the build.options section from a package entirely. Only vars with pinned values are needed at runtime, and at build time these could be moved from the build.options section into the install.requirements section.

(All examples represent build packages; not recipes.)

pkg: demo/1.0.0/O2X5LJSW
api: v0/package
build:
  options:
    - var: os
      static: linux
    - var: arch
      static: x86_64
    - var: distro
      static: rocky
    - var: rocky
      static: "9.6"
      compat: x.ab
    - var: color
      static: blue
install:
  requirements:
    - pkg: python/3.11

pkg: demo/1.0.0/O2X5LJSW
api: v1/package
requirements:  # <-- "install" would only contain "requirements" so it has been removed
  - var: os/linux
  - var: arch/x86_64
  - var: distro/rocky
  - var: rocky/9.6
    compat: x.ab
  - var: color/blue
  - pkg: python/3.11

These two examples would already have the same meaning (if "install" wasn't removed), but the change to v1/package implies that a solver doesn't have to consider build.options at all for this package. For this kind of change, v0/packages could be converted on the fly when reading them from storage.

The next part of the proposal is to remove embedded packages--specifically their requirements--from packages, and consolidate those requirements into a requirement on the stub package. This is difficult to discuss without also talking about components, because the only way for an embedded package to bring in additional requirements is through defining components¹.

pkg: demo/1.0.0/O2X5LJSW
api: v0/package
install:
  embedded:
    - pkg: embedded/1.0.0/embedded
      install:
        components:
          - name: run
            requirements:
              - pkg: python/3.11

pkg: embedded/1.0.0/embedded[demo:run/1.0.0/O2X5LJSW]
api: v0/package
install:
  components:
    - name: run
      requirements:
        - pkg: python/3.11

# embedded example, part 1

pkg: demo/1.0.0/O2X5LJSW
api: v1/package
requirements:
  - pkg: embedded/1.0.0

pkg: embedded/1.0.0/embedded[demo:run/1.0.0/O2X5LJSW]
api: v1/package
requirements:
  - pkg: python/3.11

Ignore components for the time being, for the sake of this argument. In v0/package, the solver is expected to understand embedded packages and know it needs to look for additional requirements that may be defined inside them.

$ spk env --solver-to-run resolvo demo
ERROR (link)

  × Failed to resolve: demo:run demo:run cannot be installed because there are no viable options:
  │ └─ demo:run local/demo/1.0.0/O2X5LJSW:run would require
  │    └─ python:run python:run/3.11.0, for which no candidates were found.
  │

The idea with v1/package is to turn embedded packages into pkg requirements for the packages they represent. Because the embedded stubs exist, the requirements found in the embedded stubs will naturally be handled just like every other package.

This idea raises another point about how the solver is expected to enforce that if an embedded stub is part of the solve, then its parent package is also part of the solve. Currently, the solvers explicitly check and enforce this. But this behavior could already be achieved by leveraging existing package features:

# embedded example, part 2

pkg: demo/1.0.0/O2X5LJSW
api: v1/package
requirements:
  - var: demo.fingerprint/b28f2ec2-d7cb-11f0-9f97-00155de0b192    # <-- new
  - pkg: embedded/1.0.0

pkg: embedded/1.0.0/embedded[demo:run/1.0.0/O2X5LJSW]
api: v1/package
requirements:
  - var: demo.fingerprint/b28f2ec2-d7cb-11f0-9f97-00155de0b192    # <-- new
  - pkg: demo/1.0.0                                               # <-- new
  - pkg: python/3.11

This is a throwback to #297 and adds the concept of a build "fingerprint" which is a uuid injected into the packages at build time. The embedded stub package generated also gets a runtime requirement for the package that embeds it. The fingerprint var requirement makes it so the requirement for demo can only be satisfied by the build that created the stub.

Using this approach means the concept of an embedded package is entirely hidden from the solver and all the complexity is eliminated.

The last part of the proposal is to do something similar to the embedded package idea but with components. This would take the idea of how embedded stubs are generated when packages are published and extend that idea to generating distinct packages for each component of a recipe. By doing so, the requirements of components can be expressed as standard package requirements and not require special handling.

pkg: components/1.0.0/O2X5LJSW
api: v0/package
install:
  components:
    - name: blue
      requirements:
        - pkg: python/3.11
    - name: red
      uses:
        - blue

pkg: components:build/1.0.0/O2X5LJSW
api: v1/package
requirements:
  - var: components.fingerprint/b28f2ec2-d7cb-11f0-9f97-00155de0b192

pkg: components:run/1.0.0/O2X5LJSW
api: v1/package
requirements:
  - var: components.fingerprint/b28f2ec2-d7cb-11f0-9f97-00155de0b192

pkg: components:blue/1.0.0/O2X5LJSW
api: v1/package
requirements:
  - var: components.fingerprint/b28f2ec2-d7cb-11f0-9f97-00155de0b192
  - pkg: python/3.11

pkg: components:red/1.0.0/O2X5LJSW
api: v1/package
requirements:
  - var: components.fingerprint/b28f2ec2-d7cb-11f0-9f97-00155de0b192
  - pkg: components:blue/1.0.0

Again we see that the package specs consist only of requirements, and component-specific requirements and inter-component "uses" have been re-expressed as standard package requirements. This example assumes some changes that would allow a package name in a package spec to include a component name. Otherwise, this approach also removes the need for the solver to know about components and how handle them correctly. The fingerprint enforces that only components from the same build can coexist in a single environment.

This is hopefully enough to see the intent, this should be able to cover more complex cases like mixing components and embedded packages. The complexity stays within the build logic to produce the correct set of "stubs" but then all the stub packages become a flat list of requirements that the solver can easily understand and process.

Footnotes

1. perhaps it was never intended to allow embedded packages to have additional requirements in any capacity, since spk build already rejects recipes that try to specify install.requirements in an embedded spec. However, the requirements attached to components sneaked through. ↩

[rydrman]

From the meeting today:

• RE: embedded package requirements, maybe it's better to just look at not allowing them on components either since it's likely this was never intended to work or be allowed. Are there any cases for this?
• If stubs become more complete and handled more like regular packages, should we handle cases where they are not compatible? Eg: where the parent and generated stub can't be solved together
• RE publishing and cleanup - we would be relying on expansion of the embedded stub logic that is already in place to connect these sub/attached items to the top-level build. The publish and rm commands probably need a long-overdue revamp on usability and output/reporting
• The transition to this system is a big challenge
• Would we worry about losing anything here? Maybe we introduce a sidecar piece of metadata for builds that can act as a more complete record of what was used to generate the v1 package. Crucially this would not need to be loaded or reasoned about by the solver but could be used for future recovery or migration...
• should we consider combining components into a single file in the implementation to avoid loading many small things?
• also might be worth looking at moving to flatbuffer for these since it's a fairly simple structure and a fairly clean cut from the previous version & structure on disk.
• need to double check that var requests for fingerprint have a way to actually require that it is defined, not just that if it's defined the values are the same - maybe something like a required attribute