make the immutable policy stricter

[hannesm]

addresses #29061

[avsm]

I'm in favour, although I wonder if the compiler packages will create a problem here. However one answer to that could be to move all the compiler packages to a new ocaml-opam-repo that only has compiler packages, and regularly sync them to this repo.

[hannesm]

For compiler packages, there's an ongoing discussion with @dra27 via email.

What is needed to merge this, 6 weeks later? Would including ocaml-base-compiler and ocaml-variants in the "Exceptions" make it easier to merge?

[avsm]

We need to sweep the current opam-repo to make sure there are no git urls in the current tree. A quick look reveals that many pin-depends have snuck through into the live repo!

> rg pin-depends
packages/melange-testing-library/melange-testing-library.0.1.0/opam
34:pin-depends: [

packages/melange-testing-library/melange-testing-library.0.2.0/opam
40:pin-depends: [

packages/timmy/timmy.1.0.4/opam
43:pin-depends: [

packages/docfd/docfd.3.0.0/opam
72:pin-depends: [

packages/docfd/docfd.2.1.0/opam
69:pin-depends: [

packages/docfd/docfd.2.2.0/opam
72:pin-depends: [

packages/goblint/goblint.1.1.1/opam
72:# pin-depends: [

packages/goblint/goblint.2.1.0/opam
92:# pin-depends: [

packages/goblint/goblint.2.5.0/opam
98:# pin-depends: [

packages/goblint/goblint.2.7.1/opam
102:# pin-depends: [

packages/goblint/goblint.2.2.1/opam
77:# pin-depends: [

packages/goblint/goblint.2.4.0/opam
96:# pin-depends: [

packages/goblint/goblint.2.0.0/opam
91:# pin-depends: [

packages/goblint/goblint.2.3.0/opam
77:# pin-depends: [

packages/goblint/goblint.2.6.0/opam
99:# pin-depends: [

packages/goblint/goblint.2.0.1/opam
90:# pin-depends: [

packages/lintcstubs-gen/lintcstubs-gen.0.4.7/opam
33:pin-depends: [

packages/timmy-jsoo/timmy-jsoo.1.0.4/opam
37:pin-depends: [

packages/universal-portal/universal-portal.0.1/opam
45:pin-depends: [

packages/universal-portal/universal-portal.0.2.0/opam

[hannesm]

Dear @avsm, while I appreciate your comment, my gut feeling is different, in the terms of timeline:

• we can merge this PR at any time when there's an agreement amongst the opam-repository maintainers
• at any other point in time we can "fix" the opam-repository to stick to the policies

Now, to comment on your rg dump - that includes some commented out (#) lines. The relevant packages are docfd 2.1.0, docfd 2.2.0, docfd 3.0.0, lintcstubs 0.4.7, melange-testing-library 0.1.0, melange-testing-library.0.2.0, timmy-jsoo.1.0.4, timmy.1.0.4, universal-portal.0.1, universal-portal.0.2.0.

Apart from melange-testing-library and lintcstubs there are newer versions without pin-depends in the repository -- so we can archive them or mark them as unavailable. I guess there are two underlying reasons for the merges:

• opam-repository maintainer team is understaffed
• the opam-repo-ci reports lots of warnings as errors (such as "this is not a conf package, but has depext").. maybe it is worth to revise this opam-repo-ci thingy to stick 1:1 to the policy?

But as said, I think that discussion should be held elsewhere; also I'm aware there are compiler packages which point to arbitrary git branches (and are thus not immutable). Indeed there's some need to figure out how to handle these (would a separate opam-repository for the compiler-curious devs be fine, such as opam-compilers-bleeding-edge)?

[raphael-proust]

I think the policy update is worth it. It doesn't address all the issues that can break reproducibility but it's a step in the right direction.

[mseri]

I also agree. I would push it as far as saying that the pin-depends were likely our mistakes/oversights in most cases

[hannesm]

I think the policy update is worth it. It doesn't address all the issues that can break reproducibility but it's a step in the right direction.

would you mind to elaborate which "issues that can break reproducibility" are still around?

also, i don't think we can ensure reproducibility in opam-repository easily, since any package may capture e.g. the time of the build etc.

but reproducibility is not the goal here. the goal is that a package url is immutable: pointing at ocaml.4.14.2 will always refer to the same source code.

[raphael-proust]

would you mind to elaborate which "issues that can break reproducibility" are still around?

the one i had in mind was versioning of depext tools: any executable the build process calls that's not controlled by opam (so anything outside of ocaml, dune, etc.) can differ from build to build. it can cause build errors (as was the case when gcc got upgraded by most distros) but it could also cause other strange behaviour.

it's not even just reproducibility, it's more the build environment can change. theroetically it could even change the sources that you get (e.g., if you have some malicious tar implementation which messes with the extraction process) or that you compile (e.g., if you have aa maicious configuration-time executable which messes with the files before build-time).

but as stated above: the update is welcome and we shouldn't delay it for the reason that it's incomplete.

[hannesm]

versioning of depext tools

Luckily for your scenario, there's the operating system providing the package (tar) with a specific version, and that operating system will be able to issue a security advisory with an appropriate package url. Surely, the issue remains that everything that was using the backdoored tar needs to be rebuild -- but honestly once you have such a backdoor on your machine, all you can do is to setup your computer again.

TL;DR: We should never issue advisories for the conf- packages (of course unless in the opam conf- package there's something vulnerable).