How to deal with CI

How to deal with CI

What the CI (Continuous Integration) does

The CI we run on the opam-repository checks that newly published packages and package updates can integrate correctly with the existing ecosystem of opam packages.

To this end, we run the following tests

• a lint check on every opam file
• for each packages added or modified in a significant way, check if the package builds and its tests run correctly:
  • on all "recent" OCaml compilers, as defined by Ocaml_version.recent
  • on different Linux distributions, FreeBSD and macOS
  • on different architectures (arm64, ppc64le, s390x, x86_32, arm32)
  • with different versions of opam (2.0, 2.1, 2.2, 2.3, dev)
  • with different options unabled in the compiler (flambda, frame-pointer, bytecode-only, naked-pointers, etc.)
  • when using the oldest possible versions of the specified dependencies
• check that all of a package's "reverse dependencies" (i.e., the set of packages that depend on the updated or newly published package) and build and pass tests

Here is an up to date report of all the Platforms Tested on the Opam Repository CI.

Finding the CI results

The CI results can be seen at the bottom of each PR by clicking to the opam-ci check or from a commit by clicking on the status icon (:x: or :heavy_check_mark:) and then selecting Details on the opam-ci check.

Interpreting the CI results

Each bullet point is a different log that you can read. The error(s) encountered are usually around the end of the log.

Reproducing CI builds locally

Reproduction instructions follow the line To reproduce locally: at the beginning of the logs. These commands require docker for Linux.

Note: for FreeBSD and macOS the Dockerfile is only an approximation as those platform can't run as a guest on Docker.

Dealing with CI errors

When adding new packages or updating packages, ideally the goal should be to have everything succeed.

Known errors that can be ignored

However there are some known errors that we currently ignore:

• missing system binaries from a distribution or missing system configuration. Using x-ci-accept-failures (see annex for more details) can help ignore those failures if the targetted platforms require something that can be dealt with manually but not in CI. Using the available field can also help if a certain platform will never work (e.g. uses x86_64 assembly, …).
• (failed: Sorry, resolution of the request timed out.) errors on older opam versions (2.0, 2.1, and 2.2). See https://github.com/ocurrent/opam-repo-ci/issues/354
• When modifying older packages, errors that existed before the PR can be ignored if the cause of the error stays the same.

Unexpected errors

Unexpected errors can occur. Such errors are inherently unpredictible but a couple of examples can be:

• temporary network failures or broken build machine. If this happens, please open a ticket at ocaml/infrastructure
• packages that were broken before and went unnoticed. Contributions are most welcome for this type of errors.

opam-repository CI results vs. your development CI results

If your development repository has CI enabled, you are probably only checking that your software can build and run its test on a few select platforms. The matrix of platforms that we are testing on for the opam-repository CI is generally much larger and the focus of the tests narrower: we are checking to ensure that the package can be installed with opam, on the most common supported platforms and combinations of the tool essential toll chain.

If you're noticing errors that look genuine but never noticed before on your own CI it could be because:

• your CI doesn't test the oldest versions for the dependencies (aka. lower-bounds) of your packages (check the FAQ to know how to check this locally)
• your CI doesn't test building only one packages at a time. This is common for dune users who have several packages in one repository. The typical dune workflow is to use dune build. However testing dune build -p usually reveals issues in the assignment of dune rules (by default rules belong to every packages unless (package ...) is given). Typical error will occur in tests where a test will be ran in all packages but dependencies become cycles.
• we're testing more platforms (freebsd, macos, s390x, …). If you do not have access to that platform or access to a service providing this platform, don't hesitate to ask the opam-repository maintainers for help.
• we have a different opam setup: opam-repo-ci has the sandbox unable for both build and tests, which might not be the case in your own CI (e.g. if you’re using docker). Errors are typical here if you’re using the network (e.g. during the tests)

If you have a feature request for the CI, feel free to open a ticket or a PR at ocurrent/opam-repo-ci

---

Annex

• x-ci-accept-failures is a custom extra field used in opam-repository. The syntax for this field is the following: x-ci-accept-failures: [ "<platform>" ... ], where <platform> is a certain platform tested by opam-repo-ci. The list of platforms is found under "Operating Systems" in the generated documentation for tested platforms. You can also find a platform you want to ignore from the logs of a CI job, by looking for lines similar to if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"opensuse-tumbleweed\"". In this case, opensuse-tumbleweed is the platform.
• sometimes one needs to see more verbose logs of what is going on. This can be partially achieved adding flags: verbose to the package (note that this line should be removed before merging it!)