Manage package versioning

[DuncanBloemers]

How can the versioning of packages installed via tectonic be managed? I have biber 2.18 installed but tectonic has installed biblatex 3.17 (3.18 is compatible). How could I update this?

[pkgw]

@DuncanBloemers This question comes up often, and unfortunately we don't have a clean answer for packages that combine TeX dependencies with external programs like biber. The Tectonic model is to bundle all of the TeX dependencies together into a coherent package, but by design that package doesn't update often and that model doesn't work too well with the Biblatex framework, which has that external dependency and evolves pretty quickly. I can't think of a way to square that circle that doesn't involve creating or adopting our own full-stack package manager (which we could conceivably build on something like Conda to do!).

To override Tectonic's built-in biblatex files on the TeX side, you can drop alternative versions into the directory containing your TeX input files. I don't believe that CTAN archives different versions of packages, so I think you'd have to dig into the biblatex GitHub release archive and pull out the file you need. Unfortunately I'm not sure whether one can get away with updating just biblatex.sty or whether other files have to be updated too.

If you can downgrade your version of the biber program, that might be easier. We could potentially patch the Tectonic version of the biblatex package to try a special executable name, or something, to try to give people a way to have multiple versions of biber installed, if needed.

[kpym]

@pkgw It may be a good idea to search for local files, not just in the current folder. Because if we need to replace or add some `.sty' file, there is a good chance that this will not be the case for just one project.

[kpym]

Reproducibility is a key feature, I agree. But I don't think that "reproducibility" is the enemy of "local". IMO quite the opposite.

At the moment, if I try to compile a very old project with tectonic on a new computer, it will retrieve the necessary files from the latest version of the TeXLive bundle. But it may be that the new versions will break things in that very old project. So having the option to say "use this very old local backup of the needed libraries" is a good choice.

And "reproducibility" should not be at odds with "usability". For example, in a situation where my university provides a style for my tutorial sheets that is not part of TeXLive, I would like to be able to say for all my projects "use these additional libraries".

No matter how hard you work, you will never be able to provide 100% usable defaults. So a good configuration/local system is needed.

IMO, the general rules for "good" software are

• Have very good defaults, so that it runs 99% of the time without any pre-configuration;
• but allow you to configure the behaviour for the 1% of very special cases.

[pkgw]

No disagreement there!

As to your first point, one of the points of the V2 document model is that the Tectonic.toml file encodes the specific version of the bundle that was used, so that if you go back to recompile an old document it will still use the support files that it was originally built with. The obvious extension is to add support for a "lockfile" analogous to those use by tools like NPM or Cargo, and my hope is that this sort of approach can help address issues of layering customizations on top of the bundle base.

[DuncanBloemers]

Thanks for the info I think this was a good discussion. I see the problem and I don't think it's a very easy one to solve. Long term a system like Conda sounds ideal but a lot of work. I think short term it could be useful to have something clearly labelled on https://tectonic-typesetting.github.io/ that lists commonly used external tools and what version is compatible with current versions of tectonic. This could help people do their due diligence before updating external tools.

In my case I had both biber and tectonic installed via homebrew so I (maybe naively) thought that they would be kept up to date with each other. Unfortunately homebrew doesn't support easy downgrade of biber as far as I can tell.

Thanks for all your hard work and for taking the time to give such an informative answer. Even if it wasn't quite what I was hoping to hear. I really hope to see tectonic continue to develop and succeed in the future.

[pkgw]

That's a good suggestion. I don't personally use biblatex/biber so it's not as high on my list as it is on some other people's, but some compatibility information on the website would definitely help a lot of people.

[Vigilans]

Second, there are some logistical questions about where Tectonic should look for "local files"

I am using archlinux and downgrading biber is nearly an impossible task: after downgrading biber from 2.19 to 2.17 from archives, biber 2.17 requires perl 5.34 where system's perl 5.36 will execute biber 2.17 with error.

However, archlinux's texlive-bibtexextra package provides biblatex.sty of version 3.19 that is compatible with biber 2.19. If tectonic can use the system provided biblatex.sty (located in /usr/share/texmf-dist/tex/latex/biblatex/biblatex.sty, while in debian it's /usr/share/texlive/texmf-dist/tex/latex/biblatex/biblatex.sty from texlive-bibtex-extra package), the problem is solved.

First, a major goal that I have for Tectonic is to ensure that documents can be built reproducibly and reliably.
if Tectonic just discovers that special file implicitly, it may or may not exist in the future, leading to surprising breakages.

So, isn't the expectation that "By installing both biber and biblatex.sty from system package manager, I can build my tex in a compatible way" equally important to reproducibility? By sticking with a built-in custom version instead of using system provided one, tectonic is already unreliable and leading to surprising breakages that makes user giving up tectonic.