Nix development environment?

[silky]

Issue

Goal is to get a full reproducible development environment using Nix; this will install all necessary system packages and dependencies at the correct versions.

This is a follow-up to #8565 .

Again, this PR serves mostly as a discussion piece/a way for Nix(OS) users to contribute to this project. Notably, this is built on top of the #8565 branch, so it at least proves that moving to uv seems to have no impact on the project at large; i.e. all the python dependencies installed correctly.

Description

This is an attempt to use uv2nix to define a Python development environment. It also includes tesseract.

Install Nix and configure flakes.

To start using it, either use direnv, or run:

nix develop

this opens the default "devShell". If you use direnv, it will automatically open when you cd into the folder.

From there, you can either run the scripts as pure binaries:

> test_parser --help

Or you can run linting, say, with

> uv run lint

Todo/Notes

• Two tests fail; you can run pytest to see. They are:

FAILED tests/config/test_emission_factors.py::test_all_emission_factors - assert [- snapshot] == [+ received]
FAILED electricitymap/contrib/parsers/tests/test_JP_SK.py::test_fetch_nuclear_image - assert 1000.0 == 918.0

The second one is likely due to a missing expectation on tesseract version. I'm using 5.5.1. The first one, I'm not sure yet.

• Ensure that test_parser works; seems like I need to create some tokens to even test that
• Figure out why capacity_update fails:

> capacity_update --zone FR --target_datetime 2022-01-01
Traceback (most recent call last):
  File "/nix/store/7n8dmf4lwxycgdm3si663kgrpy4hyzh3-electricitymaps-contrib-env/bin/capacity_update", line 10, in <module>
    sys.exit(capacity_update())
  File "/nix/store/7n8dmf4lwxycgdm3si663kgrpy4hyzh3-electricitymaps-contrib-env/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/7n8dmf4lwxycgdm3si663kgrpy4hyzh3-electricitymaps-contrib-env/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/nix/store/7n8dmf4lwxycgdm3si663kgrpy4hyzh3-electricitymaps-contrib-env/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/nix/store/7n8dmf4lwxycgdm3si663kgrpy4hyzh3-electricitymaps-contrib-env/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/noon/dev/electricitymaps-contrib/capacity_update.py", line 56, in capacity_update
    update_zone(zone, parsed_target_datetime, session, update_aggregate)
  File "/home/noon/dev/electricitymaps-contrib/scripts/update_capacity_configuration.py", line 44, in update_zone
    zone_capacity = parser(
  File "/home/noon/dev/electricitymaps-contrib/electricitymap/contrib/capacity_parsers/ENTSOE.py", line 64, in fetch_production_capacity
    xml_str = query_capacity(ENTSOE_DOMAIN_MAPPINGS[zone_key], session, target_datetime)
  File "/home/noon/dev/electricitymaps-contrib/electricitymap/contrib/capacity_parsers/ENTSOE.py", line 52, in query_capacity
    return query_ENTSOE(
TypeError: query_ENTSOE() got an unexpected keyword argument 'function_name'

Seems like it is just a codepath that hasn't been tested recently?

If adopted

If the maintainers wish to move to Nix, we would also like to:

• Update the CI to use the Nix derivation/work. Presently it fails, but no attempt has been made to adopt it to the new uv/Nix setup
  • Earthly
  • Various tests I'm sure
  • etc
• Also make sure the non-nix (I.e. uv-based) workflow works
• Should also setup the shell with necessary tooling for the UI

[VIKTORVAV99]

Commenting on this one as it contains all the changes:

Overall we do want to use UV and we plan on migrating to it, maybe even now in January as a part of a bigger refractor and use workspaces.

So that part is mostly okay, just need to ensure our package publishing still works if we merge it.

However there are some things I need clarity around and these are:

• Why does this use hatchling as the build backend instead of UVs build backend?
• Why do we need Nix? That is seemingly a whole other package system on top of UV, not sure we want to or can maintain that for a very limited usecase.

[silky]

Overall we do want to use UV and we plan on migrating to it, maybe even now in January as a part of a
bigger refractor and use workspaces.

So that part is mostly okay, just need to ensure our package publishing still works if we merge it.

Great! I'll close the other PR then.

Why does this use hatchling as the build backend instead of UVs build backend?

Not sure; that must just be the default that the tool migrate-to-uv picked; I didn't do anything special there.

Why do we need Nix? That is seemingly a whole other package system on top of UV, not sure we want to or can maintain that
for a very limited usecase.

Yes, very reasonable question. I'll provide perhaps too much information in response to this, and assume no prior knowledge of Nix.

Why Nix (on top of uv)?

Simply: To get all the correct versions of every dependency needed to build and run this project, on any machine.

Consistent, isolated development environments.

Nix is, in my opinion, simply the best tool to essentially remove all "Works on My Machine"-style problems a team may have when working on any project of basically any size; especially one like this with multiple languages and tools in play.

That is, uv is great for installing Python packages; but it doesn't install system dependencies (i.e. tesseract). For that, you need a system-level package manager; or, what is more common, a list of instructions for how to set up your repo.

devShells

Nix has a concept of devShells, which, when used in combination with direnv, make for a seamless transition into all the dependencies of the project when you simply cd into that folder:

# Outside this repo; a different version of Python and no pandas.
~/dev> python
Python 3.13.11 (main, Dec  5 2025, 16:06:33) [GCC 15.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import pandas
ModuleNotFoundError: No module named 'pandas'
>>>

...

# Inside this repo, the right version of Python and all dependencies
~/dev> cd electricitymaps-contrib
direnv: loading ~/dev/electricitymaps-contrib/.envrc
direnv: using flake
direnv: nix-direnv: Using cached dev shell
~/dev/electricitymaps-contrib> python
Python 3.10.19 (main, Oct  9 2025, 15:25:03) [GCC 15.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>>

This allows anyone to immediately stack hacking on the project; and all they needed to do was have Nix installed (and enable nix-flakes). Once they cd into the clone of the repo, they simply need to direnv allow the .envrc file, and the entire shell is loaded with all the appropriate dependencies; even tesseract!

# Inside the repo
~/dev/electricitymaps-contrib> tesseract --version
tesseract 5.5.1
 leptonica-1.85.0
  libgif 5.2.2 : libjpeg 6b (libjpeg-turbo 3.1.3) : libpng 1.6.52 : libtiff 4.7.1 : zlib 1.3.1 : libwebp 1.6.0 : libopenjp2 2.5.4
 Found AVX2
 Found AVX
 Found FMA
 Found SSE4.1
 Found OpenMP 201511
 Found libarchive 3.8.4 zlib/1.3.1 liblzma/5.8.1 bz2lib/1.0.8 libzstd/1.5.7 openssl/3.6.0 libb2/bundled libacl/2.3.2 libattr/2.3.2
 Found libcurl/8.17.0 OpenSSL/3.6.0 zlib/1.3.1 brotli/1.2.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.67.1 ngtcp2/1.18.0 nghttp3/1.13.1 mit-krb5/1.22.1

# Leave the repo directory ...

~/dev/electricitymaps-contrib> ..
direnv: unloading
~/dev> tesseract --version
zsh: command not found: tesseract

So, we add nodejs tools into the devShell as well, to get all the dependencies to compile the frontend, etc.

Provision of entrypoints and applications from the sourcecode

As mentioned in the PR, the scripts section of the pyproject.toml is processed to yield binaries that can be run directly inside the shell.

Moreover, you can use nix flakes applications to define very easy ways to run specific binaries with a simple invocation, even direct from GitHub! For example, the following command runs the popular "HiGHS" optimisation software directly from GitHub:

> nix run github:ERGO-Code/HiGHS -- --version
HiGHS version 1.12.0 Githash n/a. Copyright (c) 2025 HiGHS under MIT licence terms

Note

Note that because we're running it from GitHub, it needs to compile it first. But you can also run it
directly from nixpkgs, where it will just be downloaded: nix run nixpkgs#highs -- --version.

Reproducibility + caching

Nix is state-of-the-art for reproducibilty; the flake.lock file contains all the hashes of all the (nix-declared) dependencies. Given the nature of the so-called nix store, all these dependencies can be cached, and when used in combination with cachix (very good free tier), can wildly speed up development and CI; i.e. no need to re-build something that someone else has already uploaded to the cache.

Deployment

Having adopted Nix, it becomes a very simple matter to generate docker images, say, of your project, without having to re-specify build instructions or dependency-install statements (apt get install ...). You just call a certain function to build a docker image, and include all your binaries on it: an example from a previous project I've nixified.

Very wide adoption and (almost) every package ...

The nixpkgs search page shows just how many packages are already available and ready to be used (i.e. pre-built!) in your project. It's increasingly unlikely that a desired package won't be on there; but if it isn't, it's a pretty straightforward manner to define it yourself. You and also use the "nix-versions" tool to find commits of nixpkgs with older versions of the packages you need, but it's rare that this is needed.

Simplified CI

In theory, through diligent use of caching (via cachix or otherwise), CI build times can be extremely quick; and reduced in complexity because all the setup is already done by the main devShell (or specific ones for testing, if that's desired). I will say this can be an area of unbounded complexity if one wishes; but in general the approach is pretty effective; and a lot more testable than endless instructions in GHA; because you can run all the nix commands locally as well.

In conclusion ...

Hopefully this has been helpful.

It's certainly true that adopting Nix results in a bit of a management burden; and I would say one of the biggest criticisms of the Nix ecosystem is a bit of trouble finding documentation; but it's possible to overcome this by reading examples and otherwise just getting into the space, as with any new tools.

For me, the wild reduction in "Works on My Machine" is easily worth the overhead of adopting it.

Specific notes on the flake.nix here

Almost everything in the flake.nix in this PR is boilerplate from uv2nix; the two main sections are:

1. This section adding "setuptools" as a dependency to the projects that forgot to include it in their setup.py:

...
(final: prev:
  let
    inherit (final) resolveBuildSystem;
    inherit (builtins) mapAttrs;
    buildSystemOverrides = {
      # All of these were missing setuptools listed as a
      # dependency.
      demjson3.setuptools = [];
      odfpy.setuptools = [];
      pycountry.setuptools = [];
      pytesseract.setuptools = [];
      pydataxm.setuptools = [];
      sseclient.setuptools = [];
    };
  in
  mapAttrs (
    name: spec:
    prev.${name}.overrideAttrs (old: {
      nativeBuildInputs = old.nativeBuildInputs ++ resolveBuildSystem spec;
    })
  ) buildSystemOverrides
)
...

2. Adding tesseract to the list of packages available in the devShell:

...
            packages = [
              virtualenv
              pkgs.uv
              pkgs.tesseract
            ];
...

Note that, indeed, I think the main relevant error in this PR itself is the fact that the tesseract version that this project depends on is probably relevant! That's why it's picking up the wrong number for me in the tests. I.e. it's (maybe) already been useful to make this PR to try and track down the right tesseract version!

But anyway, I'm trying to show that the present flake.nix doesn't need "a lot" of maintenance at the moment anyway. The way uv2nix works is that it reads the dependencies from the pyproject.yaml. It's only when one of those is unavailable on nixpkgs that some work needs to be done.

Happy to provide more detail on any of these thoughts.

[VIKTORVAV99]

In theory that is very nice and I like the general idea, I'm just not sure if that is the right tool for us.

We already have a devContainer (albeit very limited and slim) in this repo that integrates well with VSCode and GitHub Codespaces.

I'm thinking it's probably better to expand on that than introducing other tools into the mix.

This is not a final say by any means but It's something I would have to discuss internally if we where to implement and support it, devContainers we could simply expand on if needed.

We will also be reducing the scope of this repo to focus on pure python (and Yaml config files). So multi language will be less of a concern soon.

[silky]

We already have a devContainer (albeit very limited and slim) in this repo that integrates well with
VSCode and GitHub Codespaces.

Do note that Nix is not instead of devContainers; as mentioned you can use Nix to build docker images. The problem with docker for development is that the iteration time on rebuilding a container is extremely slow. With Nix/caching, it's effectively zero; i.e. it only does work in proportion to the new dependency that is added.

This is not a final say by any means but It's something I would have to discuss internally if we where to implement and
support it, devContainers we could simply expand on if needed.

👍🏻

[VIKTORVAV99]

My main concern is that it's yet another tool that needs to be installed vs tools that are more likely to already be installed (docker).

Image speed we can speed up significantly by pre-building images if needed. If it's only python dependencies the changes will be very fast either or with UV as the package manager.

But I'll bring it up on our next team sync tomorrow to see what the others say.

[silky]

My main concern is that it's yet another tool that needs to be installed vs tools that are more likely to already be installed (docker).

Yeah; I can see this perspective.

On the other side of the fence, once you have Nix installed, you never need to install another package manager again :) I.e. note that in the Nix devShell you don't need to install uv; it is already present (in the same way as tesseract).

But I'll bring it up on our next team sync tomorrow to see what the others say.

Cool! :)

[VIKTORVAV99]

My main concern is that it's yet another tool that needs to be installed vs tools that are more likely to already be installed (docker).

Yeah; I can see this perspective.

On the other side of the fence, once you have Nix installed, you never need to install another package manager again :) I.e. note that in the Nix devShell you don't need to install uv; it is already present (in the same way as tesseract).

But I'll bring it up on our next team sync tomorrow to see what the others say.

Cool! :)

So we talked this over and we decided that in principle this is a very good idea but we won't be using Nix for it for the reasons I listed above. (Although it got a lot of love from the team).

We will however try and improve our existing Dev Container / Docker setup so it's easier to get started for everyone that wants to contribute.

I hope this makes sense!

[silky]

Hey @VIKTORVAV99 ; yes, I understand your hesitance.

In fact, this interaction inspired me to write up some thoughts more clearly: Why should my project adopt Nix? and also put together a python-uv template; if you ever want to get into it feel free to reach out :)