Immich running on Nomad orchestrator

[dani]

If anyone is interested, I've made a Nomad job to run Immich (on a complete hashicorp stack, using Nomad for the orchestrator, Consul for the service discovery and mesh, Vault for secrets management). It's available here https://git.lapiole.org/nomad/immich/src/branch/master/example

[matthiasschoger]

To add to that, I'd like to share my Immich setup on the Hashistack
https://github.com/matthiasschoger/hashilab-apps/tree/master/immich

In addition to the features listed by dani, it also includes Prometheus monitoring and the option to scale the worker and ML jobs across nodes.

[gbolo]

nice work

[matthiasschoger]

Thanks! Did learn quite a bit about Nomad while doing this, for example the ephemeral disk feature which is perfect for caching stuff like the AI models.

[ppenguin]

@dani @matthiasschoger Thanks, I'm just looking into this and already learned a lot from a quick glance. (I have 3 nomad clusters deployed over last year and it was a huge learning curve, mainly because I didn't find much clear learning resources and nomad still seems quite niche. Additionally, I run nomad on top of NixOS on bare-metal and VPS, which was an added challenge.)
Sorry in advance for hijacking this, but would you have any pointers to consistent resources about "do's and don'ts / howtos" in nomad? Also:

• proxy: nice, I was always getting by with DATABASE_URL={{ range services "svc-mypg" }}..., but I suppose using proxy could be "best practice"?
• consul DNS never worked for me, any good resources?
• out of interest: what CSI backends are you using? I'm on seaweedfs myself, but I don't yet trust it enough for important data. It's also ballooning pretty severely.

Cheers

[matthiasschoger]

Just checked the gist, thanks for posting it.

Looks like I will need to dig quite a bit deeper into Nix to understand fully what you did there.
When I was playing around, I tried to pull everything from stable, but only Nomad, Consul and CNI from unstable. Maybe your approach is the better way.

Will probably try again after my migration to Trixie.
Would be great if you could ping me when you think your full flake is ready for publication.

[ppenguin]

@matthiasschoger Even though mixed inputs (unstable/stable) is theoretically perfectly fine in NixOS, it has given me more trouble than benefit in the past (therefore I now simply equate npkgs = pkgs). For the same reason it's also advisable to use overlays sparingly or at least with their implications in mind (mainly the fact that they can cause a chain-reaction of source-built packages if you change by overlay a package that many others depend on).

As for the complexity: the main somewhat more elaborate tricks I used are only to make provisioning fully declerative by having all (or most -> WIP) certificate files included in the flake; the rest of the config is pretty standard.

I can ping if published full config, but I'm afraid it's not a priority right now. Even though I have all my secrets in sops-nix, I'm as a matter of principle a bit uncomfortable to make infrastructure (i.e. hosts, names, network structure) publicly accessible, so it would require split flakes to realize this in line with security expectations. This is on the wish list but I won't find time for that soon.

[dani]

@dani @matthiasschoger Thanks, I'm just looking into this and already learned a lot from a quick glance. (I have 3 nomad clusters deployed over last year and it was a huge learning curve, mainly because I didn't find much clear learning resources and nomad still seems quite niche.

Indeed, very few resources (compared to eg, K8S). You can take a look at my repos here https://git.lapiole.org/nomad/. In each repo, you'll find an example dir which has a full job almost ready to use (with all the needed vault policies, roles, consul policies etc.)

Additionally, I run nomad on top of NixOS on bare-metal and VPS, which was an added challenge.) Sorry in advance for hijacking this, but would you have any pointers to consistent resources about "do's and don'ts / howtos" in nomad? Also:

• proxy: nice, I was always getting by with DATABASE_URL={{ range services "svc-mypg" }}..., but I suppose using proxy could be "best practice"?

Yes, using the Consul Services Mesh is best practice.

• consul DNS never worked for me, any good resources?

In my case, all my hosts are using an unbound instance as resolver, and this unbound instance forward the .consul domain to my 3 consul servers

• out of interest: what CSI backends are you using? I'm on seaweedfs myself, but I don't yet trust it enough for important data. It's also ballooning pretty severely.

I'm using democratic-csi (with local, freenas-api NFS and iSCSI drivers), and CephCSI (both RBD and CephFS). Also played with RocketDuck for NFS which was also working, and quite trivial to setup

[ppenguin]

@dani Thanks for your elaboration. I noticed that your .nomad.hcl files seem to be somehow additionally templated ([[ ... ]]), how/by which tooling/template engine is this used?

[dani]

Yes. I'm using a custom pre-processing engine (based on gomplate) because nomad-pack is not actively developped, and far from enough for my needs. The tool itself is open, but I haven't written any documentation for it yet. Anyway, here it is : https://git.lapiole.org/dani/ctctl

It's working by relying on re-usable templates which can be imported (via git submodules), handles project dependencies, multi-level YAML config to adapt to the target env, switching between Nomad clusters etc.