About open file (descriptor) limits on Home Assistant OS

[agners]

You likely got linked here because you see Too many open files or No file descriptors available errors in the log. A bit of background why this errors suddenly appear, and what you can do about them.

If you are a user, please wait until the respective issue has been resolved. You may downgrade to HAOS 15.2 (or whichever OS version did work for you) until the issue is resolved (use ha os update --version 15.2).

If you are a developer, here is some more details on how things came about.

First off, these two messages are the same OS error with the number 24, Too many open files is the string used on glibc based systems (often add-ons), No file descriptors available you will see on musl based systems (Home Assistant Core, Supervisor or add-ons as well).

The message essentially means that the application hit its limits of maximum open files allowed by the operating system (Linux). This is enforced on a kernel level for each process. Each process has two limits: A soft limit and a hard limit. The error appears as soon as the soft limit is reached.

Recently there have been changes in Docker, and with that Home Assistant OS, what this default soft limit exactly is set at. Initially it was set to 1048576, before it got raised in HAOS 10 (which caused issues for some add-ons, see #2438) and Docker 23 to 1073741816 (due to the configuration option infinity in the containerd service file, which raises it to the maximum the Linux kernel supports), and finally in HAOS 16, with Docker 28.3.0 and containerd 2.0, the new default is 1024 (which again caused issues for some add-ons, e.g. Frigate, see #4110 and Home Assistant Core itself see #4164).

This new value is inherited by containerd 2.0, which HAOS is using. There has been some quite deep investigation into how the limits changed over time in the containerd repository, notable containerd/containerd#7566 (comment). The main change was to remove any explicit limit with containerd/containerd#8924, which essentially makes every container to start as if the application would run without containerization, just on a vanilla Linux shell. And for typical shells the soft limit has always been 1024, with variable hard limits but typically it is set at 524288.

If the application in question is unlikely to have that many files open at the same time, there might be a bug in your application (e.g. by a resource leak, where a reference to open files got kept somewhere, causing the system to have more and more files open). In this case any higher limit simply delays the resource exhausted. Obviously in this case the resource leak should be addressed.

For application simply need more file descriptor than 1024 it should raise the soft limits on it's own. This is also the official advice from the systemd blog in their File Descriptor Limit post. It seems the runtime of some programming languages (Go) do that on their own. For Python you can use the resources package. It is also possible to simply raise the soft limit in shell (e.g. in your run script of the s6-overlay, by using ulimit -Sn "524288", technically the hard limit can be different depending on environment, but 524288 is a rather safe value at this point in time).

[jymbob]

If you are a user, please wait until the respective issue has been resolved.

Given #4164 has been marked as resolved in what's basically akin to a "not our problem" philosophy, what are end users supposed to make of this?

I fully appreciate the argument that poorly written integrations leaving files open is a problem for the maintainers of those integrations, but the end result is that a change in HAOS means Home Assistant Core is getting taking down for a number of users in a manner that's not well explained, and that's a definite regression.

At a bare minimum, home assistant's responsibility is to alert the user to the issue so they can know where to start looking for fixes.

[sairon]

Given #4164 has been marked as resolved in what's basically akin to a "not our problem" philosophy, what are end users supposed to make of this?

Sorry, it's not the way I wanted to communicate it. The issue was locked because the original cause of the problem was found and addressed (to some extent). There's nothing else to be done on OS side, as the limit is now handled by HA Core (since home-assistant/core#148806 - even if the OS limits were higher, Core would them to 2048).

Snowballing issue with tons of "me too" comments is never helpful, that's why every case should be investigated individually, either in the Core issue tracker, or with the help of the community on Discord or Community Forums. There are steps in the issue you linked that can be used to identify the source of the problem, although they are quite technical - we could think of how we it can be simplified.

Last not least, as I already stated elsewhere, the problem of poorly written integrations may have shined a bad light on Home Assistant's stability in the past as well - what manifests as breakage in 1 hour with current limits becomes a breakage in 10 days with the old ones. By then users were pretty sure it's Home Assistant who's to blame, now we've brought the real issue into the light.

[zSprawl]

I feel stuck as well. I can't upgrade because of the issue but I can't identify what integration is causing it without significant effort of turning a few integrations on, and waiting to see if something happens, over the course of weeks.