Avoid Typesense resource usage explosion after multiple face-merges

[raisinbear]

Hi,

initially, I opened #3799 for that, but I slowly got to understand things are a little more complicated and this is not exactly a bug:
What I and some others describe happens for me only when performing multiple face merges in a row. And observing more closely, I think "in a row" is limited to doing another face merge (or any Typesense import) during the time it takes to reindex all faces again - a step that @alextran1502 informed me was necessary after face merge shortly after I created the issue.
So from what I understand and gather by the error logs posted in the issue, reading the code and (very small parts) of the typesense docs:

• faces are reindexed after a face merge
• when your library grows big you have a good chance of having more faces, so the above step takes longer. maybe - depending on system hardware - even a minute or more
• during that time you merge another face, the microservices container again wants to reindex all faces, attempting a lot of imports...
• ...but typesense is not ready yet, still occupied with the last reindex
• microservices gives up after the hardcoded 10s (infra.config.ts, lines 48 in current main) and tries again after 4s (same file, line above)
• according to the docs, this is undesirable and they recommend 3600s, although they don't explicitly mention, why. I got a suspicion, though, that at least some of the information from the attempted imports are saved to the then ever-growing log folder.

So I simply tried 3600 instead of 10. I think it means microservices won't give up after 10 s but will rather stand in line until typesense is ready to accept import commands again. I can now merge 5-6 or more faces (didn't have more right now ^^ but before, I could completely crash my server with only 3) in a row with 4400 faces total without seeing any errors in the logs, without typesense raft log folder growing from 60 M to >4 G, without Typesense keeping the CPU busy for almost an hour (now 1-2 minutes for 6 face merges) and thrashing my SSD with gigabytes of writes, and without the eventual and more or less final 503: Not Ready or Lagging from immich_server which would from time to time render immich unusable.
However, I don't know what else this setting is doing or if there are any downsides to it.

Any thoughts?

[Rahulsharma0810]

If anyone is facing the same issue in Kubernetes, then try to remove the Readiness liveness probe until the service is done with queuing completely, and the issue controller keeps restarting the service.

[bo0tzz]

Closing since typesense has been removed in #3605