[Feature] Automatically pause and resume Remote Machine Learning for on-and-off beefy machine

[jucor]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

I love how Immich allows to offload machine learning to a remote computer, especially useful when running Immich on a low-CPU NAS.

Would it be possible to automatically "pause" the ML queue when the remote ML computer is offline, and automatically resume the processing when the remote computer is back online?

Use case: Immich on a NAS, and remote ML computer being my beefy laptop (MacBook Pro M1 Max or Ultra). My laptop is sometimes offline, e.g. because I'm not home. If I understand the documentation correctly, this would lead to jobs crashing that I would have to manually resume. Pausing the queue instead of crashing, and polling regularly then resuming, would be terrific.

Thanks for considering!

Platform

• Server
• Web
• Mobile

[filski]

To add a slight variation too - it would be useful to be able to specify primary/backup ML instances. I.e. if the external machine is available use that, otherwise use the local ML container. That way could spin up docker/faster laptop for ML when doing a large import, but for general phone backups etc it would keep using the local ML container. The workaround is to switch the ML URL manually, so not onerous.

[Zedarius5]

Yes, this. I just spent an hour trying to figure out an nginx load balancer, but had trouble getting it to talk properly to the docker container. Would love auto-fallback onto the local ML if the external ML machine isn't reachable.

Should we submit this as a standalone feature request? I'm new to github, so I'm not sure the proper channels.

[mertalev]

Rather than pausing the queue, I think it'd be more elegant to retry failed jobs. Right now, a failed job means the admin has to manually requeue it, but it'd be better if the job scheduling were more fault tolerant. That would also help with more transient errors like SocketError, etc.

[jucor]

True, but it would mean the immich server would keep trying and failing and trying and failing in a loooong loop until the remote ML service is back on.

[mertalev]

It could probably do both, intermittently checking if the server is still up and pause/resume based on that. Retries would handle the jobs that were running when this happens along with more transient errors.

[jucor]

Dear Immich team

Would you have any visibility on this, please? That's the one blocker for me, as I'd want to run the ML tasks on my Apple Silicon laptop whenever it's plugged on my home network, with Immich itself running on my home NAS.

Sure, the alternative would be to run the initial very large import on a local Immich install on the laptop, then transfer the database to the NAS somehow, and let future ML runs be done (very slowly 😅) on the NAS whenever I upload a few more pics. Yet it seems such a waste of perfectly good compute 😅 and a bit clunky.

Alternatively, do users rent cloud GPU instances that they let run full-time for Immich? Or are there hooks to add scripts to start/stop instances when Immich needs them/when Immich is done?

Thanks for any help!

[aviv926]

Until such a feature is implemented, you can create a script that will run automatically when, for example, you connect to your home wifi network from your powerful computer. This will send an API request tostart the job you want to run using the powerful computer.

https://immich.app/docs/api/send-job-command

[jucor]

Nice idea, thanks. I guess we'd pair that with a check of the status of existing jobs? Then run those that are in failed state?

…

On Wed, 7 May 2025, 14:56 aviv926, ***@***.***> wrote: Until such a feature is implemented, you can create a script that will run automatically when, for example, you connect to your home wifi network from your powerful computer. This will send an API request tostart the job you want to run using the powerful computer. https://immich.app/docs/api/send-job-command — Reply to this email directly, view it on GitHub <#10023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFBERO5YAOZ446SQ2ILCND25IGKFAVCNFSM6AAAAAB4TNYKO2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBWGM4DMMI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[aviv926]

Nice idea, thanks. I guess we'd pair that with a check of the status of existing jobs? Then run those that are in failed state?
…
On Wed, 7 May 2025, 14:56 aviv926, @.> wrote: Until such a feature is implemented, you can create a script that will run automatically when, for example, you connect to your home wifi network from your powerful computer. This will send an API request tostart the job you want to run using the powerful computer. https://immich.app/docs/api/send-job-command — Reply to this email directly, view it on GitHub <#10023 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFBERO5YAOZ446SQ2ILCND25IGKFAVCNFSM6AAAAAB4TNYKO2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBWGM4DMMI . You are receiving this because you authored the thread.Message ID: @.>

You can enable it with force": false
{
"command": "start",
"force": false
}

so it will work in "missing" mode