On-demand instance unavailable after brief idle period

[kabilar]

Hi @asmacdo, I had the LINC instance profile running a dandi download task and it stopped after an hour or two of inactivity. See screenshot below. I thought the default timeout should be 1 day: https://www.nebari.dev/nebari-slurm/configuration#jupyterhub-idle-culler. Thanks.

[asmacdo]

@kabilar Looking at the code, IIUC the culler is based on browser activity, last_activity is the same value that shows up on the admin panel https://hub.dandiarchive.org/hub/admin

https://github.com/jupyterhub/jupyterhub-idle-culler/blob/main/jupyterhub_idle_culler/__init__.py#L220-L221

it does not check if any job is running. Im not sure what is the "minimal activity" required to trigger a last_activity update. It might be that if you just leave an open tab running thats sufficient. To confirm you could launch your job and since I can see last_activity I can confirm if that time is growing.

[kabilar]

Thanks for checking. Thats unfortunate. For our use case, we need to transfer large datasets, which would take a while and so I would need to step away from my computer. I have tried this a couple of times and it looks like if there is inactivity on the computer for a couple of hours, the server becomes unavailable. I might not be understanding the timeout vs cull_every settings. I have just started another process to download data in the hub, if you want to check.