Is there uniform way to wait all/specific queued experiments to finish?

[konn]

I'm currently working on the tool to use dvc in a supercomputer cluster and want to use parallelism to utilise available resources in grid experiments.
It seems that --jobs option of exp run won't work at the present, and we have to us queue start --jobs instead.
However, we could not find the way to wait all the queued jobs to finish (or fail) - the best approximation seems to parse the output of queue start and pass them to queue logs -f sequentially or parallely.

We execute our experiments as a batch script scheduled to supercomputer cluster and we want to achieve this behaviour automatically. Is there any better way to wait all (or specified) jobs to finish?

[dberenbaum]

@konn Have you tried to use dvc exp run --run-all? This should start the queue and block until all jobs finish.

[dberenbaum]

Hm, I'm not able to reproduce your issue. I see that dvc exp run --run-all --jobs N runs N jobs and blocks until all experiments complete. Could you provide the output of dvc doctor?

[konn]

Sorry for late reply (I was on a vacation).

Hm, I'm not able to reproduce your issue. I see that dvc exp run --run-all --jobs N runs N jobs and blocks until all experiments complete. Could you provide the output of dvc doctor?

Here is the output of dvc doctor (over poetry run), with some IP address (X.X.X.) masked:

DVC version: 2.37.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-4.18.0-305.25.1.el8_4.x86_64-x86_64-with-glibc2.28
Subprojects:
	dvc_data = 0.28.4
	dvc_objects = 0.14.0
	dvc_render = 0.0.15
	dvc_task = 0.1.6
	dvclive = 1.1.2
	scmrepo = 0.1.4
Supports:
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2022.11.0, boto3 = 1.24.59)
Cache types: hardlink, symlink
Cache directory: lustre on X.X.X.3@o2ib10:X.X.X.4@o2ib10:X.X.X.103@o2ib10:X.X.X.104@o2ib10:/lustre2/cmc
Caches: local
Remotes: s3, local
Workspace directory: lustre on X.X.X.3@o2ib10:X.X.X.4@o2ib10:X.X.X.103@o2ib10:X.X.X.104@o2ib10:/lustre2/cmc
Repo: dvc, git

[konn]

Ah, it seems that PR #8608 to exp run --run-all: re-enable --jobs concurrency gets merged in 26 Dec. Perhaps the newest release includes this fix?

[konn]

It seems that with 2.40.0, --jobs gets respected by dvc exp run, thanks!

OTOH, as dvc exp run --run-all is marked as deprecated, it might be still good to have a way to wait queued jobs to be finished.

[dberenbaum]

Please keep an eye on #8658, which should track this feature request (following the queue logs until all jobs complete).