-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Description
Describe the issue:
I'm running pymc in a condor HTC environment, where my models are sampled on compute nodes and the output is streamed or copied back to the host.
Without any options to checkpoint sampling with pymc, I want a way to check the status of the sampling even though I don't have access to stdout on the compute nodes.
Currently, we only pass refresh=True on the last iteration, so the only time condor streams stdout back to the host machine is after the last iteration. This makes me sad because no updates are streamed back to the host during sampling, so I have no way to satisfy my curiosity on the expected runtime.
In summary, the issue is that progressbar=True doesn't actually print the progress bar to stdout in such a way that condor can stream it until sampling is completed.
I want refresh=True here for all iterations, not just the last:
Lines 893 to 908 in 2842401
| self._progress.update( | |
| self.tasks[chain_idx], | |
| completed=draw, | |
| draws=draw, | |
| sampling_speed=speed, | |
| speed_unit=unit, | |
| **more_updates, | |
| ) | |
| if is_last: | |
| self._progress.update( | |
| self.tasks[chain_idx], | |
| draws=draw + 1 if not self.combined_progress else draw, | |
| **more_updates, | |
| refresh=True, | |
| ) |
Reproduceable code example:
pm.sample() # on HTCError message:
PyMC version information:
Latest
Context for the issue:
No response