Getting errors from scheduled experiment runs

[herman-rogers]

Maybe I'm missing the docs on this but is there anyway to get the errors return from an experiment? Effectively we're using sipyco to schedule an experiment to run, we then check the status, and move on when we get "finished". However, sometimes the experiment will fail but at the moment there doesn't seem to be a way to return this error:

scheduler = Client("system.app.net", "1234", target_name="schedule")

rid = scheduler.submit(
    expid=expid, pipeline_name="main", priority=2, due_date=None, flush=False
)

scheduler.get_status()
// wait for status to be finished
scheduler.check_errors() // something here where we can get the errors returned

[sbourdeauducq]

// wait for status to be finished

How?

Currently the only communication mechanism from experiments is datasets. You could set one that indicates status.

[herman-rogers]

That aspect of the code works - I was more talking about there are errors surfaced in the dashboard and I want to be able to access those errors if a scheduled experiment errors.

In terms of waiting for a status to finish if you look at the class Scheduler in artiq/artiq/master/scheduler.py you'll see this bit of code:

    def get_status(self):
        """Returns a dictionary containing information about the runs currently
        tracked by the scheduler.

        Must not be modified."""
        return self.notifier.raw_view

Which returns an object that contains the status - essentially if the status is finished then you know the job is done (so there is already a mechanism other than datasets to access information about a job).

To bring it back to the error messages - I was just curious if there was a method or way to get the error from an experiment.

Edit: we have our own servers wrapping the RPC calls that handles this:

            scheduler = Client("our-system.com", 1234, target_name="schedule")
            if rid in scheduler.get_status():
                return GetExperimentStatusResponse(rid=rid, status=ExperimentStatus.RUNNING.value)
            return GetExperimentStatusResponse(rid=rid, status=ExperimentStatus.FINISHED.value)

[sbourdeauducq]

Which returns an object that contains the status - essentially if the status is finished then you know the job is done (so there is already a mechanism other than datasets to access information about a job).

Don't they get deleted, and you have a race condition?

[herman-rogers]

It's a hash-map and if the id does not exist then the job has finished.