Update benchmark logic to be less strict

[JanssenBrm]

At this stage, the benchmarks only use the status to give a score to the service. If a benchmark failed, it gets a score of 0 and if it succeeded it gets a score of 1, resulting in a percentage that is mapped to the traffic light system.

However, this logic might be a bit too strict as the error can happen at different stages of the benchmark, such as downloading the result or during the compare phase. In some cases, it might not be a reason to mark the service as failed. Therefore, we should think of another scoring system, that is including the phase of the benchmark, which is also available in the GeoParquet file.

[JanssenBrm]

Service Score

One suggestion of tracking this, could be assign different scores (fictive examples, TBD) to the phases:

Phase	Description	Score
create-job	Failure during job creation	0
run-job	Failure during job execution	0
collect-metadata	Failure during collection of metadata	0.7
download-actual	Failure during the download of the actual results	0.6
download-reference	Failure during the download of the actual reference	0.6
compare	Failure during the comparison of the results	0.5

A successful job receives a score of 1.

The above table gives a more forgiving score whenever something happened during the metadata-collection or reference retrieval.

The final score is again a percentage resulting from the total sum divided by the number of executions.

Error Tracking

I would also suggest combining a new score with additional feedback to provide towards the user. Based on the phase we can present additional feedback based on the latest error phase that was reported.

Phase	User Feedback
create-job, run-job	We are experiencing issues with executing the service on the platform.
collect-metadata, download-actual	The service is working, but we are experiencing issues with download the results
download-reference	There is a problem with the benchmark, not the service itself.
compare	The output seems to deviate from the reference.

[JanssenBrm]

@jdries / @soxofaan - Feel free to share your thoughts on this feature.

[soxofaan]

some quick thoughts

assigning a score to each phase is quite arbitrary, wouldn't that make it harder to interpret the final average score?

note that some phases like "connect", "collect-metadata", "download-actual", "download-reference" are more about the health of the backends or benchmark system, and less about the health of the algorithm itself

Would it be an option to provide just different health scores:
e.g. also have a score:

fraction of jobs that at least run to completion with status "finished" (i.e. reach end of "run-job" phase), regardless of how actuals compare to reference

This gives indication that the algorithm at least produces something

[JanssenBrm]

Thank you for the feedback @soxofaan. The score itself in my initial example would not be shown to the user but much rather used to identify if a service is stable or not. For the end user, the error tracking part is much more useful to explain why we are marking a service as unstable.

Can you provide some example of different health scores you would seem useful to present to users of the service?

[soxofaan]

I was thinking about:

• overall score (what we currently have): fraction of runs that finish and where the actual output "equals" the expected output
• some kind of "finish" score (better name TBD): fraction of runs that finish ( without error).