Evaluations as a Service

[OAGr]

Sell evaluations for various use cases.
See this one as an example.

It doesn't have to be full-time employees doing evaluations. We could also find contractors. (Payment could either go through QURI, or direct to the contractors).

We might obviously want some way of vetting/reviewing evaluations. The meta-side to this seems tricky to get right.

Some sorts of evaluations could include:

• Standard "costs and benefits"
• Red-teaming
• Single estimates, with a few forecasters

[uvafan]

Had trouble deciding between a thumbs-up and heart here: this seems straightforwardly pretty valuable. My main concern would be scope creep, as Nuno mentioned in his evaluation. edited to add: actually another substantial concern is that QURI shouldn't be doing too many of these early days, and more once QURI has scaled up and gotten better at the meta processes of doing estimations/evaluations.

I'd be tentatively most excited about QURI doing red-teaming but standard costs and benefits also seems pretty good.

On vetting/reviewing evaluations, it seems like feedback from the community and entity being evaluated will be fairly useful signals here?

[NunoSempere]

BERI was interested in this. Could be straightforward to give it a try.