Use of APIs for GRPO reward ?

[ocramz]

Thinking of verifiable reward models: we have a program verifier that would accept Python program strings as well as pre-and post-condition predicates, and return execution traces as well as counterexamples.

We are exploring the possibility of opening up this functionality as a REST API, as a simpler integration path than reimplementing this functionality in Python for the open-r1 repo.

Questions:

• Is this kind of API integration compatible with the openness aim of open-r1?
• Does the GRPOTrainer cache function invocations?
• Related: would GRPOTrainer build a list of prompt-answer pairs therefore a dataset out of the API invocations?

Thanks!

[Muhtasham]

Seems like could be done on Modal Sandbox

[ocramz]

Here's a PoC for what's described above! Still some details to iron out namely prompting format, but the foundations are there: #379

@Muhtasham internally this uses a SMT solver so we went the docker route and deployed on Cloud Run.

[ocramz]

.. regardless of where such an API runs, I was more wondering about the "openness" requirement. I.e if all data should be public upfront in order to be used for open-r1