-
Notifications
You must be signed in to change notification settings - Fork 141
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Right now, backends are decided by a string that is internally matched with a set of accepted strings:
benchmark = Benchmark(
name="visualwebarena",
backends=["visualwebarena"],
# ...
)It would be good if we can have a class called Backend that must have a method called prepare, which is called when the backend is being prepared. This would allow anyone to submit a custom backend without having to rewrite the Benchmark class, e.g. backends=[MyBackend()]
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request