Recreate a subset of benchmarks

[svmhkk]

Hi, what would be the best way to recreate a subset of benchmarking accuracy data from the ZC json?
For ex: I would want to replicate the val accuracy obtained for the architecture 1-100 indexed from NASBench201. I understand that the val accuracies and corresponding ZC scores were open-sourced to remove the need for training, but I need to perform training for an internal evaluation. How do you suggest to go about this?
Thank you.

[abhash-er]

Would you like to create a benchmark from NASBench201 Search Space's arches or the ZC scores? You can use Trainer in naslib/defaults to run and create benchmarks. If you want to create ZeroCost benchmarks, you can simply use ZeroCost available in predictors.

from naslib.predictors import ZeroCost
pred_type = "jacov"
zc_predictor = ZeroCost(method_type=zc_pred)
score = zc_predictor.query(graph=graph, dataloader=data_loader) 
# The graph is of type NASBench201 type in your case and dataloader would be a subset of CIFAR10 data.

Let me know, if that answers your issue.