How to use Benchmark Aggregator in Timefold?

[PatrickDiallo23]

Hello Team,

I want a way to aggregate the existing benchmark results while behaving in a similar way to the BenchmarkAggregatorFrame. I don’t see any clear examples on documentation and I am not sure how to approach this.

Check this StackOverflow question as reference. Any help is appreciated. Thanks!

[triceo]

Hello @PatrickDiallo23!

You mean an automated way, without the UI? Unfortunately, there isn't one - the BenchmarkAggregatorFrame is an old piece of code and we don't use it anymore. That said, the underlying logic is still valid; with some effort and polish, an API could be exposed from it to allow to merge benchmark reports.

We do not have this on the roadmap, but contributions are welcome!

[triceo]

If memory serves (and it's been years since I last dealt with that code), the aggregator reads the original source data (so, the CSVs) of the different benchmarks, and generates a new HTML report from those CSVs. There is no parsing of HTML here, that would be very painful.

[PatrickDiallo23]

That HTML “parsing” is done by me. I am only interested in how to retrieve the original source data using those 2 parameters:

List<SingleBenchmarkResult> singleBenchmarkResultList
Map<SolverBenchmarkResult, String> solverBenchmarkResultNameMap

I am not sure if I was more clear now, but if you can give me a hint or a guidance on how to achieve the aggregator functionality, that would be great. I also provided a short code snippet in the stack overflow question. Thanks again for your kind support!

[PatrickDiallo23]

@triceo?

[triceo]

Did you consider looking into the code and figuring out how those are filled?
At this point in time, I probably know as much about this code as you do; it's not ideal, but as I said, we haven't dealt with this in years.

[PatrickDiallo23]

I did, but I didn’t understand it well at the first glance. I will deep dive into it and try to provide a solution for the community. Maybe I will add a contribution in this direction if I manage to do it. Thank you!

[PatrickDiallo23]

I managed to find a solution for aggregation. I raised a PR to add this solution in the documentation as well. I will also add the solution here:

public File aggregateAllBenchmarks() {
// Load benchmark configuration
PlannerBenchmarkConfig benchmarkConfig = PlannerBenchmarkConfig.createFromXmlResource("benchmarkConfig.xml");
File benchmarkDirectory = new File(String.valueOf(benchmarkConfig.getBenchmarkDirectory()));
if (!benchmarkDirectory.exists() || !benchmarkDirectory.isDirectory()) {
    throw new IllegalArgumentException("Benchmark directory does not exist: " + benchmarkDirectory);
}

// Read all existing benchmark results from the directory
BenchmarkResultIO benchmarkResultIO = new BenchmarkResultIO();
List<PlannerBenchmarkResult> plannerBenchmarkResults = 
        benchmarkResultIO.readPlannerBenchmarkResultList(benchmarkDirectory);

if (plannerBenchmarkResults.isEmpty()) {
    throw new IllegalArgumentException("No benchmark results found in directory: " + benchmarkDirectory);
}

// Collect all single benchmark results and preserve solver names
List<SingleBenchmarkResult> allSingleBenchmarkResults = new ArrayList<>();
Map<SolverBenchmarkResult, String> solverBenchmarkResultNameMap = new HashMap<>();

for (PlannerBenchmarkResult plannerResult : plannerBenchmarkResults) {
    for (SolverBenchmarkResult solverResult : plannerResult.getSolverBenchmarkResultList()) {
        allSingleBenchmarkResults.addAll(solverResult.getSingleBenchmarkResultList());
        solverBenchmarkResultNameMap.put(solverResult, solverResult.getName());
    }
}

// Create and configure the benchmark aggregator
BenchmarkAggregator aggregator = new BenchmarkAggregator();
aggregator.setBenchmarkDirectory(benchmarkDirectory);
aggregator.setBenchmarkReportConfig(benchmarkConfig.getBenchmarkReportConfig());

// Perform the aggregation - returns HTML report file
File htmlOverviewFile = aggregator.aggregate(allSingleBenchmarkResults, solverBenchmarkResultNameMap);

return htmlOverviewFile;
}