Return values CMASolutions for nevals / fevals are either wrong or very misleading

[berndbischl]

Hi.

0. Preliminary comment. The API docs don't make it clear what the difference between the 2 is.

https://cma-es.github.io/libcmaes/doc/html/classlibcmaes_1_1CMASolutions.html

I suspect it is kinda a simple oversight to have both of them? The code seems to handle them in the same manner.

1. What is more problematic.

As a user, I would expect to get the number of cumulative (!) fevals here. Even for restarted strategies.
But that's not true.

MRE

#include <iostream>
#include <string>
#include <vector>

#include <libcmaes/cmaes.h>
#include <libcmaes/cmasolutions.h>

using namespace libcmaes;

static int COUNTER = 0;

int main() {
  const int dim = 5;
  std::vector<double> x0(dim, 0.5);
  const double sigma = 0.3;

  // Algorithms to test (restart strategies)
  const std::vector<std::string> algos = {"ipop", "bipop", "sepipop", "sepabipop"};

  for (const std::string &alg : algos) {
    COUNTER = 0;
    CMAParameters<> params(x0, sigma);
    params.set_str_algo(alg);
    params.set_seed(42);
    params.set_mt_feval(false); // ensure single-threaded feval

    // Give enough budget to allow multiple restarts for restart strategies
    params.set_max_fevals(4000);
    params.set_restarts(10);

    FitFunc func = [&](const double *x, const int &n) -> double {
      COUNTER++;
      return 1.0;
    };

    CMASolutions sols = cmaes<GenoPheno<NoBoundStrategy>>(func, params);

    std::cout << "algo=" << alg << ", true_evals=" << COUNTER << ", sols.fevals() = " << sols.fevals()
              << ", sols.nevals() = " << sols.nevals() << std::endl;
  }

  return 0;
}

Output

algo=ipop, true_evals=4112, sols.fevals() = 160, sols.nevals() = 160
algo=bipop, true_evals=4038, sols.fevals() = 160, sols.nevals() = 160
algo=sepipop, true_evals=4112, sols.fevals() = 160, sols.nevals() = 160
algo=sepabipop, true_evals=4038, sols.fevals() = 160, sols.nevals() = 160

Likely quite related to issue #253

[beniz]

I've merged #254 and I see there's an fevals_global, so maybe you want to expose this now. You can PR as needed.

[berndbischl]

I can do a PR but the have 2 questions.

A. What about the current nevals, fevals in cmasolutions. Should they they really simply stay as they are....?

B. How do you create unit tests for such cases?

[beniz]

Thanks :

• you can add tests to tests/CMakeLists.txt
• I guess nevals et fevals can stay as they are in single-shot (no-rerun) algorithms ? fevals() returns _nevals anyways, no ?