Distinguish between target_metric and result_metrics

[nihaase]

In config/study.py the target metric is defined as

target_metric: str | list[str] = field(validator=[validate_metric])

This creates type errors like

octopus/modules/efs/core.py:333: error: Argument 1 to "get_metric_function" of "MetricsInventory" has incompatible type "str | list[str]"; expected "str" [arg-type]

Current Problems:

1. Type ambiguity: The str | list[str] union causes type errors everywhere
2. Redundancy: There are already two fields doing similar things:
   - target_metric: str | list[str] - optimization metric (+ optional extras)
   - metrics: list - metrics to calculate (defaults to target_metric)
3. Inconsistent usage: Different modules handle it differently:
   - EFS, SFS, RFE, Boruta, AutoGluon: str | list[str]
   - Training, Enssel, Bag: str only
4. Unclear intent: When target_metric=["AUCROC", "ACC"], it's not obvious that only AUCROC is used for optimization

A possible solution would be to implement something like

  @define
  class ConfigStudy:
      target_metric: str = field(validator=[validate_metric])
      """The single metric used for optimization."""

      metrics_for_results: list[str] = field(
          default=Factory(lambda self: [self.target_metric], takes_self=True),
          validator=[validators.instance_of(list), validate_metric]
      )
      """Additional metrics to calculate for reporting (includes target_metric by default)."""

[anwurl]

target_metric: the more general approach is a list of metrics to allow optimization over multiple metrics.

We added str to improve the user experience as many use-cases only optimize one metric.

Item2: current fields:

• target_metric - metric used to evaluate model performance during model optimization
• metrics - full list of metrics used for evaluation of the final model

Item 4: Currently, we don't support multi-metric optimizations. Therefore, several items in this list should lead to an error ("Multi metric optimizations are currently not supported").

Item 3: As currently none of the module and classes support multi-metric optimizations we could opt for str and then later change to list(str) if this functionality is implemented.

Is there a way to keep target_metric as str | list(str) so that we don't have to change the user interface once we implement multi-metric optimizations?

[nihaase]

I am not really a fan of having something in code that we potentially use at some point in the future. Does it have a big impact on the code to stay with str only for now and once we implement it, we can see how it can implement best.

[dasmy]

Maybe as some context: Many places in the code rely on target_metric being a string and not a list:

From my perspective it would also be OK to

• have some function get primary_metric() somewhere that just always picks the first element from the list. This would later still allow to easily find all the instances where only one metric is supported. ; or
• raise a ValueError wherever a list is not supported

I would not be a big fan of explicitly picking the [0] element in every place where a list is not supported because it will be extremely difficult to find these places later again.

[nihaase]

We could create a property for primary metric?

[nihaase]

Remove metrics input from study and just calculate all metrics available at the end. Be aware to do this not for every trial.