GH-16184 Add possibility to set auc_type in AutoML

[maurever]

Issue: #16184

[tomasfryda]

So far it looks good but could you add an R test and some documentation.

I think it would be also a good idea to expose it the same way we expose monotone_constraints .

And last but not the least, could you make sure it works with stopping_metric and that the metric actually ends up in the leaderboard? (In the multinomial case)

[maurever]

@tomasfryda, thanks for your suggestions!

It is the question if it is necessary to expose it similarly to the monotone_constraints parameter. I found out we can add algo parameters in this way, but it is nowhere documented. So, it's a good point to do that.

Why adding a new parameter to the AutoML class is better than using the algo_parameters parameter?

Definitely a good point to add an R test and test with early stopping.

[tomasfryda]

I think algo_parameters is more for "hacks" than regular use. You can change any parameter if you enable it using the algo_parameters. It's not documented because it can cause some problems so it's intended only for expert users.

[maurever]

@tomasfryda, thanks for explanation. Ok, I will do it like the monotone_contraint parameter, no problem.

I am not a fan of many parameters, so I tried to find a different solution. :)

[tomasfryda]

should be:

Suggested change

	auc_type="NONE",
	auc_type=c("NONE", "MACRO_OVO", "WEIGHTED_OVO", "MACRO_OVR", "WEIGHTED_OVR", "AUTO"),

And please see match.arg (https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/match.arg) or you can use case-insensitive version from explain module

h2o-3/h2o-r/h2o-package/R/explain.R

Lines 2 to 27 in 18341e3

	#' Works like match.arg but ignores case
	#' @param arg argument to match that should be declared as a character vector containing possible values
	#' @param choices argument to choose from (OPTIONAL)
	#' @return matched arg
	case_insensitive_match_arg <- function(arg, choices) {
	var_name <- as.character(substitute(arg))
	if (missing(choices))
	choices <- eval(formals(sys.function(-1))[[var_name]])
	orig_choices <- choices
	if (identical(arg, eval(formals(sys.function(-1))[[var_name]])))
	arg <- choices[[1]]
	choices <- tolower(choices)
	if (length(arg) != 1)
	stop(sprintf("'%s' must be of length 1", var_name), call. = FALSE)
	arg <- tolower(arg)
	i <- pmatch(arg, choices, nomatch = 0L, duplicates.ok = FALSE)
	if (all(i == 0L) || length(i) != 1)
	stop(sprintf("'%s' should be one of %s", var_name, paste(dQuote(orig_choices), collapse = ", ")), call. = FALSE)
	return(orig_choices[[i]])
	}

[maurever]

Thank @tomasfryda for the suggestion. I am still working on this PR. I converted it to a draft. I would like you to review it after I finish it. Thanks!

[tomasfryda]

I'm always happy to help with automl or R so feel free to ask if there is something unclear.