[WIP] Implement VOCS interface

[shuds13]

This implements Step 3 in #269

For first pass, keep internals the same and implement interface:

Implementation:

• Implement interface for simple generators (base/line/grid/random sampling) and tests
• Check correct inheritence of generators from standard.
• Add basic validate_vocs functions
• Update validate_vocs functions and incorporate previous check functions.
• Add tests for validate_vocs exceptions
• Handle setting type of observables
• Handle / replace extra parameter attributes in Optimas
• Check exploration_diagnostics changes
• Determine if need integer variables
• Basic update Ax generators and tests
• Update Ax generators constraints specification to use VOCS
• Update multitask gen to use _id and internal mapping - undone
• Update multitask to use discrete variable for trial_type.
• Verify that the fields passed between ingest/suggest methods are correctly derived from VOCS specifications (due to keeping the old internals, the generators work with Optimas legacy parameter handling). Indirectly this happens.
• Removed or resolved extra fields given to gens (_ignored, _ignored_reason, _status).
• Discrete variables should support strings at base class.
• Update scripts in examples dir

Checks:

• Ensure examples work (not just tests) - run on Perlmutter.
• Ensure works with external VOCS generators from xopt and libensemble (inc. APOSMM use-cases).
• Check/update docstrings and other references to old data structures
• Check for any TODO comments remaining.
• Ensure using gen standard main rather than a branch

Reviewer checks:

• Check results
• Check and update validate_vocs functions
• Check issues outlined below and solutions
• Check generator/test docstrings and comments

[shuds13]

In updating Optimas generators for the standard, how do we deal with these extra fields in VaryingParameter.

    lower_bound: float
    upper_bound: float
    is_fidelity: Optional[bool] = False
    fidelity_target_value: Optional[float] = None
    default_value: Optional[float] = None
    _is_fixed: bool = PrivateAttr(False)

re: https://github.com/optimas-org/optimas/blob/main/optimas/core/parameter.py#L79

In the standard, variables have default_value & domain, but no equivalent to the others.

https://github.com/campa-consortium/generator_standard/blob/main/generator_standard/vocs.py#L16

Do we need is_fidelity / fidelity_target_value for each parameter? Or could it be a generator option?
(fidelity_target_value is given to Ax as the fidelity we want to optimize for).

is_fixed allows fixing a parameter value between calls to run(). Generator still has to support it.

e.g.

    var1 = VaryingParameter("x0", -50.0, 5.0)
    var2 = VaryingParameter("x1", -5.0, 15.0)

    # Start with a fixed value of x0.
    var1.fix_value(-10.0)
    ...
    exploration.run(n_evals=5)
    var1.free_value()
    gen.update_parameter(var1)
    exploration.run(n_evals=5)

could be instructed directly to generator instead - for supporting gens.

e.g.

gen.fix_var('x0', -10.0)

Alternatively, we could add fields to the standard, or create subclasses to ContinuousVariable for Optimas. This may be more elegant.

Currently, I have used the following generator functions (shown with example calls) to set the Optimas internals for these, without adding to or subclassing ContinuousVariable in VOCS.

gen.fix_value("x0", -10.0)
gen.update_range("x0", -20.0, 0.0)
gen.set_fidelity_param("res", is_fidelity=True, fidelity_target_value=8.0)

These are used in test_ax_generators.py.

[shuds13]

Observables (analysed parameters) in Optimas can specify a type, but this isn’t supported in the standard interface. A type is necessary for constructing a NumPy structured array (for libEnsemble).

For example, in this test, the type must be a string.

https://github.com/optimas-org/optimas/blob/main/tests/test_env_script.py#L27

In test_template_evaluator.py and test_function_evaluator.py it stores an array and a plot object.

I have suggested we allow specifying a type for observables and constants in the standard.
campa-consortium/generator_standard#39

The standard does not support arrays generally, but I have made it possible to specify a string or a tuple for observable type, which means numpy arrays can be used.
campa-consortium/generator_standard#40

e.g.

def test_template_evaluator():
    # Define variables and objectives.
    vocs = VOCS(
        variables={"x0": [-50.0, 5.0], "x1": [-5.0, 15.0]},
        objectives={"f": "MAXIMIZE"},
        observables={
            "p0": (float, (2, 4)),
            "p1": "O",
            "fig": "O",
        },
    )

    # Define variables and objectives.
    gen = RandomSamplingGenerator(vocs=vocs)

[shuds13]

exploration_diagnostics:

exploration_diagnostics.py:
_create_exploration Still reads from a file which will contain data in the old Optimus format - this should be updated.

test_exploration_diagnostics.py:
Removes a check that now seems redundant

        # Check that all possible objective inputs give the same result.
        _, trace1 = diags.get_objective_trace()
        _, trace2 = diags.get_objective_trace("f1")
        # _, trace3 = diags.get_objective_trace(obj)  # Can be removed
        np.testing.assert_array_equal(trace1, trace2)
        # np.testing.assert_array_equal(trace1, trace3)

[shuds13]

Note that validate_vocs gets called in the generator standard base class. This makes it a bit difficult at the moment to incorporate the other functions like _check_parameters or _check_inputs as they work on the converted data types.

[shuds13]

Small point. In test_ax_generators.py constraints use <= in Optimas, where as standard uses LESS_THAN.

outcome_constraints=["p1 <= 30"],

becomes:

constraints={"p1": ["LESS_THAN", 30.0]},

[shuds13]

What do we do for test test_ax_single_fidelity_int. Do we need to support this?

    var1 = VaryingParameter("x0", -50.0, 5.0, dtype=int)
    var2 = VaryingParameter("x1", -5.0, 15.0)

In vocs this becomes:

    vocs = VOCS(
        variables={"x0": set(range(-50, 6)), "x1": [-5.0, 15.0]},
        objectives={"f": "MAXIMIZE"},
    )

I don't see anything to deal with discrete vars in Optimas, so I have converted to integer type when its a range but that seems hacky.

To support this elegantly, we could bring back IntegerVariable type in the vocs standard.

[shuds13]

I got rid of ax_trial_id and arm_name as parameters and just held them internally mapped to a single id. We discussed doing this before. But I don't think this works, as we may choose hi-fidelity points from a batch incorporated from previous history. And with this change, we lose that information. I would need to know ax_trial_id and arm_name for those historical points.

But this would not fit what vocs currently supports. ax_trial_id would be open ended integer variable, and arm is stored as a string, again open ended (not a discrete set). Maybe the could be observables, but they are created in the gen. They really just need to be known.

[shuds13]

Alt. could be for historical points, to find best points from the whole set and issue new arm_id.

[shuds13]

This is reason for:
FAILED tests/test_ax_generators.py::test_ax_multitask_with_history

[shuds13]

Following discussion, it seems we can add additional parameters to ingest (not restricted to vocs), and so these could be given to generator as extra arguments. In that case using _id may not be necessary for this gen.

[shuds13]

So trial_type (hi/low fidelity) is now declared as a discrete variable in vocs. This is a vocs variable as it is given to each simulation. As string types are not supported in the Optimus internal VaryingParameter structure, it converts to TrialParameter internally.

ax_trial_id (batch id) and arm_name (point identifier) are set up in the gen and passed back and forth as generator specific values (not part of vocs). I have continued to use TrialParameters so Optimas stores them in the history array and they can be read back in from previous histories. However, this does mean an Optimas format is being used in the generator.

[shuds13]

Is there a need for a continuous range of integers?

[RemiLehe]

I agree with removing this line (and the associated np.testing.assert_array_equal below)

[RemiLehe]

Could we raise an exception for now, saying that constraints are not supported in optimas generators?

[shuds13]

We have tests with constraints such as

optimas/tests/test_ax_generators.py

Line 158 in b760144

constraints={"p1": ["LESS_THAN", 30.0]},

[RemiLehe]

Would this work for https://github.com/optimas-org/optimas/pull/281/files#diff-54d05e7e049beb11035069e9030950c2b1080b97631d4fa5d984eb8bfe347638R541 where the discrete variables are string?

[shuds13]

The conversion of strings was added to multitask to deal with that case. So currently I think base generator class would not work with strings. Looking at that, I think we could move that conversion to the base class so it supports strings.

[shuds13]

String type variables convert to Optimas TrialParameters because, VaryingParameters require upper/lower bounds.

[shuds13]

I have moved that conversion to the generator base class now. That function can be used by any Optimas generator that requires discrete variables.