Roadmap/wishlist of intended features

[sgbaird]

Hi, great to see the release of hypervolume-aware multi-objective optimization! I have some questions about other features, whether they might already be implemented or are planned in the near-term. Let me know if there's a place with a roadmap/wishlist that I missed. Specifically, I'm wondering about the following:

• trial-level early stopping
• global-level early stopping
• NChooseK-style / sparsity-aware / cardinality optimization while still leveraging autodifferentiation benefits (i.e., without brute forcing a set of predefined candidates) (I did notice ContinuousCardinalityConstraint)
• high-dimensional BO
• analytical outcomes/objectives with autodifferentiation benefits
• arbitrary, non-linear (but still differentiable) constraints, while retaining autodifferentiation benefits
• hybrid constraints: (I saw the limitation note in constraints)
• Flexibility to swap out the surrogate model with other methods for candidate generation? (e.g., custom neural network)
• In General feedback - documentation, functionality, and software dev #101 (comment), you mentioned built-in visualizations are out-of-scope/unplanned - is that still the case?
• multi-fidelity algorithms (e.g., knowledge gradient)

Also curious if there's a central place for documented incompatibilities (e.g., like the hybrid search space one mentioned)?

[AdrianSosic]

Hi @sgbaird, thanks for checkin in. I feel that it need a little bit more context for some of the bullets but apart from that I'd say that pretty much all of that is either on the roadmap or already WIP. Let me share some details and perhaps you can comment in case I misunderstood some point:

• Trivial/Global early stopping: Perhaps you can clarify your definitions? I suspect that in you differentiate between criteria that only operate point-wise vs. such that consider the surrogate/acqf on the entire space? For the short term, we haven't planned to add anything specific here because of our hook mechanism that basically gives full control to the user to define their own stopping criteria, like done in this example. But please let me know if you have something specific in mind.
• Cardinality with auto-diff: yes, we plan to extend that framework but please note that even the current framework probably does more than what you've described. In fact, we're not brute-forcing a specific candidate set – instead, we only consider a finite set of (in-)activity assignments of the parameters. But for each of these assignments, an actual auto-diff based optimization is conducted 👍🏼
• High-dimensional BO: yes, absolutely on the list. First, by incorporating the Hvarfner priors, and second by enabling specific algorithms for it.
• Analytical outcomes/objectives: I'm not entirely sure if we mean the same thing here, but there is in fact currently a major refactoring ongoing that will entirely redesign target specifications and will give full control to the user to easily set up arbitrary transformations and composing them in various ways to define their custom objectives. Is this what you were asking for? Should hopefully be finished in the next two weeks.
• Arbitrary constraints: yes, on the roadmap, but with a little lower prio at the moment
• Hybrid constraints: no concrete plans here, partly because we haven't even had a use case for it so far. If you have one that you could share, I'd be very much interested 🙃
• Custom surrogates: Please note that we have taken care to make customization easily possible for some parts of the framework – surrogates are one of them, because we already anticipated that people might want to use their proprietary models. So yes: we have clear API defined for this in form of a lightweight protocol, to let users specify their own logic. As long as you obey to it, you can use whatever surrogate you like. Does this answer the question, though? 🤔 Writing a customization guide for our user guide is overdue 😬
• Visualizations are still not a focus area, but I wouldn't object against having some basic methods in place as long as they are generic enough to cover many use cases. Can you tell me what is it that you're hoping for?
• Multi-fidelity: yes, shouldn't be too far down the road 👍🏼

[sgbaird]

• Trivial/Global early stopping ...

E.g., for trial-level stopping, someone is growing cells and monitoring a certain metric over time. Similarly for waiting for separation with liquid-liquid extraction. Sometimes, they'll want to kill the experiment early, but still learn from that info. Likewise applicable for other types of time-series-like data.

• Cardinality with auto-diff: yes, we plan to extend that framework but please note that even the current framework probably does more than what you've described. In fact, we're not brute-forcing a specific candidate set – instead, we only consider a finite set of (in-)activity assignments of the parameters. But for each of these assignments, an actual auto-diff based optimization is conducted 👍🏼

Ah, interesting - could you point me to where I could look at this?

• High-dimensional BO: yes, absolutely on the list. First, by incorporating the Hvarfner priors, and second by enabling specific algorithms for it.

👍🏼

• Analytical outcomes/objectives: ...

Cool! A simple example is material cost which is usually an analytical outcome (known, deterministic, not black-box).

• Arbitrary constraints: yes, on the roadmap, but with a little lower prio at the moment

On a related note, I'm curious, how are the linear composition constraints implemented? https://emdgroup.github.io/baybe/stable/userguide/constraints.html#continuous-constraints --> https://emdgroup.github.io/baybe/stable/_autosummary/baybe.constraints.continuous.ContinuousLinearConstraint.html#baybe.constraints.continuous.ContinuousLinearConstraint --> https://emdgroup.github.io/baybe/stable/_modules/baybe/constraints/continuous.html#ContinuousLinearConstraint -->

baybe/baybe/constraints/continuous.py

Line 33 in b060cff

class ContinuousLinearConstraint(ContinuousConstraint):

(any suggestions on where to go from here to see how it gets passed to BoTorch?)

• Hybrid constraints: no concrete plans here, partly because we haven't even had a use case for it so far. If you have one that you could share, I'd be very much interested 🙃

Ah, I think I misunderstood originally, somehow misinterpreted this. To clarify, discrete constraints and continuous constraints can be implemented within the same campaign, independently? (Agreed about a single constraint involving both being an edge case)

• Custom surrogates: Please note that we have taken care to make customization easily possible for some parts of the framework – surrogates are one of them, because we already anticipated that people might want to use their proprietary models. So yes: we have clear API defined for this in form of a lightweight protocol, to let users specify their own logic. As long as you obey to it, you can use whatever surrogate you like. Does this answer the question, though? 🤔 Writing a customization guide for our user guide is overdue 😬

Nice! I think this addresses the question, so as long as one can provide a sampling method, it should be compatible?

• Visualizations are still not a focus area, but I wouldn't object against having some basic methods in place as long as they are generic enough to cover many use cases. Can you tell me what is it that you're hoping for?
  All good - as you mentioned previously, exposing things like inverse lengthscales, model predictions and uncertainties, and optimization traces makes it easy for others to create visualizations. There are a number of visualizations that I tend to use, some shown in https://doi.org/10.1016/j.commatsci.2022.111505.
• Multi-fidelity: yes, shouldn't be too far down the road 👍🏼
  Nice!

As an aside, I don't remember if this was here the last time I looked (great to see the trajectory of improvements to the already very good docs), but I'm noticing the non-comprehensive overview on the front page: https://emdgroup.github.io/baybe/stable/

[AdrianSosic]

E.g., for trial-level stopping, someone is growing cells and monitoring a certain metric over time. Similarly for waiting for separation with liquid-liquid extraction. Sometimes, they'll want to kill the experiment early, but still learn from that info. Likewise applicable for other types of time-series-like data.

Ah, sorry, I read "trivial" instead of "trial" 😄 So the use case is clear to me, but still, can you comment on:

• What do you mean with "global-level" early stopping and how is it different in your eyes from the "trial" one?
• For the "trial-level" early stopping, do you miss any specific functionality that is not possible with our hook mechanism?

Ah, interesting - could you point me to where I could look at this?

I don't think we actually mention the specific internals in our doc somewhere 🤔 – perhaps that can be added. The only tiny remark I currently see is here, where the max_n_subspaces parameter is mentioned. @Scienfitz: Do you see any other useful resource?

On a related note, I'm curious, how are the linear composition constraints implemented?
(any suggestions on where to go from here to see how it gets passed to BoTorch?)

Yes, you'll see that many of our baybe objects have to_botorch methods attached that take care of the translation. And so do the constraints objects, like the linear one which has this method implemented.

Ah, I think I misunderstood originally, somehow misinterpreted this. To clarify, discrete constraints and continuous constraints can be implemented within the same campaign, independently? (Agreed about a single constraint involving both being an edge case)

Yes, exactly. You can have both types of constraints in a single method, but it's not possible to have constraints that impose restrictions on continuous and discrete parameters jointly.

Nice! I think this addresses the question, so as long as one can provide a sampling method, it should be compatible?

Yes, in a sense. What you basically need to do is to write a corresponding to_botorch function that defines how your custom logic is turned into a corresponding BoTorch model object. BayBE then takes care of the rest, i.e. all pre-processing steps, model fitting, the candidate generation, etc...

As an aside, I don't remember if this was here the last time I looked (great to see the trajectory of improvements to the already very good docs), but I'm noticing the non-comprehensive overview on the front page: https://emdgroup.github.io/baybe/stable/

I've already planned an entire documentation refactoring, including migrating to another doc framework. But that will take a few weeks/months until completed 😬

Let me know if you need any more information 🙃

[sgbaird]

Sorry, just revisiting: "global" meaning relative to high-level campaign / experimental budget (total number of iterations / experiments / etc.). Trial-level meaning within a single iteration / experiment. Perhaps the best traditional ML example of trial-level early stopping is during hyperparameter tuning, terminating a specific model "early" (i.e., at fewer epochs than normal due to poor performance during early epochs).

I looked into the hook mechanism again. I think this would allow for termination (i.e., throwing away) a trial due to poor early performance, at the expense of losing any information associated with that trial. I think this answers my question though.

[Scienfitz]

Hello @sgbaird
(apologies as I'm a bit late to the thread)

Adrian already touched on many points, let me add to unmentioned things:

Analytical Outcomes
This is something we discussed recently. Our example is simply: cost (eg of reagents). This is something that should enter the acqf calculation, but does not have to be modelled, ie should not be part of the surrogate model. Should be somewhat straightforward to offer this via a new target type a la AnalyticalTarget or LookupTarget and the ability to provide a lookup via user defined functions.

Early-Stopping

• Global level: The ability to stop via hooks was pointed out and we show a working example in our docs and publication. But we have not spent a lot of time to obtain a great criterion for that. Thorough work on that would make a nice little publication on its own imo, so far I find the use of the single acqf for the criterion a bit inconsistent.
• Trial level: I think this is often referred to as pruning, eg in optuna and xgboost combo. It will probably not be easy to incorporate the info from such pruned experiments, so the only thing we currently offer for that is to mark the experiment as incomplete/failed by passing NaN (only possible with ParetoObjective), but this doesnt really use the info. Maybe a pragmatic way could be to assign "bad" target values in this pruned case, so the algo would be able to see it as "unpromising" but its not ideal (and unclear for unbounded targets).

Plots
I think we have the opportunity to provide more of this kind of functionality as part of our Insights. For instance, the SHAPInsight does the shap analysis and has an interface to the common plots for that. Looking at the publication you mentioned, it seems these are mostly regression metrics and analysis of the predictive performance. I am planning on another insight called FitInsight or RegressionInsight and it would make sense to offer all such plots as part of it. It is mostly aimed at traditional metrics, residuals, goodness of fit etc (also providing a little bridge to DOE people used to looking at these things). I would love to hear your requirements for that in detail, if you're up for it feel free to open an Issue and name the metrics and plots you consider vital and most frequently used.

Yet another idea would be a CampaignInsight providing easy access to plots about the optimization trajectory. if you have plot or postprocessing suggestions for that feel free to shoot.

Overall, visualization with baybe has already become much easier, there are now easy interfaces to calculate arbitrary acqf values or the posterior statistics.

[Scienfitz]

BayBE now maintains a rough roadmap in Milestones, including a non ETA (want and plan but no detailed ETA yet), feel free to request features via Issue which might be assigned a Milestone after completed scoping