Skip to content

Tunable constrained optimization algorithms #324

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 286 commits into
base: master
Choose a base branch
from
Draft

Tunable constrained optimization algorithms #324

wants to merge 286 commits into from

Conversation

fjwillemsen
Copy link
Collaborator

@fjwillemsen fjwillemsen commented Jul 11, 2025 •
edited
Loading

This PR combines the hyperparameter tuning with the constrained optimization algorithms, new differential evolution implementation, and support for custom strategies.
In addition, it brings several performance improvements to the handling and lookup of neighbors in the search space:

  • Simultaneous caching of various neighbor methods.
  • Specific methods for finding a random neighbor.
  • Partial neighbor caching, which works by the fact that each neighbor lookup also produces partial information on the neighbors of that neighbor.

@fjwillemsen
Refactored BO BOTorch into class structure
eba03f8
@fjwillemsen
Switched to newer fit function, more efficient model initialization b…
c6b243a 
…y reusing state
@fjwillemsen
Added option to return invalid configurations in CostFunc
1581840
@fjwillemsen
Added the handling of invalid configurations, training data is direct…
620ee60 
…ly modified by the evaluation function
@fjwillemsen
Setup structure for Tensorspace in Searchspace
009cf01
@fjwillemsen
Implemented mappings and conversions to and from tensor to parameter …
33983f7 
…configuration
@fjwillemsen
Improved efficiency of acquisition function by removing evaluated con…
f3fc81b 
…figurations
@fjwillemsen
Removed Ax, added BOTorch as dependency
a5a0471
@fjwillemsen
Convenience script for benchmarking BO
9429539
@fjwillemsen
Added objective, tuning direction and hyperparameter tuning language …
176b8f5 
…selection support
@fjwillemsen
Completed implementation of mixed-type handling and handling of inval…
196af62 
…id and evaluated configurations
@fjwillemsen
Added docstrings, improved formatting
55a5c1a
@fjwillemsen
Extended strategies test to test for ability to handle non-numeric an…
d64f783 
…d mixed parameter values
@fjwillemsen
Mixed-type parameters are not converted to numeric constraints
e95ab30
@fjwillemsen
CostFunc can now encode and decode non-numeric configurations for str…
10a6a5c 
…ategies that require only numerics
@fjwillemsen
Fixed logging statements, improved formatting
6ae3ba6
@fjwillemsen
Improved the performance of get_bounds
4873a20
@fjwillemsen
Applied non-numeric encoding in differential evolution to handle non-…
bae7e96 
…numeric parameter values
@fjwillemsen
Implemented automatic conversion to multiple types for encoded tensor…
7eb7ef7 
… parameter lookup
@fjwillemsen
Added tests for Searchspace tensor encoding and conversion
91d3ce4
@fjwillemsen
Seperated strategies and runners test cache file
80d514e
@fjwillemsen
Implemented handling of categorical parameters
a489252
@fjwillemsen
Implemented variational GP and likelihood
68aee14
@fjwillemsen
Using LogExpectedImprovement to avoid stability issues
b9c012d
@fjwillemsen
Implemented tensor space bounds in searchspace
41ce663
@fjwillemsen
Implemented normalization for input features
07ef1d4
@fjwillemsen
Tensorspace is reduced by removing inconsequential parameters
721d072
@fjwillemsen
Extended strategies tests to include single parameter value
2434b3b
@fjwillemsen
Fixed an indexing error for tensorspace bounds
1679751
@fjwillemsen
Extended searchspace tests to include single parameter value
2b816a6
fjwillemsen added 29 commits July 11, 2025 18:38
@fjwillemsen
Implemented getting a random neighbor over the full list where applic…
6e61fd6 
…able, other improvements
@fjwillemsen
Adjusted costfunc_kwargs defaults
83c4043
@fjwillemsen
Implemented partial neighbor caching, adding more information to the …
623eca4 
…overall neighbor knowledge with every lookup
@fjwillemsen
Fix to add neighbor method as argument
f671c0e
@fjwillemsen
Extended differential evolution hyperparameters to fit new algorithm …
b45f880 
…implementation, minor improvements
@fjwillemsen
Merged the changes to valid param config checking and checking matchi…
ff64ae3 
…ng problem size
@fjwillemsen
Implemented retrieving the true tunable parameters (those that are no…
f31f67c 
…t constant after restrictions) from the search space
@fjwillemsen
Implemented number of dimensions-dependent population size in diff_evo
bb152f2
@fjwillemsen
If there is a problem loading the cache, the path of the cachefile is…
b7e779e 
… given
@fjwillemsen
Implemented passing keyword arguments like meta strategy and time lim…
a89af66 
…it to hyperparameter tuning CLI
@fjwillemsen
Improved matching problem size check
8bd5991
@fjwillemsen
Implemented a discrete version of Latin Hypercube sampling that is (s…
89891a8 
…emi) constraint-aware
@fjwillemsen
Implemented efficiently representing the full search space as the ind…
b91a650 
…ices of the parameter values
@fjwillemsen
Implemented getting the numpy array as numerical values, where non-nu…
fa293cf 
…merical values are replaced with their index
@fjwillemsen
Implemented retrieving the true parameter configuration from a mixed …
0812b2b 
…numerical configuration
@fjwillemsen
Implemented parameter index-, numeric- and mixed- 2D views of the sea…
21d63bc 
…rch space
@fjwillemsen
Created additional tests for new and improved numpy views of the sear…
66f3f69 
…ch space
@fjwillemsen
Implemented a fast distributed random sample technique suitable for t…
3b1aeed 
…he discrete constraint search spaces
@fjwillemsen
Various improvements to and other searchspace functions
ce55392
@fjwillemsen
Performance improvement to __get_random_neighbor_adjacent
5ab83f8
@fjwillemsen
Implemented test for get_distributed_random_sample_indices
3a38865
@fjwillemsen
Implemented a new neighbor method for getting the neighbors clostest …
c43634b 
…to the parameter indices of the given configuration
@fjwillemsen
Wrote test for closest param indices neighbor method
0fe2ea5
@fjwillemsen
Implemented a LHS sampler, automatic adjustement to smaller integer s…
d0a5ba6 
…izes where possible for numpy arrays
@fjwillemsen
Implemented tests for the LHS sampler and the true index bounds
ce29740
@fjwillemsen
Using LHS and closest-param-indices in new diff_evo for much better p…
16c8245 
…erformance
@fjwillemsen
Disabled full validate on load for hyperparameter tuning
c044ef8
@fjwillemsen
Extended hyperparameters for optimization algorithms in paper
cde7823
@fjwillemsen
Updated required python-constraint version
f56c1fd
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
114 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.2% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@fjwillemsen @benvanwerkhoven