How to Ensure Complete Reproducibility in REINVENT4?

[bishalbudha]

Hello,

I am currently working with REINVENT4 for a project that I plan to submit for publication (targeting Q1). My workflow involves:

1. Transfer learning of the REINVENT prior using ~1000 active compounds.
2. Reinforcement learning with SAS, QED, MW, and Jaccard distance as scoring components.
3. Sampling ~1,000,000 molecules from the final RL checkpoint.
4. Scoring the sampled compounds on the same components used during RL.

My questions are:

1. Is this workflow appropriate, or is there any mismatch in the sequence I’m following?
2. Regarding reproducibility—which, as we all know, is crucial for publication—I couldn’t find an option to set a random seed in the four .toml configuration files I’m using for my project. Since reproducibility with a fixed random seed is critical, how should I handle this? Should I set the seed individually in each .toml file, or is there a global option for specifying a single seed?

I would greatly appreciate guidance, preferably with basic examples of how to correctly implement random seed settings in REINVENT4.

Thank you,
Bishal Budha

[bishalbudha]

I use -s 57 in each code used for executing run modes but each time I am getting different results.

!reinvent -l transfer_learning_parp1.log -s 57 configs/transfer_learning_parp1.toml

Where are further place where I should fix the random seeds to make complete reproducible?

[halx]

REINVENT is highly sttochastic at multiple levels and the seed only controls some aspected of this. This also means that reproducibility of a stochastic process does and can not mean faithful repeatability. It doesn't make scientifical sense to attempt to do so. Rather you would do multiple runs and provide some statistics as to the the spread of the data.

As for your protocol, point 3 (sampling after RL) is a common misconception. You already produce compounds in 2 and use 4 for postprocessing. Why would you throw away good compounds? Besides, sampling does not guarantee you compounds in line with your filtering criteria. I suggest to carry out multiple RL runs, combine the data and assess statitics.

As for the scoring componds, QED is a compound score including MW (with a certain weight) so is rather redundant. Its more typical to directly control parameters, in essence Lipinski and Veber rules plus pattern match regarding wanted and unwanted substructural features. Jaccard distance assumes that you want structural closeness. It would probably be better if you had a 3D alignment method like ROCS. The SA score is mostly a measure of complexity and typically correlates not that well with more sophisticated methods. We offer an interface to our retrosynthesis tool AiZynthFinder which may be too expensive to use with RL but post-processing is doable after filtering.

[bishalbudha]

Dear @halx Sir,

I sincerely thank you for your unwavering support and guidance. I am truly grateful. I still have a few doubts that I would kindly request your help in clarifying.

I have finalized the following code to use next for RL, based on the configuration files available in the REINVENT4 repository. Here is my code for applying RL after Transfer Learning:

run_type = "staged_learning"
device = "cuda:0"  # set torch device e.g. "cpu"
tb_logdir = "tb_logs_rl"
json_out_config = "_staged_learning.json"

[parameters]
summary_csv_prefix = "staged_learning"
use_checkpoint = false
purge_memories = false

## Reinvent
prior_file = "priors/TL_parp1_reinvent.model"
agent_file = "priors/TL_parp1_reinvent.model"

## ISIM
tb_isim = false

batch_size = 128
unique_sequences = true
randomize_smiles = false

[learning_strategy]
type = "dap"
sigma = 128
rate = 0.0001

[diversity_filter]
type = "IdenticalMurckoScaffold"
bucket_size = 30
minscore = 0.4
minsimilarity = 0.4
penalty_multiplier = 0.5

[[stage]]
chkpt_file = "RL_Final_checkpoints.chkpt"
termination = "simple"
max_score = 0.8
min_steps = 25
max_steps = 1000

[stage.scoring]
type = "geometric_mean"

[[stage.scoring.component]]
[stage.scoring.component.custom_alerts]
[[stage.scoring.component.custom_alerts.endpoint]]
name = "Unwanted SMARTS"
params.smarts = [
    "[*;r8]", "[*;r9]", "[*;r10]", "[*;r11]", "[*;r12]", "[*;r13]", "[*;r14]", "[*;r15]",
    "[*;r16]", "[*;r17]", "[#8][#8]", "[#6;+]", "[#16][#16]", "[#7;!n][S;!$(S(=O)=O)]",
    "[#7;!n][#7;!n]", "C#C", "C(=[O,S])[O,S]", "[#7;!n][C;!$(C(=[O,N])[N,O])][#16;!s]",
    "[#7;!n][C;!$(C(=[O,N])[N,O])][#7;!n]", "[#7;!n][C;!$(C(=[O,N])[N,O])][#8;!o]",
    "[#8;!o][C;!$(C(=[O,N])[N,O])][#16;!s]", "[#8;!o][C;!$(C(=[O,N])[N,O])][#8;!o]",
    "[#16;!s][C;!$(C(=[O,N])[N,O])][#16;!s]"
]




[[stage.scoring.component]]
[stage.scoring.component.QED]
[[stage.scoring.component.QED.endpoint]]
name = "QED"
weight = 4

[[stage.scoring.component]]
[stage.scoring.component.rocssimilarity]
[[stage.scoring.component.rocssimilarity.endpoint]]
name = "ROCS similarity"
weight = 4
params.color_weight = 0.5
params.shape_weight = 0.5
params.similarity_measure = "Tanimoto"
params.max_stereocenters = 4
params.ewindow = 10
params.maxconfs = 200
params.rocs_input = "YOUR_ROCS_QUERY" # can be .sdf or .sq
transform.type = "sigmoid"
transform.high = 0.6
transform.low = 0.15
transform.k = 0.5



[[stage.scoring.component]]
[stage.scoring.component.SAScore]
[[stage.scoring.component.SAScore.endpoint]]
name = "SA score"
weight = 2
transform. Type = "reverse_sigmoid"
transform.high = 6.0
transform.low = 0.1
transform.k = 0.5

My questions are as follows:

1. Diversity Filter and Custom Alerts as global filters

   • In the official REINVENT4 documentation, it is mentioned that the Diversity Filter and Custom Alerts act as global filters, where the total score is multiplied by 0 or 1. However, in the official configs provided, both are assigned weights, which suggests they are being treated like standard scoring components rather than global filters.
   • If I wish to apply them strictly as global filters, what weight should I assign (or remove) to ensure they act as true on/off gates instead of weighted components?
   • Also, since Custom Alerts is kept below alongside other scoring components in the configs (rather than grouped with the Diversity Filter), does this placement affect its role as a global filter?

2. Randomize SMILES

   • I am unclear about the option randomize_smiles = true/false. Does this mean the generated SMILES strings are randomized during training to produce new molecules?
   • If so, what is the rationale for keeping unique_sequences = true at the same time? Wouldn’t canonical SMILES ultimately map to the same unique molecules anyway?

3. Sampling vs. checkpoints

   • You previously suggested that instead of sampling only from checkpoints, I could utilize the full set of molecules generated during training. Does this mean combining molecules generated at each epoch (e.g., across 1 to 1000 steps, which would be ~128,000 molecules with batch size 128)?
   • If I need more molecules, should I extend the number of epochs, or should I adjust my usage strategy with the molecules I already have?

4. Staged scoring setup

   • My intention is to maximize QED, keep similarity balanced (neither too high nor too low), and use SAS as a decoy tool just to test its effect. I want to reward low SAS scores while gradually penalizing higher ones.
   • Is the transformation I used for SAS score appropriate?
   • For similarity (ROCS), would it be better to use a double sigmoid transformation (e.g., rewarding the range 0.4 to 0.8 while reducing reward outside this window) to achieve the “balanced similarity” objective?

5. ROCS input

   • For the params.rocs_input, should I provide a single .sdf file containing all the active molecules I want to replicate, or prepare them differently?

Thank you very much for your time and continued guidance.

With best regards,
Bishal Budha

[bishalbudha]

Sir, please help me 🙏

[halx]

1. They are filters. Weights are ignored.
2. Set the flag to true. unique_sequence is a legacy flag.
3. Run multiple independend RL runs.
4. You will need a reverse sigmoif for SA. I do not know what you plan with "balanced similarity" is.
5. You will need the OpenEye tools to create the input files. You will need a license to use OpenEye.

[bishalbudha]

Thank you @halx .

To balance similarity means I want to create new molecules that are not too similar that exactly replicate the actives and not too dissimilar that are far off from actives. Won't it accept the files if we simply create the single SDF of all actives and input it?

[halx]

Consult the OpenEye documentation on ROCS. I can't help here.

[vivianzheng-st]

Hi, if RL already generated compounds and provided scoring for designated components. Out of curiosity, why do I need to run scoring.toml for generated compound again in post-processing?

[halx]

You don't unless you want to compute additional scores That's not untypical for computationally expensive scoring components.