Reinvent transfer learning with large dataset (~500k)

[ShijiZ]

Hello,

Since reinvent was trained on Chembl dataset, I would like to retrain the reinvent model with a new dataset (~500k SMILES) that covers different chemical space as Chembl. Is transfer learning the right way to do it? For now I am testing TL from the provided reinvent.prior model, and I plan to do TL from an empty model as discussed in #87

However, I am facing an issue that huge amount of memory is needed, causing my job being killed by the system. Job report with qacct command shows:

maxvmem                  567.077G
maxrss                   438.520G

Is this normal behavior for my dataset? Since Chembl is much bigger than my dataset, how much memory was needed for the original reinvent training? Below is my input toml file:

run_type = "transfer_learning"
device = "cuda:0"
tb_logdir = "tb_TL"


[parameters]

num_epochs = 50
save_every_n_epochs = 2
batch_size = 100
num_refs = 0
sample_batch_size = 2000

input_model_file = "[path_to_priors]/reinvent.prior"
output_model_file = "TL_reinvent.model"
smiles_file = "input.smi"
# validation_smiles_file = "input.smi"
standardize_smiles = true
randomize_smiles = true
randomize_all_smiles = false
internal_diversity = true

[halx]

Hi,

there is no simple recipe for this, I think. With only 500K compounds you may struggle creating a prior generating a sufficiently high percentage of valid SMILES. This assumes that you would be using the same network size as the ChEMBL prior. We have not tested what the LSTM hyperparameters should be for "small" training sets.

What you could do, is to use the current ChEMBL prior and apply TL with your own dataset.

Your reported memory footprint does not make sense to me. As you are running on a GPU, the main determining factor is GPU memory. You should be able to train a prior of your size with less than 10GB GPU memory. I note that batch_size has significant influence on memory usage (and also conmpute time). A size of 100 is, however, rather sensible though. Also note that batch_size is a hyoer parameter and therefore influences training.

Cheers,
Hannes.

[halx]

Do you have online storage where I can drop those files?

[AyaIsmail284]

Does Box work? If so, what is an email I can use to send you a folder to upload to? Thanks

[halx]

I don't mind as long as I have access to your storage.

[AyaIsmail284]

I've sent a Box folder to your email. Let me know if there's any issue. Thank you so much for your time and support.

[halx]

Please keep in mind that those priors are purely experimental and you should have no expectations that they work well for you. Take care.

[ShijiZ]

Hi @halx , thanks for your reply. After a few more testings, I realized that the real issue might not be GPU memory usage. I have followed https://github.com/MolecularAI/REINVENT4/blob/main/notebooks/Reinvent_TLRL.py and used the provided https://github.com/MolecularAI/REINVENT4/blob/main/notebooks/data/tnks2.csv for testing, and everything worked well. However, when I use my own dataset (with only 100 SMILES for testing purpose, in .smi format), it can finish 1 epoch of transfer learning; and in the 2nd epoch, I got the following error message:

Traceback (most recent call last):
  File "/da/GENCHEM/reinvent4/env/bin/reinvent", line 8, in <module>
    sys.exit(main())
  File "/da/GENCHEM/reinvent4/repo/reinvent/Reinvent.py", line 292, in main
    runner(input_config, actual_device, tb_logdir, responder_config)
  File "/da/GENCHEM/reinvent4/repo/reinvent/runmodes/TL/run_transfer_learning.py", line 150, in run_transfer_learning
    runner.optimize()
  File "/da/GENCHEM/reinvent4/repo/reinvent/runmodes/TL/learning.py", line 215, in optimize
    self.report(
  File "/da/GENCHEM/reinvent4/repo/reinvent/runmodes/TL/learning.py", line 330, in report
    write_report(
  File "/da/GENCHEM/reinvent4/repo/reinvent/runmodes/TL/reports/tensorboard.py", line 44, in write_report
    similarities, kl_div = compute_similarity_from_sample(smilies, data.ref_fps)
  File "/da/GENCHEM/reinvent4/repo/reinvent/runmodes/TL/reports/tensorboard.py", line 107, in compute_similarity_from_sample
    missing = len(smilies) - len(similarities)
TypeError: object of type 'numpy.float64' has no len()

Is there a specific formatting requirement for the provided .smi file for transfer learning? For example, is this caused by illegal tokens in my file?

[halx]

I think, I need more details on this. Can you provide the TOML configuration file and possible the SMILES you are using or a suitable substitute, please?