Skip to content

Problem with the triton package when calling boltz predict #433

@xavgit

Description

@xavgit

Hi,
I've installed the latest version of Boltz from the sources.
I'm experiencing an error related to the triton package:

Exception: Failed to import Triton-based component: triangle_multiplicative_update:

Not Supported
Please make sure to install triton==3.3.0. Other versions may not work!
Predicting DataLoader 0: 0%| | 0/1 [00:01<?, ?it/s]

But inspecting the version of the triton package I receive the following:

(boltz2-env) xxxx@xxxx-Predator-PHN16-71:~/boltz_2_affinity_example$ pip3 list | grep triton
triton 3.3.0

Any suggestion?

Also for the warning:

"You are using a CUDA device ('NVIDIA GeForce RTX 4070 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance."

How it is possible to make the suggested settings when calling boltz?

Thanks.

Saverio

PS:
The following is the list of the used commands:

(pyenv) xxxx@xxxx-Predator-PHN16-71:$ source boltz2-env/bin/activate
(boltz2-env) xxxx@xxxx-Predator-PHN16-71:
$ boltz --help
Usage: boltz [OPTIONS] COMMAND [ARGS]...

Boltz.

Options:
--help Show this message and exit.

Commands:
predict Run predictions with Boltz.

(boltz2-env) xxxx@xxxx-Predator-PHN16-71:$ cd boltz_2_affinity_example/
(boltz2-env) xxxx@xxxx-Predator-PHN16-71:
/boltz_2_affinity_example$ boltz predict /home/xxxx/sources/boltz-2.1.1/examples/affinity.yaml --use_msa_server
Checking input data.
All inputs are already processed.
Processing 0 inputs with 0 threads.
0it [00:00, ?it/s]
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:76: Starting from v1.9.0, tensorboardX has been removed as a dependency of the pytorch_lightning package, due to potential conflicts with other packages in the ML ecosystem. For this reason, logger=True will use CSVLogger as the default logger, unless the tensorboard or tensorboardX packages are found. Please pip install lightning[extra] or one of them to enable TensorBoard support by default
Running structure prediction for 1 input.
/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/utilities/migration/utils.py:56: The loaded checkpoint was produced with Lightning v2.5.0.post0, which is newer than your current Lightning version: v2.5.0
You are using a CUDA device ('NVIDIA GeForce RTX 4070 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting DataLoader 0: 0%| | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/xxxx/boltz2-env/bin/boltz", line 8, in
sys.exit(cli())
^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/main.py", line 1190, in predict
trainer.predict(
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 859, in predict
return call._call_and_handle_interrupt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/trainer/call.py", line 47, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 898, in _predict_impl
results = self._run(model, ckpt_path=ckpt_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in _run
results = self._run_stage()
^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 1021, in _run_stage
return self.predict_loop.run()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/loops/utilities.py", line 179, in _decorator
return loop_run(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/loops/prediction_loop.py", line 125, in run
self._predict_step(batch, batch_idx, dataloader_idx, dataloader_iter)
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/loops/prediction_loop.py", line 255, in _predict_step
predictions = call._call_strategy_hook(trainer, "predict_step", *step_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/trainer/call.py", line 323, in _call_strategy_hook
output = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/pytorch_lightning/strategies/strategy.py", line 438, in predict_step
return self.lightning_module.predict_step(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/model/models/boltz2.py", line 1055, in predict_step
out = self(
^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/model/models/boltz2.py", line 464, in forward
z = z + template_module(
^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/model/modules/trunkv2.py", line 498, in forward
v = v + self.pairformer(v, pair_mask, use_kernels=use_kernels)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/model/layers/pairformer.py", line 331, in forward
z = layer(
^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/model/layers/pairformer.py", line 245, in forward
z = z + dropout * self.tri_mul_out(
^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/model/layers/triangular_mult.py", line 92, in forward
return kernel_triangular_mult(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/boltz/model/layers/triangular_mult.py", line 23, in kernel_triangular_mult
return triangle_multiplicative_update(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/cuequivariance_torch/primitives/triangle.py", line 223, in triangle_multiplicative_update
return f(
^^
File "/home/xxxx/boltz2-env/lib/python3.12/site-packages/cuequivariance_ops_torch/init.py", line 68, in triangle_multiplicative_update
raise Exception(f"Failed to import Triton-based component: triangle_multiplicative_update:\n{IMPORT_EXCEPTION}\n"
Exception: Failed to import Triton-based component: triangle_multiplicative_update:
Not Supported
Please make sure to install triton==3.3.0. Other versions may not work!
Predicting DataLoader 0: 0%| | 0/1 [00:01<?, ?it/s]
(boltz2-env) xxxx@xxxx-Predator-PHN16-71:~/boltz_2_affinity_example$ pip3 list | grep triton
triton 3.3.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions