Torchscript Model can't have bfloat16 inputs / outputs in 24.09

[MatthieuToulemont]

I am using the triton container from 24.09 and want to serve torscript models with BFLOAT16 inputs but I am getting the following error:

failed to load 'UnflattenSequence' version 1: Internal: unsupported datatype TYPE_BF16 for input 'latent_as_sequence_input__0' for model 'UnflattenSequence'"

Given that torchscript supports bfloat16, shouldn't this be supported as well ?

[rmccorm4]

Hi @MatthieuToulemont,

We have limited backend support in our framework matrix today: https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#datatypes

I think it may be relatively simple to add support if the libtorch C++ library supports it, I think you would just need to map between Triton's TYPE_BF16 and the corresponding libtorch type in this file: https://github.com/triton-inference-server/pytorch_backend/blob/main/src/libtorch_utils.cc#L32.

Would you be interested in trying that out and contributing support?

[phuhung273]

Hi @rmccorm4, I created a PR triton-inference-server/pytorch_backend#154 as your instruction. Can you help advise if the change looks good ? Thanks a lot.

[whoisj]

Hi @rmccorm4, I created a PR triton-inference-server/pytorch_backend#154 as your instruction. Can you help advise if the change looks good ? Thanks a lot.

Apologies for the delay. I've asked a couple of people more familiar with the codebase to review your change.