Is there any plan to support new pyTorch(==1.5) model archive format?

[rtrobin]

With pyTorch release v1.5, it comes a new feature TorchServe, whick provides model serve like trtis. Also it introduces a new model format model archive, which is used in TorchServe.

Currently, we export pyTorch model to torchscript module for trtis deployment. Sometime it needs code adjustment to fulfill torchscript need. I haven't done much investment in new archive format, but it says it supports torch model state_dict eager mode and torchscript model. It may leads to an easy way to archive the pyTorch model.

Have you guys read anything about TorchServe? Do you have a plan to support new pyTorch model format? Thanks.

[deadeyegoodwin]

From the documentation it is unclear if the archive format is meant to be a general purpose format that any inference server can use or something specific for torchserver. Torchscript files (*.pt) currently have insufficient meta-data for an inference server to completely understand inputs and outputs of the model (this is why --strict-model-config=false works so poorly for torchscript models relative to other models). It would be nice if that was fixed (independent of the archive format), but again in the documentation we don't see any indication of that.

But if the archive format is helpful and meant to be general purpose then we will likely support it in the future.

[rtrobin]

@deadeyegoodwin Thanks for the sharing. It would be great if trtis could support pyTorch archive format, if this new format actually could solve current flaw of pyTorch in deployment.

[mvpel]

On a related note, I'm interested to know if there's plans to support the Keras HDF5 model archive format used by its save() and load_model() functions. Currently we've been exporting the models to TF SavedModel format to bring them under Triton, but direct support for h5 files would be handy.

[dyastremsky]

Thank you for filing this ticket. We have opted not to implement this feature, as it does not seem to be general purpose and we have not seen it requested often recently.

[kimminw00]

It is necessary to add support for the new PyTorch model archive format to replace deprecated TorchScript, which is no longer in active development.