Can you load onnx files with external dependency weight files? (DirectML problem)

[elephantpanda]

I downloaded an Onnx which had external weight and bias files.

Can c# onnxruntime load them if I just load the onnx file like this:

InferenceSession("unet.onnx")

I tried it but I'm getting a

Exception(5) tid(24b4) 80070057 The parameter is incorrect.

BTW the one I'm trying to load is here:
https://huggingface.co/bes-dev/stable-diffusion-v1-4-onnx/tree/main
unet.tar.gz
(inside this tar file is an unet.onnx and lots of weights and bias files).

If it is not possible, is there a tool to combine all these files into a single onnx?

[xadupre]

onnx file are limited to 2Gb. It is a limitation of protobuf. Which version of onnxruntime are you using?

[elephantpanda]

onnx file are limited to 2Gb. It is a limitation of protobuf. Which version of onnxruntime are you using?

Hi, I can confirm that it loaded the onnx successfully in CPU mode. But it failed in DirectML mode. (Even though other onnx load in DirectML mode. So I don't know why this one doesn't). It is failing on the InferenceSession("unet.onnx") function.

The complete error message is:

Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:RuntimeException] Exception during initialization: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\GraphKernelHelper.cpp(78)\onnxruntime.dll!00007FFA23FD6CA3: (caller: 00007FFA23FD7445) Exception(3) tid(46b0) 80070057 The parameter is incorrect.

  at Microsoft.ML.OnnxRuntime.NativeApiStatus.VerifySuccess (System.IntPtr nativeStatus) [0x0002c] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at Microsoft.ML.OnnxRuntime.InferenceSession.Init (System.String modelPath, Microsoft.ML.OnnxRuntime.SessionOptions options, Microsoft.ML.OnnxRuntime.PrePackedWeightsContainer prepackedWeightsContainer) [0x0002f] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at Microsoft.ML.OnnxRuntime.InferenceSession..ctor (System.String modelPath, Microsoft.ML.OnnxRuntime.SessionOptions options) [0x0002c] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at ONNXRuntimeTest.CreateNamedSession (System.String path) [0x0004a] in C:\Users\Shadow\Dropbox\My Stable Diffusion\Assets\ONNXRuntimeTest.cs:84 
UnityEngine.Debug:Log (object)
ONNXRuntimeTest:CreateNamedSession (string) (at Assets/ONNXRuntimeTest.cs:91)
ONNXRuntimeTest:Unet () (at Assets/ONNXRuntimeTest.cs:124)
ONNXRuntimeTest:GO () (at Assets/ONNXRuntimeTest.cs:100)
UnityEngine.EventSystems.EventSystem:Update () (at Library/PackageCache/[email protected]/Runtime/EventSystem/EventSystem.cs:377)

My nuget packages are:
DirectML.1.10.0
OnnxRuntime.DirectML.1.13.1
OnnxRuntime.Managed.1.13.1

using netstandard2.0

I am running it on a Shadow PC with Nvidia Quadro P5000. I am using the Unity 3D engine.

Unfortunately although the unet.onnx loads in CPU mode. When I run the inference I get the error:

Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:RuntimeException] Non-zero status code returned while running Mul node. Name:'Mul_187' Status Message: D:\a\_work\1\s\onnxruntime\core/providers/cpu/math/element_wise_ops.h:503 onnxruntime::BroadcastIterator::Init axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 2 by 160

  at Microsoft.ML.OnnxRuntime.NativeApiStatus.VerifySuccess (System.IntPtr nativeStatus) [0x0002c] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at Microsoft.ML.OnnxRuntime.InferenceSession.RunImpl (Microsoft.ML.OnnxRuntime.RunOptions options, System.IntPtr[] inputNames, System.IntPtr[] inputValues, System.IntPtr[] outputNames, Microsoft.ML.OnnxRuntime.DisposableList`1[T] cleanupList) [0x0004c] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at Microsoft.ML.OnnxRuntime.InferenceSession.Run (System.Collections.Generic.IReadOnlyCollection`1[T] inputs, System.Collections.Generic.IReadOnlyCollection`1[T] outputNames, Microsoft.ML.OnnxRuntime.RunOptions options) [0x00061] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at Microsoft.ML.OnnxRuntime.InferenceSession.Run (System.Collections.Generic.IReadOnlyCollection`1[T] inputs, System.Collections.Generic.IReadOnlyCollection`1[T] outputNames) [0x00001] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at Microsoft.ML.OnnxRuntime.InferenceSession.Run (System.Collections.Generic.IReadOnlyCollection`1[T] inputs) [0x00025] in <1e02e1e264404d07be919d65b1f5ddc7>:0 
  at ONNXRuntimeTest.RunIt2 (System.Collections.Generic.List`1[T] inputs) [0x0000a] in C:\Users\Shadow\Dropbox\My Stable Diffusion\Assets\ONNXRuntimeTest.cs:476 
UnityEngine.Debug:Log (object)
ONNXRuntimeTest:RunIt2 (System.Collections.Generic.List`1<Microsoft.ML.OnnxRuntime.NamedOnnxValue>) (at Assets/ONNXRuntimeTest.cs:480)
ONNXRuntimeTest:Unet () (at Assets/ONNXRuntimeTest.cs:140)
ONNXRuntimeTest:GO () (at Assets/ONNXRuntimeTest.cs:105)
UnityEngine.EventSystems.EventSystem:Update () (at Library/PackageCache/[email protected]/Runtime/EventSystem/EventSystem.cs:377)

I am defining a scalar like this: Not sure if correct:
var t = new DenseTensor(1);
t[0] = 0.5f;

The inputs are:
"latent_model_input", (2, 4, 64, 64)
"encoder_hidden_states", (2, 77, 768)

[elephantpanda]

I updated this into an "issue".