Tritonserver serving Tensorrt model with cuda graph results in weird unconsistent outputs.

[WingEdge777]

Hi TensorRT team.

I'm not sure which component should be blamed for this problem exactly. So I filed a ticket here as well as triton-inference-server/server#8550.

In conclusion, when we send requests sequentially to TritonServer, in the AAAAABBBBBAAAABBBB pattern. In every A(B)‘s requesting round, the first few A(B) requests probably return the last period round B(A)'s results, which is absurdly wrong for the current input.

Details can be found in the issue I posted above.

Thanks for your product. Looking forward to your response.

[WingEdge777]

closing this since the root cause is tritonserver side.