Unity trained onnx model behaves a bit wrong when infer with onnxruntime in python.

[Norite-dev]

Intro

Hi!

I am an RL researcher and use MuJoCo in Unity for my personal projects.

My setup

Unity 6, MuJoCo 3.2.6, Python 3.10

My question

I've made and trained a simple inverted pendulum in Unity with MuJoCo, the pendulum works great.

Then I built the same digital twin and exchange observations/action with a python program that performs infer of the same .onnx model but using ONNX runtime library. And it behaves a bit differently.

Dear @Balint-H , could you suggest the correct way to infer a model trained in Unity using onnxruntime in python?

[Balint-H]

Hello,

Inverted pendulum / cartpole policies can definitely transfer in either directions. Issues I ran into when I tried it the first time was ensuring the observation spaces match identically between MuJoCo and Unity, including their order. If the MJCFs match (you can export the unity scene as MJCF), checking the observations would be my first advice.

[Balint-H]

How different are the behaviours? Just a timy amount or it fails completely?

[Norite-dev]

Significant I would say. Here it is on video, first - normal infer with mlagents. Second part - infer with onnxruntime and sending to Unity with ZeroMQ (very fast data exchange, so it is not time delays problem).

https://youtu.be/nBOq-N_ACcA

[Balint-H]

Interesting setup! Is the external inference done fully in sync with the unity env? (1 action for each ctrl step) or slightly out of sync (some delay, action reuse, etc)

[Norite-dev]

Thanks! And thanks for asking probably crucial thing here. Cause ZeroMQ server starts in a separate independent thread and works out of sync for sure. I'll check it.

[Balint-H]

Definitely, unless accounted for in domain randomisation etc., delays and timing offsets can be huge sources of performance drop in most RL control settings. Let me know if you manage to fix it! If you need inspiration, check out ML-Agents python package, where the python runtime directly steps the Unity scene. Are you aware you can wrap Unity builds into gym-style environments you can step directly with python?