Online submission fails

[alanxuefei]

Description

In local PC, the following two cmd are ran with identical docker-compose.yaml

docker compose -f docker/docker-compose.yaml build model
docker compose -f docker/docker-compose.yaml up

There are two issues:
1, The progress is very slow without enabling Gazebo and Rviz GUI.
2, The local PC produces the following output, while the online submission fails and does not appear on the leaderboard.
If model validation succeeds, should it should succeed and appear on the leaderboard?

eval-1   | [aic_engine-5] [INFO] [1775445054.247755059] [aic_engine]: ╔════════════════════════════════════════╗
eval-1   | [aic_engine-5] [INFO] [1775445054.247763255] [aic_engine]: ║   ✓ All Trials Processed!              ║
eval-1   | [aic_engine-5] [INFO] [1775445054.247769229] [aic_engine]: ║   Successful: 2  Failed: 1             ║
eval-1   | [aic_engine-5] [INFO] [1775445054.247777472] [aic_engine]: ║   Total Score: 74.357                     ║
eval-1   | [aic_engine-5] [INFO] [1775445054.247785274] [aic_engine]: ╚════════════════════════════════════════╝
eval-1   | [aic_engine-5] [INFO] [1775445054.247791458] [aic_engine]:  
eval-1   | [aic_engine-5] [INFO] [1775445054.248284289] [aic_engine]: ╔════════════════════════════════════════╗
eval-1   | [aic_engine-5] [INFO] [1775445054.248294950] [aic_engine]: ║        Complete Scoring Results        ║
eval-1   | [aic_engine-5] [INFO] [1775445054.248301354] [aic_engine]: ╚════════════════════════════════════════╝
eval-1   | [aic_engine-5] [INFO] [1775445054.248309004] [aic_engine]: total: 74.356843602937644
eval-1   | [aic_engine-5] [INFO] [1775445054.248315676] [aic_engine]: trial_1:
eval-1   | [aic_engine-5] [INFO] [1775445054.248321581] [aic_engine]:   tier_1:
eval-1   | [aic_engine-5] [INFO] [1775445054.248327193] [aic_engine]:     score: 1
eval-1   | [aic_engine-5] [INFO] [1775445054.248332991] [aic_engine]:     message: Model validation succeeded.
eval-1   | [aic_engine-5] [INFO] [1775445054.248338730] [aic_engine]:   tier_2:
eval-1   | [aic_engine-5] [INFO] [1775445054.248344180] [aic_engine]:     score: 16.596355082134728
eval-1   | [aic_engine-5] [INFO] [1775445054.248349800] [aic_engine]:     message: Scoring succeeded.
eval-1   | [aic_engine-5] [INFO] [1775445054.248355074] [aic_engine]:     categories:
eval-1   | [aic_engine-5] [INFO] [1775445054.248360539] [aic_engine]:       contacts:
eval-1   | [aic_engine-5] [INFO] [1775445054.248365959] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248371371] [aic_engine]:         message: No contact detected.
eval-1   | [aic_engine-5] [INFO] [1775445054.248377176] [aic_engine]:       duration:
eval-1   | [aic_engine-5] [INFO] [1775445054.248382729] [aic_engine]:         score: 5.6718545454545461
eval-1   | [aic_engine-5] [INFO] [1775445054.248388287] [aic_engine]:         message: "Task duration: 34.00 seconds."
eval-1   | [aic_engine-5] [INFO] [1775445054.248393978] [aic_engine]:       insertion force:
eval-1   | [aic_engine-5] [INFO] [1775445054.248399263] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248404768] [aic_engine]:         message: No excessive force detected
eval-1   | [aic_engine-5] [INFO] [1775445054.248410723] [aic_engine]:       trajectory efficiency:
eval-1   | [aic_engine-5] [INFO] [1775445054.248416278] [aic_engine]:         score: 6
eval-1   | [aic_engine-5] [INFO] [1775445054.248422079] [aic_engine]:         message: "Total end-effector path length: 0.00 m, initial plug-port distance: 0.13 m"
eval-1   | [aic_engine-5] [INFO] [1775445054.248428567] [aic_engine]:       trajectory smoothness:
eval-1   | [aic_engine-5] [INFO] [1775445054.248434049] [aic_engine]:         score: 4.9245005366801831
eval-1   | [aic_engine-5] [INFO] [1775445054.248439443] [aic_engine]:         message: "Average linear jerk magnitude of the end effector: 8.96 m/s^3"
eval-1   | [aic_engine-5] [INFO] [1775445054.248445506] [aic_engine]:   tier_3:
eval-1   | [aic_engine-5] [INFO] [1775445054.248451304] [aic_engine]:     score: 24.909813450499676
eval-1   | [aic_engine-5] [INFO] [1775445054.248457014] [aic_engine]:     message: "No insertion detected. Final plug port distance: 0.05m."
eval-1   | [aic_engine-5] [INFO] [1775445054.248462486] [aic_engine]: trial_2:
eval-1   | [aic_engine-5] [INFO] [1775445054.248468295] [aic_engine]:   tier_1:
eval-1   | [aic_engine-5] [INFO] [1775445054.248474010] [aic_engine]:     score: 1
eval-1   | [aic_engine-5] [INFO] [1775445054.248479424] [aic_engine]:     message: Model validation succeeded.
eval-1   | [aic_engine-5] [INFO] [1775445054.248484682] [aic_engine]:   tier_2:
eval-1   | [aic_engine-5] [INFO] [1775445054.248490126] [aic_engine]:     score: 16.866134849521274
eval-1   | [aic_engine-5] [INFO] [1775445054.248495652] [aic_engine]:     message: Scoring succeeded.
eval-1   | [aic_engine-5] [INFO] [1775445054.248501083] [aic_engine]:     categories:
eval-1   | [aic_engine-5] [INFO] [1775445054.248520621] [aic_engine]:       contacts:
eval-1   | [aic_engine-5] [INFO] [1775445054.248527519] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248533307] [aic_engine]:         message: No contact detected.
eval-1   | [aic_engine-5] [INFO] [1775445054.248539061] [aic_engine]:       duration:
eval-1   | [aic_engine-5] [INFO] [1775445054.248544597] [aic_engine]:         score: 5.6727272727272728
eval-1   | [aic_engine-5] [INFO] [1775445054.248550145] [aic_engine]:         message: "Task duration: 34.00 seconds."
eval-1   | [aic_engine-5] [INFO] [1775445054.248555731] [aic_engine]:       insertion force:
eval-1   | [aic_engine-5] [INFO] [1775445054.248561462] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248566868] [aic_engine]:         message: No excessive force detected
eval-1   | [aic_engine-5] [INFO] [1775445054.248572380] [aic_engine]:       trajectory efficiency:
eval-1   | [aic_engine-5] [INFO] [1775445054.248578003] [aic_engine]:         score: 6
eval-1   | [aic_engine-5] [INFO] [1775445054.248583753] [aic_engine]:         message: "Total end-effector path length: 0.00 m, initial plug-port distance: 0.14 m"
eval-1   | [aic_engine-5] [INFO] [1775445054.248590194] [aic_engine]:       trajectory smoothness:
eval-1   | [aic_engine-5] [INFO] [1775445054.248595825] [aic_engine]:         score: 5.1934075767940033
eval-1   | [aic_engine-5] [INFO] [1775445054.248601339] [aic_engine]:         message: "Average linear jerk magnitude of the end effector: 6.72 m/s^3"
eval-1   | [aic_engine-5] [INFO] [1775445054.248607459] [aic_engine]:   tier_3:
eval-1   | [aic_engine-5] [INFO] [1775445054.248612764] [aic_engine]:     score: 24.984540220781966
eval-1   | [aic_engine-5] [INFO] [1775445054.248618253] [aic_engine]:     message: "No insertion detected. Final plug port distance: 0.05m."
eval-1   | [aic_engine-5] [INFO] [1775445054.248623844] [aic_engine]: trial_3:
eval-1   | [aic_engine-5] [INFO] [1775445054.248629470] [aic_engine]:   tier_1:
eval-1   | [aic_engine-5] [INFO] [1775445054.248635052] [aic_engine]:     score: 1
eval-1   | [aic_engine-5] [INFO] [1775445054.248640517] [aic_engine]:     message: Model validation succeeded.
eval-1   | [aic_engine-5] [INFO] [1775445054.248646175] [aic_engine]:   tier_2:
eval-1   | [aic_engine-5] [INFO] [1775445054.248651696] [aic_engine]:     score: -12
eval-1   | [aic_engine-5] [INFO] [1775445054.248657188] [aic_engine]:     message: Scoring succeeded.
eval-1   | [aic_engine-5] [INFO] [1775445054.248662917] [aic_engine]:     categories:
eval-1   | [aic_engine-5] [INFO] [1775445054.248668455] [aic_engine]:       contacts:
eval-1   | [aic_engine-5] [INFO] [1775445054.248673987] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248679546] [aic_engine]:         message: No contact detected.
eval-1   | [aic_engine-5] [INFO] [1775445054.248685078] [aic_engine]:       duration:
eval-1   | [aic_engine-5] [INFO] [1775445054.248690578] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248711313] [aic_engine]:         message: Task not completed.
eval-1   | [aic_engine-5] [INFO] [1775445054.248717331] [aic_engine]:       insertion force:
eval-1   | [aic_engine-5] [INFO] [1775445054.248728064] [aic_engine]:         score: -12
eval-1   | [aic_engine-5] [INFO] [1775445054.248734784] [aic_engine]:         message: "Insertion force above 20.00 N, detected for a time of 1.73 seconds. Max detected force: 49.46N. This is above the threshold of 1.00 seconds. Penalty applied."
eval-1   | [aic_engine-5] [INFO] [1775445054.248741843] [aic_engine]:       trajectory efficiency:
eval-1   | [aic_engine-5] [INFO] [1775445054.248747490] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248753544] [aic_engine]:         message: Task not completed.
eval-1   | [aic_engine-5] [INFO] [1775445054.248759102] [aic_engine]:       trajectory smoothness:
eval-1   | [aic_engine-5] [INFO] [1775445054.248764997] [aic_engine]:         score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248770373] [aic_engine]:         message: Task not completed.
eval-1   | [aic_engine-5] [INFO] [1775445054.248776091] [aic_engine]:   tier_3:
eval-1   | [aic_engine-5] [INFO] [1775445054.248781694] [aic_engine]:     score: 0
eval-1   | [aic_engine-5] [INFO] [1775445054.248787685] [aic_engine]:     message: Task not completed.

Setup method

Docker + Distrobox + Pixi (recommended)

OS

Ubuntu 24.04

System specs

CPU: Intel i9
Memory: 32G
GPU: 4090 24G
NVIDIA-SMI 580.126.09
Driver Version: 580.126.09
CUDA Version: 13.0

Steps to reproduce

docker compose -f docker/docker-compose.yaml build model
docker compose -f docker/docker-compose.yaml up

Relevant log output

Additional context

No response

[iche033]

1, The progress is very slow without enabling Gazebo and Rviz GUI.

Can you elaborate a little bit? What progress is slow?

2, The local PC produces the following output, while the online submission fails and does not appear on the leaderboard.
If model validation succeeds, should it should succeed and appear on the leaderboard?

Yes if model validation succeeds, it should appear on the leaderboard. You can try submitting an image built with one of the example policy and see if submission is successful, in which case you should see 3 points on the leaderboard.

[alanxuefei]

1, During the descent, the gripper moves down at about one second per step. In Gazebo and RViz2, the speed is normal.

[alanxuefei]

A video is as below

The following fix solved the issue.
Forwarding the host graphics devices into the eval container enabled GPU-accelerated Ogre2 rendering and fixed the slowdown.
Will it affect submission?

# Merge on top of docker-compose.yaml to keep Gazebo headless while forwarding
# the same local graphics stack used by docker-compose.gui.yaml.
#
# This is helpful on developer machines where the default headless eval path is
# noticeably slower because Ogre2 camera rendering falls back to a slower
# graphics path.
#
# Headless local GPU rendering:
#   docker compose \
#     -f docker/docker-compose.yaml \
#     -f docker/docker-compose.headless-local.yaml \
#     up

services:
  eval:
    # Keep the same eval args as the default compose file, but preserve the
    # local graphics passthrough used by the GUI overlay.
    command: gazebo_gui:=false launch_rviz:=false ground_truth:=false start_aic_engine:=true shutdown_on_aic_engine_exit:=true aic_engine_config_file:=/ws_aic/install/share/aic_engine/config/sample_config.yaml
    environment:
      DISPLAY: ${DISPLAY}
      XAUTHORITY: /tmp/.docker.xauth
      XDG_RUNTIME_DIR: /tmp/runtime-root
      QT_X11_NO_MITSHM: ${QT_X11_NO_MITSHM:-1}
      NVIDIA_DRIVER_CAPABILITIES: ${NVIDIA_DRIVER_CAPABILITIES:-all}
    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix:rw
      - ${XAUTHORITY}:/tmp/.docker.xauth:ro
      - /dev/dri:/dev/dri:rw

[alanxuefei]

The submission still fails.

Is it GPU issue? Is it possible to see log in the evaluation server?

Feature	NVIDIA L4	RTX 4090
VRAM	24 GB	24 GB
Type	Server GPU	Desktop GPU
Speed	Slower	Faster
Power	~72W	~450W

[iche033]

Resolving this submission issue offline. Please check the recommended action and see issue persists.

[alanxuefei]

Thanks for the help. I believe that the issue is solved.