The issues with the evaluation script.

[llbll]

1. When using Docker for evaluation, the WARNING shown in the above image appeared: 2025-12-24 05:48:49,034 - compute_metrics_parallel - Error evaluating constraint is_visible. Is this normal?

2. The evaluation metrics we finally obtained have a large gap compared to those in the paper（

）. Is there something wrong with our configuration? The evaluation results are as follows:
{
"language_adherence_success": 15.171428571428573,
"global_constraint_accuracy": 45.08521418701059,
"complete_placement_success": 8.542857142857143,
"subgroup_metrics": {
"spatial": 0.5661352865144335,
"rotational": 0.17298735861610112,
"visibility": 0.0,
"physical": 0.5731428571428572
},
"per_constraint_metrics": {
"on": {
"accuracy": 0.9378238341968912,
"satisfied": 1267,
"total": 1351
},
"below": {
"accuracy": 0.2553191489361702,
"satisfied": 48,
"total": 188
},
"is_visible": {
"accuracy": 0.0,
"satisfied": 0,
"total": 996
},
"near": {
"accuracy": 0.49556650246305417,
"satisfied": 1006,
"total": 2030
},
"above": {
"accuracy": 0.16822429906542055,
"satisfied": 18,
"total": 107
},
"not_visible": {
"accuracy": 0.0,
"satisfied": 0,
"total": 214
},
"plausible": {
"accuracy": 0.5731428571428572,
"satisfied": 2006,
"total": 3500
},
"facing": {
"accuracy": 0.17298735861610112,
"satisfied": 260,
"total": 1503
},
"adjacent": {
"accuracy": 0.4088397790055249,
"satisfied": 74,
"total": 181
},
"between": {
"accuracy": 0.27388535031847133,
"satisfied": 215,
"total": 785
}
},
"metrics_by_constraint_count": {
"2": {
"total_placements": 1481,
"at_least_one_correct": 944,
"all_correct": 203
},
"1": {
"total_placements": 878,
"at_least_one_correct": 258,
"all_correct": 258
},
"3": {
"total_placements": 1049,
"at_least_one_correct": 848,
"all_correct": 70
},
"4": {
"total_placements": 92,
"at_least_one_correct": 89,
"all_correct": 0
}
}
}

[Samir55]

Hi @llbll,

Thank you for your interest in PlaceIt3D and for sharing the results.

1. The warning Error evaluating constraint is_visible is not expected. It is related to the evaluation of the visibility constraint, which requires offscreen rendering. Our evaluation code uses pyrender.OffscreenRenderer for this step.

2. Since the visibility metrics are reported as 0.0, the visibility evaluation did not run affecting the global metrics. The other sub_group metrics are similar. Please try running the following offscreen rendering example inside your Docker container to make sure it is working: https://pyrender.readthedocs.io/en/latest/examples/offscreen.html

[llbll]

Hi @llbll,

Thank you for your interest in PlaceIt3D and for sharing the results.

1. The warning Error evaluating constraint is_visible is not expected. It is related to the evaluation of the visibility constraint, which requires offscreen rendering. Our evaluation code uses pyrender.OffscreenRenderer for this step.
2. Since the visibility metrics are reported as 0.0, the visibility evaluation did not run affecting the global metrics. The other sub_group metrics are similar. Please try running the following offscreen rendering example inside your Docker container to make sure it is working: https://pyrender.readthedocs.io/en/latest/examples/offscreen.html

Hello, I installed pyrender using pip install pyrender and set PYOPENGL_PLATFORM=egl. I also initialized the renderer with:
r = pyrender.OffscreenRenderer(viewport_width=640, viewport_height=480, point_size=1.0),
and everything runs without error. However, when I subsequently run the evaluation script run_benchmark.bin, I still get the error:
"Error evaluating constraint is_visible".

Could this be caused by an inconsistency between my current environment and the environment used when run_benchmark.bin is executed? For example, does run_benchmark.bin load functions from a local pyrender folder instead of the one installed via pip?

Additionally, at the very beginning of execution, there was another issue:
/tmp/nianticspatial/placeit3d/1.0/dateutil/zoneinfo/__init__.py:26: UserWarning: I/O error(2): No such file or directory.
I didn’t notice it at first—could this missing file be critical, or what should I do to resolve it?