[Question] Generate failed examples as successful data sets

[cyclexfusion]

I’ve added teleoperation equipment to my Isaac lab, controlling a Franka robotic arm to perform 10 successful stacking demonstrations, then using automatic labeling. However, when I use this dataset to generate mini-batches of examples, I’m finding that it’s misclassifying failed examples as successful ones. This happens even when all the blocks are scattered across the table. I don’t understand why this happens, and could someone tell me how to troubleshoot the issue? Thanks!

cmd：
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py
–device cuda --num_envs 10 --generation_num_trials 10
–input_file ./datasets/annotated_dataset.hdf5 --output_file ./datasets/generated_dataset_small.hdf5

environment：
Isaac sim 4.5
isaac lab 2.1
ubuntu 22.04

[RandomOakForest]

Thank you for posting this. May it be that you are not using enough data to train? This is a great post for our Discussions section. I'll move it there for our team and others to follow up.

[peterd-NV]

Hi, thanks for your question. To better help understand what's causing the issue could you please specify if the misclassification is occurring in one of the following ways?

1. You run the cmd to generate a dataset. While generating the data, you are seeing that failed demonstrations (cubes are not stacked at the end of the demo) are being classified as successful.
2. You run the cmd to generate a dataset. After generation is finished, you are replaying the demos in generated_dataset_small.hdf5 and seeing that some demonstrations are not successful.

[cyclexfusion]

My situation is 2, because 25 environments are generated in parallel, I cannot clearly observe whether the generation process fails.

[peterd-NV]

Thanks for clarifying. Unfortunately, full determinism of env.step() when using parallel envs currently cannot be guaranteed due to the sharing of the underlying simulation PhysX buffers. This can result in trajectories that were successful during original data generation to exhibit slightly different behaviour when visualized with the replay script (See the note here). Determinism for single environment recording and play back via the annotation script is achieved by performing a full sim buffer flush after each sim reset, however this flush cannot be done when environments are running in parallel and reset asynchronously.

While this non determinism can be observed using the replay.py visualization tool, the actual data saved to the HDF5 data is accurate and should only contain state/observations of successful demos.

[cyclexfusion]

Batch generation is run in a parallel environment. If the complete determinism of env.step() cannot be guaranteed, the accuracy of the batch-generated dataset will be unreliable？