Errors when training with lmdb

[ZhangCYG]

Hello, I use your preprocess/cocoify_obman.py and generate lmdb file, but when I train the pose_kpt task, it seems there's some error of imdb file. Speficically, It seems that some images are written to NoneType in lmdb file. Could you please help me solve this problem, since I try to find some bugs but failed?

[ZhangCYG]

But when I try to train without lmdb, it seems nothing error.

[ZhangCYG]

So I also wonder how or whether using lmdb influences training process? Thanks a lot!

[zerchen]

Hi,

Thanks for your issues. I will look into this next week.
In my server, data loading is a bottleneck, so I use lmdb to accelerate the process. If your server is good enough for data loading, I think it is fine to train without lmdb.

[ZhangCYG]

Thanks for helping, I think there may remain some bugs in lmdb reading or writing.

[wyl2077]

Hi,

I experienced a same issue during the training of hand pose estimation model. I discovered that the cause is linked to a non-existent index in the RGB LMDB. Specifically, the discrepancy arises because the RGB LMDB is generated using train_87k.json, whereas the pose training configuration is set to use 138k. This mismatch is the source of the problem. You can find the relevant code in gSDF/preprocess/cocoify_obman.py at line 178, and in gSDF/playground/pose_kpt/config.py at line 32. I hope this information proves helpful in resolving the issue.

[QvQKing]

Hi,

I experienced a same issue during the training of hand pose estimation model. I discovered that the cause is linked to a non-existent index in the RGB LMDB. Specifically, the discrepancy arises because the RGB LMDB is generated using train_87k.json, whereas the pose training configuration is set to use 138k. This mismatch is the source of the problem. You can find the relevant code in gSDF/preprocess/cocoify_obman.py at line 178, and in gSDF/playground/pose_kpt/config.py at line 32. I hope this information proves helpful in resolving the issue.

Hello, Have you solved this problem?