Encoder Pretrained Weight

[Leonard-Zeng]

Hello,

Thank you for your excellent work on this project! I have a question regarding the released weights: do they include the VAE encoder? When I ran the model, it appeared that the encoder only generated random latent representations.

I also attempted to retrain the encoder while keeping the decoder frozen for some downstream tasks, but I ran into difficulties reducing the near-surface BCE loss (around 0.3 to 0.4). Could you please share any additional training details or insights that might help resolve this issue?

Thank you for your time and assistance!

[yuedajiong]

非官方回答：
感觉你没有搞懂这种VAE+Diffuse配套工作的基本概念，和如何训练。

这种架构：
第一步是用大量的3D模型，提取出point-cloud和mesh(这个不同的算法有小差别,也可能是几乎等价于mesh的表示形式）；然后开始训练VAE，points-encode-latent-decode-......-mesh，核心这里是想把decoder给后来的流程使用。
第二步是用上面3D对应的不同相机位姿的渲染的图片，训练得到对应的latent。 （训练策略可以不同，可以更端到端一点，但我这里把最本质的绕不开的步骤描述了。） 得到latent后，用前面vae里的decoder来解码到某种表示然后继续得到mesh（不同算法这里有小差别）。
最后，推理的时候，也就是作者，还有大量目前sota给show出来的infer部分。 image - diffuse -latent - decoder@vae - xxx-representantion -... - mesh

如果你非要独立的玩vae中的encoder，那么encoder要给一个有意义的point-cloud， decoder部分就更需要一个有意义的latent了。 如果你random一个latent就去decode，真正学到的有意义的latent空间只是random全空间的一个很小的子空间，几乎不是一个normal/uniform就能撞上一个有意义的mesh出来。

非官方回答！

[ech0-7]

I also met this problem. I provided the point cloud with normals as input, but it didn’t produce correct results. I also tried different xyz orders, but it still didn’t work. If the pretrain_encoder weights are fine, what' s the problem?

[Leonard-Zeng]

I also met this problem. I provided the point cloud with normals as input, but it didn’t produce correct results. I also tried different xyz orders, but it still didn’t work. If the pretrain_encoder weights are fine, what' s the problem?

I instead finetune the whole VAE in the end, but I don't think the diffusion model will work again in this case.

[ech0-7]

I also met this problem. I provided the point cloud with normals as input, but it didn’t produce correct results. I also tried different xyz orders, but it still didn’t work. If the pretrain_encoder weights are fine, what' s the problem?

I instead finetune the whole VAE in the end, but I don't think the diffusion model will work again in this case.

They released the VAE.ckpt. You can try with it.