关于Cross-chunk latent stitching问题

[Binn37]

def generate_avc(...):
    ......        
    # 5. Prepare latent variables
    video = self.video_processor.preprocess_video(self.video_processor, video, height=height, width=width, resize_mode=resize_mode)
    video = video.to(device=device, dtype=prompt_embeds.dtype) 
    cond_videos = video[:, :, -num_cond_frames:]
    cond_videos_latents = retrieve_latents(self.vae.encode(cond_videos), generator, sample_mode="argmax")
    cond_videos_latents = self.normalize_latents(cond_videos_latents)
    .......

Narration in paper:
However, during the generation of each continuation, obtaining the context latents requires first decoding the previously
predicted chunk via the VAE and then re-encoding the selected context frames. This repetitive VAE decode-encode
cycle introduces two major issues. First, the repeated encoding and decoding can result in information loss and error
accumulation within each chunk. Second, it significantly reduces inference efficiency.

问题是：理解多个片段视频生产，代码是实现还是使用旧的方案（就是技术报告图7 (a)），还是进行对前一个片段视频的motion帧进行编码再引入。请问这里是不是有误？请解释一下？谢谢！

[GFENGG]

def generate_avc(...):
    ......        
    # 5. Prepare latent variables
    video = self.video_processor.preprocess_video(self.video_processor, video, height=height, width=width, resize_mode=resize_mode)
    video = video.to(device=device, dtype=prompt_embeds.dtype) 
    cond_videos = video[:, :, -num_cond_frames:]
    cond_videos_latents = retrieve_latents(self.vae.encode(cond_videos), generator, sample_mode="argmax")
    cond_videos_latents = self.normalize_latents(cond_videos_latents)
    .......

Narration in paper: However, during the generation of each continuation, obtaining the context latents requires first decoding the previously predicted chunk via the VAE and then re-encoding the selected context frames. This repetitive VAE decode-encode cycle introduces two major issues. First, the repeated encoding and decoding can result in information loss and error accumulation within each chunk. Second, it significantly reduces inference efficiency.

问题是：理解多个片段视频生产，代码是实现还是使用旧的方案（就是技术报告图7 (a)），还是进行对前一个片段视频的motion帧进行编码再引入。请问这里是不是有误？请解释一下？谢谢！

因为 VAE 的解码模式是 Image latent + video latent + video latent + ...，所以需要先编码上一个clip最后的 num_cond_frames，拼到当前clip的latent之前，再解码出当前的 video frames

真正避免重编码，在 latent 上续写的部分在 code，逻辑是用上一个clip最后几个latent 作为当前clip的conditions

[Binn37]

def generate_avc(...):
    ......        
    # 5. Prepare latent variables
    video = self.video_processor.preprocess_video(self.video_processor, video, height=height, width=width, resize_mode=resize_mode)
    video = video.to(device=device, dtype=prompt_embeds.dtype) 
    cond_videos = video[:, :, -num_cond_frames:]
    cond_videos_latents = retrieve_latents(self.vae.encode(cond_videos), generator, sample_mode="argmax")
    cond_videos_latents = self.normalize_latents(cond_videos_latents)
    .......

Narration in paper: However, during the generation of each continuation, obtaining the context latents requires first decoding the previously predicted chunk via the VAE and then re-encoding the selected context frames. This repetitive VAE decode-encode cycle introduces two major issues. First, the repeated encoding and decoding can result in information loss and error accumulation within each chunk. Second, it significantly reduces inference efficiency.
问题是：理解多个片段视频生产，代码是实现还是使用旧的方案（就是技术报告图7 (a)），还是进行对前一个片段视频的motion帧进行编码再引入。请问这里是不是有误？请解释一下？谢谢！

因为 VAE 的解码模式是 Image latent + video latent + video latent + ...，所以需要先编码上一个clip最后的 num_cond_frames，拼到当前clip的latent之前，再解码出当前的 video frames

真正避免重编码，在 latent 上续写的部分在 code，逻辑是用上一个clip最后几个latent 作为当前clip的conditions

好的，理解了，谢谢！
那么是否可以直接使用上一个clip的latents最后几个时序帧作为num_cond_frames的代表，不在经过VAE的decoder+encoder，直接拼到当前要生产的clip的latents中？即是避开 # 5. Prepare latent variables 这个过程，谢谢！

[GFENGG]

def generate_avc(...):
    ......        
    # 5. Prepare latent variables
    video = self.video_processor.preprocess_video(self.video_processor, video, height=height, width=width, resize_mode=resize_mode)
    video = video.to(device=device, dtype=prompt_embeds.dtype) 
    cond_videos = video[:, :, -num_cond_frames:]
    cond_videos_latents = retrieve_latents(self.vae.encode(cond_videos), generator, sample_mode="argmax")
    cond_videos_latents = self.normalize_latents(cond_videos_latents)
    .......

Narration in paper: However, during the generation of each continuation, obtaining the context latents requires first decoding the previously predicted chunk via the VAE and then re-encoding the selected context frames. This repetitive VAE decode-encode cycle introduces two major issues. First, the repeated encoding and decoding can result in information loss and error accumulation within each chunk. Second, it significantly reduces inference efficiency.
问题是：理解多个片段视频生产，代码是实现还是使用旧的方案（就是技术报告图7 (a)），还是进行对前一个片段视频的motion帧进行编码再引入。请问这里是不是有误？请解释一下？谢谢！

因为 VAE 的解码模式是 Image latent + video latent + video latent + ...，所以需要先编码上一个clip最后的 num_cond_frames，拼到当前clip的latent之前，再解码出当前的 video frames
真正避免重编码，在 latent 上续写的部分在 code，逻辑是用上一个clip最后几个latent 作为当前clip的conditions

好的，理解了，谢谢！ 那么是否可以直接使用上一个clip的latents最后几个时序帧作为num_cond_frames的代表，不在经过VAE的decoder+encoder，直接拼到当前要生产的clip的latents中？即是避开 # 5. Prepare latent variables 这个过程，谢谢！

正如之前提到的 “VAE 的解码模式是 Image latent + video latent + video latent + ...”，所以如果直接拼接上一个clip的末尾latents，会导致解码出的视频出现 arfficts，影响效果；所以最合适的方式是：# 5. Prepare latent variables