On the Generality and Training-Free Nature of the Feature Pipeline

[67L1]

Hi authors,
Thank you for your excellent work on MovieChat. It's a very clever and effective approach for long video understanding. I'm studying your method and have a few technical questions about the feature processing pipeline. I would be grateful for your clarification.

1. Generality of the Memory Mechanism
Conceptually, your memory consolidation process seems highly adaptable. I was wondering if it could be applied to other MLLM architectures, especially those that use a simpler projector (e.g., a single MLP layer) instead of a Q-Former?

2. Q-Former's Handling of "Synthetic" Features
Your method merges features from several frames (from EVA-CLIP) into a "synthetic" feature before feeding it to the BLIP-2 Q-Former. Since the Q-Former was pre-trained on original, non-merged features from a vision encoder, how does it generalize to correctly understand these "mixed" features created by your merging process?

3. The Training-Free Projector/Connection to the LLM
My main question is about the final connection to the LLM. In standard MLLMs, the projector is a critical component that is explicitly trained to map features from a specific vision encoder into the LLM's semantic space.
Usually, if the input visual features are altered, one would need to retrain the projector and fine-tune the LLM to align the new feature distribution. Your framework impressively bypasses this entire training step.
Could you elaborate on what makes this possible? After the features go through the complex chain (EVA-CLIP ViT -> Merging -> BLIP-2 Q-Former), what is the key insight that allows Vicuna to understand them without a dedicated training phase for the connection? Is it that the BLIP-2 Q-Former's output is a "universal" representation that is already well-aligned with the input space of most LLMs?
Thank you for your time and for sharing this fantastic research.

[Espere-1119-Song]

Thank you very much for your thoughtful and detailed questions. We’re glad to hear that you’ve been studying MovieChat in depth. Please find our clarifications below:

1. Both MovieChat and MovieChat+ are training-free frameworks, and we have successfully migrated our memory consolidation mechanism from VideoLLaMA to LLaVA-OneVision and InternVL, consistently observing performance improvements.

2. The merging process in our framework does not change the underlying feature space — it performs a weighted aggregation within the same embedding space of the original vision encoder. Therefore, even though the merged features combine information from multiple frames, they still lie within the same distribution that the Q-Former was originally trained on. As a result, the pre-trained Q-Former can correctly interpret these “synthetic” embeddings without additional tuning. Of course, if fine-tuning resources are available, further alignment could bring additional gains.

3. Our design ensures that the semantic structure of the visual features is preserved throughout the entire pipeline. Because of this consistency, the output representation produced by the BLIP-2 Q-Former remains well-aligned with the language model’s input space, allowing direct integration with LLMs such as Vicuna without retraining. In essence, the BLIP-2 Q-Former serves as a universal projector that already bridges the visual and language modalities effectively.

Thank you again for your interest and for raising such insightful questions!

[67L1]

Hi ,

Thank you again for your previous response, which was very helpful. I've been diving deeper into the implementation, and I have another question regarding the memory mechanism.

In the paper , it is mentioned that after memory consolidation, the output video feature is used to augment the long-term memory and also to "reinitialize the short-term memory with this feature." The paper suggests this is to "communicate information between different sliding windows."

However, upon reviewing the encode_short_memory_frame function in moviechat.py, I noticed that after the memory consolidation loop, the short-term memory buffer seems to be completely cleared, rather than being reinitialized with the merged frames:

# ... memory consolidation loop finishes ...
# The merged frames are moved to the long-term memory
for frame in self.short_memory_buffer:
    self.long_memory_buffer.append(frame)

# The short-term memory is then cleared
self.short_memory_buffer = [] 

This implementation appears to differ from the description in the paper. I was wondering if you could provide some clarification on this design choice.

My question is: Was the re-initialization step omitted from the codebase intentionally? For instance, did you perhaps find during experiments that clearing the buffer completely yielded better or more stable results compared to re-initializing it with the consolidated features?

I'm very interested in understanding the rationale behind this, as it seems like a key detail in how context is passed between video segments.

Thank you for your time and for sharing this fantastic work

[Espere-1119-Song]

Thank you for pointing this out. You are correct that according to the paper, the short-term memory should be reinitialized with the merged features after consolidation. It seems that this step was accidentally omitted during the ablation experiments and was not reverted in the final release.

Following the approach described in the paper, where the short-term memory is reinitialized with the merged features, indeed gives the best results in our experiments.

Thank you again for your careful reading and for catching this detail.