10k frame memory?

[dcompgriff]

How can I configure the model to use a video with 10k frames, sampled at 1 frame per second? I'd like to have a model that can answer questions across video data aggregated across a 2 hour video at 1 fps.

[Espere-1119-Song]

Setting short-term memory length to 34, and long-term memory length to 512 can achieve 8k frames. The total frame number = (short-term memory length - 2 ) * (long-term memory length / 2). Please note that it will take a pretty long time to process these frames (maybe over 1h).

[dcompgriff]

It seems to me after reviewing the paper and thinking about the code that the memory method proposed could be incorporated into any LLaVA style VLM model we choose, provided we get rid of the minigpt-4 Q-former, and just use the pre-trained ViT, Pre-trained Projection Layer, and Pre-Trained LM decoder. Is that a correct statement, or should I expect to have to do fine-tuning with a standard (non memory augmented) VLM for using the movie-chat memory method?

[rese1f]

@dcompgriff hi yes! Actually the motivation is quite similar to Native Sparse Attention by deepseek. You can have a try!

[dcompgriff]

Why is the total number of frames dependent on (long-term memory length / 2) and not just long-term memory length? The paper talks about a fixed RL frames to keep in long term memory, but doesn't talk about this /2 factor.

[Espere-1119-Song]

In every memory consolidation operation, the frames in the short-term memory will be merged into 2 frames. This means that, every 2 frames in long-term memory represents the original frames in short-term memory.

We have performed ablation study on the number of merged frames, and 2 merged frames performs best.

[dcompgriff]

If frames are merged in memory consolidation from pairs of 2 averaged into 1, then put into long term memory, wouldn't that mean the opposite is true? Every 1 entry in long term memory represents a consolidated 2-frame from short term memory?

…

On Thu, Feb 27, 2025, 8:06 PM EnxinSong ***@***.***> wrote: In every memory consolidation operation, the frames in the short-term memory will be merged into 2 frames. This means that, every 2 frames in long-term memory represents the original frames in short-term memory. We have performed ablation study on the number of merged frames, and 2 merged frames performs best. — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVJKJNHOFF5HO4Q7T275OT2R7OFPAVCNFSM6AAAAABVUBLC3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZGY3DONJZG4> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: Espere-1119-Song]*Espere-1119-Song* left a comment (rese1f/MovieChat#89) <#89 (comment)> In every memory consolidation operation, the frames in the short-term memory will be merged into 2 frames. This means that, every 2 frames in long-term memory represents the original frames in short-term memory. We have performed ablation study on the number of merged frames, and 2 merged frames performs best. — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVJKJNHOFF5HO4Q7T275OT2R7OFPAVCNFSM6AAAAABVUBLC3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZGY3DONJZG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dcompgriff]

Unless you mean because frame are duplicated during pairing? So 1, 2, 3, 4 becomes 12, 23, 34, 45, ... which is why there's a doubling in long term memory, this halving the effective number of frames that cam be held.

…

On Thu, Feb 27, 2025, 8:27 PM D G ***@***.***> wrote: If frames are merged in memory consolidation from pairs of 2 averaged into 1, then put into long term memory, wouldn't that mean the opposite is true? Every 1 entry in long term memory represents a consolidated 2-frame from short term memory? On Thu, Feb 27, 2025, 8:06 PM EnxinSong ***@***.***> wrote: > In every memory consolidation operation, the frames in the short-term > memory will be merged into 2 frames. This means that, every 2 frames in > long-term memory represents the original frames in short-term memory. > > We have performed ablation study on the number of merged frames, and 2 > merged frames performs best. > > — > Reply to this email directly, view it on GitHub > <#89 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACVJKJNHOFF5HO4Q7T275OT2R7OFPAVCNFSM6AAAAABVUBLC3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZGY3DONJZG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > [image: Espere-1119-Song]*Espere-1119-Song* left a comment > (rese1f/MovieChat#89) > <#89 (comment)> > > In every memory consolidation operation, the frames in the short-term > memory will be merged into 2 frames. This means that, every 2 frames in > long-term memory represents the original frames in short-term memory. > > We have performed ablation study on the number of merged frames, and 2 > merged frames performs best. > > — > Reply to this email directly, view it on GitHub > <#89 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACVJKJNHOFF5HO4Q7T275OT2R7OFPAVCNFSM6AAAAABVUBLC3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZGY3DONJZG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[Espere-1119-Song]

Imagine that the short-term memory length is 18, after memory consolidation, we maintain two merged-frames to initialize the short-term memory, which means that the short-term memory read 16 new frames each time. During each memory consolidation operation, the 18 frames in short-term memory will be merged into 2 frames and sent to long-term memory. Therefore, if the long-term memory length is 256, it contains 128 consolidation operation.

[dcompgriff]

Ok, got it 👍

…

On Thu, Feb 27, 2025, 9:42 PM EnxinSong ***@***.***> wrote: Imagine that the short-term memory length is 18, after memory consolidation, we maintain two merged-frames to initialize the short-term memory, which means that the short-term memory read 16 new frames each time. During each memory consolidation operation, the 18 frames in short-term memory will be merged into 2 frames and sent to long-term memory. Therefore, if the long-term memory length is 256, it contains 128 consolidation operation. — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVJKJNUDRYLNRSRUVJYWNL2R7ZN5AVCNFSM6AAAAABVUBLC3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZG43TOOBQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: Espere-1119-Song]*Espere-1119-Song* left a comment (rese1f/MovieChat#89) <#89 (comment)> Imagine that the short-term memory length is 18, after memory consolidation, we maintain two merged-frames to initialize the short-term memory, which means that the short-term memory read 16 new frames each time. During each memory consolidation operation, the 18 frames in short-term memory will be merged into 2 frames and sent to long-term memory. Therefore, if the long-term memory length is 256, it contains 128 consolidation operation. — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVJKJNUDRYLNRSRUVJYWNL2R7ZN5AVCNFSM6AAAAABVUBLC3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZG43TOOBQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MATTbomerts]

Setting short-term memory length to 34, and long-term memory length to 512 can achieve 8k frames. The total frame number = (short-term memory length - 2 ) * (long-term memory length / 2). Please note that it will take a pretty long time to process these frames (maybe over 1h).

Is that mean when i set short-term memory length to 8 and long-term memory length to 40, then the total frames i processed in one video is 120? and should i set N_SAMPLES to other values or keep it to 128?

[Espere-1119-Song]

Sure, if the target merged frames number is 2, then the processed frame number is 120. And the N_SAMPLES should be set to long-term memory length / 2.