can i provide frame-index-level dialogue information when organizing my videodata?

[jamesbond20181031]

hi! great job!
i want to finetune my videodata,but when i check the dataset prepare part, the annos are like this:

{
        "video": ["videos/xxx.mp4"],
        "conversations": [
            {
                "from": "human",
                "value": "<video>\nWhat are the main activities that take place in the video?"
            },
            {
                "from": "gpt",
                "value": "The main activities that take place in the video are the preparation of camera equipment by a man, a group of men riding a helicopter, and a man sailing a boat through the water."
            },
            ...
        ]
    },

these annos are video-level conversations,just tell us what the video says, but i want to know what happened in the video and also when does it happen.
so can i provide frame-index-level dialogue information when organizing my data?
For example: Does the person in the video pick up any object? If so, at what time segments does it happen?
further more,maybe some objects` bbox are needed(does the man holding a phone? if so,tell me the cords of the phone!)

[2U1]

You could but, you should set the caculate the max frames and fps for every video and set the dialogue to it.
Becuase the code samples the video with setting the max frames and fps. So, you should calculate how much frames the video will take and how what frame has the activity you want.

[jamesbond20181031]

@2U1
in the readme part,

• if you want to train both the language model and the vision model with LoRA:
• bash scripts/finetune_lora_vision.sh
• Train with video dataset
• You can train the model using a video dataset. However, SmolVLm processes videos as a sequence of images, so you’ll need to select specific frames and treat them as multiple images for training. You can set LoRA configs and use for LoRA too.
• bash scripts/finetune_video.sh
• Note: When training with video, it just as multi-image so you should adjust the max_pixels for maximum resolution and fps based on the available VRAM.

However, in my scenario, I have already split the video data into many frames, and when constructing the dataset, I build the conversation using several consecutive frames together, as shown in the example below:

  {
    "id": "0",
    "image": [
      "/datasets/rawframes/011DF2DAD94641E3C54F62C2C45FD9EF_1/img_00112.jpg",
      "/datasets/rawframes/011DF2DAD94641E3C54F62C2C45FD9EF_1/img_00113.jpg",
      "/datasets/rawframes/011DF2DAD94641E3C54F62C2C45FD9EF_1/img_00114.jpg",
      "/datasets/rawframes/011DF2DAD94641E3C54F62C2C45FD9EF_1/img_00115.jpg"
    ],
    "conversations": [
      {
        "from": "human",
        "value": "<image>\n<image>\n<image>\n<image>\nFor every frame, give the action category and location of each interacted object."
      },
      {
        "from": "gpt",
        "value": "Frame 1: object-0: taking_at a bbox(457,472,574,502)\nFrame 2: object-0: taking_at a bbox(402,461,546,496)\nFrame 3: object-0: taking_at a bbox(361,451,489,484)\nFrame 4: object-0: taking_at a bbox(313,441,418,473)"
      }
    ]
  },

So I should fine-tune it in a multi-image format rather than as a video, right?

But the other problem is, if I fine-tune it as a multi-image input, won’t that lose the temporal information? After all, these frames are consecutive and all belong to the same action class—this sequence is a complete action.
If I fine-tune it as a video-format and let the conversation include temporal cues—like when the action occurs and where its bounding box is at each moment—the training should make far more sense.

Looking forward to your reply!

[2U1]

I think that could be used with smolvlm2 and it is supported by the repo now, but I havent' updated the README.

When using smolvlm2, there are now way to give an temporal information, so you need to give an index for each image in the prompt and finetune it.