Image format for Axolotl multimodal training

[HamidGadirov]

Hi all, I'm training Mistral-small-24B using Axolotl for multi-GPU fine-tuning. I already have my dataset prepared in .jsonl format with prompts and responses. I’m including images alongside text prompts, and my current sample looks like this:

{"messages": [ {"role": "user", "content": "<image>\nWhat is shown on this page?"}, {"role": "assistant", "content": "This is a sample response describing the document content."}], "images": ["data/page9.png"]}

My question is: Should the image be in .png (like above), or in a base64 format, or something else for Axolotl to work correctly? I assume .jsonl format is correct, if not please correct me.

Thanks in advance!

[NanoCode012]

Hey! We have doc on dataset sample format here https://docs.axolotl.ai/docs/multimodal.html#dataset-format

The images should be placed within the message key. The format you used would also work, but it’s an old format.

[HamidGadirov]

Thanks @NanoCode012 for the reply! Do I understand correctly that you recommend to use the fooling format (here image is within the message key:

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "image", "path": "data/page9.png" },
        { "type": "text",  "text": "What do you see on this page?" }
      ]
    },
    {
      "role": "assistant",
      "content": [
        { "type": "text",  "text": "## 1. This is a sample response describing the document content …" }
      ]
    }
  ]
}

? Then I will prepare the dataset to be in this format for fine-tuning.

[NanoCode012]

Yes

[NanoCode012]

@HamidGadirov , just following up on this, did the above solved your problem?