How to do inference?

[Himanshunitrr]

I just want do inference on few images.
I followed these (kindly correct me if I am wrong):

• git clone pa-llava
• git clone xtuner
• downloaded the weights of instruction_tuning_weight.pth from hugginface.
• moved pa-llava inside xtuner
• so:
  -- xtuner
  -- -- pa-llava
  -- -- tools
  -- -- xtuner

and then copied all the files from tool_add (in pa-llava/tool_add) to tools in xtuner which is inside xtuner.

After this I had these doubts:

• Can you explain or give an example for oscc.json in --data-path absolute_path/OSCC/oscc.json?
• Also I have assumed that ./instruction_tuning_weight_ft is instruction_tuning_weight.pth folder
• whats --work-dir?

NPROC_PER_NODE=1 xtuner zero_shot meta-llama/Meta-Llama-3-8B-Instruct --visual-encoder PLIP --llava ./instruction_tuning_weight_ft --prompt-template llama3_chat --data-path absolute_path/OSCC/oscc.json --work-dir absolute_path/logs/oscc --launcher pytorch --anyres-image

[Himanshunitrr]

update:
I used empty captions:
temp_data.json:
[ { "image": "test.png", "caption": "" } ] temp_data_structured.json
and used dataformat.py to get the corresponding .json file
{"id": "test", "image": "test.png", "conversations": [{"from": "human", "value": "<image>\nRelay an account of the picture shown."}, {"from": "gpt", "value": ""}]}
which has questions.

But I am getting:

NPROC_PER_NODE=1 xtuner zero_shot meta-llama/Meta-Llama-3-8B-Instruct --visual-encoder PLIP --llava instruction_tuning_weight.pth --prompt-template llama3_chat --data-path temp_data_structured.json --work-dir temp_data_output --launcher pytorch --anyres-image
02/14 22:09:29 - mmengine - WARNING - WARNING: command error!
02/14 22:09:29 - mmengine - WARNING -
Arguments received: ['xtuner', 'zero_shot', 'meta-llama/Meta-Llama-3-8B-Instruct', '--visual-encoder', 'PLIP', '--llava', 'instruction_tuning_weight.pth', '--prompt-template', 'llama3_chat', '--data-path', 'temp_data_structured.json', '--work-dir', 'temp_data_output', '--launcher', 'pytorch', '--anyres-image']. xtuner commands use the following syntax:

    xtuner MODE MODE_ARGS ARGS

    Where   MODE (required) is one of ('list-cfg', 'copy-cfg', 'log-dataset', 'check-custom-dataset', 'train', 'test', 'chat', 'convert', 'preprocess', 'mmbench', 'eval_refcoco')
            MODE_ARG (optional) is the argument for specific mode
            ARGS (optional) are the arguments for specific command

Some usages for xtuner commands: (See more by using -h for specific command!)

    1. List all predefined configs:
        xtuner list-cfg
    2. Copy a predefined config to a given path:
        xtuner copy-cfg $CONFIG $SAVE_FILE
    3-1. Fine-tune LLMs by a single GPU:
        xtuner train $CONFIG
    3-2. Fine-tune LLMs by multiple GPUs:
        NPROC_PER_NODE=$NGPUS NNODES=$NNODES NODE_RANK=$NODE_RANK PORT=$PORT ADDR=$ADDR xtuner dist_train $CONFIG $GPUS
    4-1. Convert the pth model to HuggingFace's model:
        xtuner convert pth_to_hf $CONFIG $PATH_TO_PTH_MODEL $SAVE_PATH_TO_HF_MODEL
    4-2. Merge the HuggingFace's adapter to the pretrained base model:
        xtuner convert merge $LLM $ADAPTER $SAVE_PATH
        xtuner convert merge $CLIP $ADAPTER $SAVE_PATH --is-clip
    4-3. Split HuggingFace's LLM to the smallest sharded one:
        xtuner convert split $LLM $SAVE_PATH
    5-1. Chat with LLMs with HuggingFace's model and adapter:
        xtuner chat $LLM --adapter $ADAPTER --prompt-template $PROMPT_TEMPLATE --system-template $SYSTEM_TEMPLATE
    5-2. Chat with VLMs with HuggingFace's model and LLaVA:
        xtuner chat $LLM --llava $LLAVA --visual-encoder $VISUAL_ENCODER --image $IMAGE --prompt-template $PROMPT_TEMPLATE --system-template $SYSTEM_TEMPLATE
    6-1. Preprocess arxiv dataset:
        xtuner preprocess arxiv $SRC_FILE $DST_FILE --start-date $START_DATE --categories $CATEGORIES
    6-2. Preprocess refcoco dataset:
        xtuner preprocess refcoco --ann-path $RefCOCO_ANN_PATH --image-path $COCO_IMAGE_PATH --save-path $SAVE_PATH
    7-1. Log processed dataset:
        xtuner log-dataset $CONFIG
    7-2. Verify the correctness of the config file for the custom dataset:
        xtuner check-custom-dataset $CONFIG
    8. MMBench evaluation:
        xtuner mmbench $LLM --llava $LLAVA --visual-encoder $VISUAL_ENCODER --prompt-template $PROMPT_TEMPLATE --data-path $MMBENCH_DATA_PATH
    9. Refcoco evaluation:
        xtuner eval_refcoco $LLM --llava $LLAVA --visual-encoder $VISUAL_ENCODER --prompt-template $PROMPT_TEMPLATE --data-path $REFCOCO_DATA_PATH
    10. List all dataset formats which are supported in XTuner

Run special commands:

    xtuner help
    xtuner version

GitHub: https://github.com/InternLM/xtuner

For some reason, the zero shot is not coming in xtuner, can you tell me how can I do that?

[aruiplex]

I am also confused, could you provide a more detailed explanation? @ddw2AIGROUP2CQUPT I will cite your paper, thanks.

[ddw2AIGROUP2CQUPT]

Replace xtuner's native entry_point.py with our supplied entry_point.py