InternVL3.5-1B NPU

User:<image>Describe the image.

Answer: The image depicts an astronaut on the moon, holding a green bottle of beer and sitting next to a green cooler with some writing on it. The background shows Earth from space, highlighting the contrast between the moon's barren surface and the planet below.

---

InternVL3.5-1B for RK3588 NPU (Rock 5, Orange Pi 5).

Paper: InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

Hugging face: https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct

---

Introduction

LLMs (Large Language Models) are neural networks trained on large text datasets to understand and generate language.
VLMs (Vision-Language Models) add a visual encoder so the model can process images and text together.
A combined VLM+LLM system is often referred to as a multimodal model.

These models can be large—hundreds of millions to billions of parameters—which impacts accuracy, memory use, and runtime speed.
On edge devices like the RK3588, available RAM and compute are limited, and even the NPU has strict constraints on supported operations.
Because of this, models typically need to be quantised or simplified to fit.

Performance is usually expressed in tokens (words) per second.
Once converted to RKNN, parts of the model can run on the NPU, improving speed.
Despite these limits, models like Qwen3-2B run well on the RK3588 because the NPU efficiently accelerates the heavy math, and the vision encoder can be optimised. This makes advanced multimodal AI feasible on small, power-efficient devices.

---

Model performance benchmark (FPS)

All models, with C++ examples, can be found on the Q-engineering GitHub.

All LLM models are quantized to w8a8, while the VLM vision encoders use fp16.

model	RAM (GB)1	llm cold sec2	llm warm sec3	vlm cold sec2	vlm warm sec3	Resolution	Tokens/s
Qwen3-2B	3.1	21.9	2.6	10.0	0.9	448 x 448	11.5
Qwen3-4B	8.7	49.6	5.6	10.6	1.1	448 x 448	5.7
InternVL3.5-1B	1.9	8.3	8.0	1.5	0.8	448 x 448	24
InternVL3.5-2B	3.0	22	8.0	2.7	0.8	448 x 448	11.2
InternVL3.5-4B	5.4	50	8.0	5.9	0.8	448 x 448	5
InternVL3.5-8B	8.8	92	8.0	50.5	5.8	448 x 448	3.5
Qwen2.5-3B	4.8	48.3	4.0	17.9	1.8	392 x 392	7.0
Qwen2-7B	8.7	86.6	34.5	37.1	20.7	392 x 392	3.7
Qwen2-2.2B	3.3	29.1	2.5	17.1	1.7	392 x 392	12.5
InternVL3-1B	1.3	6.8	1.1	7.8	0.75	448 x 448	30
SmolVLM2-2.2B	3.4	21.2	2.6	10.5	0.9	384 x 384	11
SmolVLM2-500M	0.8	4.8	0.7	2.5	0.25	384 x 384	31
SmolVLM2-256M	0.5	1.1	0.4	2.5	0.25	384 x 384	54

¹ The total used memory; LLM plus the VLM.
² When an llm/vlm model is loaded for the first time from your disk to RAM or NPU, it is called a cold start.
The duration depends on your OS, I/O transfer rate, and memory mapping.
³ Subsequent loading (warm start) takes advantage of the already mapped data in RAM. Mostly, only a few pointers need to be restored.

---

Dependencies.

To run the application, you have to:

• OpenCV 64-bit installed.
• rkllm library.
• rknn library.
• Optional: Code::Blocks. ($ sudo apt-get install codeblocks)

Installing the dependencies.

Start with the usual

$ sudo apt-get update 
$ sudo apt-get upgrade
$ sudo apt-get install cmake wget curl

OpenCV

To install OpenCV on your SBC, follow the Raspberry Pi 4 guide.

Or, when you have no intentions to program code:

$ sudo apt-get install libopencv-dev 

---

Installing the app.

$ git clone https://github.com/Qengineering/Qwen3-VL-2B-NPU

RKLLM, RKNN

To run InternVL3, you need to have the rkllm-runtime library version 1.2.3 (or higher) installed, as well as the rknpu driver version 0.9.8.
If you don't have these on your machine, or if you have a lower version, you need to install them.
We have provided the correct versions in the repo.

$ cd ./Qwen3-VL-2B-NPU/aarch64/library
$ sudo cp ./*.so /usr/local/lib
$ cd ../include
$ sudo cp ./*.h /usr/local/include

Download the LLM and VLM model.

The next step is downloading the two needed files (1.5 GB) from our Sync.com server:
internvl3_5-1b-instruct_w8a8_rk3588.rkllm and internvl3_5-1b_vision_rk3588.rknn
Copy both into this folder.

Building the app.

Once you have the two models, it is time to build your application.
You can use Code::Blocks.

• Load the project file *.cbp in Code::Blocks.
• Select Release, not Debug.
• Compile and run with F9.
• You can alter command line arguments with Project -> Set programs arguments...

Or use Cmake.

$ mkdir build
$ cd build
$ cmake ..
$ make -j4

Running the app.

The app has the following arguments.

VLM_NPU Picture RKNN_model RKLLM_model NewTokens ContextLength

Argument	Comment
picture	The image. Provide a dummy if you don't want to use an image
RKNN_model	The visual encoder model (vlm)
RKLLM_model	The large language model (llm)
NewTokens	This sets the maximum number of new tokens. Optional, default 2048
ContextLength	This specifies the maximum total number of tokens the model can process. Optional, default 4096

In the context of the Rockchip RK3588 LLM (Large Language Model) library, the parameters NewTokens and ContextLength both control different limits for text generation, and they're typical in LLM workflows.
NewTokens
This sets the maximum number of tokens (pieces of text, typically sub-word units) that the model is allowed to generate in response to a prompt during a single inference round. For example, if set to 300, the model will not return more than 300 tokens as output, regardless of the prompt length. It's important for controlling generation length to avoid too-short or too-long responses, helping manage resource use and output size.
ContextLength
This specifies the maximum total number of tokens the model can process in one go, which includes both the prompt (input) tokens and all generated tokens. For example, if set to 2048 and your prompt already uses 500 tokens, the model can generate up to 2048-500 = 1548 new tokens. This is a hardware and architecture constraint set during model conversion and deployment, as the context window cannot exceed the model's design limit (for instance, 4096 or 8192 tokens depending on the model variant).

A typical command line can be:

./VLM_NPU ./Moon.jpg ./models/internvl3_5-1b_vision_rk3588.rknn ./models/internvl3_5-1b-instruct_w8a8_rk3588.rkllm 2048 4096

The NewTokens (2048) and ContextLength (4096) are optional and can be omitted.

Using the app.

Using the application is simple. Once you provide the image and the models, you can ask everything you want.
Remember, we are on a bare Rock5C, so don't expect the same quality answers as ChatGPT can provide.
On the other hand, when you see the examples below, the app performs amazingly well.

If you want to talk about the picture, you need to include the token <image> in your prompt once.
The app remembers the dialogue until you give the token <clear>.
With <exit>, you leave the application.

C++ code.

Below, you find the surprisingly little code of main.cpp.

#include "RK35llm.h"

int main(int argc, char** argv)
{
    std::string input_str;
    std::string output_str;
    RK35llm RKLLM;

    RKLLM.SetInfo(true);            //yes, you may give me additional model information
    RKLLM.SetSilence(false);        //you may print the incremental text chunks on the terminal

    if     (argc< 4) {std::cerr << "Usage: " << argv[0] << " image vlm_model llm_model [option]NewTokens [option]ContextLength\n"; return -1;}
    else if(argc==4) RKLLM.LoadModel(argv[2],argv[3]);
    else if(argc==5) RKLLM.LoadModel(argv[2],argv[3],std::atoi(argv[4]));
    else if(argc> 5) RKLLM.LoadModel(argv[2],argv[3],std::atoi(argv[4]),std::atoi(argv[5]));

    cv::Mat Pic = cv::imread(argv[1]);
    RKLLM.LoadImage(Pic);

    while(true) {
        printf("\n");
        printf("User: ");

        std::getline(std::cin, input_str);
        if (input_str == "exit") break;

        output_str = RKLLM.Ask(input_str);
//        std::cout << "\nLLM Reply: " << output_str << std::endl;
    }

    return 0;
}

Most code speaks for itself. One remark.
The LLM generates the answer to your request in little pieces of text. You see them as if the LLM is typing on your terminal.
RKLLM.SetSilence controls this behaviour. When set to true, it will suppress this output. RKLLM.Ask() also returns the complete answer.
In main.cpp uncomment this line:

std::cout << "\nLLM Answer: " << output_str << std::endl;

---

VLM examples.

Image	Dialog
	User: <image>Which city is this? Answer: This image depicts the skyline of Singapore, known for its iconic Marina Bay Sands hotel and the Singapore Flyer Ferris wheel. The cityscape features several notable landmarks, including the distinctive ArtScience Museum shaped like an open shell, which is a unique architectural feature in Singapore.
	User: <image>Tell me something about the history of this place. Answer: The Great Wall of China is one of the most iconic and historically significant structures in the world, spanning over 13,000 miles across several Chinese dynasties. Constructed primarily during the Qin Dynasty (221-206 BCE) to protect against invasions by nomadic tribes such as the Xiongnu, it has been expanded and rebuilt multiple times throughout history. The wall's construction was a response to various threats, including raids from the northern tribes and internal rebellions. It served not only as a defensive barrier but also as a symbol of Chinese power and unity. The Great Wall is now a UNESCO World Heritage Site and attracts millions of visitors each year who come to admire its grandeur and historical significance. Despite being over 2,000 years old, the wall has endured through various eras of conflict and natural disasters, including earthquakes and floods. Today, it stands as a testament to human ingenuity and resilience in the face of adversity.
	User: <image>What are the ingredients I need to make this? Answer: To make a delicious pizza, you'll need these basic ingredients: 1. Pizza Dough: You can use scratchy or regular dough. 2. Tomato Sauce: A mixture of tomato sauce and water. 3. Cheese: Mozzarella cheese is the classic choice. 4. Vegetables: Fresh basil leaves are a great addition, but you could also add other vegetables like bell peppers, onions, mushrooms, or spinach. 5. Filling: You can fill your dough with any combination of your favorite toppings. Here's a simple recipe to get you started: Ingredients: - 1 ½ cups all-purpose flour - 3 tablespoons active dry yeast - 2 teaspoons sugar - 1 teaspoon salt - 4 cups warm water (around 100°F) - 1 cup tomato sauce - ¾ cup shredded mozzarella cheese - Fresh basil leaves Instructions: 1. Preheat Oven: Preheat your oven to 450°F (230°C). 2. Make the Dough: In a large bowl, combine flour and yeast. Add sugar and salt. Mix until the dough comes together in a ball. 3. Let Rise: Place the dough in a greased bowl, cover it with a damp cloth, and let it rise in a warm place for about 1 hour or until doubled in size. 4. Prepare Sauce: In a separate bowl, mix together tomato sauce and water until smooth. 5. Assemble the Pizza: Punch down the risen dough and roll it out to your desired thickness on a floured surface. 6. Add Toppings: Spread the tomato sauce over the dough, then add your favorite vegetables and cheese. 7. Bake: Place the pizza in the preheated oven and bake for 10-12 minutes or until the crust is golden brown and the cheese is bubbly. Enjoy your homemade pizza!
	User: <image>Which movie is this? Answer: This image appears to be from the movie "The Princess Bride." The characters in the scene, along with their attire and expressions, are reminiscent of the iconic characters from that film.

LLM example.

Notice the nice Markdown makeup of the answer!

User: Can you make me a C++ call that embeds the Pythagorean theorem?

Answer: Certainly! The Pythagorean theorem states that in a right-angled triangle, the square of the length of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the lengths of the other two sides. Mathematically, this can be expressed as:

[ c^2 = a^2 + b^2 ]

where ( c ) is the length of the hypotenuse, and ( a ) and ( b ) are the lengths of the other two sides.

To embed this theorem in a C++ program, you would typically use the cmath library for mathematical operations. Here's an example of how you might implement this:

#include <iostream>
#include <cmath> // For sqrt

int main() {
    double a = 3.0;
    double b = 4.0;
    double c;

    // Calculate the hypotenuse using the Pythagorean theorem
    c = std::sqrt(a * a + b * b);

    // Output the result
    std::cout << "The length of the hypotenuse is: " << c << std::endl;

    return 0;
}

In this example:

• We include the iostream library for input and output operations.
• We include the cmath library for mathematical functions like sqrt.
• In the main function, we define two variables a and b with values of 3.0 and 4.0 respectively.
• We calculate the hypotenuse c using the formula ( c = \sqrt{a^2 + b^2} ).
• Finally, we output the result to the console.

This program will print out the length of the hypotenuse for a right-angled triangle with sides 3 and 4.

---

Rock5GPT

To get a taste, try our professional Qwen3 AI-chatbot running on a Rock 5C: https://rock5gpt.qengineering.eu

---