Can a slightly modified multimodal model achieve inference through the tensorrt llm pipeline?

[bleedingfight]

I tried replacing the llm of the llava model with llama3, and now I want to try using tensorrt llm to complete the entire inference. I have a few questions that I don't quite understand and would like to ask for advice:

1. The official llava model's llm is llama2, and the tensorrt conversion uses the hf model. Therefore, it is necessary to first rename the partial key of the official llava's llama weight, then fill 32000 to 32064 to generate the hf model, and then convert the hf model to the trt model. I want to know if I still need to fill in data to 32064 if I replace it with llama3?
2. My architecture consists of three parts: vision_tower, mmprojector（changed and add some conv op）, and llama3. My network is not a transformer native supported architecture (huggingface Auto), and I have my own Forword. I just want to briefly explain that replacing vision_tower and mmprojector with trt engine can achieve the use of trt llm to complete the entire algorithm pipeline