Set model loader to ExLlamav2_HF or ExLlamav2?

[Kaszebe]

As the title states, do we need to set the model loader to ExLlamav2_HF or ExLlamav2?

The documentation says:

**ExLlamav2**
The same as ExLlamav2_HF but using the internal samplers of ExLlamav2 instead of the ones in the Transformers library.

I tried doing just that and got an error message:

11:10:48-588864 INFO     Loading "turboderp_Mistral-Large-Instruct-2407-123B-exl2"                        
11:10:48-593031 ERROR    Failed to load the model.                                                        
Traceback (most recent call last):
  File "/opt/text-generation-webui/modules/ui_model_menu.py", line 231, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/modules/models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/modules/models.py", line 312, in ExLlamav2_loader
    from modules.exllamav2 import Exllamav2Model
  File "/opt/text-generation-webui/modules/exllamav2.py", line 13, in <module>
    from exllamav2 import (
ImportError: cannot import name 'ExLlamaV2Cache_TP' from 'exllamav2' (/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllamav2/__init__.py)

And when I selected turboderp_Mistral-Large-Instruct-2407-123B-exl2 in the dropdown menu, it automatically defaulted to ExLlamav2_HF as the loader.

[RandomInternetPreson]

You should be able to use either one.

Questions for you:

Are you able to get the non-hf version working?
Do you have enough vram to run the model? I have 7x24gb gpus and can only run the 8bit version using exllamav2. The model you named doesn't look to be quantized?

[Kaszebe]

No, I was unable to get the non-hf version working. I have four 3090s and one 4090 (120GB VRAM). I can also throw a 4080 into the mix, but I haven't seen the need because I'm running the 6.0bpw quant, which is about 92GB:

https://huggingface.co/turboderp/Mistral-Large-Instruct-2407-123B-exl2

I set max_seq_len to 15678 (I pulled this number out of thin air). Aside from gpu-split, I leave all other settings as is.

I recall reading somewhere that I should balance the load across all GPUs, and for gpu-split I'm using 19,19,19,19,19. It resulted in the below:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:01:00.0 Off |                  N/A |
|100%   25C    P8             33W /  390W |   20365MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:45:00.0  On |                  N/A |
|100%   28C    P8             43W /  420W |   20637MiB /  24576MiB |      7%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 4090        Off |   00000000:81:00.0 Off |                  Off |
| 80%   28C    P8             19W /  450W |   21053MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        Off |   00000000:C1:00.0 Off |                  N/A |
|100%   39C    P0            108W /  350W |   20704MiB /  24576MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA GeForce RTX 3090        Off |   00000000:C2:00.0 Off |                  N/A |
|100%   27C    P8             25W /  350W |   17725MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      4878      G   /usr/lib/xorg/Xorg                              4MiB |
|    0   N/A  N/A      8520      C   python                                      20342MiB |
|    1   N/A  N/A      4878      G   /usr/lib/xorg/Xorg                             97MiB |
|    1   N/A  N/A      5028      G   /usr/bin/gnome-shell                          109MiB |
|    1   N/A  N/A      7574      G   /usr/lib/firefox/firefox-bin                   13MiB |
|    1   N/A  N/A      8520      C   python                                      20398MiB |
|    2   N/A  N/A      4878      G   /usr/lib/xorg/Xorg                              4MiB |
|    2   N/A  N/A      8520      C   python                                      21030MiB |
|    3   N/A  N/A      4878      G   /usr/lib/xorg/Xorg                              4MiB |
|    3   N/A  N/A      4974    C+G   ...libexec/gnome-remote-desktop-daemon        339MiB |
|    3   N/A  N/A      8520      C   python                                      20330MiB |
|    4   N/A  N/A      4878      G   /usr/lib/xorg/Xorg                              4MiB |
|    4   N/A  N/A      8520      C   python                                      17702MiB |
+-----------------------------------------------------------------------------------------+

Another contributing factor is that I am a bit of a n00b to running EXL2 files. I have mostly used GGUF. I only switched over because I read one of your posts of Reddit (at least I think that was you) and I wanted to see if TP could speed up the relatively slow tokens/s.

11:40:23-503358 INFO     Loading "turboderp_Mistral-Large-Instruct-2407-123B-exl2"                        
11:41:38-237431 INFO     Loaded "turboderp_Mistral-Large-Instruct-2407-123B-exl2" in 74.73 seconds.       
11:41:38-247545 INFO     LOADER: "ExLlamav2_HF"                                                           
11:41:38-248082 INFO     TRUNCATION LENGTH: 15768                                                         
11:41:38-248629 INFO     INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)"                    
Output generated in 21.45 seconds (7.55 tokens/s, 162 tokens, context 665, seed 119388714)
Output generated in 12.84 seconds (7.95 tokens/s, 102 tokens, context 848, seed 1770587562)
Output generated in 17.67 seconds (8.20 tokens/s, 145 tokens, context 957, seed 521931597)
Output generated in 60.15 seconds (8.51 tokens/s, 512 tokens, context 1159, seed 1616749695)
Output generated in 18.59 seconds (7.85 tokens/s, 146 tokens, context 1679, seed 954773575)
Output generated in 16.03 seconds (8.11 tokens/s, 130 tokens, context 1840, seed 861351508)
Output generated in 23.25 seconds (8.21 tokens/s, 191 tokens, context 2079, seed 1221962969)
Output generated in 28.64 seconds (8.20 tokens/s, 235 tokens, context 2362, seed 441691756)
Output generated in 25.94 seconds (6.52 tokens/s, 169 tokens, context 4619, seed 590644056)
Output generated in 21.05 seconds (6.60 tokens/s, 139 tokens, context 6104, seed 1911249069)
Output generated in 56.00 seconds (8.21 tokens/s, 460 tokens, context 6281, seed 515814499)
Output generated in 61.02 seconds (8.10 tokens/s, 494 tokens, context 6809, seed 836205733)
Output generated in 19.29 seconds (7.41 tokens/s, 143 tokens, context 7362, seed 604702441)
Output generated in 59.21 seconds (8.12 tokens/s, 481 tokens, context 7550, seed 256036150)
Output generated in 9.06 seconds (6.07 tokens/s, 55 tokens, context 8136, seed 179916045)
Output generated in 7.77 seconds (4.89 tokens/s, 38 tokens, context 1718, seed 192809925)
Output generated in 5.42 seconds (6.64 tokens/s, 36 tokens, context 1800, seed 295348403)
Output generated in 4.85 seconds (6.80 tokens/s, 33 tokens, context 1843, seed 77058945)
12:20:22-261443 INFO     Deleted "logs/chat/Copywriter/20240905-12-14-30.json".                           
Output generated in 10.75 seconds (7.63 tokens/s, 82 tokens, context 632, seed 1780877154)
Output generated in 4.54 seconds (6.17 tokens/s, 28 tokens, context 741, seed 1366624299)

If it matters, I am a copywriter (this is what I do for a living). I 'pretend' that the LLM is another human being sitting next to me. I ask it questions and bounce ideas off of it. As such, I need the most intelligent LLM out there...one that has an in-depth understanding of the subtle nuances of the King's and how they affect human persuasion, emotion, etc.

My chats with LLMs usually grow very long over the course of a project. For example, I may spend upwards of one hour tweaking a 15-word value proposition. The LLM needs the ability to follow along (as best as possible...considering they really don't have a 'memory' as such) and be able to recall back to things we talked about 30 minutes ago.

To that point—I have installed the simple_memory Oobabooga extension in the high hopes it will help the AI 'remember' things I tell it, such as grammar rules that it needs to follow (e.g. Write in active voice only).

[RandomInternetPreson]

Gotacha!

Okay, I have a new question for you, did you use the update wizard to install any extensions? If you use the update wizard it will undo all of the stuff from the separate exllamav2 install instructions from the repo. The update wizard will reinstall everything (including exllamav2) even if you are just installing an extension.

When you do the separate installation of exllamav2 via the start_linuxh.sh terminal, do you get confirmation that exllama2 completed building? after the pip install . step completes you should see some type of confirmation that things are complete.

Just to make sure TP is working, could you try and do a new different v1.14 install of textgen in a separate folder (every folder you unzip the release into will be a separate install), when complete follow the steps on the repo, then try using the --enable_tp in the CMD_Flags.txt file before doing anything else to the installation.

I used to use exllamav2 all the time, but recently switched to ggufs, and now I'm back to exllamav2 because of tensor parallelism too!

The 6bit exllamav2 quant is good (I quantized locally but it is likely the same), it takes almost exactly 6x24gpus to fully load the 6bit quant of mistral large 2 with all of the context available (using tensor parallelism I don't know the vram is more efficiently used, I think you can get more use of vram for context with TP but I'm not 100% sure).

I use my model to discuss scientific ideas and to help me write code. I don't know about copywriting specifically, but the model can recall things very well in my experience. I think the context retention might not be as good if you quantize the context (an option in textgen) but I may be wrong.

[RandomInternetPreson]

https://www.reddit.com/r/LocalLLaMA/comments/1efg2wv/llama_31_405b_exl2_quant_results/

This post is a good resource, except for one big problem, the guy who made the post quantized the context to 4bit :c so the mixtral context length gets all goofy around 64k. I have conversations that are well over 100k and the model still recalls things well.

The using 8bit context (cach_8bit) might not have such an effect and it might let you fit more context onto your cards.

[Kaszebe]

EDIT: I originally had a CUDA mismatch error. Managed to solve it and was able to successfully install everything. As such, I Just edited this post to remove that question. Just need to figure out the two files question I have below.

Hi, apologies for the delay---I screwed something up so I had to yank the CMOS battery and short the jumpers and re-install popos. I also found an unreleased BIOS from the mfg of my motherboard that supposedly solves many PCIE issues. It's taken me a week to get the server back to the way it was.

With that being said---I'm in the process of installing your TP code. The instructions at the bottom of the page here: https://github.com/RandomInternetPreson/TextGenTips

Say to replace the exllamav2.py file and shared.py (located in the modules folder) file with these: https://github.com/RandomInternetPreson/TextGenTips/tree/main/ExllamaV2_TensorParallel_Files

When I go to that link there are four files:

20240830-16-23-25.json
exllamav2.py
exllamav2_hf.py
shared.py

Do I need to copy all four of those files and overwrite the files in the /modules/ folder? Or just exllamav2.py file and shared.py?

And I do not use update_wizard_linux.sh. I don't have any extensions. I may add that "short context" one to see if that can help the model remember my strict grammar rules.

Also, I removed a 3090 because it was giving off a ton of PCIE errors (or so I thought). I replaced it with an old 4080 I had lying around. I then got a moment of inspiration and decided to deshroud the 3090---as it had an accident and two fans no longer worked. I then added two 120mm fans to sit on top of it (open air mining rig) and temps are anywhere between 21c to 30c when idle and running inference. I also power limited all GPUs to 70% of their original wattage. I have been running the 3090 for 5? days now and not a single PCIE error and it's running extremely cool. I might deshroud all other GPUs as well.

Thanks!