You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+53-45Lines changed: 53 additions & 45 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# Croco.Cpp (CCPP) :
1
+
# Croco.Cpp (CCPP) - Readme to be updated :
2
2
3
3
<details>
4
4
<summary>Unroll DISCLAIMER:</summary>
@@ -8,7 +8,7 @@ The namechange is due to my boredom with the Frankenstein marker I myself initia
8
8
As usual, the Croco.Cpp builds are NOT supported by the KoboldCPP (KCPP) team, Github, or Discord channel.
9
9
They are for greedy-test and amusement only.
10
10
Any potential support found them is a courtesy, not a due.
11
-
My CCPP version number bumps as soon as the version number in the official experimental branch bumps in the following way x.xxx : (KCPP)x.xx.x.(CCPP)xx.
11
+
My CCPP version number bumps as soon as the version number in the official experimental branch bumps in the following way x.xxx (ex : 1.80.1) : (KCPP)x.xxx.(CCPP)xx.
12
12
They are not "upgrades" over the official version. And they might be bugged at time: only the official KCPP releases are to be considered correctly numbered, reliable and "fixed".
13
13
The LllamaCPP version + the additional PRs integrated follow my CCPP versioning in the title, so everybody knows what version they deal with.
14
14
Important : New models sometimes integrated in my builds (like recently Mistral Nemo, which posed problems for several users) are for personal testing only, and CAN'T be fixed if they fail because their support come from third party PRs coming from LlamaCPP merged "savagely" in my builds, sometimes before even being merged on LlamaCPP master.
@@ -17,18 +17,28 @@ Important : New models sometimes integrated in my builds (like recently Mistral
17
17
Presentation :
18
18
19
19
Croco.Cpp (CCPP) is a fork of the experimental branch of KoboldCPP (KCPP), mainly aimed at NVidia Cuda users (I'm myself using Ampere GPUs, it MIGHT support the other backends also, everything is compîled but Hipblas/ROCm, but it's not tested), with a few modifications accordingly to my own needs :
20
-
- More context steps in GUI, as well as more Blas Batch Size (supports MMVQ 1-8 for example)
21
-
- 26 different modes of quantization for the context cache (F16, 20 KV modes with Flash Attention, 5 K modes without Flash Attention for models like Gemma)
20
+
- A more cluttered GUI that I had to enlarge to put all my mess.
21
+
- More context steps in GUI, as well as more Blas Batch Size (supports MMVQ 1-8 for example).
22
+
- Physical Blas Batch Size Exposed and configurable.
23
+
- 22 or so different modes of quantization for the context cache (F16, around 15 KV modes with Flash Attention, 7 quantum legacy K cache modes without Flash Attention for models like Gemma).
24
+
- KV cache supports IQ4_NL and Q6_0 (except for Gemma), thanks to Ikawrakow.
25
+
- Supports inference for B16 models in Cuda (thanks Ikawrakow).
26
+
- Supports inference for new quants made by Ikawrakow (Q6_0 legacy for irregularly shaped tensors ; IQ_2K, 3K, 4K, 5K, 6K (first gen) ; IQ2_KS, 4_KSS, 4_KS (second gen, working with IK's reworked MMVQ template) ; IQ2_KT, 3_KT, 4_KT (Trellis, working with a restored DMMV kernel).
27
+
- A dozen or so commits taken from Ikawrakow's IK_Llama.CPP for performances (notably on Gemma). That includes a few more GGML ops.
22
28
- A slightly different benchmark (one flag per column instead of a single flag space).
23
-
-8 Stories slots instead of 6 in the web-interface (KLite).
24
-
- Often some PRs unsupported/not yet supported in KCPP.
29
+
-10 Stories slots instead of 6 in the web-interface (KLite).
30
+
- Often some PRs unsupported/not yet supported in KCPP (I look especially at Cuda and KV cache related PRs).
25
31
- More infos displayed in the CLI, without activating debug mode.
26
-
- Smartcontext instead of contextshift by default in GUI for compatibility with Gemma
27
-
- Since 1.71010, an enhanced model layers autoloader on GPU, based on Concedo's code and Pyroserenus formulas, but different from Henky's subsequent commit on KCPP-official. It's compatible with KV_Quants, works in single and multi-GPU, is accessible in CLI and GUI modes, and can be configured easily in tandem with tensor split for an entirely customized loading accordingly to one's rig and needs.
32
+
- Smartcontext instead of contextshift by default in GUI for compatibility with Gemma.
33
+
- Support the edition of NORM_EPS_RMS value.
34
+
- More logging out of debug mode.
35
+
- Support EmphasisFSM by Yoshku to handle the "" and ** formatting in KCPP and SillyTavern (mostly, if you have troubles of chat (thoughts, actions, dialogues) formatting, and anti-slop doesn't cut it for your needs somehow).
36
+
- Since 1.71010, an enhanced model layers autoloader on GPU (which is less and less cluttered and bugged lol), based on Concedo's code and Pyroserenus formulas, but different from Henky's subsequent commit on KCPP-official. It's compatible with KV_Quants, accounts for FA, MMQ, LowVram, works in single and multi-GPU (up to 16?), is accessible in CLI and GUI modes, and can be configured easily in tandem with tensor split for an entirely customized loading accordingly to one's rig and needs.
37
+
28
38
29
39
Recommanded settings for Commande Line Interface / GUI :
30
40
```
31
-
--flashattention (except for Gemma)
41
+
--flashattention (except for Gemma?)
32
42
--blastbatchsize 128 (256 for Gemma)
33
43
--usecublas mmq (for NVidia users, MMQ mode is faster)
34
44
```
@@ -41,20 +51,23 @@ Check the help section (koboldcpp.exe --help or python koboldcpp.py --help) for
41
51
42
52
With Flash Attention :
43
53
- F16 -> Fullproof (the usual KV quant since the beginning of LCPP/KCPP)
44
-
- K F16 with : V Q8_0, Q5_1, Q5_0, Q4_1, Q4_0
45
-
- K Q8_0 with : V F16, Q8_0 (stable, my current main, part of the LCPP/KCPP main triplet), Q5_1 (maybe unstable), Q5_0 (maybe unstable), Q4_1 (maybe stable), the rest is untested beyond benches), Q4_0 (maybe stable)
46
-
- K Q5_1 with : V Q5_1, Q5_0, Q4_1, Q4_0
47
-
- K Q5_0 with : V Q5_0, Q4_1, V Q4_0
48
-
- K Q4_1 with : V Q4_1 (stable), Q4_0 (maybe stable)
54
+
- BF16 (experimental)
55
+
- K F16 with : V Q8_0, Q6_0 (experimental), Q5_1, Q5_0, iq4_nl
56
+
- K Q8_0 with : V Q8_0 (stable, part of the LCPP/KCPP main triplet), Q6_0 (experimental), Q5_1 (maybe unstable), Q5_0 (maybe unstable), iq4_nl (maybe stable), Q4_0 (maybe stable)
57
+
- K Q6_0 with : V Q6_0, Q5_0, iq4_nl
58
+
- K Q5_1 with : V Q5_0, iq4_nl
59
+
- K Q5_0 with : V iq4_nl
49
60
- KV Q4_0 (quite stable, if we consider that it's part of the LCPP/KCPP main triplet)
50
61
Works in command line, normally also via the GUI, and normally saves on .KCPPS config files.
62
+
- KV iq4_nl (with -1% perplexity compared to Q4_0).
51
63
52
64
Without Flash Attention nor MMQ (for models like Gemma) :
53
-
- V F16 with KQ8_0, Q5_1, Q5_0, Q4_1, and Q4_0.
65
+
- V F16 with K Q8_0, Q5_1, Q5_0, Q4_1, and Q4_0.
66
+
- K Q6_0 and IQ4_NL to be tested, might not work.
54
67
</details>
55
68
56
69
<details>
57
-
<summary>Unroll the options to set KV Quants</summary>
70
+
<summary>Unroll the options to set KV Quants (obsolete)</summary>
58
71
59
72
KCPP official KV quantized modes (modes 1 and 2 require Flash Attention) :
60
73
@@ -64,35 +77,30 @@ KCPP official KV quantized modes (modes 1 and 2 require Flash Attention) :
0 commit comments