Skip to content

Commit 642c2b0

Browse files
ThireusThireus
authored
Merge branch 'ikawrakow:main' into main
2 parents 607e01e + 1b05210 commit 642c2b0

File tree

7 files changed

+205
-21
lines changed

7 files changed

+205
-21
lines changed

‎README.md‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,15 @@ This repository is a fork of [llama.cpp](https://github.com/ggerganov/llama.cpp)
1010

1111
### Model Support
1212

13-
LlaMA-3-Nemotron [PR 377](https://github.com/ikawrakow/ik_llama.cpp/pull/377), Qwen3 [PR 355](https://github.com/ikawrakow/ik_llama.cpp/pull/355), GLM-4 [PR 344](https://github.com/ikawrakow/ik_llama.cpp/pull/344), Command-A [PR 341](https://github.com/ikawrakow/ik_llama.cpp/pull/341), bitnet-b1.58-2B-4T [PR 337](https://github.com/ikawrakow/ik_llama.cpp/pull/337), LLaMA-4 [PR 321](https://github.com/ikawrakow/ik_llama.cpp/pull/321), Gemma3 [PR 276](https://github.com/ikawrakow/ik_llama.cpp/pull/276), DeepSeek-V3 [PR 176](https://github.com/ikawrakow/ik_llama.cpp/pull/176)
13+
LlaMA-3-Nemotron [PR 377](https://github.com/ikawrakow/ik_llama.cpp/pull/377), Qwen3 [PR 355](https://github.com/ikawrakow/ik_llama.cpp/pull/355), GLM-4 [PR 344](https://github.com/ikawrakow/ik_llama.cpp/pull/344), Command-A [PR 341](https://github.com/ikawrakow/ik_llama.cpp/pull/341), bitnet-b1.58-2B-4T [PR 337](https://github.com/ikawrakow/ik_llama.cpp/pull/337), LLaMA-4 [PR 321](https://github.com/ikawrakow/ik_llama.cpp/pull/321), Gemma3 [PR 276](https://github.com/ikawrakow/ik_llama.cpp/pull/276), DeepSeek-V3 [PR 176](https://github.com/ikawrakow/ik_llama.cpp/pull/176), Kimi-2 [PR 609](https://github.com/ikawrakow/ik_llama.cpp/pull/609), dots.llm1 [PR 573](https://github.com/ikawrakow/ik_llama.cpp/pull/573), Hunyuan [PR 565](https://github.com/ikawrakow/ik_llama.cpp/pull/565)
1414

1515
### Quantization
1616

1717
#### Quantization additions
1818

19-
##### Trellis quants (`IQ2_KT`, `IQ3_KT`, `IQ4_KT`)
19+
##### Trellis quants (`IQ1_KT`, `IQ2_KT`, `IQ3_KT`, `IQ4_KT`)
2020

21-
Information and the original CUDA implementation in [PR 113](https://github.com/ikawrakow/ik_llama.cpp/pull/113). Additional implementations: Metal [PR 475](https://github.com/ikawrakow/ik_llama.cpp/pull/475), Neon [PR 471](https://github.com/ikawrakow/ik_llama.cpp/pull/471), CPU [PR 441](https://github.com/ikawrakow/ik_llama.cpp/pull/441)
21+
Information and the original CUDA implementation in [PR 113](https://github.com/ikawrakow/ik_llama.cpp/pull/113). Additional implementations: Metal [PR 475](https://github.com/ikawrakow/ik_llama.cpp/pull/475), Neon [PR 471](https://github.com/ikawrakow/ik_llama.cpp/pull/471), CPU [PR 441](https://github.com/ikawrakow/ik_llama.cpp/pull/441). `IQ1_KT` was added more recently in [PR 616](https://github.com/ikawrakow/ik_llama.cpp/pull/616). Note: these are base on a novel, integer-base trellis, which allows to achieve reasonable CPU performance, see [PR 529](https://github.com/ikawrakow/ik_llama.cpp/pull/529) and PRs quoted there for details.
2222

2323
##### IQK quants
2424

@@ -28,12 +28,16 @@ Initial implementations (Zen4, AVX2, NEON): `IQ5_KS_R4` [PR 426](https://github.
2828

2929
Cuda implementations: `IQ4_KS_R4` and `IQ5_KS_R4` [PR 493](https://github.com/ikawrakow/ik_llama.cpp/pull/493), `IQ1_S_R4` [PR 492](https://github.com/ikawrakow/ik_llama.cpp/pull/492), `IQ1_M_R4` [PR 494](https://github.com/ikawrakow/ik_llama.cpp/pull/494). `IQ4_KS_R4` and `IQ5_KS_R4` [PR 462](https://github.com/ikawrakow/ik_llama.cpp/pull/462), `IQ2_K_R4`, `IQ3_K_R4`, `IQ4_K_R4`, `IQ5_K_R4` [PR 461](https://github.com/ikawrakow/ik_llama.cpp/pull/461), `IQ4_K, IQ5_K, IQ6_K` [PR 417](https://github.com/ikawrakow/ik_llama.cpp/pull/417), `IQ2_KS, IQ2_K, IQ3_K` [PR 418](https://github.com/ikawrakow/ik_llama.cpp/pull/417)
3030

31+
`IQ2_KL` is a more recent addition in [PR 602](https://github.com/ikawrakow/ik_llama.cpp/pull/602)
32+
3133
#### Quantization improvements
3234

3335
`IQ1_M` [PR 327](https://github.com/ikawrakow/ik_llama.cpp/pull/327), `IQ2_XS` [PR 312](https://github.com/ikawrakow/ik_llama.cpp/pull/312), `Q2_K, Q4_K, Q5_K, Q4_1, Q5_1` [PR 302](https://github.com/ikawrakow/ik_llama.cpp/pull/302), `Q4_0, Q5_0, Q6_0, Q3_K, Q6_K, IQ4_XS, IQ4_NL` [PR 295](https://github.com/ikawrakow/ik_llama.cpp/pull/295)
3436

3537
#### Quantization performance improvements
3638

39+
* Much faster CPU prompt processing for all non-interleaved quants. Initial idea in [PR 515](https://github.com/ikawrakow/ik_llama.cpp/pull/515) and [PR 531](https://github.com/ikawrakow/ik_llama.cpp/pull/531), with many follow up PRs to apply to all quantization types for the 3 supported CPU platforms.
40+
* All quantization types now have quantized matrix multiplication CUDA kernels, see [PR 557](https://github.com/ikawrakow/ik_llama.cpp/pull/515) and several others
3741
* Faster CPU prompt processing for Trellis quants and MoE models. [PR 488](https://github.com/ikawrakow/ik_llama.cpp/pull/488)
3842
* Trellis quants: faster CPU prompt processing [PR 482](https://github.com/ikawrakow/ik_llama.cpp/pull/482).
3943
* Minor (~2%) `iq2_ks` TG performance improvement on CUDA [PR 468](https://github.com/ikawrakow/ik_llama.cpp/pull/468)
@@ -43,6 +47,8 @@ Cuda implementations: `IQ4_KS_R4` and `IQ5_KS_R4` [PR 493](https://github.com/i
4347

4448
### Features
4549

50+
* Function call support [PR 628](https://github.com/ikawrakow/ik_llama.cpp/pull/628)
51+
* Webui: New Features for Conversations, Settings, and Chat Messages [PR 618](https://github.com/ikawrakow/ik_llama.cpp/pull/618)
4652
* Legacy quants conversion schemes in `convert_hf_to_gguf.py` [PR 449](https://github.com/ikawrakow/ik_llama.cpp/pull/449), `Q6_0` in [PR 483](https://github.com/ikawrakow/ik_llama.cpp/pull/483)
4753
* June 8 2025: Webui updated (legacy still available when `--path ./examples/server/public_legacy` is passed) [PR 481](https://github.com/ikawrakow/ik_llama.cpp/pull/481)
4854
* June 8 2025: RPC improvements [PR 480](https://github.com/ikawrakow/ik_llama.cpp/pull/480)
@@ -62,6 +68,7 @@ Cuda implementations: `IQ4_KS_R4` and `IQ5_KS_R4` [PR 493](https://github.com/i
6268

6369
### Performance improvements
6470

71+
* Better GPU offload strategy for MoE models when using hybrid HPU/CPU inference, see [PR 520](https://github.com/ikawrakow/ik_llama.cpp/pull/520)
6572
* May 13 2025: Better CPU FA performance for DeepSeek-Lite. [PR 410](https://github.com/ikawrakow/ik_llama.cpp/pull/410)
6673
* May 11 2025: Slightly faster flash attention for DeepSeek models on CUDA, along with extending compatibility to Touring or newer GPUs. [PR 408](https://github.com/ikawrakow/ik_llama.cpp/pull/408)
6774
* May 4 2025: Significant token generation performance improvement on CUDA with Flash Attention for GQA models. For details and benchmarks. [PR 370](https://github.com/ikawrakow/ik_llama.cpp/pull/370)

‎ggml/src/ggml-cuda/mmq.cu‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,9 @@ void ggml_cuda_op_mul_mat_q(
100100
case GGML_TYPE_IQ3_KS:
101101
mul_mat_q_case<GGML_TYPE_IQ3_KS>(ctx, args, stream);
102102
break;
103+
case GGML_TYPE_IQ4_KSS:
104+
mul_mat_q_case<GGML_TYPE_IQ4_KSS>(ctx, args, stream);
105+
break;
103106
case GGML_TYPE_IQ4_KS:
104107
mul_mat_q_case<GGML_TYPE_IQ4_KS>(ctx, args, stream);
105108
break;
@@ -209,6 +212,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11) {
209212
case GGML_TYPE_IQ4_NL:
210213
case GGML_TYPE_IQ2_KL:
211214
case GGML_TYPE_IQ3_KS:
215+
case GGML_TYPE_IQ4_KSS:
212216
case GGML_TYPE_IQ4_KS:
213217
case GGML_TYPE_IQ4_KS_R4:
214218
case GGML_TYPE_IQ5_KS:

‎ggml/src/ggml-cuda/mmq.cuh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,7 @@ static mmq_q8_1_ds_layout mmq_get_q8_1_ds_layout(const ggml_type type_x) {
9191
case GGML_TYPE_IQ2_KL:
9292
case GGML_TYPE_IQ3_KS:
9393
case GGML_TYPE_IQ3_K_R4:
94+
case GGML_TYPE_IQ4_KSS:
9495
case GGML_TYPE_IQ4_KS:
9596
case GGML_TYPE_IQ4_KS_R4:
9697
case GGML_TYPE_IQ4_K:
@@ -205,6 +206,7 @@ static constexpr __host__ __device__ tile_x_sizes mmq_get_dp4a_tile_x_sizes(ggml
205206
case GGML_TYPE_IQ4_NL : return MMQ_DP4A_TXS_Q8_0;
206207
case GGML_TYPE_IQ2_KL : return MMQ_DP4A_TXS_Q8_0;
207208
case GGML_TYPE_IQ3_KS : return MMQ_DP4A_TXS_Q8_0;
209+
case GGML_TYPE_IQ4_KSS : return MMQ_DP4A_TXS_Q8_0;
208210
case GGML_TYPE_IQ4_KS : return MMQ_DP4A_TXS_Q8_0;
209211
case GGML_TYPE_IQ4_KS_R4 : return MMQ_DP4A_TXS_Q8_0;
210212
case GGML_TYPE_IQ5_KS : return MMQ_DP4A_TXS_Q8_0;
@@ -263,6 +265,7 @@ static constexpr __host__ __device__ int mmq_get_mma_tile_x_k(ggml_type type) {
263265
case GGML_TYPE_IQ4_NL : return MMQ_MMA_TILE_X_K_Q8_0;
264266
case GGML_TYPE_IQ2_KL : return MMQ_MMA_TILE_X_K_Q8_0;
265267
case GGML_TYPE_IQ3_KS : return MMQ_MMA_TILE_X_K_Q8_0;
268+
case GGML_TYPE_IQ4_KSS : return MMQ_MMA_TILE_X_K_Q8_0;
266269
case GGML_TYPE_IQ4_KS : return MMQ_MMA_TILE_X_K_Q8_0;
267270
case GGML_TYPE_IQ4_KS_R4 : return MMQ_MMA_TILE_X_K_Q8_0;
268271
case GGML_TYPE_IQ5_KS : return MMQ_MMA_TILE_X_K_Q8_0;
@@ -4164,6 +4167,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_IQ4_NL);
41644167
extern DECL_MMQ_CASE(GGML_TYPE_IQ4_XS);
41654168
extern DECL_MMQ_CASE(GGML_TYPE_IQ2_KL);
41664169
extern DECL_MMQ_CASE(GGML_TYPE_IQ3_KS);
4170+
extern DECL_MMQ_CASE(GGML_TYPE_IQ4_KSS);
41674171
extern DECL_MMQ_CASE(GGML_TYPE_IQ4_KS);
41684172
extern DECL_MMQ_CASE(GGML_TYPE_IQ4_KS_R4);
41694173
extern DECL_MMQ_CASE(GGML_TYPE_IQ5_KS_R4);

‎ggml/src/ggml-cuda/template-instances/mmq-instance-iq4_kss.cu‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
#include "../mmq.cuh"
2+
3+
template <int mmq_y, int nwarps, bool need_check> static __device__ __forceinline__ void load_tiles_iq4_kss(
4+
const char * __restrict__ x, int * __restrict__ x_tile, const int & kbx0, const int & i_max, const int & stride) {
5+
6+
#ifdef INT8_MMA_AVAILABLE
7+
int * x_qs = (int *) x_tile;
8+
float * x_df = (float *) (x_qs + WARP_SIZE*2);
9+
#else
10+
constexpr tile_x_sizes txs = mmq_get_dp4a_tile_x_sizes(GGML_TYPE_IQ4_XS, mmq_y);
11+
int * x_qs = (int *) x_tile;
12+
float * x_df = (float *) (x_qs + txs.qs);
13+
#endif // INT8_MMA_AVAILABLE
14+
15+
const int kqsx = threadIdx.x / 4;
16+
17+
uint32_t aux32[2];
18+
auto a8 = (const uint8_t *)aux32;
19+
20+
#pragma unroll
21+
for (int i0 = 0; i0 < mmq_y; i0 += 4*nwarps) {
22+
int i = i0 + 4*threadIdx.y + threadIdx.x%4;
23+
24+
if (need_check) {
25+
i = min(i, i_max);
26+
}
27+
28+
const float * dptr = (const float *)(x + i*stride);
29+
const block_iq4_kss * bxi = (const block_iq4_kss *)(dptr + 1) + kbx0;
30+
const uint32_t * q4 = bxi->qs + 4*kqsx;
31+
uint32_t s32 = (q4[0] & 0x00010001) | ((q4[1] & 0x00010001) << 2) | ((q4[2] & 0x00010001) << 4) | ((q4[3] & 0x00010001) << 6);
32+
uint8_t ls = (s32 | (s32 >> 15)) & 0xff;
33+
34+
auto values = iq4k_table + ((ls & 1) << 8);
35+
36+
#pragma unroll
37+
for (int j = 0; j < 4; ++j) {
38+
uint32_t val = q4[j] & 0xfffefffe;
39+
val = val ^ (val >> 1);
40+
aux32[0] = (val >> 0) & 0x0f0f0f0f;
41+
aux32[1] = (val >> 4) & 0x0f0f0f0f;
42+
#ifdef INT8_MMA_AVAILABLE
43+
x_qs[i*MMQ_MMA_TILE_X_K_Q8_0 + 8*kqsx + j + 0] = int_from_table_x(a8+0, values);
44+
x_qs[i*MMQ_MMA_TILE_X_K_Q8_0 + 8*kqsx + j + 4] = int_from_table_x(a8+4, values);
45+
#else
46+
x_qs[i*(2*WARP_SIZE + 1) + 8*kqsx + j + 0] = int_from_table_x(a8+0, values);
47+
x_qs[i*(2*WARP_SIZE + 1) + 8*kqsx + j + 4] = int_from_table_x(a8+4, values);
48+
#endif // INT8_MMA_AVAILABLE
49+
}
50+
#ifdef INT8_MMA_AVAILABLE
51+
x_df[i*MMQ_MMA_TILE_X_K_Q8_0 + kqsx] = dptr[0] * ((ls & 254) - 127);
52+
#else
53+
x_df[i*(WARP_SIZE/4) + i/4 + kqsx] = dptr[0] * ((ls & 254) - 127);
54+
#endif // INT8_MMA_AVAILABLE
55+
}
56+
57+
}
58+
59+
60+
template <int mmq_x, int mmq_y, int nwarps, bool need_check>
61+
struct mmq_type_traits<mmq_x, mmq_y, nwarps, need_check, GGML_TYPE_IQ4_KSS> {
62+
static constexpr load_tiles_mmq_t load_tiles = load_tiles_iq4_kss<mmq_y, nwarps, need_check>;
63+
static constexpr vec_dot_mmq_t vec_dot_mma = vec_dot_q8_0_q8_1_mma<mmq_x, mmq_y, nwarps, MMQ_Q8_1_DS_LAYOUT_D4>;
64+
static constexpr vec_dot_mmq_t vec_dot_dp4a = vec_dot_q8_0_q8_1_dp4a<mmq_x, mmq_y, nwarps>;
65+
};
66+
67+
DECL_MMQ_CASE(GGML_TYPE_IQ4_KSS);
68+

‎ggml/src/iqk/iqk_gemm_iqk_quants.cpp‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2706,6 +2706,60 @@ void iqk_convert_iq3_ks_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
27062706
}
27072707
}
27082708

2709+
void iqk_convert_iq4_kss_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int nrc_x) {
2710+
GGML_ASSERT(n%QK_K == 0);
2711+
GGML_ASSERT(nrc_x%8 == 0);
2712+
2713+
int nb = n/QK_K;
2714+
2715+
const block_iq4_kss * x8[8];
2716+
2717+
block_q8_k_r8 * y = (block_q8_k_r8 *)vy;
2718+
2719+
__m256i values[2];
2720+
{
2721+
auto v1 = _mm_loadu_si128((const __m128i *)iq4k_values+0);
2722+
auto v2 = _mm_loadu_si128((const __m128i *)iq4k_values+1);
2723+
values[0] = MM256_SET_M128I(v1, v1);
2724+
values[1] = MM256_SET_M128I(v2, v2);
2725+
}
2726+
2727+
float drow[8];
2728+
float dnew[8];
2729+
int16_t ls[16];
2730+
2731+
__m256i xv[8];
2732+
uint32_t block[8];
2733+
2734+
for (int ix = 0; ix < nrc_x; ix += 8) {
2735+
for (int k = 0; k < 8; ++k) {
2736+
const float * dptr = (const float *)((const char *)vx + (ix + k)*bx);
2737+
drow[k] = dptr[0];
2738+
x8[k] = (const block_iq4_kss *)(dptr + 1);
2739+
}
2740+
auto vd = _mm256_loadu_ps(drow);
2741+
for (int i = 0; i < nb; ++i) {
2742+
for (int k = 0; k < 8; ++k) {
2743+
for (int ib32 = 0; ib32 < 8; ++ib32) {
2744+
auto val = _mm_loadu_si128((const __m128i *)x8[k][i].qs+ib32);
2745+
auto val_q = _mm_and_si128(val, _mm_set1_epi32(0xfffefffe));
2746+
val_q = _mm_xor_si128(val_q, _mm_srli_epi16(val_q, 1));
2747+
xv[ib32] = _mm256_and_si256(MM256_SET_M128I(_mm_srli_epi16(val_q, 4), val_q), _mm256_set1_epi8(0xf));
2748+
auto q4 = x8[k][i].qs + 4*ib32;
2749+
uint32_t s32 = (q4[0] & 0x00010001) | ((q4[1] & 0x00010001) << 2) | ((q4[2] & 0x00010001) << 4) | ((q4[3] & 0x00010001) << 6);
2750+
uint8_t s8 = (s32 | (s32 >> 15)) & 0xff;
2751+
//auto val_s = _mm_madd_epi16(_mm_and_si128(val, _mm_set1_epi32(0x00010001)), _mm_set1_epi64x(0x0008000400020001));
2752+
ls[2*ib32+0] = ls[2*ib32+1] = ((s8 & 254) - 127);
2753+
xv[ib32] = _mm256_shuffle_epi8(values[s8 & 1], xv[ib32]);
2754+
}
2755+
dnew[k] = convert_to_q8_k_r8(k, 1.f/127, xv, ls, block, y[i].qs);
2756+
}
2757+
_mm_storeu_si128((__m128i *)y[i].d, _mm256_cvtps_ph(_mm256_mul_ps(vd, _mm256_loadu_ps(dnew)), _MM_ROUND_NEAREST));
2758+
}
2759+
y += nb;
2760+
}
2761+
}
2762+
27092763
void iqk_convert_iq4_ks_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int nrc_x) {
27102764
GGML_ASSERT(n%QK_K == 0);
27112765
GGML_ASSERT(nrc_x%8 == 0);
@@ -3132,6 +3186,7 @@ bool iqk_convert_iqk_quants_q80_r8(int type, int n, const void * vx, size_t bx,
31323186
case GGML_TYPE_IQ2_KL : iqk_convert_iq2_kl_q8_k_r8(n, vx, bx, vy, nrc_x); break;
31333187
case GGML_TYPE_IQ3_KS : iqk_convert_iq3_ks_q8_k_r8(n, vx, bx, vy, nrc_x); break;
31343188
case GGML_TYPE_IQ3_K : iqk_convert_iq3_k_q8_k_r8 (n, vx, bx, vy, nrc_x); break;
3189+
case GGML_TYPE_IQ4_KSS: iqk_convert_iq4_kss_q8_k_r8(n, vx, bx, vy, nrc_x); break;
31353190
case GGML_TYPE_IQ4_KS : iqk_convert_iq4_ks_q8_k_r8(n, vx, bx, vy, nrc_x); break;
31363191
case GGML_TYPE_IQ4_K : iqk_convert_iq4_k_q8_k_r8 (n, vx, bx, vy, nrc_x); break;
31373192
case GGML_TYPE_IQ5_KS : iqk_convert_iq5_ks_q8_k_r8(n, vx, bx, vy, nrc_x); break;
@@ -4718,6 +4773,57 @@ void iqk_convert_iq2_kl_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
47184773
}
47194774
}
47204775

4776+
void iqk_convert_iq4_kss_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int nrc_x) {
4777+
GGML_ASSERT(n%QK_K == 0);
4778+
GGML_ASSERT(nrc_x%8 == 0);
4779+
4780+
int nb = n/QK_K;
4781+
4782+
const block_iq4_kss * x8[8];
4783+
4784+
block_q8_k_r8 * y = (block_q8_k_r8 *)vy;
4785+
4786+
auto values = vld1q_s8_x2(iq4k_values);
4787+
4788+
float drow[8];
4789+
float dnew[8];
4790+
int8_t ls[16];
4791+
4792+
int8x16x2_t xv[8];
4793+
uint32_t block[8];
4794+
4795+
for (int ix = 0; ix < nrc_x; ix += 8) {
4796+
for (int k = 0; k < 8; ++k) {
4797+
const float * dptr = (const float *)((const char *)vx + (ix + k)*bx);
4798+
drow[k] = dptr[0];
4799+
x8[k] = (const block_iq4_kss *)(dptr + 1);
4800+
}
4801+
auto vd = vld1q_f32_x2(drow);
4802+
for (int i = 0; i < nb; ++i) {
4803+
for (int k = 0; k < 8; ++k) {
4804+
for (int ib32 = 0; ib32 < 8; ++ib32) {
4805+
auto q4 = x8[k][i].qs + 4*ib32;
4806+
uint32_t s32 = (q4[0] & 0x00010001) | ((q4[1] & 0x00010001) << 2) | ((q4[2] & 0x00010001) << 4) | ((q4[3] & 0x00010001) << 6);
4807+
uint8_t s8 = (s32 | (s32 >> 15)) & 0xff;
4808+
ls[2*ib32+0] = ls[2*ib32+1] = ((s8 & 254) - 127);
4809+
auto val16 = vandq_u16(vld1q_u16((const uint16_t *)q4), vdupq_n_u16(0xfffe));
4810+
auto val8 = vreinterpretq_u8_u16(veorq_u16(val16, vshrq_n_u16(val16, 1)));
4811+
auto& block_values = values.val[s8 & 1];
4812+
xv[ib32].val[0] = vqtbl1q_s8(block_values, vandq_u8(val8, vdupq_n_u8(0xf)));
4813+
xv[ib32].val[1] = vqtbl1q_s8(block_values, vshrq_n_u8(val8, 4));
4814+
}
4815+
dnew[k] = convert_to_q8_k_r8(1.f/127, xv, ls, block, (uint32_t *)y[i].qs + k);
4816+
}
4817+
auto d = vld1q_f32_x2(dnew);
4818+
d.val[0] = vmulq_f32(d.val[0], vd.val[0]);
4819+
d.val[1] = vmulq_f32(d.val[1], vd.val[1]);
4820+
vst1_f16((float16_t *)y[i].d + 0, vcvt_f16_f32(d.val[0]));
4821+
vst1_f16((float16_t *)y[i].d + 4, vcvt_f16_f32(d.val[1]));
4822+
}
4823+
y += nb;
4824+
}
4825+
}
4826+
47214827
void iqk_convert_iq4_ks_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int nrc_x) {
47224828
GGML_ASSERT(n%QK_K == 0);
47234829
GGML_ASSERT(nrc_x%8 == 0);
@@ -5163,6 +5269,7 @@ bool iqk_convert_iqk_quants_q80_r8(int type, int n, const void * vx, size_t bx,
51635269
case GGML_TYPE_IQ2_KL : iqk_convert_iq2_kl_q8_k_r8(n, vx, bx, vy, nrc_x); break;
51645270
case GGML_TYPE_IQ3_KS : iqk_convert_iq3_ks_q8_k_r8(n, vx, bx, vy, nrc_x); break;
51655271
case GGML_TYPE_IQ3_K : iqk_convert_iq3_k_q8_k_r8 (n, vx, bx, vy, nrc_x); break;
5272+
case GGML_TYPE_IQ4_KSS: iqk_convert_iq4_kss_q8_k_r8(n, vx, bx, vy, nrc_x); break;
51665273
case GGML_TYPE_IQ4_KS : iqk_convert_iq4_ks_q8_k_r8(n, vx, bx, vy, nrc_x); break;
51675274
case GGML_TYPE_IQ4_K : iqk_convert_iq4_k_q8_k_r8 (n, vx, bx, vy, nrc_x); break;
51685275
case GGML_TYPE_IQ5_KS : iqk_convert_iq5_ks_q8_k_r8(n, vx, bx, vy, nrc_x); break;

‎ggml/src/iqk/iqk_mul_mat.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,7 @@ struct MulMat {
255255
case GGML_TYPE_IQ3_KS : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
256256
case GGML_TYPE_IQ3_K : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
257257
case GGML_TYPE_IQ4_KS : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
258+
case GGML_TYPE_IQ4_KSS: return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
258259
case GGML_TYPE_IQ4_K : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
259260
case GGML_TYPE_IQ5_KS : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
260261
case GGML_TYPE_IQ5_K : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
@@ -301,6 +302,7 @@ struct MulMat {
301302
case GGML_TYPE_IQ2_KS : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
302303
case GGML_TYPE_IQ2_KL : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
303304
case GGML_TYPE_IQ3_KS : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
305+
case GGML_TYPE_IQ4_KSS: return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
304306
case GGML_TYPE_IQ4_KS : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
305307
case GGML_TYPE_IQ5_KS : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;
306308
case GGML_TYPE_IQ2_K : return nrc_y >= 32 ? GGML_TYPE_Q8_K_R8 : type;

0 commit comments

Comments
 (0)