Skip to content

Draft: Integrate TQ2_0 into vulkan #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 23 commits into
base: temp-latest
Choose a base branch
from
Open

Draft: Integrate TQ2_0 into vulkan #22

wants to merge 23 commits into from

Conversation

makaveli10
Copy link

Convert 1bitLLM/bitnet_b1_58-large to TQ2_0:

python convert_hf_to_gguf.py 1bitLLM/bitnet_b1_58-large --remote --outtype f32
./build_vulkan/bin/llama-quantize 1bitLLM-bitnet_b1_58-large-f32.gguf TQ2_0

Run inference on vulkan with:

./build_vulkan/bin/llama-cli -m ggml-model-TQ2_0.gguf -if -p "Hello" -ngl 999

makaveli10 and others added 23 commits July 30, 2025 16:58
@makaveli10
Add lora finetuning from adapter
d8ba38a
@makaveli10
Add: create new lora adapter for target modules to finetune if no lor…
c8ea7b2 
…a is provided
@makaveli10
Fix identical loss over epochs; fix garbage lora initization
69c464a 
Signed-off-by: vineet <[email protected]>
@makaveli10
Remove lora training from finetune.cpp
ab0dae2 
Signed-off-by: vineet <[email protected]>
@makaveli10
Add adapter saving & other lora target modules
1b6e22d 
Signed-off-by: vineet <[email protected]>
@makaveli10
Add finetune-lora for lora finetuning in examples
c61a5af 
Signed-off-by: vineet <[email protected]>
@makaveli10
Add dequantization to out_prod cuda kernel
7799375 
Signed-off-by: vineet <[email protected]>
@makaveli10
Update README with finetune-lora
cbd4e1f 
Signed-off-by: vineet <[email protected]>
Vulkan: add support for fp32 OUT_PROD op
cde18d5
CPU: add support for fp16_fp32 OUT_PROD op
b6b30de
Vulkan: add support for f16_f32 OUT_PROD op
5d4d767
@zoq
Vulkan: Add Q4_0/Q8_0 OUT_PROD Vulkan support
e55225a
@makaveli10
vulkan: Add initial cross entropy loss backward shader
c81f689 
Signed-off-by: vineet <[email protected]>
@makaveli10
vulkan: Fix cross-entropy-loss-back dispatch size and wg denominator
fc3ae66 
Signed-off-by: vineet <[email protected]>
@makaveli10
vulkan: Change uint32 cast to int32 for outprod; allows android compi…
055e30f 
…lation

Signed-off-by: vineet <[email protected]>
@makaveli10
vulkan: Deallocate memory after destroying buffer
ae5ae40
@makaveli10
vulkan: Set specialization constants to { 0 } for out_prod
794c37f 
This fixes the vkDeviceLostError on Mali
@makaveli10
vulkan: Set out_prod pipeline disable_robustness to true
36cfb76
@makaveli10
Fix out_prod; vulkan ci issues
ea37326
@zoq @makaveli10
Add GEGLU backward (Vulkan) to enable Gemma training.
4614f2d
@makaveli10
ggml-vulkan: Add TQ2_0 dequantize and mul_mat vec
7d0f6db
@makaveli10
ggml-vulkan: Enable coopmat support for Android
794f94c
@makaveli10
ggml-vulkan: Add mul_mm path for TQ2_0
e05c62a
@makaveli10 makaveli10 changed the title Integrate TQ2_0 into vulkan Draft: Integrate TQ2_0 into vulkan Sep 29, 2025
@zoq zoq mentioned this pull request Oct 9, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

examples ggml Nvidia GPU testing Vulkan

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@makaveli10 @zoq