Skip to content

Conversation

@createthis
Copy link
Contributor

Not working properly, but inference starts at least.

@github-actions github-actions bot added the python python script changes label Aug 20, 2025
@createthis
Copy link
Contributor Author

Apparently this does work properly, it's just how base models behave. Example usage:

./build/bin/llama-cli \
    --model /data2/DeepSeek-V3.1-Base-GGUF/q4_k_m/DeepSeek-V3.1-Base-Q4_K_M-00001-of-00009.gguf \
    --numa numactl \
    --threads 32 \
    --ctx-size 163840 \
    --n-gpu-layers 62 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
    -ot exps=CPU \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 0.3 \
    --min-p 0.0 \
    --log-colors \
    --flash-attn \
    --jinja \
    --mlock \
    --prio 2 \
    -p "You are a helpful assistant.<|User|>Hello, how are you?<|Assistant|>I'm doing well thanks! Yourself?<|User|>I'm doing great! Can you explain the laws of thermodynamics?<|Assistant|>" \
    -no-cnv \
    -ngl 0

Stolen from the model card at https://huggingface.co/bartowski/deepseek-ai_DeepSeek-V3.1-Base-Q4_K_M-GGUF

TIL. Closing, probably not useful until the Instruct comes out.

@createthis createthis closed this Aug 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python python script changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant