Kronk

This project lets you use Go for hardware accelerated local inference with llama.cpp directly integrated into your applications via the yzma module. Kronk provides a high-level API that feels similar to using an OpenAI compatible API.

Copyright 2025 Ardan Labs
[email protected]

Owner Information

Name:    Bill Kennedy
Company: Ardan Labs
Title:   Managing Partner
Email:   [email protected]
Twitter: goinggodotnet

Models

Kronk uses models in the GGUF format supported by llama.cpp. You can find many models in GGUF format on Hugging Face (over 147k at last count):

https://huggingface.co/models?library=gguf&sort=trending

Support

Kronk currently has support for over 94% of llama.cpp functionality thanks to yzma. See the yzma ROADMAP.md for the complete list.

You can use multimodal models (image/audio) and text language models with full hardware acceleration on Linux, on macOS, and on Windows.

OS	CPU	GPU
Linux	amd64, arm64	CUDA, Vulkan, HIP, ROCm, SYCL
macOS	arm64	Metal
Windows	amd64	CUDA, Vulkan, HIP, SYCL, OpenCL

Whenever there is a new release of llama.cpp, the tests for yzma are run automatically. Kronk runs tests once a day and will check for updates to llama.cpp. This helps us stay up to date with the latest code and models.

Examples

There are examples in the examples direction:

The first time you run these programs the system will download and install the model and libraries.

CHAT - This example shows you how to create a simple chat application against an inference model using kronk. Thanks to Kronk and yzma, reasoning and tool calling is enabled.
$ make example-chat

EMBEDDING - This example shows you how to use an embedding model.
$ make example-embedding

QUESTION - This example shows you a basic program of using Kronk to ask a model a question.
$ make example-question

VISION - This example shows you how to execute a simple prompt against a vision model.
$ make example-vision

WEB - This example shows you a web service that provides a chat endpoint for asking questions to a model with a browser based chat UI.
$ make example-web

You can find more examples in the ArdanLabs AI training repo at Example13.

Sample - Question Example

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"github.com/ardanlabs/kronk"
	"github.com/ardanlabs/kronk/examples/install"
	"github.com/ardanlabs/kronk/model"
	"github.com/hybridgroup/yzma/pkg/download"
)

const (
	modelURL       = "https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q8_0.gguf?download=true"
	libPath        = "tests/libraries"
	modelPath      = "tests/models"
	modelInstances = 1
)

func main() {
	if err := run(); err != nil {
		fmt.Printf("\nERROR: %s\n", err)
		os.Exit(1)
	}
}

func run() error {
	modelFile, err := installSystem()
	if err != nil {
		return fmt.Errorf("unable to installation system: %w", err)
	}

	if err := kronk.Init(libPath, kronk.LogSilent); err != nil {
		return fmt.Errorf("unable to init kronk: %w", err)
	}

	krn, err := kronk.New(modelInstances, model.Config{
		ModelFile:  modelFile,
	})

	if err != nil {
		return fmt.Errorf("unable to create inference model: %w", err)
	}

	defer func() {
		fmt.Println("\nUnloading Kronk")
		if err := krn.Unload(context.Background()); err != nil {
			fmt.Printf("failed to unload model: %v", err)
		}
	}()

	// -------------------------------------------------------------------------

	ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
	defer cancel()

	question := "Hello model"

	fmt.Println()
	fmt.Println("QUESTION:", question)
	fmt.Println()

	d := model.D{
		"messages": model.DocumentArray(
			model.ChatMessage("user", question),
		),
	}

	params := model.Params{
		Temperature: 0.7,
		TopP:        0.9,
		TopK:        40,
		MaxTokens:   2048,
	}

	ch, err := krn.ChatStreaming(ctx, params, d)
	if err != nil {
		return fmt.Errorf("chat streaming: %w", err)
	}

	// -------------------------------------------------------------------------

	var reasoning bool

	for resp := range ch {
		switch resp.Choice[0].FinishReason {
		case model.FinishReasonError:
			return fmt.Errorf("error from model: %s", resp.Choice[0].Delta.Content)

		case model.FinishReasonStop:
			return nil

		default:
			if resp.Choice[0].Delta.Reasoning != "" {
				reasoning = true
				fmt.Printf("\u001b[91m%s\u001b[0m", resp.Choice[0].Delta.Reasoning)
				continue
			}

			if reasoning {
				reasoning = false
				fmt.Println()
				continue
			}

			fmt.Printf("%s", resp.Choice[0].Delta.Content)
		}
	}

	return nil
}

func installSystem() (string, error) {
	if err := install.llama.cpp(libPath, download.CPU, true); err != nil {
		return "", fmt.Errorf("unable to install llama.cpp: %w", err)
	}

	modelFile, err := install.Model(modelURL, modelPath)
	if err != nil {
		return "", fmt.Errorf("unable to install model: %w", err)
	}

	return modelFile, nil
}

This example can produce the following output:

$ make example-question
export LD_LIBRARY_PATH=:tests/libraries && \
	CGO_ENABLED=0 go run examples/question/main.go

Output:

- check llama.cpp installation: ✓
  - latest version : b7211
  - current version: b7211
- check "Qwen3-8B-Q8_0" installation: ✓

QUESTION: Hello model

Okay, the user said "Hello model." I need to respond appropriately. First, I should acknowledge their greeting. Since they mentioned "model," maybe they're referring to me as a language model. I should clarify that I'm Qwen, a large language model developed by Alibaba Cloud. I should keep the response friendly and open-ended to encourage them to ask questions or share what they need help with. Let me make sure the tone is welcoming and not too formal. Also, check for any possible misunderstandings. They might be testing if I recognize the term "model," so confirming my identity as Qwen is important. Alright, time to put it all together in a concise and friendly manner.

! I'm Qwen, a large language model developed by Alibaba Cloud. How can I assist you today? 😊
Unloading Kronk