Support for generating test cases

[emou]

Hi,

I've been thinking of modifying gotests to support using the testing/quick package to generate test cases
and optimize them for code coverage.

Have you thought about something similar before and is this something you would accept as a feature in the project, or is it better to fork it?

My high-level idea is to:

• Use gotests to generate tests
• Use testing/quick to generate test inputs
• Measure the code coverage while running each test inputs
• Shrink the test inputs to a minimal set that provides high coverage

Not sure how complex generating test cases for complex/nested structs will be, but it should be fairly simple for basic types.

[cweill]

@emou - I like your idea. Generating test cases fits nicely with the goal of gotests: to make writing tests easier. If you could outline a design of how this would work, I would be happy to critique it and provide guidance.

One of the core challenges is figuring out how to seamlessly get testing/quick to generate test cases and then pass them to gotests. It's not immediately clear how we can get gotests to execute test code, since it's a binary separate from your code which just performs static analysis right now.

[cweill]

Proposal: LLM-Based Test Case Generation

I'd like to propose an approach to address this issue using local LLM models to generate actual test cases, complementing gotests' existing scaffolding.

The Problem

Currently, gotests generates excellent table-driven test scaffolding but leaves test case creation to developers:

tests := []struct {
    name string
    args args
    want int
}{
    // TODO: Add test cases.  ← Developer must fill this manually
}

Proposed Solution

Add an optional -ai flag that generates test cases using local LLM models:

$ gotests -ai -w math.go

Generated output:

tests := []struct {
    name string
    args args
    want int
}{
    {
        name: "positive_numbers",
        args: args{a: 2, b: 3},
        want: 5,
    },
    {
        name: "zero_values",
        args: args{a: 0, b: 0},
        want: 0,
    },
    {
        name: "negative_numbers",
        args: args{a: -1, b: -2},
        want: -3,
    },
}

Why This Approach?

Unlike property-based testing (original issue proposal), this:

• ✅ Stays within gotests' static analysis model - No runtime execution required
• ✅ Completely free - Uses local open-source models via Ollama
• ✅ Privacy-first - Code never leaves your machine
• ✅ Opt-in - Doesn't change existing behavior without the flag
• ✅ Fast - Tiny models generate cases in 2-3 seconds
• ✅ Works offline - No internet required after model download

---

MVP Design

Core Features

Single Provider: Ollama (local LLM runtime)
Default Model: qwen2.5-coder:0.5b (500MB, runs on any laptop)
Test Cases: 3 per function (normal, edge, error cases)
Validation: In-memory compilation check with retry logic

End-User Workflow

First-time setup (one-time):

# Install Ollama
$ curl -fsSL https://ollama.com/install.sh | sh

# Pull tiny code model (~400MB)
$ ollama pull qwen2.5-coder:0.5b

Ongoing usage:

# Same as before, just add -ai flag
$ gotests -ai -w myfile.go

Example: Error Handling

Source code:

func Divide(n, d int) (int, error) {
    if d == 0 {
        return 0, errors.New("division by zero")
    }
    return n / d, nil
}

Generated test:

func TestDivide(t *testing.T) {
    tests := []struct {
        name    string
        args    args
        want    int
        wantErr bool
    }{
        {
            name: "normal_division",
            args: args{n: 10, d: 2},
            want: 5,
            wantErr: false,
        },
        {
            name: "division_by_zero",
            args: args{n: 10, d: 0},
            want: 0,
            wantErr: true,  // AI correctly expects error
        },
        {
            name: "negative_divisor",
            args: args{n: 10, d: -2},
            want: -5,
            wantErr: false,
        },
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := Divide(tt.args.n, tt.args.d)
            if (err != nil) != tt.wantErr {
                t.Errorf("Divide() error = %v, wantErr %v", err, tt.wantErr)
                return
            }
            if got != tt.want {
                t.Errorf("Divide() = %v, want %v", got, tt.want)
            }
        })
    }
}

---

Architecture

Provider Interface (Extensible)

// internal/ai/provider.go
type Provider interface {
    GenerateTestCases(ctx context.Context, fn *models.Function) ([]TestCase, error)
    IsAvailable() bool
}

type TestCase struct {
    Name    string
    Args    map[string]string  // param name -> value
    Want    map[string]string  // return name -> expected value  
    WantErr bool
}

Compilation Validation

Uses Go's built-in go/parser and go/types packages for in-memory validation:

func ValidateGeneratedTest(testCode, pkgName string) error {
    fset := token.NewFileSet()
    
    // Parse
    file, err := parser.ParseFile(fset, "test.go", testCode, parser.AllErrors)
    if err != nil {
        return fmt.Errorf("parse error: %w", err)
    }
    
    // Type-check
    conf := types.Config{Importer: importer.Default()}
    _, err = conf.Check(pkgName, fset, []*ast.File{file}, nil)
    return err
}

Retry logic with feedback:

Attempt 1: LLM generates code
         → Validator: "undefined variable 'x'"
         → Retry with error feedback
Attempt 2: LLM fixes issue
         → Validation passes ✓

File Structure

internal/ai/
├── provider.go      # Provider interface (30 lines)
├── ollama.go        # Ollama implementation (200 lines)
├── prompt.go        # Prompt templates (50 lines)
├── parser.go        # Parse JSON responses (20 lines)
└── validator.go     # Compilation checking (50 lines)

Total new code: ~375 lines
Modified code: ~45 lines (flags, template injection)

Template Changes

Minimal modification to internal/render/templates/function.tmpl:

tests := []struct {
    // ... fields ...
}{
    {{if .AIGeneratedCases}}
    {{range .AIGeneratedCases}}
    {
        name: "{{.Name}}",
        args: args{ {{range .Args}}{{.Key}}: {{.Value}}, {{end}} },
        want: {{.Want}},
        wantErr: {{.WantErr}},
    },
    {{end}}
    {{else}}
    // TODO: Add test cases.
    {{end}}
}

---

Recommended Models

All run locally via Ollama:

Model	Size	Speed	Quality	Use Case
qwen2.5-coder:0.5b	400MB	Very Fast	Good	Default (runs anywhere)
qwen2.5-coder:1.5b	1.5GB	Fast	Better	Balance
qwen2.5-coder:7b	4.4GB	Medium	Best	High quality (needs GPU)
phi3:mini	2.2GB	Fast	Good	Alternative (Microsoft)
deepseek-coder:6.7b	3.8GB	Medium	Best	Code-specialized

Users can switch models with -ai-model=qwen2.5-coder:7b

---

What It Handles (90% Use Cases)

✅ Simple functions (math, string manipulation)
✅ Functions returning errors
✅ Basic types (int, string, bool, []byte)
✅ Edge cases (zero, nil, negative values)
✅ Boundary conditions
✅ Error scenarios

Future enhancements (remaining 10%):

• Complex custom types
• Functions with 10+ parameters
• Mocking/dependency injection
• Integration test scenarios

---

Implementation Phases

Phase 1: MVP (This Proposal)

• Provider interface
• Ollama implementation
• -ai flag
• In-memory validation
• Basic prompt engineering

Deliverable: Working -ai flag with qwen2.5-coder:0.5b

Phase 2: Multi-Provider

• Generic OpenAI-compatible provider
• OpenAI API support
• Claude API support
• Provider auto-detection

Phase 3: Quality & Polish

• Enhanced prompts for complex types
• Better error handling
• Unit tests for AI integration
• Documentation and examples

Phase 4: Advanced Features

• Interactive review mode
• Caching generated cases
• Fine-tuned prompts per construct
• Existing test enhancement

---

Why Ollama + Tiny Models?

Ollama is the Docker of LLMs:

• Easy installation (one curl command)
• Manages models locally
• Simple API
• Active development (Go-based project)
• Large model library

Tiny models (0.5B-7B parameters):

• Run on CPU (no GPU required for 0.5B)
• Fast inference (2-3 seconds)
• Good enough for basic test cases
• Completely free and open source
• Recent (2024-2025), trained on modern code

[s-kostyaev]

Why only 3 cases? It's not enough for complex functions with multiple branches. I think it should be customizable. Or 3 cases only for MVP?

[cweill]

@s-kostyaev Great question. The AI feature supports customizable test case counts. We provide two
flags:

• -ai-min-cases (default: 3) - minimum number of test cases to generate
• -ai-max-cases (default: 10) - maximum number of test cases to generate

The LLM determines how many test cases to generate within this range based on the function's
complexity. For complex functions with multiple branches, you can increase these limits:

gotests -ai -ai-min-cases=5 -ai-max-cases=20 -all path/to/your/file.go

This allows the AI to generate anywhere from 5-20 test cases depending on what it determines is
appropriate for the function. The validation limits are:

• Minimum: at least 1 case
• Maximum: up to 100 cases

So yes, it's fully customizable! The default 3-10 range was chosen as a reasonable starting point
that balances coverage with generation time, but you can adjust it based on your needs.