refactor: Replace static SQLite with dynamic linking and add JSONL default storage (#346)

Replaces the statically linked SQLite dependency with dynamic linking
using `github.com/mattn/go-sqlite3`. Introduces JSONL as the new default
storage backend for conversations, providing zero-dependency
persistence. Updates CI/CD workflows and documentation accordingly.

## Key Changes:

1. **SQLite Migration**: Switched from `modernc.org/sqlite` (pure-Go) to
`github.com/mattn/go-sqlite3` (CGO-based)
2. **New Default Storage**: JSONL storage is now the default instead of
SQLite
3. **Build Changes**: Added CGO_ENABLED=1 and system dependencies for
SQLite
4. **Documentation**: Updated docs to reflect JSONL as the recommended
default
5. **Fallback Handling**: Improved error messages when storage backends
are unavailable

## Benefits:

- **JSONL Storage**: Zero-dependency, simple file-based storage that
works out of the box
- **Dynamic SQLite**: Better performance and compatibility with existing
SQLite tools
- **Flexibility**: Users can still choose SQLite, PostgreSQL, or Redis
as alternatives
- **Simplified Setup**: No SQLite compilation or static linking required
for most users

## Migration Notes:

- Existing SQLite databases will continue to work
- New installations will default to JSONL storage
- SQLite now requires CGO and system SQLite libraries
- JSONL files are human-readable and easy to backup/migrate

The JSONL storage provides a simpler, dependency-free solution for most
users while maintaining SQLite, PostgreSQL, and Redis as alternative
options.

---------

Signed-off-by: Eden Reich <[email protected]>

Author: edenreich

.github/workflows/artifacts.yml

@@ -20,11 +20,11 @@ jobs:
           - os: ubuntu-24.04
             goos: linux
             goarch: amd64
-            cgo: "0"
+            cgo: "1"
           - os: ubuntu-24.04
             goos: linux
             goarch: arm64
-            cgo: "0"
+            cgo: "1"
           - os: macos-15-intel
             goos: darwin
             goarch: amd64
@@ -46,6 +46,12 @@ jobs:
           go-version-file: 'go.mod'
           cache: true
 
+      - name: Install build dependencies (Linux)
+        if: matrix.goos == 'linux'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y gcc libsqlite3-dev
+
       - name: Get version info
         id: version
         run: |

.github/workflows/ci.yml

@@ -97,7 +97,14 @@ jobs:
         go-version-file: 'go.mod'
         cache: true
 
+    - name: Install build dependencies
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y gcc libsqlite3-dev
+
     - name: Build project
+      env:
+        CGO_ENABLED: 1
       run: |
         go build \
           -ldflags "\
@@ -120,5 +127,12 @@ jobs:
         go-version-file: 'go.mod'
         cache: true
 
+    - name: Install build dependencies
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y gcc libsqlite3-dev
+
     - name: Run tests
+      env:
+        CGO_ENABLED: 1
       run: go test ./...

.infer/.gitignore

@@ -3,5 +3,6 @@ logs/*.log
 history
 chat_export_*
 conversations.db
+conversations
 bin/
 tmp/

.infer/config.yaml

@@ -290,7 +290,7 @@ scm:
     delete_local_branch: false
 storage:
   enabled: true
-  type: sqlite
+  type: jsonl
   sqlite:
     path: .infer/conversations.db
   postgres:
@@ -305,6 +305,8 @@ storage:
     port: 6379
     password: ""
     db: 0
+  jsonl:
+    path: .infer/conversations
 conversation:
   title_generation:
     enabled: true

CLAUDE.md

@@ -78,7 +78,7 @@ The codebase follows Clean Architecture principles with clear separation:
    - Orchestrates domain services
 
 3. **Infrastructure Layer** (`internal/infra/`, `internal/services/`): External concerns
-   - Storage implementations (SQLite, PostgreSQL, Redis)
+   - Storage implementations (JSONL, SQLite, PostgreSQL, Redis)
    - Tool implementations (Bash, Read, Write, Grep, etc.)
    - SDK integration for model communication
 
@@ -343,7 +343,7 @@ func setupTestConfig() *config.Config {
 Storage implementations abstracted behind interfaces:
 
 - Interface: `domain.ConversationRepository`
-- Implementations: `storage.SQLiteStorage`, `storage.PostgresStorage`, `storage.RedisStorage`
+- Implementations: `storage.JsonlStorage`, `storage.SQLiteStorage`, `storage.PostgresStorage`, `storage.RedisStorage`
 - Factory: `storage.NewConversationStorage(cfg)`
 
 ### Strategy Pattern
@@ -352,7 +352,7 @@ Tools and services use strategy pattern for different implementations:
 
 - Grep backend: ripgrep vs Go implementation
 - Gateway mode: Docker vs binary mode
-- Storage backend: SQLite vs PostgreSQL vs Redis
+- Storage backend: JSONL (default) vs SQLite vs PostgreSQL vs Redis
 
 ### Observer Pattern
 
@@ -511,7 +511,7 @@ alongside interactive chat sessions.
 
 **Behavior:**
 
-- Agent conversations are saved by default to the configured storage backend (SQLite/PostgreSQL/Redis)
+- Agent conversations are saved by default to the configured storage backend (JSONL/SQLite/PostgreSQL/Redis)
 - Each message, tool execution, and token usage is persisted automatically
 - Agent sessions appear in `infer conversations list` alongside chat conversations
 - Auto-save uses async goroutines to prevent blocking execution
@@ -540,7 +540,7 @@ infer conversations list
 - Token usage tracking per model
 - Cost calculation integrated with pricing service
 - Session completion summary with statistics
-- Works with all storage backends (SQLite, PostgreSQL, Redis)
+- Works with all storage backends (JSONL, SQLite, PostgreSQL, Redis)
 
 **Implementation Files:**
 

Dockerfile

@@ -17,7 +17,7 @@ LABEL org.opencontainers.image.version="${VERSION}"
 LABEL org.opencontainers.image.revision="${REVISION}"
 LABEL org.opencontainers.image.created="${BUILD_DATE}"
 
-RUN apk --no-cache --no-scripts add ca-certificates jq bash git
+RUN apk --no-cache --no-scripts add ca-certificates jq bash git sqlite-libs
 RUN addgroup -g 1000 infer && \
     adduser -u 1000 -G infer -h /home/infer -s /bin/bash -D infer
 ARG TARGETARCH

Taskfile.yml

@@ -19,7 +19,13 @@ tasks:
   build:
     desc: Build the CLI binary
     cmds:
-      - go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o {{.BINARY_NAME}} {{.MAIN_PACKAGE}}
+      - |
+        if ! command -v gcc >/dev/null 2>&1 && ! command -v clang >/dev/null 2>&1; then
+          echo "ERROR: C compiler not found. Required for SQLite support."
+          echo "Install: build-essential (Linux) or Xcode CLI Tools (macOS)"
+          exit 1
+        fi
+      - CGO_ENABLED=1 go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o {{.BINARY_NAME}} {{.MAIN_PACKAGE}}
     sources:
       - "**/*.go"
       - go.mod
@@ -30,7 +36,7 @@ tasks:
   install:
     desc: Install the CLI from source
     cmds:
-      - go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o $(go env GOPATH)/bin/{{.BINARY_NAME}} {{.MAIN_PACKAGE}}
+      - CGO_ENABLED=1 go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o $(go env GOPATH)/bin/{{.BINARY_NAME}} {{.MAIN_PACKAGE}}
 
   clean:
     desc: Clean build artifacts
@@ -41,17 +47,17 @@ tasks:
   test:
     desc: Run all tests
     cmds:
-      - go test ./...
+      - CGO_ENABLED=1 go test ./...
 
   test:verbose:
     desc: Run tests with verbose output
     cmds:
-      - go test -v ./...
+      - CGO_ENABLED=1 go test -v ./...
 
   test:coverage:
     desc: Run tests with coverage report
     cmds:
-      - go test -cover ./...
+      - CGO_ENABLED=1 go test -cover ./...
 
   lint:
     desc: Run linter (requires golangci-lint)
@@ -79,6 +85,36 @@ tasks:
     cmds:
       - go mod download
 
+  verify:cgo:
+    desc: Verify CGO and SQLite dependencies
+    cmds:
+      - |
+        echo "Checking C compiler..."
+        if command -v gcc >/dev/null 2>&1; then
+          echo "✓ GCC found: $(gcc --version | head -n1)"
+        elif command -v clang >/dev/null 2>&1; then
+          echo "✓ Clang found: $(clang --version | head -n1)"
+        else
+          echo "✗ No C compiler found"
+          echo "Install: build-essential (Linux) or Xcode CLI Tools (macOS)"
+          exit 1
+        fi
+
+        echo "Checking SQLite library..."
+        if pkg-config --exists sqlite3 2>/dev/null; then
+          echo "✓ SQLite found: $(pkg-config --modversion sqlite3)"
+        else
+          echo "⚠ SQLite not detected via pkg-config (may still work if installed)"
+        fi
+
+        echo "Checking CGO..."
+        CGO_STATUS=$(go env CGO_ENABLED)
+        if [ "$CGO_STATUS" = "1" ]; then
+          echo "✓ CGO enabled"
+        else
+          echo "⚠ CGO disabled (will be enabled for build tasks)"
+        fi
+
   run:
     desc: Run the CLI locally
     cmds:
@@ -118,8 +154,8 @@ tasks:
     desc: Build release binaries for multiple platforms (Linux only - macOS builds require native runners)
     cmds:
       - mkdir -p dist
-      - CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o dist/{{.BINARY_NAME}}-linux-amd64 {{.MAIN_PACKAGE}}
-      - CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o dist/{{.BINARY_NAME}}-linux-arm64 {{.MAIN_PACKAGE}}
+      - CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o dist/{{.BINARY_NAME}}-linux-amd64 {{.MAIN_PACKAGE}}
+      - CGO_ENABLED=1 GOOS=linux GOARCH=arm64 go build -ldflags "-X github.com/inference-gateway/cli/cmd.version={{.VERSION}} -X github.com/inference-gateway/cli/cmd.commit={{.COMMIT}} -X github.com/inference-gateway/cli/cmd.date={{.DATE}}" -o dist/{{.BINARY_NAME}}-linux-arm64 {{.MAIN_PACKAGE}}
       - cd dist && sha256sum {{.BINARY_NAME}}-* > checksums.txt
       - |
         if command -v cosign >/dev/null 2>&1; then

cmd/init.go

@@ -80,6 +80,7 @@ logs/*.log
 history
 chat_export_*
 conversations.db
+conversations
 bin/
 tmp/
 `

config/config.go

@@ -415,6 +415,7 @@ const (
 	StorageTypeSQLite   StorageType = "sqlite"
 	StorageTypePostgres StorageType = "postgres"
 	StorageTypeRedis    StorageType = "redis"
+	StorageTypeJsonl    StorageType = "jsonl"
 )
 
 // StorageConfig contains storage backend configuration
@@ -424,6 +425,7 @@ type StorageConfig struct {
 	SQLite   SQLiteStorageConfig   `yaml:"sqlite,omitempty" mapstructure:"sqlite,omitempty"`
 	Postgres PostgresStorageConfig `yaml:"postgres,omitempty" mapstructure:"postgres,omitempty"`
 	Redis    RedisStorageConfig    `yaml:"redis,omitempty" mapstructure:"redis,omitempty"`
+	Jsonl    JsonlStorageConfig    `yaml:"jsonl,omitempty" mapstructure:"jsonl,omitempty"`
 }
 
 // SQLiteStorageConfig contains SQLite-specific configuration
@@ -449,6 +451,11 @@ type RedisStorageConfig struct {
 	DB       int    `yaml:"db" mapstructure:"db"`
 }
 
+// JsonlStorageConfig contains JSONL-specific configuration
+type JsonlStorageConfig struct {
+	Path string `yaml:"path" mapstructure:"path"`
+}
+
 // A2AAgentInfo contains information about an A2A agent connection
 type A2AAgentInfo struct {
 	Name        string            `yaml:"name" mapstructure:"name"`
@@ -797,7 +804,10 @@ Respond with ONLY the commit message, no quotes or explanation.`,
 		},
 		Storage: StorageConfig{
 			Enabled: true,
-			Type:    "sqlite",
+			Type:    "jsonl",
+			Jsonl: JsonlStorageConfig{
+				Path: ConfigDirName + "/conversations",
+			},
 			SQLite: SQLiteStorageConfig{
 				Path: ConfigDirName + "/conversations.db",
 			},