Project-level secret management

[rachmlenig]

User Story

As a LlamaFarm user, I want to securely store and manage API keys and secrets for my projects without hardcoding them in llamafarm.yaml, so that I can safely commit my configuration to version control and collaborate with my team.

Problem

Currently, users must hardcode API keys and secrets directly in llamafarm.yaml, which creates security risks when committing to git. Additionally, the dual-location nature of llamafarm.yaml (user project directory and ~/.llamafarm/projects) makes file-based secret management (like .env files) confusing and error-prone.

Proposed Solution

Implement a project-level variable/secret store that LlamaFarm manages internally. Secrets would be set via API and stored securely, with initial support for local filesystem storage.

MVP Scope

API Endpoints

• POST /api/projects/{project_id}/secrets - Set a secret
• GET /api/projects/{project_id}/secrets - List secret names (not values)
• DELETE /api/projects/{project_id}/secrets/{name} - Delete a secret

Configuration Syntax

Support ${secret:VAR_NAME} syntax in llamafarm.yaml:

runtime:
  models:
    - name: gpt4
      provider: openai
      model: gpt-4
      api_key: ${secret:OPENAI_API_KEY}

Storage

• Store secrets encrypted in ~/.llamafarm/projects/{project_id}/secrets/
• Use a simple encryption mechanism (can be enhanced later)
• Secrets are project-scoped and isolated

CLI Support

llamafarm secrets set OPENAI_API_KEY sk-abc123...
llamafarm secrets list
llamafarm secrets delete OPENAI_API_KEY

Out of Scope for MVP

• Cloud secret providers (AWS Secrets Manager, Vault, etc.)
• Kubernetes secret integration
• Secret rotation/expiration
• Access control/permissions
• Audit logging
• Secret sharing across projects

Acceptance Criteria

• Users can set secrets via API/CLI without exposing them in config files
• llamafarm.yaml can reference secrets using ${secret:VAR_NAME} syntax
• Secrets are encrypted at rest
• Project configs load successfully with secret substitution
• Error messages don't leak secret values
• Documentation explains how to use the secret management system

Future Enhancements

• Support for multiple secret providers (filesystem, Kubernetes, cloud services)
• Secret rotation and expiration policies
• Team-based access control
• Audit logging for secret access

[rachmlenig]

Secret Management Implementation Options

Executive Summary

Three approaches evaluated for storing project secrets in LlamaFarm. Recommendation: Option A (keyring) for MVP due to OS-native security, zero key management overhead, and alignment with local laptop deployment model.

---

Option A: OS Keychain Integration (keyring)

Overview

Leverage OS-native credential storage via the keyring Python library. Secrets stored in macOS Keychain, Windows Credential Locker, or Linux Secret Service.

How Encryption Key Management Works

No manual key management required - the OS handles encryption/decryption automatically using the user's login credentials. Keys are tied to the OS user account.

Implementation

import keyring
Store secret
keyring.set_password("llamafarm", f"{project_id}_{secret_name}", secret_value)
Retrieve secret
secret = keyring.get_password("llamafarm", f"{project_id}_{secret_name}")
Delete secret
keyring.delete_password("llamafarm", f"{project_id}_{secret_name}")

Pros

• ✅ Zero key management overhead - OS handles it
• ✅ Battle-tested, mature library (13+ years, 500+ stars)
• ✅ Cross-platform (macOS, Windows, Linux)
• ✅ Integrates with existing OS security infrastructure
• ✅ No separate files to gitignore or manage
• ✅ Secrets tied to user's OS account (good security boundary)
• ✅ Simple API - only 3 methods needed
• ✅ No additional service or daemon required

Cons

• ⚠️ Linux users may need to install gnome-keyring or kwallet
• ⚠️ Secrets not portable across machines (must re-enter on new laptop)
• ⚠️ Harder to backup/restore secrets (OS-dependent process)
• ⚠️ Testing requires mock backend to avoid polluting dev keychain

Storage Location

• macOS: Keychain.app (~/Library/Keychains/)
• Windows: Credential Manager (Windows Vault)
• Linux: Secret Service API (freedesktop.org standard)

Cross-Platform Support

Excellent - Works on all major platforms with graceful fallbacks

Library Maturity

• PyPI: 25M+ downloads/month
• Maintained: Active development
• Python versions: 3.8+

---

Option B: Encrypted JSON File (Fernet + cryptography)

Overview

Store secrets in an encrypted JSON file (~/.llamafarm/projects/{project_id}/secrets.json.enc) using symmetric encryption (Fernet from cryptography library).

How Encryption Key Management Works

Master key derived from machine-specific identifiers (MAC address, hostname, user ID) or stored in ~/.llamafarm/master.key. Key must be accessible to server process.

Implementation

from cryptography.fernet import Fernet
import json
Generate/load master key
def get_master_key():

key_file = Path.home() / ".llamafarm" / "master.key"

if not key_file.exists():

key = Fernet.generate_key()

key_file.write_bytes(key)

return key_file.read_bytes()
Encrypt and store
fernet = Fernet(get_master_key())

secrets = {"OPENAI_API_KEY": "sk-abc123..."}

encrypted = fernet.encrypt(json.dumps(secrets).encode())

Path("secrets.json.enc").write_bytes(encrypted)
Decrypt and retrieve
encrypted = Path("secrets.json.enc").read_bytes()

decrypted = fernet.decrypt(encrypted)

secrets = json.loads(decrypted)

Pros

• ✅ Full control over storage format
• ✅ Easy to backup/restore (just copy the file + key)
• ✅ Portable across machines (with key)
• ✅ Works identically on all platforms
• ✅ Simple to test (use temp directories)
• ✅ Built on well-regarded cryptography library

Cons

• ⚠️ Must manage encryption key ourselves
• ⚠️ Key file (master.key) is a single point of failure
• ⚠️ If key is compromised, all secrets are exposed
• ⚠️ Key derivation from machine ID is fragile (changes on hardware/OS changes)
• ⚠️ Must gitignore key file and encrypted secrets
• ⚠️ More code to maintain (encryption/decryption logic)

Storage Location

• Secrets: ~/.llamafarm/projects/{project_id}/secrets.json.enc
• Key: ~/.llamafarm/master.key (or derived from machine)

Cross-Platform Support

Excellent - Pure Python, works everywhere

Library Maturity

• PyPI: 150M+ downloads/month
• Maintained: PyCA (Python Cryptographic Authority)
• Python versions: 3.7+

---

Option C: Simple Obfuscation (Base64) - MVP Fast Path

Overview

Not secure encryption - just base64 encoding to keep secrets out of plain text. Clear warnings that this is not cryptographically secure.

How "Encryption" Key Management Works

No encryption key - uses base64 encoding only. Security through obscurity (prevents casual viewing, not actual security).

Implementation

import base64
import json
"Encrypt" (encode)
secrets = {"OPENAI_API_KEY": "sk-abc123..."}

encoded = base64.b64encode(json.dumps(secrets).encode())

Path("secrets.json.b64").write_bytes(encoded)
"Decrypt" (decode)
encoded = Path("secrets.json.b64").read_bytes()

secrets = json.loads(base64.b64decode(encoded))

Pros

• ✅ Fastest to implement (ships MVP quickly)
• ✅ Zero dependencies beyond stdlib
• ✅ No key management complexity
• ✅ Gets API and ${secret:VAR} syntax working immediately
• ✅ Can upgrade to real encryption in v2
• ✅ Easy to test and debug

Cons

• ❌ NOT SECURE - anyone with file access can decode
• ❌ Must prominently warn users
• ❌ Not suitable for production secrets
• ❌ May damage reputation if users don't understand limitations
• ❌ Creates technical debt (must migrate to real encryption)

Storage Location

• ~/.llamafarm/projects/{project_id}/secrets.json.b64

Cross-Platform Support

Perfect - stdlib only

Library Maturity

N/A - Python stdlib

---

Comparison Matrix

Criteria	Option A: keyring	Option B: Fernet	Option C: Base64
Security	⭐⭐⭐⭐⭐ OS-native	⭐⭐⭐⭐ Strong if key secured	⭐ None (obfuscation only)
Key Management	⭐⭐⭐⭐⭐ Automatic	⭐⭐ Manual	⭐⭐⭐⭐⭐ N/A
Implementation Complexity	⭐⭐⭐⭐⭐ Very simple	⭐⭐⭐ Moderate	⭐⭐⭐⭐⭐ Trivial
Cross-Platform	⭐⭐⭐⭐ Linux needs setup	⭐⭐⭐⭐⭐ Perfect	⭐⭐⭐⭐⭐ Perfect
Portability	⭐⭐ OS-bound	⭐⭐⭐⭐⭐ Fully portable	⭐⭐⭐⭐⭐ Fully portable
Testing	⭐⭐⭐ Needs mocking	⭐⭐⭐⭐⭐ Easy	⭐⭐⭐⭐⭐ Easy
Maintenance	⭐⭐⭐⭐⭐ Minimal	⭐⭐⭐ Ongoing	⭐⭐⭐⭐ Minimal
Time to MVP	⭐⭐⭐⭐ 1-2 days	⭐⭐⭐ 2-3 days	⭐⭐⭐⭐⭐ Hours

---

Recommendation: Option A (keyring)

Why keyring Wins

1. Solves the key management challenge - OS handles it automatically
2. Aligns with "local laptop" deployment - leverages existing OS security
3. Battle-tested and mature - 13 years in production use
4. Minimal code to maintain - simple 3-method API
5. Better security than DIY encryption - don't roll your own crypto
6. No separate service required - addresses the concern about avoiding new service dependencies

Linux Consideration

For Linux users without keyring backends, we can:

• Detect missing backend on first use
• Provide clear installation instructions
• Fall back to keyring's encrypted file backend (better than Option B since we still don't manage the key)

Migration Path

If we need Option B features later (portability, backup), we can add export/import:

llamafarm secrets export > secrets.json  # encrypted with user-provided passphrase
llamafarm secrets import secrets.json    # re-import on new machine

---

Prototype Code (keyring)

# server/services/secret_service.py
import keyring
from pathlib import Path
from typing import Dict, Optional
class SecretService:

SERVICE_NAME = "llamafarm"
@classmethod
def _make_key(cls, project_id: str, secret_name: str) -> str:
    """Create keyring username from project and secret name."""
    return f"project_{project_id}_{secret_name}"

@classmethod
def set_secret(cls, project_id: str, name: str, value: str) -> None:
    """Store a secret in the OS keychain."""
    key = cls._make_key(project_id, name)
    keyring.set_password(cls.SERVICE_NAME, key, value)

@classmethod
def get_secret(cls, project_id: str, name: str) -> Optional[str]:
    """Retrieve a secret from the OS keychain."""
    key = cls._make_key(project_id, name)
    return keyring.get_password(cls.SERVICE_NAME, key)

@classmethod
def delete_secret(cls, project_id: str, name: str) -> bool:
    """Delete a secret from the OS keychain."""
    key = cls._make_key(project_id, name)
    try:
        keyring.delete_password(cls.SERVICE_NAME, key)
        return True
    except keyring.errors.PasswordDeleteError:
        return False

@classmethod
def list_secrets(cls, project_id: str) -> list[str]:
    """List secret names for a project.
    
    Note: keyring doesn't provide a list operation, so we need
    to track secret names separately (in a non-secret metadata file).
    """
    # TODO: Implement metadata tracking
    raise NotImplementedError()

Config Substitution

# In existing config loading code
import re
SECRET_PATTERN = re.compile(r'${secret:([^}]+)}')
def substitute_secrets(config_dict: dict, project_id: str) -> dict:

"""Recursively substitute ${secret:VAR} with values from keyring."""
def replacer(match: re.Match, pid: str) -> str:
    secret_name = match.group(1)
    value = SecretService.get_secret(pid, secret_name)
    if value is None:
        raise ValueError(f"Secret not found: {secret_name}")
    return value

def process_value(val, pid: str):
    if isinstance(val, str):
        return SECRET_PATTERN.sub(lambda m: replacer(m, pid), val)
    elif isinstance(val, dict):
        return {k: process_value(v, pid) for k, v in val.items()}
    elif isinstance(val, list):
        return [process_value(item, pid) for item in val]
    return val

return process_value(config_dict, project_id)

---

Addressing Feedback Concerns

✅ Use Existing Secret Manager Libraries

Using keyring - a mature, widely-adopted library rather than implementing encryption from scratch.

✅ Avoid New Service Dependencies

No separate service needed - keyring is a Python library that interfaces with OS-native credential stores. No daemon or background process required.

✅ Variable Substitution Pattern

Implementing ${secret:VAR_NAME} syntax in llamafarm.yaml for clean, readable configuration.

✅ Key Management

OS handles all key management automatically. No master keys to store or manage. Encryption keys are tied to the user's OS login credentials.

---

Next Steps

1. Get team feedback on keyring approach
2. Spike: Test keyring on all platforms (macOS, Windows, Linux VM)
3. Address list_secrets limitation - decide on metadata storage strategy
4. Update issue with chosen approach
5. Start implementation if approved

---

Questions for Discussion

1. Is OS-native keychain acceptable for LlamaFarm's security model?
2. How should we handle list_secrets (keyring doesn't provide enumeration)?
3. Should we support secret export/import for machine portability in MVP?
4. What's the priority order: macOS > Linux > Windows or all equal?