RFC: Ollama OpenTelemetry Instrumentation Simulator Options

[aygp-dr]

Abstract

This RFC proposes options for creating a simulator that demonstrates OpenTelemetry instrumentation patterns for Ollama interactions. The simulator will help developers understand and validate OpenTelemetry semantic conventions for generative AI systems without requiring actual LLM execution. We present three language options - Scheme, Common Lisp, and Python - each with their tradeoffs.

Background

OpenTelemetry's semantic conventions for generative AI define standards for events, metrics, and spans. While Ollama provides a real API for LLM interactions, we need a simulator to:

1. Demonstrate correct OpenTelemetry instrumentation patterns
2. Test telemetry collection without running actual models
3. Validate semantic convention compliance
4. Provide a reference implementation for instrumentation

Simulator Options

1. Guile Scheme Simulator

Advantages

• Native support for asynchronous programming through futures and promises
• Strong metaprogramming capabilities via macros
• Seamless integration with Emacs via Geiser
• Clean record type definitions through SRFI-9
• Built-in JSON support

Disadvantages

• Smaller ecosystem for HTTP/networking libraries
• Less common in production environments
• Fewer developers familiar with the language

Example Structure

(define-record-type <ollama-client>
  (make-ollama-client base-url)
  ollama-client?
  (base-url client-base-url))

(define-record-type <span>
  (make-span name start-time end-time attributes events)
  span?
  (name span-name)
  (start-time span-start-time)
  (end-time span-end-time)
  (attributes span-attributes)
  (events span-events))

2. Common Lisp Implementation

Advantages

• Robust CLOS object system
• Excellent condition system for error handling
• Strong support for multiprocessing via bordeaux-threads
• Rich ecosystem via Quicklisp
• Interactive development with SLIME/SLY

Disadvantages

• More complex setup compared to Python
• Learning curve for CLOS and conditions
• Larger runtime footprint

Example Structure

(defclass ollama-client ()
  ((base-url :initarg :base-url
             :reader base-url)))

(defclass span ()
  ((name :initarg :name :reader span-name)
   (start-time :initarg :start-time :reader start-time)
   (end-time :initarg :end-time :reader end-time)
   (attributes :initarg :attributes :reader attributes)
   (events :initarg :events :reader events)))

3. Python Implementation

Advantages

• Rich ecosystem for HTTP clients and async programming
• Widespread familiarity and adoption
• Strong typing support via type hints
• Simple integration with existing OpenTelemetry libraries
• Easy deployment and packaging

Disadvantages

• Less powerful metaprogramming capabilities
• Requires managing virtual environments
• GIL limitations for true parallelism

Example Structure

import os
import logging
from dataclasses import dataclass, field
from typing import Dict, List, Any
import requests
import urllib3

# Set up debug logging
logging.basicConfig(level=logging.DEBUG)
requests_log = logging.getLogger("urllib3")

@dataclass
class Span:
    name: str
    start_time: int
    end_time: int
    attributes: Dict[str, Any]
    events: List[Event] = field(default_factory=list)

class OllamaTelemetrySimulator:
    def __init__(self):
        """Initialize simulator for OpenTelemetry demonstration"""
        self.simulated_latency = 0.1  # seconds

Development Environment Options

1. Org-mode with Babel

• Supports all three languages
• Literate programming approach
• Interactive development
• Easy documentation
• Can tangle to source files

Example header:

#+PROPERTY: header-args:python :session ollama-demo :results output
#+PROPERTY: header-args:scheme :session ollama-demo :results output
#+PROPERTY: header-args:lisp :session ollama-demo :results output

2. Traditional Editor/IDE

• VSCode with language extensions
• Emacs with language-specific modes
• PyCharm/IntelliJ for Python
• Separate source files
• Traditional debugging tools

Recommendation

All three languages are viable options. The choice should be based on:

1. Primary Use Case:

   • Interactive development/research: Scheme or Common Lisp with Org-mode
   • Production deployment: Python
   • Educational purposes: Any of the three

2. Team Experience:

   • Lisp experience: Scheme or Common Lisp
   • Python experience: Python
   • Learning focus: Any (with Org-mode for documentation)

3. Integration Requirements:

   • Existing OpenTelemetry setup: Python
   • Emacs ecosystem: Scheme
   • Complex error handling: Common Lisp

Proxy and Debugging Considerations

All implementations must support:

• HTTP_PROXY and HTTPS_PROXY environment variables
• Proxy authentication if required
• Debug logging of HTTP requests/responses
• SSL/TLS verification configuration
• Local proxy tools (e.g., mitmproxy, Charles Proxy)

Language-specific Proxy Handling

Scheme Implementation

(define (make-ollama-client #:key (host "localhost") (port 11434))
  (let ((http-proxy (getenv "HTTP_PROXY"))
        (https-proxy (getenv "HTTPS_PROXY")))
    (make-ollama-client-internal 
      #:base-url (string-append "http://" host ":" (number->string port) "/api")
      #:http-proxy http-proxy
      #:https-proxy https-proxy)))

Common Lisp Implementation

(defclass ollama-client ()
  ((base-url :initarg :base-url :reader base-url)
   (http-proxy :initform (uiop:getenv "HTTP_PROXY") :reader http-proxy)
   (https-proxy :initform (uiop:getenv "HTTPS_PROXY") :reader https-proxy)))

Python Implementation

class OllamaClient:
    def __init__(self, host="localhost", port=11434):
        self.base_url = f"http://{host}:{port}/api"
        # Requests will automatically use HTTP(S)_PROXY env vars
        self.session = requests.Session()
        
        # Optional proxy configuration
        if os.environ.get('HTTP_PROXY'):
            self.session.proxies['http'] = os.environ['HTTP_PROXY']
        if os.environ.get('HTTPS_PROXY'):
            self.session.proxies['https'] = os.environ['HTTPS_PROXY']
            
        # Debug logging
        if os.environ.get('DEBUG'):
            import http.client
            http.client.HTTPConnection.debuglevel = 1

Debug Logging Format

All implementations should log:

• Full HTTP request headers and body
• Response headers and streaming chunks
• Timing information
• Proxy connection details
• TLS/SSL handshake information (when relevant)

Example debug output:

-> POST http://localhost:11434/api/generate
-> Proxy: http://localhost:8080
-> Headers: {...}
-> Body: {"model": "codellama", "prompt": "..."}
<- Status: 200 OK
<- Headers: {...}
<- Streaming chunks: [...]
<- Total time: 2.3s

Security Considerations

All implementations must handle:

• Secure HTTP connections
• Input validation
• Error handling for streaming responses
• Resource cleanup
• Token/credential management

Performance Considerations

All implementations should:

• Use streaming responses effectively
• Minimize memory usage for large responses
• Handle concurrent requests if needed
• Implement proper cleanup of resources

Testing Strategy

All implementations should include:

• Unit tests for telemetry data structures
• Integration tests with Ollama
• Response streaming tests
• Error handling tests
• Performance benchmarks

Migration Strategy

The implementation should be structured to allow:

• Easy switching between languages
• Consistent API across implementations
• Clear upgrade path for OpenTelemetry changes
• Documentation of differences between implementations

References

1. OpenTelemetry Semantic Conventions for GenAI
2. Ollama API Documentation
3. Language-specific OpenTelemetry SDKs
4. Org-mode Babel Documentation

Next Steps

1. Choose implementation language based on requirements
2. Set up development environment
3. Implement core functionality
4. Add OpenTelemetry instrumentation
5. Create documentation and examples
6. Add tests
7. Create deployment/packaging strategy

[jwalsh]

Technical Review: Ollama OpenTelemetry Simulator Implementation

Author: Staff+ Engineer Technical Review
Date: 2024-02-07

Overview

I've reviewed the proposed Ollama OpenTelemetry simulator implementation. Here are my key observations and recommendations for resources and implementation details.

Key Resources

Documentation

• OpenTelemetry Semantic Conventions for GenAI - Essential for understanding event types
• Ollama API Reference - For understanding real API behavior to simulate
• OpenTelemetry Metrics SDK - For proper metric implementation

Testing Resources

SAMPLE_RESPONSES = {
    "hello_world": {
        "prompt": "Write hello world in Python",
        "expected_tokens": 15,
        "expected_duration": 0.5
    },
    "fibonacci": {
        "prompt": "Write fibonacci sequence",
        "expected_tokens": 45,
        "expected_duration": 1.2
    }
}

Critical Implementation Notes

1. Event Generation

Be precise with event timestamps:

def create_event(name, timestamp, attributes):
    """Events should have millisecond precision"""
    return {
        "name": name,
        "timestamp": int(timestamp * 1000),  # Convert to ms
        "attributes": attributes
    }

2. Span Context

Remember to propagate span context:

def simulate_generation(model, prompt):
    with tracer.start_as_current_span("ollama.generate") as span:
        # Set span attributes
        span.set_attribute("gen_ai.system", "ollama")
        span.set_attribute("gen_ai.request.model", model)
        
        # All child operations inherit context
        simulate_tokenization(prompt)
        simulate_response_generation()

3. Metric Bucketing

Use appropriate histogram buckets:

DURATION_BUCKETS = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0]
TOKEN_BUCKETS = [10, 50, 100, 500, 1000, 5000, 10000]

Implementation Priorities

1. Core Simulation

   • Events must follow exact OpenTelemetry semantic conventions
   • Timing should be realistic but accelerated
   • Token counts should match typical model behavior

2. Telemetry Accuracy

   • All required attributes must be present
   • Optional attributes should be included when relevant
   • Metric units must be correct

3. Error Simulation

   • Include common error patterns
   • Test error propagation
   • Validate error attributes

Common Pitfalls to Avoid

1. Event Timing

   # WRONG
   events = [create_event("start"), create_event("end")]
   
   # RIGHT
   start_time = time.time()
   events = [
       create_event("start", start_time),
       create_event("end", start_time + duration)
   ]

2. Attribute Types

   # WRONG
   span.set_attribute("gen_ai.request.temperature", "0.7")
   
   # RIGHT
   span.set_attribute("gen_ai.request.temperature", 0.7)

3. Event Ordering

   # WRONG: Events out of order
   events = [assistant_msg, system_msg, user_msg]
   
   # RIGHT: Maintain chronological order
   events = [system_msg, user_msg, assistant_msg]

Testing Strategy

1. Semantic Convention Validation

   def test_event_schema():
       """Validate event follows semantic conventions"""
       event = simulate_user_message("test")
       assert "gen_ai.system" in event.attributes
       assert event.name == "gen_ai.user.message"
       assert isinstance(event.timestamp, int)

2. Timing Validation

   def test_response_timing():
       """Verify events have realistic timing"""
       start = time.time()
       result = simulate_generation("test")
       duration = time.time() - start
       assert 0.1 <= duration <= 2.0  # Reasonable range

3. Error Case Testing

   def test_error_handling():
       """Verify error propagation"""
       with pytest.raises(SimulatedModelError):
           simulate_generation("invalid_model")

Performance Considerations

1. Memory Usage

   • Don't store full response history
   • Stream events when possible
   • Clean up simulation artifacts

2. CPU Usage

   • Avoid unnecessary sleep() calls
   • Use efficient time.monotonic() for timing
   • Batch metric creation

Migration Path

1. Version 1: Basic Simulation

   • Core events and spans
   • Simple metric recording
   • Basic error simulation

2. Version 2: Enhanced Fidelity

   • Realistic token counts
   • Proper streaming simulation
   • Advanced error cases

3. Version 3: Production Ready

   • Full semantic convention compliance
   • Comprehensive testing
   • Documentation and examples

Future Considerations

1. New OpenTelemetry Features

   • Watch for semantic convention updates
   • Plan for new metric types
   • Consider exemplars support

2. Integration Testing

   • Add OpenTelemetry collector testing
   • Validate export formats
   • Test visualization tools

3. Documentation

   • Maintain examples
   • Update test cases
   • Add troubleshooting guide

Final Recommendations

1. Start Simple

   • Begin with core events
   • Add complexity gradually
   • Test thoroughly

2. Focus on Correctness

   • Follow conventions exactly
   • Validate all attributes
   • Test edge cases

3. Plan for Growth

   • Design for extensibility
   • Document assumptions
   • Consider future use cases

Review Sign-off Checklist

• Semantic conventions fully implemented
• Event timing is realistic
• Metric types are correct
• Error handling is comprehensive
• Tests are thorough
• Documentation is complete
• Code is maintainable

Please address these points before proceeding with implementation. Happy to discuss any questions or concerns.