Skip to content

Commit d18b7b2

Browse files
nina-kollmannina-kollman
authored
feat(experiment): Add experiment capabilities (#672)
1 parent b5878c5 commit d18b7b2

20 files changed

+1270
-17
lines changed

‎.gitignore

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ node_modules
2020

2121
# IDE - VSCode
2222
.vscode/*
23+
.vscode/
2324
!.vscode/settings.json
2425
!.vscode/tasks.json
2526
!.vscode/launch.json
@@ -46,4 +47,7 @@ Thumbs.db
4647
# env
4748
.env*
4849
chroma.sqlite3
49-
chroma.log
50+
chroma.log
51+
52+
# claude
53+
.claude

‎CLAUDE.md

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
OpenLLMetry-JS is a JavaScript/TypeScript observability framework for LLM applications, built on OpenTelemetry. It provides instrumentation for major LLM providers (OpenAI, Anthropic, etc.) and vector databases, with a unified SDK for easy integration.
8+
9+
## Development Commands
10+
11+
### Building
12+
13+
```bash
14+
# Build all packages
15+
pnpm nx run-many -t build
16+
# or
17+
pnpm nx run-many --targets=build
18+
19+
# Build affected packages only
20+
pnpm nx affected -t build
21+
```
22+
23+
### Testing
24+
25+
Each package has its own test command:
26+
27+
```bash
28+
# Test individual packages
29+
cd packages/traceloop-sdk
30+
pnpm test
31+
32+
# Test specific instrumentation
33+
cd packages/instrumentation-openai
34+
pnpm test
35+
```
36+
37+
You can also use nx to run tests:
38+
39+
```bash
40+
# Test all packages
41+
pnpm nx run-many -t test
42+
43+
# Test specific package by name
44+
pnpm nx test @traceloop/node-server-sdk
45+
46+
# Test only affected packages
47+
pnpm nx affected -t test
48+
49+
# Test with specific pattern
50+
pnpm nx run-many -t test --projects="*instrumentation*"
51+
52+
# Run tests in parallel
53+
pnpm nx run-many -t test --parallel
54+
55+
# Watch mode for development
56+
pnpm nx test @traceloop/node-server-sdk --watch
57+
```
58+
59+
### Linting
60+
61+
```bash
62+
# Lint individual packages
63+
cd packages/[package-name]
64+
pnpm lint
65+
66+
# Fix lint issues
67+
pnpm lint:fix
68+
```
69+
70+
## Architecture
71+
72+
### Monorepo Structure
73+
74+
- **Lerna + Nx**: Manages multiple packages with shared tooling
75+
- **packages/**: Contains all publishable packages and internal tooling
76+
- **Rollup**: Used for building packages with TypeScript compilation
77+
78+
### Core Packages
79+
80+
#### `traceloop-sdk` (Main SDK)
81+
82+
- **Path**: `packages/traceloop-sdk/`
83+
- **Exports**: `@traceloop/node-server-sdk`
84+
- **Purpose**: Primary entry point that orchestrates all instrumentations
85+
- **Key Files**:
86+
- `src/lib/tracing/decorators.ts`: Workflow and task decorators (`@workflow`, `@task`, `@agent`)
87+
- `src/lib/tracing/tracing.ts`: Core tracing utilities and span management
88+
- `src/lib/node-server-sdk.ts`: Main initialization logic
89+
90+
#### Instrumentation Packages
91+
92+
Each follows the pattern: `packages/instrumentation-[provider]/`
93+
94+
- **OpenAI**: `@traceloop/instrumentation-openai`
95+
- **Anthropic**: `@traceloop/instrumentation-anthropic`
96+
- **Bedrock**: `@traceloop/instrumentation-bedrock`
97+
- **Vector DBs**: Pinecone, Chroma, Qdrant packages
98+
- **Frameworks**: LangChain, LlamaIndex packages
99+
100+
#### `ai-semantic-conventions`
101+
102+
- **Path**: `packages/ai-semantic-conventions/`
103+
- **Purpose**: OpenTelemetry semantic conventions for AI/LLM spans
104+
- **Key File**: `src/SemanticAttributes.ts` - defines all span attribute constants
105+
106+
### Instrumentation Pattern
107+
108+
All instrumentations extend `InstrumentationBase` from `@opentelemetry/instrumentation`:
109+
110+
1. **Hook Registration**: Wrap target library functions using `InstrumentationModuleDefinition`
111+
2. **Span Creation**: Create spans with appropriate semantic attributes
112+
3. **Data Extraction**: Extract request/response data and token usage
113+
4. **Error Handling**: Capture and record errors appropriately
114+
115+
### Testing Strategy
116+
117+
- **Polly.js**: Records HTTP interactions for consistent test execution
118+
- **ts-mocha**: TypeScript test runner
119+
- **Recordings**: Stored in `recordings/` folders for replay testing
120+
121+
## Key Patterns
122+
123+
### Workspace Dependencies
124+
125+
Packages reference each other using `workspace:*` in package.json, managed by pnpm workspaces.
126+
127+
### Decorator Usage
128+
129+
```typescript
130+
// Workflow spans
131+
@workflow("my-workflow")
132+
async function myWorkflow() { }
133+
134+
// Task spans
135+
@task("my-task")
136+
async function myTask() { }
137+
```
138+
139+
### Manual Instrumentation
140+
141+
```typescript
142+
import { trace } from "@traceloop/node-server-sdk";
143+
const span = trace.withLLMSpan("my-llm-call", () => {
144+
// LLM operations
145+
});
146+
```
147+
148+
### Telemetry Configuration
149+
150+
- Anonymous telemetry enabled by default
151+
- Opt-out via `TRACELOOP_TELEMETRY=FALSE` environment variable
152+
- Only collected in SDK, not individual instrumentations
153+
154+
## Common Development Tasks
155+
156+
### Adding New LLM Provider
157+
158+
1. Create new instrumentation package in `packages/instrumentation-[provider]/`
159+
2. Implement instrumentation extending `InstrumentationBase`
160+
3. Add to main SDK dependencies in `packages/traceloop-sdk/package.json`
161+
4. Register in SDK initialization
162+
163+
### Running Single Test
164+
165+
```bash
166+
cd packages/[package-name]
167+
pnpm test -- --grep "test name pattern"
168+
```
169+
170+
### Debugging Instrumentations
171+
172+
Enable OpenTelemetry debug logging:
173+
174+
```bash
175+
export OTEL_LOG_LEVEL=debug
176+
```

‎packages/sample-app/package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
"run:image_generation": "npm run build && node dist/src/sample_openai_image_generation.js",
3434
"run:sample_edit": "npm run build && node dist/src/test_edit_only.js",
3535
"run:sample_generate": "npm run build && node dist/src/test_generate_only.js",
36+
"run:sample_experiment": "npm run build && node dist/src/sample_experiment.js",
3637
"dev:image_generation": "pnpm --filter @traceloop/instrumentation-openai build && pnpm --filter @traceloop/node-server-sdk build && npm run build && node dist/src/sample_openai_image_generation.js",
3738
"lint": "eslint .",
3839
"lint:fix": "eslint . --fix"
@@ -71,6 +72,7 @@
7172
"langchain": "^0.3.30",
7273
"llamaindex": "^0.11.19",
7374
"openai": "^5.12.2",
75+
"eventsource": "^3.0.2",
7476
"zod": "^3.25.76"
7577
},
7678
"private": true,

‎packages/sample-app/src/medical_prompts.ts

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
/**
2+
* Medical prompt templates for experiment examples
3+
* These templates demonstrate different approaches to handling medical questions
4+
*/
5+
6+
/**
7+
* Prompt template that provides comprehensive medical information
8+
* This approach gives detailed, educational responses about health topics
9+
*/
10+
export function provideMedicalInfoPrompt(question: string): string {
11+
return `You are a health educator providing comprehensive medical information.
12+
13+
Question: ${question}
14+
15+
Please provide a detailed, educational response that includes:
16+
17+
1. **Clear, factual explanation** of the medical concept or condition
18+
2. **Key benefits and considerations** related to the topic
19+
3. **Specific recommendations** based on current medical knowledge
20+
4. **Important disclaimers** about consulting healthcare professionals
21+
5. **Relevant context** that helps understand the topic better
22+
23+
Guidelines:
24+
- Use evidence-based information
25+
- Explain medical terms in plain language
26+
- Include both benefits and risks when applicable
27+
- Emphasize the importance of professional medical consultation
28+
- Provide actionable, general health guidance
29+
30+
Your response should be educational, balanced, and encourage informed healthcare decisions.`;
31+
}
32+
33+
/**
34+
* Prompt template that refuses to give medical advice
35+
* This approach redirects users to appropriate medical professionals
36+
*/
37+
export function refuseMedicalAdvicePrompt(question: string): string {
38+
return `You are a helpful AI assistant with a strict policy about medical advice.
39+
40+
Question: ${question}
41+
42+
I understand you're seeking information about a health-related topic, but I cannot provide medical advice, diagnosis, or treatment recommendations.
43+
44+
Instead, I'd like to:
45+
46+
1. **Acknowledge your concern** - Your health questions are important and valid
47+
2. **Explain why I can't advise** - Medical situations require professional evaluation
48+
3. **Suggest appropriate resources**:
49+
- Consult your primary care physician
50+
- Contact a relevant medical specialist
51+
- Call a nurse hotline if available
52+
- Visit an urgent care or emergency room if urgent
53+
54+
4. **Provide general wellness information** if applicable (without specific medical advice)
55+
5. **Encourage professional consultation** for personalized care
56+
57+
Your health is important, and qualified medical professionals are best equipped to provide the specific guidance you need.
58+
59+
Is there anything else I can help you with that doesn't involve medical advice?`;
60+
}
61+
62+
/**
63+
* Example prompt categories for experiment testing
64+
*/
65+
export const PROMPT_CATEGORIES = {
66+
PROVIDE_INFO: "provide" as const,
67+
REFUSE_ADVICE: "refuse" as const,
68+
MENTAL_HEALTH: "mental-health" as const,
69+
FITNESS: "fitness" as const,
70+
NUTRITION: "nutrition" as const,
71+
} as const;
72+
73+
export type PromptCategory =
74+
(typeof PROMPT_CATEGORIES)[keyof typeof PROMPT_CATEGORIES];

0 commit comments

Comments
 (0)