Chat Generation & Analysis System

An intelligent system that learns from example chats to generate new conversations and provides comprehensive analysis with beautiful email reports using the OpenAI Agents SDK.

Features

Chat Generation & Evaluation

• Policy Extraction: Analyzes example chats to extract conversation patterns, topics, and style
• Chat Generation: Creates new chats based on extracted policies and user prompts
• Intelligent Evaluation: Evaluates generated chats against examples with detailed scoring
• Policy Refinement: Automatically refines policies based on evaluation feedback
• Prompt Optimization: Improves user prompts for better generation results
• Iterative Improvement: Multiple generation rounds with feedback-driven improvements

Analysis & Reporting

• Chat Analysis: Extracts insights about common topics, frequently asked questions, and time saved
• Category-Specific Analysis: Separate analysis for different lawyer types (Privacy, Commercial Contracts)
• Email Reports: Beautiful HTML email reports using React Email
• Visual Elements: Progress bars, metric cards, and color-coded sections
• Time Estimation: Research, drafting, and analysis time estimates
• Email-Optimized: Works across all major email clients

Quick Start

Environment Variables

export OPENAI_API_KEY=sk-...
OPENAI_THINKING_MODEL=o4-mini

Generate Email from Latest Analysis

bun generate-email-report.ts

This will:

1. Find the most recent combined analysis report
2. Generate a beautiful HTML email from the analysis
3. Save the HTML file with timestamp

Generate New Analysis

bun generate-analysis.ts

This will:

1. Run the chat analysis on your generated chats
2. Create JSON reports for each category
3. Save analysis files with timestamps

Architecture

Agents

1. Policy Extractor Agent (src/agents/policy-extractor.ts)

   • Reads example chats from /example-chats
   • Extracts patterns, topics, and conversation structure
   • Saves policy as JSON

2. Chat Generator Agent (src/agents/chat-generator.ts)

   • Loads extracted policies
   • Generates complete chat conversations
   • Follows XML format from examples

3. Chat Evaluator Agent (src/agents/chat-evaluator.ts)

   • Compares generated chats with examples
   • Provides detailed scoring and feedback
   • Suggests policy and prompt refinements

4. Policy Refiner Agent (src/agents/policy-refiner.ts)

   • Applies refinements to existing policies
   • Maintains version history
   • Improves generation quality over iterations

5. Prompt Optimizer Agent (src/agents/prompt-optimizer.ts)

   • Analyzes prompt effectiveness
   • Refines prompts for better alignment
   • Improves generation results

6. Chat Analysis Agent (src/agents/chat-analysis.ts)

   • Analyzes generated chats for insights
   • Extracts topics, questions, and time saved estimates
   • Provides category-specific analysis

Orchestrator

The main orchestrator (src/orchestrator.ts) coordinates all agents and manages the iterative improvement process.

Usage

Basic Chat Generation

import { runChatGeneration } from './src/orchestrator';

// Generate with default settings
await runChatGeneration();

// Generate with custom prompt
await runChatGeneration('Generate a chat between a lawyer and a client');

Advanced Chat Generation

await runChatGeneration(
  'Generate a technical support chat',
  {
    maxIterations: 5,        // Maximum refinement iterations
    targetScore: 90,         // Target evaluation score (0-100)
    enablePolicyRefinement: true,   // Enable automatic policy refinement
    enablePromptOptimization: true  // Enable prompt optimization
  }
);

Email Report Generation

import { generateEmailReport } from './generate-email-report';

// Generate email from latest analysis
const { emailPath, reportPath } = await generateEmailReport();

console.log(`Email ready: ${emailPath}`);
console.log(`Data available: ${reportPath}`);

Send via Email Service

// Example with Resend
import { Resend } from 'resend';
import { readFileSync } from 'fs';

const resend = new Resend('your-api-key');
const emailHtml = readFileSync('reports/combined-analysis-123456.html', 'utf-8');

await resend.emails.send({
  from: 'reports@yourcompany.com',
  to: 'team@yourcompany.com',
  subject: 'AI Assistant Performance Report',
  html: emailHtml,
});

Setup

1. Install dependencies:

   bun install

2. Set your OpenAI API key:

   export OPENAI_API_KEY=sk-...

3. Run the system:

   bun run generate-chats.ts

Testing

Run the test script to verify the system:

bun run test-system.ts

Directory Structure

gcai/
├── example-chats/       # Example chat files
├── generated-chats/     # Generated chat outputs
├── policies/           # Extracted and refined policies
├── prompts/           # XML prompt templates
├── reports/           # Analysis reports and email HTML
│   ├── privacy-lawyer-analysis-{timestamp}.json
│   ├── commercial-contracts-lawyer-analysis-{timestamp}.json
│   ├── combined-analysis-{timestamp}.json
│   └── combined-analysis-{timestamp}.html  # 📧 Email HTML
├── src/
│   ├── agents/        # Agent implementations
│   ├── emails/        # React Email templates
│   ├── types/         # TypeScript type definitions
│   ├── utils/         # Utility functions
│   └── orchestrator.ts # Main orchestration logic
├── generate-chats.ts   # Chat generation entry point
├── generate-analysis.ts # Analysis generation entry point
└── generate-email-report.ts # Email report generation

How It Works

Chat Generation Pipeline

1. Policy Extraction: The system first analyzes all example chats to understand patterns
2. Initial Generation: Creates a chat based on the policy and user prompt
3. Evaluation: Evaluates the generated chat against examples
4. Refinement: If the score is below target, refines both policy and prompt
5. Iteration: Repeats generation with improved policy/prompt until target is reached

Analysis & Reporting Pipeline

1. Chat Analysis: Analyzes all generated chats for insights and patterns
2. Category Separation: Creates separate analysis for each lawyer type
3. Time Estimation: Estimates time saved based on task complexity
4. Email Generation: Creates beautiful HTML email reports using React Email

Email Template Features

Header Section

• 🤖 AI Assistant Performance Report title
• Generation date and model used
• Professional branding

Overview Dashboard

• Total Chats - Number of conversations analyzed
• Time Saved - Estimated time saved for users
• Average Score - Quality score across all categories

Category Sections

Each lawyer category gets its own section with:

• Category Metrics - Chats, scores, time saved per category
• Top Topics - Most discussed topics with frequency
• Key Insights - Important findings and patterns
• Time Breakdown - Visual breakdown of research/drafting/analysis time

Visual Elements

• Progress Bars - For time breakdowns and scores
• Metric Cards - Clean, professional data presentation
• Color Coding - Blue for Privacy Law, Orange for Commercial Contracts
• Icons - 🔒 for Privacy, 📄 for Contracts

Evaluation Criteria

• Structure (0-100): XML format and message pattern matching
• Topic Coherence (0-100): Natural topic introduction and development
• Conversation Flow (0-100): Logical progression
• Style Match (0-100): Tone and style similarity to examples
• Policy Adherence (0-100): Following extracted patterns

Data Structure

The email template expects this data structure from your analysis:

interface CombinedAnalysisReport {
  generatedAt: string;
  model: string;
  categories: string[];
  analyses: Record<string, {
    chatCount: number;
    analysis: CategoryAnalysis; // JSON string or object
  }>;
}

Each CategoryAnalysis includes:

• Summary metrics (chats, scores, time saved)
• Common topics with frequency
• Key insights and evidence
• Time breakdown (research/drafting/analysis)
• Quality metrics and improvement areas

License

MIT