Agent Reviewer is a powerful application designed to evaluate call center personnel by analyzing audio recordings. It provides in-depth assessments using a combination of built-in and user customizable metrics, delivering actionable business insights from every conversation.
- Audio Processing: Upload audio files directly, no pre-processing or conversion required.
- Format Support: Compatible with all common audio formats (mp3, wav, etc.).
- Automated Evaluation: Assess agent performance with powerful LLM analytics.
- Custom Metrics: Supply additional evaluation criteria to fit your business case.
- Business Insights: Extract trends, sentiment, and customer satisfaction indicators.
- Quality assurance for customer service teams
- Agent performance benchmarking
- Identification of training opportunities
- Measuring customer sentiment and engagement
- Upload a call recording through the app interface.
- Process the audio using built-in or user-supplied metrics.
- Review an automatically generated evaluation report.
- Analyze employee trends and insights across multiple recordings.
-
API Key
Agree to the Pyannote.audio terms and generate Hugging Face API key https://github.com/pyannote/pyannote-audio?tab=readme-ov-file#tldr -
Clone
git clone https://github.com/nthanapaisal/agent-reviewer -
Env
cd agent-reviewer/backend/src/
create file.env
add API key to .env fileHUGGING_FACE="your_api_key_here" -
Compose
cd agent-reviewer
docker compose up --build -d
Further documentation located in BUILD.md
- Speaker Diarization: pyannote.audio
- Audio Transcription: openai-whisper
- Prompt Construction: Spacy
- Analysis: Mistral-7B
- Trend Generation: Numpy
Relevance: Evaluates whether the agent's response addresses the user's question
Clarity: Evaluates the clarity of the agent's response
Sentiment Score: Analyzes the sentiment polarity of the agent's tone (positive, neutral, negative)
Completeness: Evaluates whether the agent's response contains complete information
Consistency: Evaluates whether the agent's response is consistent with the context
User Satisfaction: Measures the user's satisfaction with the agent's response
Engagement Level: Evaluates the depth of the user's interaction (e.g., simple response vs. asking more questions)
Problem Solved: Indicates whether the agent successfully solved the user's problem
Context Awareness: Evaluates whether the agent correctly understands the conversation context
You are an agent evaluator.
Evaluation Task: Evaluate the agent in this conversation: "{transcription}" using these metrics: "{metrics}". Additional user metrics: "{user_prompt}".
Evaluation Behavior Instruction: Be flexible and reasonable in your evaluation—do not apply overly strict standards. Consider the agent’s intent, overall helpfulness, and adaptability when scoring. Take into consideration that some callers may be irrational and unfair; sometimes, it is out of the agent’s control. Note that the transcription speaker labels may be inaccurate; you may reassess them when evaluating.
Scoring and Expected Output: There are two requirements: 1. For each metric_name: give a score on a scale out of 5 along with the reason. 2. Write a 9–15 sentence paragraph summarizing the agent's overall performance. Include specific quotes from the transcription that significantly influenced your evaluation, and explain why they were important. If the agent performed well, offer praise to encourage a positive learning environment. If the agent did not perform well, suggest what they could have done better.
Formatting Instruction:
Return your response strictly as a valid JSON object using double quotes for all keys and strings, like this:
{{"report": [["metric_name", score, "reason"]], "summary": "Your summary here. don't forget quotes from transcription"}}
