batch cookbook v2

Author: siddharthsambharia-portkey

docs.json

@@ -842,6 +842,7 @@
             "group": "Use Cases",
             "pages": [
               "guides/use-cases",
+              "guides/use-cases/run-batch-evals",
               "guides/use-cases/few-shot-prompting",
               "guides/use-cases/enforcing-json-schema-with-anyscale-and-together",
               "guides/use-cases/emotions-with-gpt-4o",

guides/use-cases.mdx

@@ -3,6 +3,7 @@ title: Overview
 ---
 
 <CardGroup>
+  <Card title="How to Run Structure Output Evals on Portkey at Scale" href="/guides/use-cases/run-batch-evals" />
   <Card title="Few-Shot Prompting" href="/guides/use-cases/few-shot-prompting" />
   <Card title="Enforcing JSON Schema with Anyscale & Together" href="/guides/use-cases/enforcing-json-schema-with-anyscale-and-together" />
   <Card title="Detecting Emotions with GPT-4o" href="/guides/use-cases/emotions-with-gpt-4o" />

guides/use-cases/run-batch-evals.mdx

@@ -1,9 +1,7 @@
 ---
-title: "How to Strctured Output Evals with Portkey"
-## Why Evaluations Matter
+title: "How to Run Strctured Output Evals at Scale"
 ---
 
-
 Building production-ready AI applications requires more than just choosing the right model and infrastructure. Just like traditional software development relies on comprehensive testing, AI applications need rigorous evaluation to ensure they perform reliably in the real world.
 
 Consider a customer support chatbot for a food delivery service. If it incorrectly processes a refund request, your company bears the financial cost. If it misunderstands a complaint about food allergies, the consequences could be even more severe. These high-stakes scenarios demand systematic evaluation before deployment.
@@ -33,16 +31,9 @@ We'll work through a practical example: an AI agent that analyzes Amazon product
 ### Our Evaluation Framework
 Using real Amazon review data, we'll build evaluations that check:
 1. **Format compliance**: Does the output match our required JSON schema `{"sentiment": "positive"}`?
-2. **PII detection**: Deny Request with PII using Guardrails
-3. **Sentiment classification accuracy**: Do the classifications match human labels?
-
-By the end of this guide, you'll have a reusable framework for building evaluations specific to your own AI applications.
-
-
-
-
 
 
+By the end of this guide, you'll have a reusable framework for building evaluations specific to your own AI applications.
 
 
 
@@ -501,13 +492,70 @@ print("\nDetailed Results:")
 print(tabulate(display_df, headers='keys', tablefmt='simple', showindex=False))
 ```
 
+
 ## Understanding the Results
 
+The script outputs a CSV with your evaluation results. Here's what matters:
+
+- **status_code 246**: Request passed JSON schema validation ✅
+- **status_code 446**: Request failed JSON schema validation ❌
+
+Example output:
+```
+Summary:
+Total requests: 10
+Successful: 8
+Schema Validation Passed: 8
+Schema Validation Failed: 2
+
+Sentiment Distribution:
+positive    5
+negative    3
+parse_error 2
+```
+
+## What to Do Next
 
+<Steps>
+<Step title="Fix Schema Failures">
+If you see 446 status codes, check:
+- Is your prompt output format clear?
+- Did you include enough examples?
+- Is the JSON structure in your prompt exactly right?
+</Step>
 
+<Step title="Scale Up">
+Change `length: 10` to `length: 100` or `length: 1000` to test more reviews
+</Step>
+
+<Step title="Add More Guardrails">
+Create guardrails for:
+- Checking if negative reviews mention refunds
+- Validating positive reviews aren't too short
+- Flagging reviews that need human review
+</Step>
+</Steps>
 
 ## Conclusion
 
+You now have:
+- A working evaluation pipeline
+- Automatic JSON validation
+- Batch processing for scale
+- Clear pass/fail metrics
+
+Start with 100 reviews, fix any issues, then scale to thousands.
+
+## Resources
+
+- [Portkey Docs](https://docs.portkey.ai) - API reference
+- [Discord](https://discord.gg/portkey) - Get help
+- Support: [email protected]
+
+---
+
+
+
 ### Complete Script
 
 Here's the complete script you can save and run: