Skip to content

Commit b2d7693

Browse files
authored
docs: quickstart guide with interactive LLM and project structure (#2380)
## Changes Made ### Documentation (docs/getstarted/evals.md): - Added `quickstart` cmd to follow through guide with an example project. - Updated "Custom Evaluation with LLMs" to reference DiscreteMetric from generated code - Replaced static examples with modern `llm_factory` API - Changed "Using Pre-Built Metrics" to AspectCritic with modern async/await syntax - Updated "Evaluating on a Dataset" to use ragas.Dataset API ### Build Configuration (mkdocs.yml): - Made social plugin conditional: `enabled: !ENV [MKDOCS_CI, true]` ### Makefile: - Added explicit `MKDOCS_CI=false` to serve-docs target. This avoids social plugin error in macos if in case `cairosvg` is not found.
1 parent cf7e8a7 commit b2d7693

File tree

17 files changed

+555
-215
lines changed

17 files changed

+555
-215
lines changed

Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,7 @@ build-docs: ## Build all documentation
161161
@echo "Converting ipynb notebooks to md files..."
162162
$(Q)MKDOCS_CI=true uv run python $(GIT_ROOT)/docs/ipynb_to_md.py
163163
@echo "Building ragas documentation..."
164-
$(Q)uv run --group docs mkdocs build
164+
$(Q)MKDOCS_CI=false uv run --group docs mkdocs build
165165

166166
serve-docs: ## Build and serve documentation locally
167-
$(Q)uv run --group docs mkdocs serve --dirtyreload
167+
$(Q)MKDOCS_CI=false uv run --group docs mkdocs serve --dirtyreload

README.md

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -97,21 +97,39 @@ Available templates:
9797

9898
### Evaluate your LLM App
9999

100-
This is 5 main lines:
100+
This is a simple example evaluating a summary for accuracy:
101101

102102
```python
103-
from ragas import SingleTurnSample
104-
from ragas.metrics import AspectCritic
103+
import asyncio
104+
from ragas.metrics.collections import AspectCritic
105+
from ragas.llms import llm_factory
105106

107+
# Setup your LLM
108+
llm = llm_factory("gpt-4o")
109+
110+
# Create a metric
111+
metric = AspectCritic(
112+
name="summary_accuracy",
113+
definition="Verify if the summary is accurate and captures key information.",
114+
llm=llm
115+
)
116+
117+
# Evaluate
106118
test_data = {
107119
"user_input": "summarise given text\nThe company reported an 8% rise in Q3 2024, driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. Analysts attribute this success to strategic marketing and product localization. The positive trend in the Asian market is expected to continue into the next quarter.",
108120
"response": "The company experienced an 8% increase in Q3 2024, largely due to effective marketing strategies and product adaptation, with expectations of continued growth in the coming quarter.",
109121
}
110-
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
111-
metric = AspectCritic(name="summary_accuracy",llm=evaluator_llm, definition="Verify if the summary is accurate.")
112-
await metric.single_turn_ascore(SingleTurnSample(**test_data))
122+
123+
score = await metric.ascore(
124+
user_input=test_data["user_input"],
125+
response=test_data["response"]
126+
)
127+
print(f"Score: {score.value}")
128+
print(f"Reason: {score.reason}")
113129
```
114130

131+
> **Note**: Make sure your `OPENAI_API_KEY` environment variable is set.
132+
115133
Find the complete [Quickstart Guide](https://docs.ragas.io/en/latest/getstarted/evals)
116134

117135
## Want help in improving your AI application using evals?
60.5 KB
Loading

0 commit comments

Comments
 (0)