Skip to content

Commit 4e732cf

Browse files
authored
Improve question generation efficiency In Response Relevancy metrics (#1810)
**Description**: This update refactors the _ascore method to use asyncio.gather for parallel execution of question generation tasks. Key improvements include: **Issue** : When ResponseRelevancy evaluation loops based on the strictness variable (default 3), which is taking longer than expected **Performance Enhancement**: By processing all tasks concurrently, the overall execution time is significantly reduced, especially for high strictness values. **Code Simplification**: The refactored code is more concise and aligns better with Python's asynchronous programming patterns. **Scalability**: The new implementation handles a larger number of tasks without blocking, making it more suitable for use cases requiring high concurrency. **Backward Compatibility**: All existing functionality remains intact, with no changes required to external interfaces. This change improves the efficiency and maintainability of the codebase, aligning with best practices for asynchronous programming in Python. Result after resolution : Average 1 run of Example code based on tutorial code (10 runs total) reduced to 4.35s => 2.82s
1 parent c455c38 commit 4e732cf

File tree

1 file changed

+6
-4
lines changed

1 file changed

+6
-4
lines changed

src/ragas/metrics/_answer_relevance.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
SingleTurnMetric,
1717
)
1818
from ragas.prompt import PydanticPrompt
19+
import asyncio
1920

2021
logger = logging.getLogger(__name__)
2122

@@ -136,14 +137,15 @@ async def _ascore(self, row: t.Dict, callbacks: Callbacks) -> float:
136137
assert self.llm is not None, "LLM is not set"
137138

138139
prompt_input = ResponseRelevanceInput(response=row["response"])
139-
responses = []
140-
for _ in range(self.strictness):
141-
response = await self.question_generation.generate(
140+
tasks = [
141+
self.question_generation.generate(
142142
data=prompt_input,
143143
llm=self.llm,
144144
callbacks=callbacks,
145145
)
146-
responses.append(response)
146+
for _ in range(self.strictness)
147+
]
148+
responses = await asyncio.gather(*tasks)
147149

148150
return self._calculate_score(responses, row)
149151

0 commit comments

Comments
 (0)