After upgrading the version from 0.2.15 to 0.3.6, there was an error message displayed.

[jietang00723-cloud]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
After upgrading my ragas version from 0.2.15 to 0.3.6, I encountered an error message. As follows: LM returned 1 generations instead of requested 3. Proceeding with 1 generations. What should I do?
Code Examples

创建LLM实例

llm = ChatOpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
    model="deepseek-chat",
    logprobs=True
)




# 构建评估数据集
eval_data = {
    'user_input': [
        '人工智能的三个主要分支是什么？',
        '什么是向量数据库？它在RAG系统中有什么作用？',
        '解释一下大语言模型的微调过程'
    ],
    'retrieved_contexts': [
        ['人工智能主要包括机器学习、深度学习、自然语言处理和计算机视觉等领域。'],
        ['向量数据库是专门用于向量数据存储和检索的系统。在RAG应用中，向量数据库存储文档嵌入，通过计算相似度快速找到匹配查询的文档。'],
        ['大语言模型微调是在已预训练的模型基础上，使用特定数据集进行额外训练的过程，目的是使模型适应特定任务或领域。']
    ],
    'response': [
        '人工智能的三个主要分支是机器学习、深度学习和自然语言处理。',
        '向量数据库是专门存储和检索向量嵌入的数据库系统，在RAG系统中用于高效地检索语义相似的文档。',
        '大语言模型的微调是在预训练模型的基础上，使用特定任务的数据进一步训练模型的过程。'
    ],
    'reference': [
        '人工智能的三个主要分支是机器学习、自然语言处理和计算机视觉。',
        '向量数据库是专门设计用于存储和检索向量嵌入的数据库系统。在RAG系统中，它用于存储文档的向量表示，并通过相似性搜索快速检索与查询最相关的文档。',
        '大语言模型的微调是一个过程，其中预训练的基础模型使用特定领域或任务的数据进行额外训练，以优化其在特定应用场景的性能。'
    ],
    
}
comprehensive_dataset = Dataset.from_dict(eval_data)


from ragas.metrics import (
    context_recall,
    context_precision,
    answer_correctness,
    answer_similarity,
    answer_relevancy,
    faithfulness
)

# 定义评估指标
metrics = [
    faithfulness,
    answer_relevancy,
    answer_similarity,
    answer_correctness,
    context_precision,
    context_recall,

]

# 综合评估
comprehensive_score = evaluate(
    comprehensive_dataset,
    metrics=metrics,
    llm=llm,
    embeddings=embeddings
    # embeddings=jina_embedding
)
# 结果分析
result_df = comprehensive_score.to_pandas()

Additional context
After upgrading the version from 0.2.15 to 0.3.6, there was an error message displayed.

[anistark]

@jietang00723-cloud Do you mean the deprecation warning for LangChain LLM?

Could you share the error message exactly?

[jietang00723-cloud]

@jietang00723-cloud Do you mean the deprecation warning for LangChain LLM?你是否指的是 LangChain LLM 的弃用警告？

Could you share the error message exactly?你能准确分享错误信息吗？
Hello, I'm glad you replied. The problem I encountered was merely a warning message. Although this information doesn't have much impact on my usage, I still want it to disappear. The warning message is as follows:
‘’LM returned 1 generations instead of requested 3. Proceeding with 1 generations.‘’

[dcaled]

I'm also facing the same problem, on the same ragas version (0.3.6).
I believe this message comes from here:

ragas/src/ragas/prompt/pydantic_prompt.py

Lines 267 to 271 in 8b190e1

	if actual_n < n:
	logger.warning(
	f"LLM returned {actual_n} generations instead of requested {n}. "
	f"Proceeding with {actual_n} generations."
	)

Also, I'm using langchain_openai's AzureOpenAIEmbeddings.

[anistark]

The issue most likely is that some LLM providers/configurations may not support or may limit the number of concurrent batched requests, returning only 1 generation instead of the requested 3.

You can either, set a lower strictness, answer_relevancy.strictness = 1

or migrate to Ragas's native LLM interface for better support:

from ragas.llms import llm_factory

llm = llm_factory(
    "openai",
    api_key=openai_api_key,
    base_url=openai_api_base,
    model="deepseek-chat"
)

comprehensive_score = evaluate(
    comprehensive_dataset,
    metrics=metrics,
    llm=llm,
    embeddings=embeddings
)

[jietang00723-cloud]

Thank you. I think I have a general idea of what the problem is. By the way, I'd like to mention that the "faithfulness" indicator is relatively insensitive to the logical reasoning contained in the answers and often makes incorrect judgments. For instance, the model performs context-based refinement, but the judgment of fidelity is not accurate.

[anistark]

Thanks for bringing it up @jietang00723-cloud

Would you like to discuss faithfulness in a separate issue?

If this issue is resolved for you, we can close it then.

[danielaskdd]

The test code test_bypass_n_true_async_does_not_pass_n in tests/unit/llms/test_llm.py demonstrates a robust workaround for this issue. The recommended approach is to wrap the LangChain LLM using LangchainLLMWrapper and pass the bypass_n=True parameter within the wrapper.

sample code looks like:

from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# 1. create an origin LangChain LLM（without  bypass_n）
self.eval_llm = ChatOpenAI(**llm_kwargs)
self.eval_embeddings = OpenAIEmbeddings(**embedding_kwargs)

# 2. Use LangchainLLMWrapper to wrap LLM，an set bypass_n=True
self.ragas_llm = LangchainLLMWrapper(
    langchain_llm=self.eval_llm, 
    bypass_n=True  # Set bypass_n here
)

# 3. Use wrapped LLM in evaluate function
eval_results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
    llm=self.ragas_llm,  # 使用包装后的 LLM
    embeddings=self.eval_embeddings,
)

[paperClub-hub]

Unsupported provider: 'deepseek'. Supported: openai, anthropic, google, litellm

llm = llm_factory(
"openai",
api_key=deepseek_api_key,
base_url=deekseek_base_url,
model="deepseek-chat"
)

[anistark]

@paperClub-hub

Please open a separate issue for this.