Why do I get the warning "Failed to set LM usage" when using MLflow to track the DSPy execution process?

[Turbolento]

The warning does not appear when I remove MLflow. The code is as follows:

import os
import dspy
from dotenv import load_dotenv
from dspy.evaluate import SemanticF1
import mlflow

# Load environment variables
load_dotenv()

# Configure MLflow
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("DSPy_cot")
mlflow.autolog()

# Configure DSPy with the language model
model_name = os.getenv('MODEL_NAME')
api_base_url = os.getenv('API_URL')
api_key = os.getenv('API_KEY')

llm = dspy.LM(model=f'{model_name}', api_key=api_key, base_url=api_base_url, temperature=0)
dspy.configure(lm=llm)

# Create a simple test example
class TestExample:
    def __init__(self, question, response):
        self.question = question
        self.response = response
    def inputs(self):
        return {'question': self.question}

example = TestExample(
    question="What is the capital of France?",
    response="The capital of France is Paris."
)

# Create a simple ChainOfThought module
cot = dspy.ChainOfThought('question -> response')

# Generate a prediction
pred = cot(question=example.question)
print(f"Question: {example.question}")
print(f"Gold Response: {example.response}")
print(f"Predicted Response: {pred.response}")

# Test SemanticF1
metric = SemanticF1(decompositional=True)
score = metric(example, pred)
print(f"Semantic F1 Score: {score:.2f}")

print("\nTest completed successfully!")

Some of the results printed in the console are as follows:

2026/02/08 22:39:19 WARNING dspy.primitives.module: Failed to set LM usage. Please return dspy.Prediction object from dspy.Module to enable usage tracking.

[TomeHirata]

Thanks for the report, I could repro the issue. The root cause is that the SemanticF1 class is a Module, but doesn't return dspy.Prediction. Let me file a PR to fix.