Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ public Flux<ChatResponse> internalStream(Prompt prompt, ChatResponse previousCha
// FIXME: bounded elastic needs to be used since tool calling
// is currently only synchronous
return Flux.deferContextual((ctx) -> {
// TODO: factor out the tool execution logic with setting context into a uitlity.
// TODO: factor out the tool execution logic with setting context into a utility.
ToolExecutionResult toolExecutionResult;
try {
ToolCallReactiveContextHolder.setContext(ctx);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ void multiModalityEmbeddedImage(String modelName) throws IOException {
@ValueSource(strings = { "claude-3-7-sonnet-latest", "claude-sonnet-4-0" })
void multiModalityImageUrl(String modelName) throws IOException {

// TODO: add url method that wrapps the checked exception.
// TODO: add url method that wraps the checked exception.
URL url = new URL("https://docs.spring.io/spring-ai/reference/_images/multimodal.test.png");

// @formatter:off
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ void beanStreamOutputConverterRecords() {
@Test
void multiModalityImageUrl() throws IOException {

// TODO: add url method that wrapps the checked exception.
// TODO: add url method that wraps the checked exception.
URL url = new URL("https://docs.spring.io/spring-ai/reference/_images/multimodal.test.png");

// @formatter:off
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,7 @@ void multiModalityEmbeddedImage(String modelName) throws IOException {
@ValueSource(strings = { "anthropic.claude-3-5-sonnet-20240620-v1:0" })
void multiModalityImageUrl2(String modelName) throws IOException {

// TODO: add url method that wrapps the checked exception.
// TODO: add url method that wraps the checked exception.
URL url = new URL("https://docs.spring.io/spring-ai/reference/_images/multimodal.test.png");

// @formatter:off
Expand All @@ -401,7 +401,7 @@ void multiModalityImageUrl2(String modelName) throws IOException {
@ValueSource(strings = { "anthropic.claude-3-5-sonnet-20240620-v1:0" })
void multiModalityImageUrl(String modelName) throws IOException {

// TODO: add url method that wrapps the checked exception.
// TODO: add url method that wraps the checked exception.
URL url = new URL("https://docs.spring.io/spring-ai/reference/_images/multimodal.test.png");

// @formatter:off
Expand All @@ -420,7 +420,7 @@ void multiModalityImageUrl(String modelName) throws IOException {
@Test
void streamingMultiModalityImageUrl() throws IOException {

// TODO: add url method that wrapps the checked exception.
// TODO: add url method that wraps the checked exception.
URL url = new URL("https://docs.spring.io/spring-ai/reference/_images/multimodal.test.png");

// @formatter:off
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ private void validate(ChatResponseMetadata responseMetadata) {
.doesNotHaveAnyRemainingCurrentObservation()
.hasObservationWithNameEqualTo(DefaultChatModelObservationConvention.DEFAULT_NAME)
.that()
// TODO - this condition occasionall fails.
// TODO - this condition occasionally fails.
// .hasContextualNameEqualTo("chat " +
// OpenAiApi.ChatModel.GPT_4_O_MINI.getValue())
.hasLowCardinalityKeyValue(LowCardinalityKeyNames.AI_OPERATION_TYPE.asString(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ void multiModalityEmbeddedImage(String modelName) throws IOException {
@ValueSource(strings = { "gpt-4o" })
void multiModalityImageUrl(String modelName) throws IOException {

// TODO: add url method that wrapps the checked exception.
// TODO: add url method that wraps the checked exception.
URL url = new URL("https://docs.spring.io/spring-ai/reference/_images/multimodal.test.png");

// @formatter:off
Expand All @@ -343,7 +343,7 @@ void multiModalityImageUrl(String modelName) throws IOException {
@Test
void streamingMultiModalityImageUrl() throws IOException {

// TODO: add url method that wrapps the checked exception.
// TODO: add url method that wraps the checked exception.
URL url = new URL("https://docs.spring.io/spring-ai/reference/_images/multimodal.test.png");

// @formatter:off
Expand Down
50 changes: 48 additions & 2 deletions spring-ai-test/README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,48 @@
TODO:
Documentation and sample tests using the `BasicEvaluationTest`.
# Spring AI Test

The Spring AI Test module provides utilities and base classes for testing AI applications built with Spring AI.

## Features

- **BasicEvaluationTest**: A base test class for evaluating question-answer quality using AI models
- **Vector Store Testing**: Utilities for testing vector store implementations
- **Audio Testing**: Utilities for testing audio-related functionality

## BasicEvaluationTest

The `BasicEvaluationTest` class provides a framework for evaluating the quality and relevance of AI-generated answers to questions.

### Usage

Extend the `BasicEvaluationTest` class in your test classes:

```java
@SpringBootTest
public class MyAiEvaluationTest extends BasicEvaluationTest {

@Test
public void testQuestionAnswerAccuracy() {
String question = "What is the capital of France?";
String answer = "The capital of France is Paris.";

// Evaluate if the answer is accurate and related to the question
evaluateQuestionAndAnswer(question, answer, true);
}
}
```

### Configuration

The test requires:
- A `ChatModel` bean (typically OpenAI)
- Evaluation prompt templates located in `classpath:/prompts/spring/test/evaluation/`

### Evaluation Types

- **Fact-based evaluation**: Use `factBased = true` for questions requiring factual accuracy
- **General evaluation**: Use `factBased = false` for more subjective questions

The evaluation process:
1. Checks if the answer is related to the question
2. Evaluates the accuracy/appropriateness of the answer
3. Fails the test with detailed feedback if the answer is inadequate
Loading