feat: Orchestration image support (#294)

* Call Orchestration with image and multiString via executeRequestFromJsonModuleConfig

* Work in Progress

* Align System- nad AssintantMessage with UserMessage

* Improve exceptions

* Improve tests

* Prepare draft

* - Restrict Multi- and ImageContent to appropriate classes;
- Make content() return MessageContent object

* Rename addTextMessages() to addText()

* Remove or hide unneeded constructors, use Message.user() etc. instead

* Add simple e2e tests

* Prepare draft

* Small change

* Refactor newly introduced classes

* WIP

* add test for base64 image

* Small cleanups, no more unnecessary Exceptions

* Small changes: ImageItem.DetailLevel is not mandatory by spec, refactor MessageContent construction

* Delete MessageContent.toString()

* Bit of clean up

* Add/improve javadocs

* Add/improve javadocs some more

* Add annotations

* Add explicit allArgs constructor to ImageItem

* Fix codestyle etc.

* Fix order in add methods

* Improve unit test

* Improve e2e test

* Small fixes after merge

* Add unit test for message construction

* Fix sample app after merge

* Simplify multiMessage unit test

* Formatting

* Add constructor for multiple strings for UserMessage and MessageContent

* Small changes

* Add documentation and release notes

* Improve documentation

* Update orchestration/src/main/java/com/sap/ai/sdk/orchestration/MessageContent.java

Co-authored-by: Matthias Kuhr <[email protected]>

* Minor changes

* Make `.content()` @beta

* change method names from `addXYZ()` to `andXYZ()`

* Delete unnecessary @nonnull from ImageItem constructor

* Improve tests

* increase coverage

* We hate Jacoco

* Update docs/release-notes/release_notes.md

Co-authored-by: Charles Dubois <[email protected]>

* Simplify logic

* Small fixes

* Reduce and streamline amount of public API

* Rename convenience methods to `withXyz()`

* Simplify code and adapt jacoco coverage

* Small change

* Add release number to javadocs

* Fit jacoco coverage

* Rename MessageContent.contentItemList to MessageContent.items

* Update docs

---------

Co-authored-by: Jonas Israel <[email protected]>
Co-authored-by: SAP Cloud SDK Bot <[email protected]>
Co-authored-by: Matthias Kuhr <[email protected]>
Co-authored-by: I538344 <[email protected]>
Co-authored-by: Charles Dubois <[email protected]>

Author: 4 people

docs/guides/ORCHESTRATION_CHAT_COMPLETION.md

@@ -259,6 +259,47 @@ try (Stream<String> stream = client.streamChatCompletion(prompt, config)) {
 Please find [an example in our Spring Boot application](../../sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).
 It shows the usage of Spring Boot's `ResponseBodyEmitter` to stream the chat completion delta messages to the frontend in real-time.
 
+
+## Add images and multiple text inputs to a message
+
+It's possible to add images and multiple text inputs to a message.
+
+### Add images to a message
+
+An image can be added to a message as follows.
+
+```java
+var message = Message.user("Describe the following image");
+var newMessage = message.withImage("https://url.to/image.jpg");
+```
+
+You can also construct a message with an image directly, using the `ImageItem` class.
+
+```java
+var message = Message.user(new ImageItem("https://url.to/image.jpg"));
+```
+
+Some AI models, like GPT 4o, support additionally setting the detail level with which the image is read. This can be set via the `DetailLevel` parameter.
+
+```java
+var newMessage = message.withImage("https://url.to/image.jpg", ImageItem.DetailLevel.LOW);
+```
+Note, that currently only user messages are supported for image attachments.
+
+### Add multiple text inputs to a message
+
+It's also possible to add multiple text inputs to a message. This can be useful for providing additional context to the AI model. You can add additional text inputs as follows.
+
+```java
+var message = Message.user("What is chess about?");
+var newMessage = message.withText("Answer in two sentences.");
+```
+
+Note, that only user and system messages are supported for multiple text inputs.
+
+Please find [an example in our Spring Boot application](../../sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).
+
+
 ## Set model parameters
 
 Change your LLM configuration to add model parameters:

docs/release-notes/release_notes.md

@@ -8,12 +8,14 @@
 
 ### 🔧 Compatibility Notes
 
--
+- `Message.content()` returns a `ContentItem` now instead of a `String`. Use `((TextItem) Message.content().items().get(0)).text()` if the corresponding `ContentItem` is a `TextItem` and the string representation is needed.
 
 ### ✨ New Functionality
 
 - Upgrade to release 2502a of AI Core.
-- [Add Orchestration `LlamaGuardFilter`](../guides/ORCHESTRATION_CHAT_COMPLETION.md#chat-completion-filter).
+- Orchestration:
+  - [Add `LlamaGuardFilter`](../guides/ORCHESTRATION_CHAT_COMPLETION.md#chat-completion-filter).
+  - [Convenient methods to create messages containing images and multiple text inputs](../guides/ORCHESTRATION_CHAT_COMPLETION.md#add-images-and-multiple-text-inputs-to-a-message)
 
 ### 📈 Improvements
 

orchestration/pom.xml

@@ -31,11 +31,11 @@
   </developers>
   <properties>
     <project.rootdir>${project.basedir}/../</project.rootdir>
-    <coverage.complexity>77%</coverage.complexity>
+    <coverage.complexity>80%</coverage.complexity>
     <coverage.line>92%</coverage.line>
-    <coverage.instruction>92%</coverage.instruction>
-    <coverage.branch>70%</coverage.branch>
-    <coverage.method>92%</coverage.method>
+    <coverage.instruction>93%</coverage.instruction>
+    <coverage.branch>71%</coverage.branch>
+    <coverage.method>95%</coverage.method>
     <coverage.class>100%</coverage.class>
   </properties>
 

orchestration/src/main/java/com/sap/ai/sdk/orchestration/AssistantMessage.java

@@ -1,6 +1,9 @@
 package com.sap.ai.sdk.orchestration;
 
+import com.google.common.annotations.Beta;
+import java.util.List;
 import javax.annotation.Nonnull;
+import lombok.Getter;
 import lombok.Value;
 import lombok.experimental.Accessors;
 
@@ -13,5 +16,16 @@ public class AssistantMessage implements Message {
   @Nonnull String role = "assistant";
 
   /** The content of the message. */
-  @Nonnull String content;
+  @Nonnull
+  @Getter(onMethod_ = @Beta)
+  MessageContent content;
+
+  /**
+   * Creates a new assistant message with the given single message.
+   *
+   * @param singleMessage the single message.
+   */
+  public AssistantMessage(@Nonnull final String singleMessage) {
+    content = new MessageContent(List.of(new TextItem(singleMessage)));
+  }
 }

orchestration/src/main/java/com/sap/ai/sdk/orchestration/ContentItem.java

@@ -0,0 +1,8 @@
+package com.sap.ai.sdk.orchestration;
+
+/**
+ * Represents an item in a {@link MessageContent} object.
+ *
+ * @since 1.3.0
+ */
+public sealed interface ContentItem permits TextItem, ImageItem {}

orchestration/src/main/java/com/sap/ai/sdk/orchestration/ImageItem.java

@@ -0,0 +1,68 @@
+package com.sap.ai.sdk.orchestration;
+
+import java.util.Locale;
+import javax.annotation.Nonnull;
+
+/**
+ * Represents an image item in a {@link MessageContent} object.
+ *
+ * @param imageUrl the URL of the image
+ * @param detailLevel the detail level of the image (optional)
+ * @since 1.3.0
+ */
+public record ImageItem(@Nonnull String imageUrl, @Nonnull DetailLevel detailLevel)
+    implements ContentItem {
+
+  /**
+   * Creates a new image item with the given image URL.
+   *
+   * @param imageUrl the URL of the image
+   * @since 1.3.0
+   */
+  public ImageItem(@Nonnull final String imageUrl) {
+    this(imageUrl, DetailLevel.AUTO);
+  }
+
+  /**
+   * The detail level of the image.
+   *
+   * @since 1.3.0
+   */
+  public enum DetailLevel {
+    /** Low detail level. */
+    LOW("low"),
+    /** High detail level. */
+    HIGH("high"),
+    /** Automatic detail level. */
+    AUTO("auto");
+
+    private final String level;
+
+    /**
+     * Converts a string to a detail level.
+     *
+     * @param str the string to convert
+     * @return the detail level
+     * @since 1.3.0
+     */
+    @Nonnull
+    static DetailLevel fromString(@Nonnull final String str) {
+      return DetailLevel.valueOf(str.toUpperCase(Locale.ENGLISH));
+    }
+
+    /**
+     * Get the string representation of the DetailLevel
+     *
+     * @return the DetailLevel as string
+     * @since 1.3.0
+     */
+    @Nonnull
+    public String toString() {
+      return level;
+    }
+
+    DetailLevel(@Nonnull final String level) {
+      this.level = level;
+    }
+  }
+}

orchestration/src/main/java/com/sap/ai/sdk/orchestration/Message.java

@@ -2,43 +2,62 @@
 
 import com.google.common.annotations.Beta;
 import com.sap.ai.sdk.orchestration.model.ChatMessage;
+import com.sap.ai.sdk.orchestration.model.ImageContent;
+import com.sap.ai.sdk.orchestration.model.ImageContentImageUrl;
+import com.sap.ai.sdk.orchestration.model.MultiChatMessage;
+import com.sap.ai.sdk.orchestration.model.MultiChatMessageContent;
 import com.sap.ai.sdk.orchestration.model.SingleChatMessage;
+import com.sap.ai.sdk.orchestration.model.TextContent;
+import java.util.LinkedList;
+import java.util.List;
 import javax.annotation.Nonnull;
 
 /** Interface representing convenience wrappers of chat message to the orchestration service. */
 public sealed interface Message permits UserMessage, AssistantMessage, SystemMessage {
 
   /**
-   * A convenience method to create a user message.
+   * A convenience method to create a user message from a string.
    *
-   * @param msg the message content.
+   * @param message the message content.
    * @return the user message.
    */
   @Nonnull
-  static UserMessage user(@Nonnull final String msg) {
-    return new UserMessage(msg);
+  static UserMessage user(@Nonnull final String message) {
+    return new UserMessage(message);
+  }
+
+  /**
+   * A convenience method to create a user message containing only an image.
+   *
+   * @param imageItem the message content.
+   * @return the user message.
+   * @since 1.3.0
+   */
+  @Nonnull
+  static UserMessage user(@Nonnull final ImageItem imageItem) {
+    return new UserMessage(new MessageContent(List.of(imageItem)));
   }
 
   /**
    * A convenience method to create an assistant message.
    *
-   * @param msg the message content.
+   * @param message the message content.
    * @return the assistant message.
    */
   @Nonnull
-  static AssistantMessage assistant(@Nonnull final String msg) {
-    return new AssistantMessage(msg);
+  static AssistantMessage assistant(@Nonnull final String message) {
+    return new AssistantMessage(message);
   }
 
   /**
-   * A convenience method to create a system message.
+   * A convenience method to create a system message from a string.
    *
-   * @param msg the message content.
+   * @param message the message content.
    * @return the system message.
    */
   @Nonnull
-  static SystemMessage system(@Nonnull final String msg) {
-    return new SystemMessage(msg);
+  static SystemMessage system(@Nonnull final String message) {
+    return new SystemMessage(message);
   }
 
   /**
@@ -48,7 +67,21 @@ static SystemMessage system(@Nonnull final String msg) {
    */
   @Nonnull
   default ChatMessage createChatMessage() {
-    return SingleChatMessage.create().role(role()).content(content());
+    final var itemList = this.content().items();
+    if (itemList.size() == 1 && itemList.get(0) instanceof TextItem textItem) {
+      return SingleChatMessage.create().role(role()).content(textItem.text());
+    }
+    final var contentList = new LinkedList<MultiChatMessageContent>();
+    for (final ContentItem item : itemList) {
+      if (item instanceof TextItem textItem) {
+        contentList.add(TextContent.create().type(TextContent.TypeEnum.TEXT).text(textItem.text()));
+      } else if (item instanceof ImageItem imageItem) {
+        final var detail = imageItem.detailLevel().toString();
+        final var img = ImageContentImageUrl.create().url(imageItem.imageUrl()).detail(detail);
+        contentList.add(ImageContent.create().type(ImageContent.TypeEnum.IMAGE_URL).imageUrl(img));
+      }
+    }
+    return MultiChatMessage.create().role(role()).content(contentList);
   }
 
   /**
@@ -66,5 +99,5 @@ default ChatMessage createChatMessage() {
    */
   @Nonnull
   @Beta
-  String content();
+  MessageContent content();
 }

orchestration/src/main/java/com/sap/ai/sdk/orchestration/MessageContent.java

@@ -0,0 +1,36 @@
+package com.sap.ai.sdk.orchestration;
+
+import com.sap.ai.sdk.orchestration.model.ImageContent;
+import com.sap.ai.sdk.orchestration.model.MultiChatMessageContent;
+import com.sap.ai.sdk.orchestration.model.TextContent;
+import java.util.List;
+import javax.annotation.Nonnull;
+
+/**
+ * Represents the content of a chat message.
+ *
+ * @param items a list of the content items
+ * @since 1.3.0
+ */
+public record MessageContent(@Nonnull List<ContentItem> items) {
+  @Nonnull
+  static MessageContent fromMCMContentList(
+      @Nonnull final List<MultiChatMessageContent> mCMContentList) {
+    final var itemList =
+        mCMContentList.stream()
+            .map(
+                content -> {
+                  if (content instanceof TextContent text) {
+                    return new TextItem(text.getText());
+                  } else {
+                    final var imageUrl = ((ImageContent) content).getImageUrl();
+                    return (ContentItem)
+                        new ImageItem(
+                            imageUrl.getUrl(),
+                            ImageItem.DetailLevel.fromString(imageUrl.getDetail()));
+                  }
+                })
+            .toList();
+    return new MessageContent(itemList);
+  }
+}

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationChatResponse.java

@@ -6,6 +6,7 @@
 import com.sap.ai.sdk.orchestration.model.CompletionPostResponse;
 import com.sap.ai.sdk.orchestration.model.LLMChoice;
 import com.sap.ai.sdk.orchestration.model.LLMModuleResultSynchronous;
+import com.sap.ai.sdk.orchestration.model.MultiChatMessage;
 import com.sap.ai.sdk.orchestration.model.SingleChatMessage;
 import com.sap.ai.sdk.orchestration.model.TokenUsage;
 import java.util.ArrayList;
@@ -51,33 +52,47 @@ public TokenUsage getTokenUsage() {
   /**
    * Get all messages. This can be used for subsequent prompts as a message history.
    *
-   * @throws UnsupportedOperationException if the MultiChatMessage type message in chat.
+   * @throws IllegalArgumentException if the MultiChatMessage type message in chat.
    * @return A list of all messages.
    */
   @Nonnull
-  public List<Message> getAllMessages() throws UnsupportedOperationException {
+  public List<Message> getAllMessages() throws IllegalArgumentException {
     final var messages = new ArrayList<Message>();
-
     for (final ChatMessage chatMessage : originalResponse.getModuleResults().getTemplating()) {
       if (chatMessage instanceof SingleChatMessage simpleMsg) {
-        final var message =
-            switch (simpleMsg.getRole()) {
-              case "user" -> new UserMessage(simpleMsg.getContent());
-              case "assistant" -> new AssistantMessage(simpleMsg.getContent());
-              case "system" -> new SystemMessage(simpleMsg.getContent());
-              default -> throw new IllegalStateException("Unexpected role: " + simpleMsg.getRole());
-            };
-        messages.add(message);
+        messages.add(chatMessageIntoMessage(simpleMsg));
+      } else if (chatMessage instanceof MultiChatMessage mCMessage) {
+        messages.add(chatMessageIntoMessage(mCMessage));
       } else {
-        throw new UnsupportedOperationException(
-            "Messages of MultiChatMessage type not supported by convenience API");
+        throw new IllegalArgumentException(
+            "Messages of type " + chatMessage.getClass() + " are not supported by convenience API");
       }
     }
-
-    messages.add(new AssistantMessage(getChoice().getMessage().getContent()));
+    messages.add(Message.assistant(getChoice().getMessage().getContent()));
     return messages;
   }
 
+  @Nonnull
+  private Message chatMessageIntoMessage(@Nonnull final SingleChatMessage simpleMsg) {
+    return switch (simpleMsg.getRole()) {
+      case "user" -> Message.user(simpleMsg.getContent());
+      case "assistant" -> Message.assistant(simpleMsg.getContent());
+      case "system" -> Message.system(simpleMsg.getContent());
+      default -> throw new IllegalStateException("Unexpected role: " + simpleMsg.getRole());
+    };
+  }
+
+  @Nonnull
+  private Message chatMessageIntoMessage(@Nonnull final MultiChatMessage mCMessage) {
+    return switch (mCMessage.getRole()) {
+      case "user" -> new UserMessage(MessageContent.fromMCMContentList(mCMessage.getContent()));
+      case "system" -> new SystemMessage(MessageContent.fromMCMContentList(mCMessage.getContent()));
+      default ->
+          throw new IllegalStateException(
+              "Unexpected role with complex message: " + mCMessage.getRole());
+    };
+  }
+
   /**
    * Get the LLM response. Useful for accessing the finish reason or further data like logprobs.
    *