Merge branch 'refs/heads/main' into bean

# Conflicts:
#	docs/release-notes/release_notes.md

Author: CharlesDuboisSAP

.github/workflows/deploy-snapshot.yaml

@@ -37,3 +37,14 @@ jobs:
         env:
           DEPLOYMENT_USER: ${{ secrets.ARTIFACTORY_COMMON_USER }}
           DEPLOYMENT_PASS: ${{ secrets.ARTIFACTORY_COMMON_PASSWORD }}
+
+      - name: "Slack Notification"
+        if: failure()
+        uses: slackapi/slack-github-action@v2.0.0
+        with:
+          webhook: ${{ secrets.SLACK_WEBHOOK }}
+          webhook-type: incoming-webhook
+          payload: |
+            {
+              "text": "⚠️ Snapshot Deployment failed! 😬 Please inspect & fix by clicking <https://github.com/SAP/ai-sdk-java/actions/runs/${{ github.run_id }}|here>"
+            }

README.md

@@ -21,6 +21,7 @@ The SDK simplifies the setup and interaction with SAP AI Core, allowing you to f
     - [Run the Application Locally](#run-the-application-locally)
     - [Explore Further Capabilities](#explore-further-capabilities)
 - [Documentation](#documentation)
+- [Build the Project](#build-the-project)
 - [FAQs](#faqs)
 - [Contribute, Support and Feedback](#contribute-support-and-feedback)
 - [Security / Disclosure](#security--disclosure)
@@ -137,6 +138,18 @@ For more detailed information and advanced usage, please refer to the following:
 
 For updating versions, please refer to the [**Release Notes**](docs/release-notes/release-notes-0-to-14.md).
 
+## Build the Project
+
+You can build the project using Maven:
+
+```shell
+mvn clean install -DskipTests
+```
+
+This will install the current `SNAPSHOT` version of the project into your local Maven repository.
+
+For SAP internal development, you can also use `SNAPSHOT` builds from the [internal](https://int.repositories.cloud.sap/ui/repos/tree/General/proxy-build-snapshots-cloudsdk/com/sap/ai/sdk) and [internet-facing](https://common.repositories.cloud.sap/artifactory/build-snapshots-cloudsdk/com/sap/ai/sdk/) Artifactory.
+
 ## FAQs
 
 ### _"How to add a custom header to AI Core requests?"_

docs/adrs/005-release-strategy.md

@@ -27,11 +27,12 @@ Since most new features will rely on generated code, the options for Java are li
 Testing can be performed against multiple landscapes and can vary based on the type of tests.
 
 - Unit tests may use test data from any landscape (including development landscapes) manually tested with e.g. Bruno.
-- E2E tests may use canary or production landscapes.
+- E2E tests may use integration, canary or production landscapes.
   In case of multiple landscapes:
    - GitHub matrix builds can be used to easily testing against multiple landscapes
    - For any differences between landscapes, test toggles need to be considered (e.g. `@EnabledIfSystemProperty`)
    - Such toggles come with a bit of additional maintenance cost, as they need to be removed once the feature is released to all landscapes
+   - Alternatively, E2E tests may be executed manually during development against any landscape
 
 ## Release
 
@@ -56,47 +57,45 @@ However, here there would be more freedom in when a feature toggle is enabled fo
 
 We decide as follows:
 
-> 1. New AI SDK versions are released roughly every 2 weeks, shortly after new AI Core versions have been released to _production EU10_.
-> 2. Any _publicly available release_ of the AI SDKs must only contain _public API_ for AI Core features available in _production EU10_ under the service plan _extended_.
-> 3. E2E tests run against canary EU12 and production EU10 using test toggles. 
+> 1. New AI SDK versions are released roughly every 2 weeks, shortly after new AI Core versions have been released to **all** landscapes.
+> 2. Any _publicly available release_ of the AI SDKs must only contain _public API_ for AI Core features available in **any** landscape under the service plan _extended_.
+> 3. E2E tests run automatically against canary only (EU12 for Java and JS, EU10 for Python). Production EU10 can be used for manual test runs. 
 
 Further explanations and notes:
 
 - Any features released exclusively under the `sap-internal` plan are not supported.
-  Similarly, any features released only to specific landscapes (other than prod EU10) are not supported.
-- There will be no releases of the SDKs to internal artifactory.
+- There will be no stable releases of the SDKs to internal artifactory.
+  For Java, we deliver SNAPSHOT versions to Artifactory, for JS we deliver canary releases to NPM.
 - Please note that the following is allowed:
   - Public API in an unreleased SDK version for unreleased AI Core features.
     Notably, this will **block** the release of the SDK until the AI Core feature is released publicly.
   - Internal code in a released SDK version for unreleased AI Core features.
+- Testing against additional landscapes is possible, but requires maintaining test toggles.
 
 ### An Example Development Lifecycle Iteration
 
 The following depicts a development flow where the AI SDK development steps are performed as soon as possible. 
 
 1. A new AI Core feature is being developed.
-2. A PR is raised on the AI SDK with a corresponding implementation, but so far not E2E-tested.
-   - Potentially aided by unit tests based on test data manually copied from e.g. Bruno.
+   - (optional) A preview of the changes is created on-demand on the AI SDK.
    - Generated code is created from a development version of the relevant spec file.
-3. (+2 weeks later) The feature is released to EU12 canary landscape.
-4. The AI SDK PR is enhanced:
-   - With an updated spec file.
-   - With an E2E test against canary.
-5. The AI SDK PR is merged.
-6. (+1 week later) The feature is released to EU10 production landscape.
-7. The E2E test is updated to now also run against production.
-8. The AI SDK is released publicly.
+   - Potentially aided by unit tests based on test data manually copied from e.g. Bruno.
+2. (+2 weeks later) The feature is released to EU12 canary landscape.
+3. An AI SDK PR is automatically raised with an updated spec file.
+   - If required, E2E tests are enhanced to cover the additional feature scope.
+4. The AI SDK PR is merged.
+5. (+1 week later) The feature is released to EU10 production landscape.
+6. The AI SDK is released publicly.
 
 In case of delays in the release process of AI Core:
 
-- If step (3) is delayed, the open PR may be closed and re-opened later
-- If step (5) is delayed, we consider 3 options:
+- If step (6) is delayed, we consider 3 options:
   1. The PR is reverted, together with potentially other related PRs
   2. The AI SDK release is delayed equally
   3. The AI SDK is released anyway with exceptional PO approval
 
 ## Further Links
 
 - The single source of truth for all landscapes is in [mlf-gitops](https://github.tools.sap/MLF-prod/mlf-gitops-prod)
-  - In particular, we care about the [version of orchestration in Prod EU10](https://github.tools.sap/MLF-prod/mlf-gitops-prod/blob/aws.eu-central-1.prod-eu/current/services/llm-orchestration/source/Chart.yaml) 
-- [This JIRA ticket](https://jira.tools.sap/browse/AI-44024) tracks releases
+  - In particular, we care about the [version of orchestration](https://github.tools.sap/MLF-prod/mlf-gitops-prod/blob/aws.eu-central-1.prod-eu/current/services/llm-orchestration/source/Chart.yaml)
+- [This JIRA ticket](https://jira.tools.sap/browse/AI-44024) tracks releases

docs/release-notes/release_notes.md

@@ -8,15 +8,20 @@
 
 ### 🔧 Compatibility Notes
 
+- `ChatMessage`, as well as new `MultiChatMessage`, are now subtypes of new interface `ChatMessagesInner`.
+  Most variables or methods previously typed as `ChatMessage` in `model` package are now typed as `ChatMessagesInner`. 
 - Add missing `@Beta` annotations to all `com.sap.ai.sdk.core.client` and `com.sap.ai.sdk.core.model` classes.
 
 ### ✨ New Functionality
 
+- Orchestration supports images as input in newly introduced `MultiChatMessage`.
+- `MultiChatMessage` also allows for multiple content items (text or image) in one object.
+- Grounding input can be masked with `DPIConfig`.
 - [Integrated the Orchestration client in Spring AI.](../guides/ORCHESTRATION_CHAT_COMPLETION.md#spring-ai-integration)
 
 ### 📈 Improvements
 
--
+- Update Orchestration client to version 0.43.0 (2412a)
 
 ### 🐛 Fixed Issues
 

orchestration/src/main/java/com/sap/ai/sdk/orchestration/ConfigToRequestTransformer.java

@@ -1,6 +1,6 @@
 package com.sap.ai.sdk.orchestration;
 
-import com.sap.ai.sdk.orchestration.model.ChatMessage;
+import com.sap.ai.sdk.orchestration.model.ChatMessagesInner;
 import com.sap.ai.sdk.orchestration.model.CompletionPostRequest;
 import com.sap.ai.sdk.orchestration.model.ModuleConfigs;
 import com.sap.ai.sdk.orchestration.model.OrchestrationConfig;
@@ -32,7 +32,10 @@ static CompletionPostRequest toCompletionPostRequest(
             OrchestrationConfig.create().moduleConfigurations(toModuleConfigs(configCopy)))
         .inputParams(prompt.getTemplateParameters())
         .messagesHistory(
-            prompt.getMessagesHistory().stream().map(Message::createChatMessage).toList());
+            prompt.getMessagesHistory().stream()
+                .map(Message::createChatMessage)
+                .map(ChatMessagesInner.class::cast)
+                .toList());
   }
 
   @Nonnull
@@ -45,7 +48,7 @@ static TemplatingModuleConfig toTemplateModuleConfig(
      * In this case, the request will fail, since the templating module will try to resolve the parameter.
      * To be fixed with https://github.tools.sap/AI/llm-orchestration/issues/662
      */
-    val messages = template instanceof Template t ? t.getTemplate() : List.<ChatMessage>of();
+    val messages = template instanceof Template t ? t.getTemplate() : List.<ChatMessagesInner>of();
     val messagesWithPrompt = new ArrayList<>(messages);
     messagesWithPrompt.addAll(
         prompt.getMessages().stream().map(Message::createChatMessage).toList());

orchestration/src/main/java/com/sap/ai/sdk/orchestration/JacksonMixins.java

@@ -18,4 +18,7 @@ interface LLMModuleResultMixIn {}
   @JsonTypeInfo(use = JsonTypeInfo.Id.NONE)
   @JsonDeserialize(as = LLMChoice.class)
   interface ModuleResultsOutputUnmaskingInnerMixIn {}
+
+  @JsonTypeInfo(use = JsonTypeInfo.Id.NONE)
+  interface NoneTypeInfoMixin {}
 }

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationChatResponse.java

@@ -3,6 +3,7 @@
 import static lombok.AccessLevel.PACKAGE;
 
 import com.sap.ai.sdk.orchestration.model.ChatMessage;
+import com.sap.ai.sdk.orchestration.model.ChatMessagesInner;
 import com.sap.ai.sdk.orchestration.model.CompletionPostResponse;
 import com.sap.ai.sdk.orchestration.model.LLMChoice;
 import com.sap.ai.sdk.orchestration.model.LLMModuleResultSynchronous;
@@ -50,21 +51,28 @@ public TokenUsage getTokenUsage() {
   /**
    * Get all messages. This can be used for subsequent prompts as a message history.
    *
+   * @throws UnsupportedOperationException if the MultiChatMessage type message in chat.
    * @return A list of all messages.
    */
   @Nonnull
-  public List<Message> getAllMessages() {
+  public List<Message> getAllMessages() throws UnsupportedOperationException {
     final var messages = new ArrayList<Message>();
 
-    for (final ChatMessage chatMessage : originalResponse.getModuleResults().getTemplating()) {
-      final var message =
-          switch (chatMessage.getRole()) {
-            case "user" -> new UserMessage(chatMessage.getContent());
-            case "assistant" -> new AssistantMessage(chatMessage.getContent());
-            case "system" -> new SystemMessage(chatMessage.getContent());
-            default -> throw new IllegalStateException("Unexpected role: " + chatMessage.getRole());
-          };
-      messages.add(message);
+    for (final ChatMessagesInner chatMessage :
+        originalResponse.getModuleResults().getTemplating()) {
+      if (chatMessage instanceof ChatMessage simpleMsg) {
+        final var message =
+            switch (simpleMsg.getRole()) {
+              case "user" -> new UserMessage(simpleMsg.getContent());
+              case "assistant" -> new AssistantMessage(simpleMsg.getContent());
+              case "system" -> new SystemMessage(simpleMsg.getContent());
+              default -> throw new IllegalStateException("Unexpected role: " + simpleMsg.getRole());
+            };
+        messages.add(message);
+      } else {
+        throw new UnsupportedOperationException(
+            "Messages of MultiChatMessage type not supported by convenience API");
+      }
     }
 
     messages.add(new AssistantMessage(getChoice().getMessage().getContent()));

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationClient.java

@@ -5,13 +5,15 @@
 import com.fasterxml.jackson.core.JsonProcessingException;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
 import com.fasterxml.jackson.databind.node.ObjectNode;
 import com.google.common.annotations.Beta;
 import com.sap.ai.sdk.core.AiCoreService;
 import com.sap.ai.sdk.core.DeploymentResolutionException;
 import com.sap.ai.sdk.core.common.ClientResponseHandler;
 import com.sap.ai.sdk.core.common.ClientStreamingHandler;
 import com.sap.ai.sdk.core.common.StreamedDelta;
+import com.sap.ai.sdk.orchestration.model.ChatMessagesInner;
 import com.sap.ai.sdk.orchestration.model.CompletionPostRequest;
 import com.sap.ai.sdk.orchestration.model.CompletionPostResponse;
 import com.sap.ai.sdk.orchestration.model.LLMModuleResult;
@@ -49,6 +51,14 @@ public class OrchestrationClient {
     JACKSON.addMixIn(
         ModuleResultsOutputUnmaskingInner.class,
         JacksonMixins.ModuleResultsOutputUnmaskingInnerMixIn.class);
+
+    final var module =
+        new SimpleModule()
+            .addDeserializer(
+                ChatMessagesInner.class,
+                PolymorphicFallbackDeserializer.fromJsonSubTypes(ChatMessagesInner.class))
+            .setMixInAnnotation(ChatMessagesInner.class, JacksonMixins.NoneTypeInfoMixin.class);
+    JACKSON.registerModule(module);
   }
 
   @Nonnull private final Supplier<HttpDestination> destinationSupplier;

orchestration/src/main/java/com/sap/ai/sdk/orchestration/PolymorphicFallbackDeserializer.java

@@ -0,0 +1,106 @@
+package com.sap.ai.sdk.orchestration;
+
+import com.fasterxml.jackson.annotation.JsonSubTypes;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.google.common.annotations.Beta;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import javax.annotation.Nonnull;
+import lombok.AccessLevel;
+import lombok.AllArgsConstructor;
+
+/**
+ * Handles polymorphic deserialization for a base class or interface.
+ *
+ * <p>This deserializer uses a fallback strategy by attempting to deserialize the JSON into known
+ * subtypes. Subtypes are either discovered using the {@link JsonSubTypes} annotation or provided
+ * explicitly. If deserialization fails for all candidates, a {@link JsonMappingException} is thrown
+ * with suppressed exceptions.
+ *
+ * @since 1.2.0
+ * @param <T> The base type for deserialization.
+ */
+@Beta
+@AllArgsConstructor(access = AccessLevel.PROTECTED)
+class PolymorphicFallbackDeserializer<T> extends JsonDeserializer<T> {
+
+  @Nonnull private final Class<T> baseClass;
+  @Nonnull private final List<Class<? extends T>> candidates;
+
+  /**
+   * Constructs the deserializer using candidates inferred from the {@link JsonSubTypes} annotation.
+   *
+   * @param baseClass The base class or interface to be resolved.
+   * @throws IllegalStateException If no subtypes are found.
+   */
+  @Nonnull
+  protected static <T> PolymorphicFallbackDeserializer<T> fromJsonSubTypes(
+      @Nonnull final Class<T> baseClass) {
+    final var subTypes = baseClass.getAnnotation(JsonSubTypes.class);
+    if (subTypes == null || subTypes.value().length == 0) {
+      throw new IllegalStateException("No subtypes found for " + baseClass.getName());
+    }
+
+    final var candidates = new ArrayList<Class<? extends T>>();
+    for (final var subType : subTypes.value()) {
+      candidates.add((Class<? extends T>) subType.value());
+    }
+
+    return PolymorphicFallbackDeserializer.fromCandidates(baseClass, candidates);
+  }
+
+  /**
+   * Constructs the deserializer with an explicit given list of candidate types.
+   *
+   * @param baseClass The base class or interface to be resolved.
+   * @param candidates A list of candidate classes to try deserialization.
+   */
+  @Nonnull
+  protected static <T> PolymorphicFallbackDeserializer<T> fromCandidates(
+      @Nonnull final Class<T> baseClass, @Nonnull final List<Class<? extends T>> candidates) {
+    return new PolymorphicFallbackDeserializer<>(baseClass, candidates);
+  }
+
+  /**
+   * Deserializes the JSON into the first matching candidate type.
+   *
+   * @param jsonParser The parser providing the JSON.
+   * @param deserializationContext The deserialization context.
+   * @return The deserialized object of a matching candidate type.
+   * @throws JsonMappingException If deserialization fails for all candidates.
+   * @throws IOException If json content cannot be consumed.
+   */
+  @Nonnull
+  @Override
+  public T deserialize(
+      @Nonnull final JsonParser jsonParser,
+      @Nonnull final DeserializationContext deserializationContext)
+      throws IOException {
+
+    final var root = jsonParser.readValueAsTree();
+    final var suppressed = new ArrayList<JsonMappingException>();
+
+    for (final var candidate : candidates) {
+      try {
+        return jsonParser.getCodec().treeToValue(root, candidate);
+      } catch (JsonMappingException e) {
+        suppressed.add(e);
+      }
+    }
+
+    final var aggregateException =
+        JsonMappingException.from(
+            jsonParser,
+            "PolymorphicFallbackDeserializer failed to deserialize "
+                + this.baseClass.getName()
+                + ". Attempted candidates: "
+                + candidates.stream().map(Class::getName).toList());
+
+    suppressed.forEach(aggregateException::addSuppressed);
+    throw aggregateException;
+  }
+}

orchestration/src/main/java/com/sap/ai/sdk/orchestration/model/AzureContentSafety.java

@@ -2,7 +2,7 @@
  * Orchestration
  * Orchestration is an inference service which provides common additional capabilities for business AI scenarios, such as content filtering and data masking. At the core of the service is the LLM module which allows for an easy, harmonized access to the language models of gen AI hub. The service is designed to be modular and extensible, allowing for the addition of new modules in the future. Each module can be configured independently and at runtime, allowing for a high degree of flexibility in the orchestration of AI services.
  *
- * The version of the OpenAPI document: 0.36.1
+ * The version of the OpenAPI document: 0.43.0
  *
  *
  * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).