Add AWS Bedrock AI support #174

tzolov · 2023-12-17T22:34:06Z

Add spring-ai-bedrock project with support for Cohere, Llama2, Ai21 Jurassic 2, Titan and Anthropic LLM models.
Add native API clients for CohereChat CohereEmbedding , Llama2Chat, JurassicChat, TitanChat and TitanEmbedding models, supporting both single shot and streaming completions (for the models that allows it)
Add ITs tests for the native API clients.
Implement Chat (AiClient) and ChatStreaming (AiStreamingClient) and EmbeddingClients (according to the models’ support for those) for Cohere, Llama2, and Anthropica. Titan and Jurassic2 are WIP.
Add ITs for the ChatClient, ChatStreamingClient and EmbeddingClient implementations.
Add Spring Boot Auto-configurations with flexible properties for the Llama2, Anthropic and Cohere modes + ITs
Add Spring Boot Starter configurations for all Bedrock models.
Add README documentations for all models.

Resolves #66

markpollack · 2023-12-18T17:05:00Z

spring-ai-bedrock/README.md

@@ -0,0 +1,83 @@
+# Bedrock AI Chat and Embedding Clients


The documentation for these base clients still needs to go into our reference documentation, which is in adoc format. I will create a separate issue for that.

markpollack · 2023-12-18T17:07:46Z

spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/MessageToPromptStrategy.java

+/**
+ * Converts a list of messages to a prompt for particular model.
+ *
+ * TODO: consider factoring out this into an interface and provide different


Create an issue instead of a TODO. Will do on merge.

Rename MessageToPromptStrategy to MessageToPromptConverter

markpollack · 2023-12-18T17:09:28Z

spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/MessageToPromptStrategy.java

+			.map(this::messageToString)
+			.collect(Collectors.joining("\n"));
+
+		final String prompt = String.format("%s\n\n%s\n%s", systemMessages, userMessages, ASSISTANT_PROMPT);


use %n instead of \n and can return directly instead of storing in a string first.

markpollack · 2023-12-18T17:12:54Z

spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/api/AbstractBedrockApi.java

+				.build();
+
+		InvokeModelResponse response = this.client.invokeModel(invokeRequest);
+		// BedrockRuntimeResponseMetadata metadata = response.responseMetadata();


remove commented line

markpollack · 2023-12-18T17:15:08Z

...drock/src/main/java/org/springframework/ai/bedrock/cohere/api/CohereEmbeddingBedrockApi.java

+			 * In search use-cases, use search_document when you encode documents for embeddings that you store in a
+			 * vector database.
+			 */
+			search_document,


shouldn't these all be upper case names?

markpollack · 2023-12-18T17:19:22Z

...ai-bedrock/src/main/java/org/springframework/ai/bedrock/llama2/api/Llama2ChatBedrockApi.java

+			/**
+			 * The model has finished generating text for the input prompt.
+			 */
+			stop,


should these be upper case to follow existing conventions?

markpollack · 2023-12-18T17:20:37Z

...bedrock/src/main/java/org/springframework/ai/bedrock/titan/api/TitanEmbeddingBedrockApi.java

+	 *
+	 * @param embedding The embedding vector.
+	 * @param inputTextTokenCount The number of tokens in the input text.
+	 * @param message TODO


extraneous TODO

markpollack · 2023-12-18T17:23:44Z

spring-ai-bedrock/src/test/resources/doc/Bedrock Cohere Chat API.jpg

these jpg should to into some spot in the antora docs. ATM we don't yet have many diagrams in the docs, e.g. ETL pipeline, which would greatly enhance the docs.

markpollack · 2023-12-18T17:25:29Z

spring-ai-huggingface/pom.xml

@@ -42,6 +42,11 @@


 		<!-- Spring Framework -->
+		<dependency>


why is this dependency necessary to add?

markpollack · 2023-12-18T17:28:11Z

vector-stores/spring-ai-chroma/pom.xml

@@ -27,6 +27,11 @@
 			<version>${parent.version}</version>
 		</dependency>

+		<dependency>


why was this added?

- Add spring-ai-bedrock project with support for Cohere, Llama2, Ai21 Jurassic 2, Titan and Anthropic LLM models. - Add native API clients for CohereChat CohereEmbedding , Llama2Chat, JurassicChat, TitanChat and TitanEmbedding models, supporting both single shot and streaming completions (for the models that allows it) - Add ITs tests for the native API clients. - Implement Chat (AiClient) and ChatStreaming (AiStreamingClient) and EmbeddingClients (according to the models’ support for those) for Cohere, Llama2, and Anthropica. Titan and Jurassic2 are WIP. - Add ITs for the ChatClient, ChatStreamingClient and EmbeddingClient implementations. - Add Spring Boot Auto-configurations with flexible properties for the Llama2, Anthropic and Cohere modes + ITs - Add Spring Boot Starter configurations for all Bedrock models. - Add README documentations for all models. Resolves spring-projects#66

tzolov · 2023-12-18T18:32:03Z

spring-ai-bedrock/README.md

@@ -0,0 +1,83 @@
+# Bedrock AI Chat and Embedding Clients


tzolov · 2023-12-18T18:39:09Z

spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/MessageToPromptStrategy.java

+/**
+ * Converts a list of messages to a prompt for particular model.
+ *
+ * TODO: consider factoring out this into an interface and provide different


Rename MessageToPromptStrategy to MessageToPromptConverter

tzolov · 2023-12-18T18:51:01Z

...ck/src/test/java/org/springframework/ai/bedrock/anthropic/api/AnthropicChatBedrockApiIT.java

+			.withTemperature(0.8f)
+			.withMaxTokensToSample(300)
+			.withTopK(10)
+			// .withStopSequences(List.of("\n\nHuman:"))


markpollack · 2023-12-18T23:23:46Z

merged in 0e3192a

tzolov force-pushed the bedrok-support branch 2 times, most recently from 368346c to 810d2be Compare December 18, 2023 07:40

tzolov mentioned this pull request Dec 18, 2023

Support for AWS Bedrock & TItan #67

Closed

tzolov force-pushed the bedrok-support branch from 59dc188 to 6d0364f Compare December 18, 2023 13:39

markpollack reviewed Dec 18, 2023

View reviewed changes

tzolov added 6 commits December 18, 2023 19:20

Remove json-starter in core and fix OpenAI and Vertex API changes

f26e41d

Add BedrockAi APIs AOT hints

1e4c10b

Add Ai21Jurassic2ChatBedrockApi test

55287d0

Add TitanEmbeddingBedrockApi test

13c1323

Add TitanChatBedrockApi test

289bc8d

tzolov commented Dec 18, 2023

View reviewed changes

tzolov force-pushed the bedrok-support branch from 6d0364f to 289bc8d Compare December 18, 2023 19:05

address review comments

9c6c81f

markpollack closed this Dec 18, 2023

markpollack added this to the 0.8.0 milestone Jan 2, 2024

Add AWS Bedrock AI support #174

Add AWS Bedrock AI support #174

Uh oh!

Conversation

tzolov commented Dec 17, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markpollack commented Dec 18, 2023

Uh oh!

Uh oh!