Add Bedrock Converse API chat model support #1650
Conversation
| this.observationRegistry) | ||
| .observe(() -> { | ||
|
|
||
| ConverseResponse converseResponse = this.bedrockRuntimeClient.converse(converseRequest); |
There was a problem hiding this comment.
Should we add a retry mechanism via RetryTemplate in case something might happen?
There was a problem hiding this comment.
I've noticed that the AWS SDK has a built-in retry utilities.
But we need to expose and configure them hopefully they will support the async model too.
There was a problem hiding this comment.
| private String model = "anthropic.claude-3-5-sonnet-20240620-v1:0"; | ||
|
|
||
| @NestedConfigurationProperty | ||
| private PortableFunctionCallingOptions options = PortableFunctionCallingOptions.builder() |
There was a problem hiding this comment.
With the Bedrock Converse API, different models have different options such as stop sequences etc. I wonder if we should add something to set default options for models.
Maybe we could add a new class BedrockConversePortableFunctionCallingOptions with a key-value map dynamic field and convert them to Bedrock Converse API's additionalModelRequestFields?
There was a problem hiding this comment.
Currently the PortableChatOptions and the PortableFunctionCallingOptions allow setting stopSequences and common chat and tool calling options that should work with all models.
The BedrockConversePortableFunctionCallingOptions with a key-value map dynamic field could be easiest, dynamic, approach to allow adding additionalModelRequestFields.
We can add it a follow up PR.
And in the future we can still add a type-safe options (e.g. dedicated per-model option class).
There was a problem hiding this comment.
Also we should explore if we can :
- implement additional metrics handler for the Bedrock Converse metrics.
- provide support for the guard features.
- review the cumbersome streaming tools calling impl.
Introduces support for Amazon Bedrock Converse API through a new BedrockProxyChatModel implementation. This enables integration with Bedrock's conversation models with features including: - Support for sync/async chat completions - Stream response handling - Tool/function calling capabilities - System message support - Image input support - Observation and metrics integration - Configurable model parameters and AWS credentials Adds core support classes: - BedrockUsage: Implements Usage interface for token tracking - ConverseApiUtils: Utility class for handling Bedrock API responses including: - Tool use event aggregation and processing - Chat response transformation from stream outputs - Model options conversion - Support for metadata aggregation - URLValidator: Utility for URL validation and normalization with support for: - Basic and strict URL validation - URL normalization - Multimodal input handling - Enhanced FunctionCallingOptionsBuilder with merge capabilities for both ChatOptions and FunctionCallingOptions - Added BEDROCK_CONVERSE to AiProvider enum for metrics tracking - Extended AWS credentials support with session token capability - Added configurable session token property to BedrockAwsConnectionProperties Adds new auto-configuration support: - BedrockConverseProxyChatAutoConfiguration for automatic setup of the Bedrock Converse chat model - BedrockConverseProxyChatProperties for configuration including: - Model selection (defaults to Claude 3 Sonnet) - Timeout settings (defaults to 5 minutes) - Temperature and token control - Top-K and Top-P sampling parameters - Integration with existing BedrockAwsConnectionConfiguration for AWS credentials Updates to testing infrastructure: - Adds comprehensive test suite for Bedrock Converse properties and auto-configuration - Integration tests for chat completion and streaming scenarios - Property validation tests for configuration options - Temporarily disabled other Bedrock tests due to AWS quota limitations - Added ObjectMapper configuration for proper JSON handling Added new spring-ai-bedrock-converse-spring-boot-starter module Updates module configuration in parent POM and BOM to include new bedrock-converse modules and starters. Adds necessary auto-configuration imports for seamless integration with Spring Boot applications. Unrelated changes: - Disabled several Bedrock model tests (Jurassic2, Llama, Titan) due to AWS quota limitations - Disabled PaLM2 tests due to API decommissioning by Google Resolves spring-projects#809, spring-projects#802 Co-authored-by: maxjiang153 <[email protected]>
- Move timeout configuration from chat properties to connection properties - Add comprehensive documentation for Bedrock Converse API usage and configuration - Update tests to reflect configuration changes Co-authored-by: maxjiang153 <[email protected]>
tzolov
force-pushed
the
add-aws-bedrock-converse-api
branch
from
ac4697f to
bf0f1b4
Compare
|
|
||
| if (!CollectionUtils.isEmpty(userMessage.getMedia())) { | ||
| List<ContentBlock> mediaContent = userMessage.getMedia().stream().map(media -> { | ||
| ContentBlock cb = ContentBlock.fromImage(ImageBlock.builder() |
There was a problem hiding this comment.
Bedrock also supports documents, it would be nice to have this feature implemented too. In my use-case I solved it with such convertMediaToContentBlock method:
if (IMAGE.equals(mimeType.getType())) {
return ContentBlock.fromImage(image -> image.format(
media.getMimeType().getSubtype())
.source(ImageSource.fromBytes(SdkBytes.fromByteArray(getContentMediaData(media.getData())))));
} else {
String fileName = mimeType.getParameter(FILE_NAME_KEY);
return ContentBlock.fromDocument(document -> document.format(
FileNameUtils.detectDocFormatByFileName(fileName))
.name(FilenameUtils.getBaseName(fileName))
.source(DocumentSource.fromBytes(SdkBytes.fromByteArray(getContentMediaData(media.getData())))));
}
possibly passing the filename in the mimeType is not the best way, but I did not find better option.
public static String detectDocFormatByFileName(String fileName) {
FileTypeValidator.validateAllowedDocumentType(fileName);
String extension = FilenameUtils.getExtension(fileName);
if (FileTypeValidator.isAcceptedDocumentFormat(extension)) {
return extension;
}
return "txt";
}
Note that the api accepts very few file formats, so types like .json, .java etc can be sent as txt
PDF, MD, TXT, DOC, DOCX, HTML, CSV, XLS, XLSX
| @ValueSource(strings = { "anthropic.claude-3-5-sonnet-20240620-v1:0" }) | ||
| void multiModalityImageUrl(String modelName) throws IOException { | ||
|
|
||
| // TODO: add url method that wrapps the checked exception. |
| metadataEvent.usage().outputTokens().longValue(), | ||
| metadataEvent.usage().totalTokens().longValue()); | ||
|
|
||
| // TODO |
There was a problem hiding this comment.
Create an issue on what 'TODO'
I suppose then we keep the delta of converse and the existing bedrock support in the project and deprecate and eventually remove the areas of overlap? |
|
What about |
|
rebased, squashed and merged at 0d2d4b7 |
4 participants
Introduces support for Amazon Bedrock Converse API through a new BedrockProxyChatModel implementation. This enables integration with Bedrock's conversation models with features including:
Adds core support classes:
Adds new auto-configuration support:
Updates to testing infrastructure:
Added new spring-ai-bedrock-converse-spring-boot-starter module
Updates module configuration in parent POM and BOM to include new bedrock-converse modules and starters. Adds necessary auto-configuration imports for seamless integration with Spring Boot applications.
Unrelated changes:
Resolves #809, #802