[BUG] Batch inference failed: not all arguments converted during string formatting

[dferguson992]

Checks

• I have updated to the lastest minor and patch version of Strands
• I have checked the documentation and this is not expected behavior
• I have searched ./issues and there are no duplicates of my issue

Strands Version

1.3.0

Python Version

3.12.9

Operating System

macOS15.5

Installation Method

pip

Steps to Reproduce

1. Create a SageMakerAIModel object with an endpoint configuration featuring an "InService" inference component hosted on an "InService" SageMaker Inference Endpoint.

model = SageMakerAIModel(
    endpoint_config={
        'endpoint_name': 'endpoint-name',
        'region_name': 'endpoint-region',
        'inference_component_name': 'inference-component-name,
    },
    payload_config={
        'max_tokens': 256,
        'stream': False
    },
    boto_session=boto_session
)

2. Create an Agent using that new model:

system_prompt = "You are a helpful assistant that provides concise answers."
agent = Agent(
    model=model,
    system_prompt=system_prompt,
    tools=[calculator]
)

3. Submit a query to the agent object:

result = agent("Calculate the square root of -1. Briefly explain the significance of the result.")
print(result)

Expected Behavior

It is expected that the model will receive the request and respond comparable to the below smoke test:

smr_client = boto3.client("sagemaker-runtime") 
response_model = smr_client.invoke_endpoint(
    InferenceComponentName='inference-component-name',
    EndpointName='endpoint-name',
    Body=json.dumps({
      "inputs": "Calculate the square root of -1. Briefly explain the significance of the result.",
      "parameters": {
        "do_sample": True,
        "max_new_tokens": 256,
      }
    }),
    ContentType="application/json"
)
response = json.loads(response_model['Body'].read().decode('utf-8'))
response['generated_text']

" Likewise the field of complex numbers contains analogues of the above concepts. It also describes complex number multiplication, equality and ordering for complex numbers. By properties of the exponent and radicals, we get , where . Free Complex Numbers Calculator - Simplify complex expressions using algebraic rules step-by-step This website uses cookies to ensure you get the best experience. Contents All Basic Concepts. Thus we write as 2 + 3 i (not 2 + 3 j. ) z 1 and z 2, we need to know how to add, multiply, subtract, and divide complex numbers * Definition of the Complex Numbers. r is called the absolute value (or modulus ) of the complex number z, just as |d| is the absolute value of the real number d. The absolute value is a real number that is the distance between 0 and z. The conjugate is sometimes denoted using the notation z = a + ib Example: 1 + i 2 (In Python 3, complex numbers are implemented using the complex class in the standard library. Or, If you're using Microsoft Excel, you can type =COMPLEX(X,Y,Phase) for rectangular values. We will now talk about another way to represent a set of numbers which falls within the"

Actual Behavior

Strands responds with a code 424, and an error "Batch inference failed."

Here are the relevant logs from the agent invocation:

DEBUG | strands.models.sagemaker | invoking model
INFO | strands.models.sagemaker | response=<{
"code": 424,
"message": "Batch inference failed",
"properties": {},
"content": {
"keys": [],
"values": []
},
"cancelled": false
}>
DEBUG | strands.agent.conversation_manager.sliding_window_conversation_manager | message_count=<1>, window_size=<40> | skipping context reduction

Here is the screenshot of the stack trace error:

Here are the CloudWatch logs for this invocation (relevant lines bolded):

[WARN ] RollingBatch - Batch inference failed: prediction failure
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: _inputs, _param = parse_chat_completions_request(
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/tmp/.djl.ai/python/0.27.0/djl_python/chat_completions/chat_utils.py", line 31, in parse_chat_completions_request
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: chat_params = ChatProperties(**inputs)
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 171, in init
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: self.pydantic_validator.validate_python(data, self_instance=self)
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatProperties
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:messages.1.content
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: Input should be a valid string [type=string_type, input_value=[{'text': 'Calculate the ...sult.', 'type': 'text'}], input_type=list]
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: For further information visit https://errors.pydantic.dev/2.6/v/string_type
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:During handling of the above exception, another exception occurred:
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:Traceback (most recent call last):
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/usr/lib/python3.10/logging/init.py", line 1100, in emit
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: msg = self.format(record)
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/usr/lib/python3.10/logging/init.py", line 943, in format
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: return fmt.format(record)
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/usr/lib/python3.10/logging/init.py", line 678, in format
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: record.message = record.getMessage()
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/usr/lib/python3.10/logging/init.py", line 368, in getMessage
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: msg = msg % self.args
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:TypeError: not all arguments converted during string formatting
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:Call stack:
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/tmp/.djl.ai/python/0.27.0/djl_python_engine.py", line 195, in
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: main()
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/tmp/.djl.ai/python/0.27.0/djl_python_engine.py", line 166, in main
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: engine.run_server()
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: File "/tmp/.djl.ai/python/0.27.0/djl_python_engine.py", line 128, in run_server
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: logging.exception("Failed invoke service.invoke_handler()", e)
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:Message: 'Failed invoke service.invoke_handler()'
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:Arguments: (1 validation error for ChatProperties
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]:messages.1.content
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: Input should be a valid string [type=string_type, input_value=[{'text': 'Calculate the ...sult.', 'type': 'text'}], input_type=list]
[WARN ] PyProcess - W-248-85426b809e310c5-stderr: [1,0]: For further information visit https://errors.pydantic.dev/2.6/v/string_type,)

Additional Context

The API of this model is not OpenAI compatible which is a requirement for the SageMakerAIModel provider.

Possible Solution

Pull Request to incorporate a model provider flag which formats requests to the format required by the specified model provider.

Related Issues

No response

[pgrayy]

Is the API of this model OpenAI compatible? This is a requirement with the SageMakerModel provider (docs).

[dferguson992]

Yes you are correct. Am working on a PR to address this for SageMakerAIModel objects.

[pgrayy]

If you would like the SageMakerModel provider to support additional model APIs, I would recommend opening a feature request. However, for this particular case, we are likely to recommend that you either implement a custom model provider that you maintain on your end or perform the necessary request/response formatting within your SageMaker handler. The reason is that the team already has quite a large number of providers to support, each of which requires slightly different integration test and maintenance strategies. To ensure we keep up with support, we want to first implement better community contribution strategies before accepting new providers.