paws.common > 0.8.5 breaks dynamodb updates to numeric values inside "map" attributes

[dleopold]

I recently migrated a project to a new system and updated packages, including paws. Previously working functions carrying out DynamoDB updates are now failing when updating a numeric value inside a map attribute. For example:

paws.database::dynamodb()$update_item(
    Key = list(
      pk = list(S = pk),
      sk = list(S = sk)
    ),
    ExpressionAttributeValues = list(
      `:v1` = list(
        M = list(
          numeric_attribute = list(N = 123)
        )
      ) 
    ),
    UpdateExpression = "SET test_attribute = :v1",
    TableName = "my-table"
  )

will fail with ! com.amazon.coral.service (HTTP 400). NUMBER_VALUE cannot be converted to String
Providing the number to a string allows a successful update, but should not be necessary and was not needed before:

 ExpressionAttributeValues = list(
      `:v1` = list(
        M = list(
          numeric_attribute = list(N = "123")
        )
      ) 
    )

the issue does not apply to updates of numeric attributes that are not inside map attributes. For example, this works:

paws.database::dynamodb()$update_item(
    Key = list(
      pk = list(S = pk),
      sk = list(S = sk)
    ),
    ExpressionAttributeValues = list(
      `:v1` = list(N = 123)
    ),
    UpdateExpression = "SET test_attribute = :v1",
    TableName = "my-table"
  )

When debugging this problem I noticed that my migration/update advanced me to paws.common 0.8.7 (which appears to be the current version on CRAN), yet the github release is 0.8.5. Reverting to 0.8.5 fixed the issue. I also tested 0.8.6 and that version also contains the bug.

[DyfanJones]

Ah sorry about that, can you provide the structure of your dynamodb table so I can reproduce the issue in my account please.

[DyfanJones]

Can this be reproduced in boto3? I am wondering if this was a bug in 0.8.5 and now we have fixed this behaviour but I want to make sure first :D

From looking at boto3.dynamodb.update_item and paws.dynamodb.update_item.

It looks like N: "string" and not N: 123. Which suggest there might be a bug in the previous version of paws. However I don't want to say for sure without some boto3 comparison and testing on my side :)

However the map attribute does raise a concern and a potential bug.

[DyfanJones]

Creating a dummy table so that I can test out the boto3.

library(paws)

client <- dynamodb(config(credentials(profile = "paws")))

# Create the DynamoDB table
client$create_table(
  TableName = "my-table",
  KeySchema = list(
    list(
      AttributeName = "pk",
      KeyType = "HASH"  # Partition key
    ),
    list(
      AttributeName = "sk",
      KeyType = "RANGE"  # Sort key
    )
  ),
  AttributeDefinitions = list(
    list(
      AttributeName = "pk",
      AttributeType = "S"  # String
    ),
    list(
      AttributeName = "sk",
      AttributeType = "S"  # String
    )
  ),
  BillingMode = "PAY_PER_REQUEST"
)

import boto3

# Create DynamoDB client
session = boto3.Session(profile_name='paws')
client = session.client('dynamodb')

# Update item
response = client.update_item(
    TableName='my-table',
    Key={
        'pk': {'S': pk},
        'sk': {'S': sk}
    },
    UpdateExpression='SET test_attribute = :v1',
    ExpressionAttributeValues={
        ':v1': {
            'M': {
                'numeric_attribute': {'N': 123}
            }
        }
    }
)

ParamValidationError: Parameter validation failed:
Invalid type for parameter ExpressionAttributeValues.:v1.M.numeric_attribute.N, value: 123, type: <class 'int'>, valid types: <class 'str'>
Cell In[5], line 1
----> 1 response = client.update_item(
      2     TableName='my-table',
      3     Key={
      4         'pk': {'S': pk},
      5         'sk': {'S': sk}
      6     },
      7     UpdateExpression='SET test_attribute = :v1',
      8     ExpressionAttributeValues={
      9         ':v1': {
     10             'M': {
     11                 'numeric_attribute': {'N': 123}
     12             }
     13         }
     14     }
     15 )
Show Traceback

It looks like boto3 returns a similar error. I believe boto3 captures the error before sending it to the AWS.

client.update_item(
    Key={
        'pk': {'S': pk},
        'sk': {'S': sk}
    },
    ExpressionAttributeValues = {
      ":v1": {"N": 123}
    },
    UpdateExpression = "SET test_attribute = :v1",
    TableName = "my-table"
  )

ParamValidationError: Parameter validation failed:
Invalid type for parameter ExpressionAttributeValues.:v1.N, value: 123, type: <class 'int'>, valid types: <class 'str'>
Cell In[6], line 1
----> 1 client.update_item(
      2     Key={
      3         'pk': {'S': pk},
      4         'sk': {'S': sk}
      5     },
      6     ExpressionAttributeValues = {
      7       ":v1": {"N": 123}
      8     },
      9     UpdateExpression = "SET test_attribute = :v1",
     10     TableName = "my-table"
     11   )

It looks the same for not being inside map attributes.

I believe the new behaviour is correct with the N: "string". However it looks like we have a bug when not in the map attribute. I will have a little look at that the see what is the issue

[DyfanJones]

I think the real issue is in the populate function - it's not correctly populating the attributes based on the method input. This is causing problems now that we've tightened up the JSON builder.
Before, the builder was pretty relaxed and just defaulted to string parsing when the attitude "type" wasn't there. That was fine in some ways, but it broke things for other services that actually needed R objects parsed correctly based on their class.
So the stricter parsing we have now is actually right, but it's exposing that populate isn't setting up the attributes correctly based on what the method is getting as input.

[DyfanJones]

@dleopold I believe I fixed it in my dev branch:

remotes::install_github("DyfanJones/paws/paws.common", ref = "feature/old_r")

Just need to clean it up and it should be ready to go.