fix: assertion of DynamoType failing

[verysamuel]

This is a complete test which reproduces the issue.

The reload function is sourced from this Stack Overflow response. Our tests currently rely on this behaviour. The use of this function contributes to the occurrence of this bug.

import importlib
import sys

import boto3
import pytest
from boto3.dynamodb.types import TypeSerializer
from moto import mock_dynamodb

serializer = TypeSerializer()


@pytest.fixture(autouse=True)
def reload(monkeypatch):
    importlib.reload(sys.modules["moto"])
    to_reload = [m for m in sys.modules if m.startswith("moto.")]
    for m in to_reload:
        importlib.reload(sys.modules[m])


@pytest.fixture
def setup_dynamo():
    with mock_dynamodb():
        dynamodb = boto3.client("dynamodb", region_name="eu-west-1")
        dynamodb.create_table(
            TableName="SomeTable",
            AttributeDefinitions=[
                {"AttributeName": "SomeKey", "AttributeType": "S"},
            ],
            KeySchema=[
                {"AttributeName": "SomeKey", "KeyType": "HASH"},
            ],
            BillingMode="PAY_PER_REQUEST",
        )
        yield dynamodb


@pytest.mark.parametrize(
    "value",
    [
        serializer.serialize(value)
        for value in [
            {"This test case passes": "A"},
            {"This test case fails": "B"},
        ]
    ],
)
def test_reproduce_dynamo_type_assertion_error(setup_dynamo, value):
    config_key = "SomeKeyValue"

    setup_dynamo.put_item(
        TableName="SomeTable",
        Item={"SomeKey": {"S": config_key}, "ConfigValue": serializer.serialize({})},
    )

    setup_dynamo.update_item(
        TableName="SomeTable",
        Key={
            "SomeKey": {"S": config_key},
        },
        UpdateExpression="SET SomeValue = :some_value",
        ExpressionAttributeValues={":some_value": value},
    )

    dynamo_item = setup_dynamo.get_item(
        TableName="SomeTable",
        Key={"SomeKey": {"S": config_key}},
    )["Item"]["SomeValue"]

    assert dynamo_item == value

Running the above code results in this assertion failing:

moto/moto/dynamodb/parsing/ast_nodes.py

Line 330 in bfac8a8

assert isinstance(value, DynamoType), "DDBTypedValue must be of DynamoType"

It seems something along the lines of this is going on: the reload function combined with the inconsistent imports means two different instances of DynamoType coexist and are in use.

Because the reload function is called between test invocations, this error only occurs when more than one test runs in a single pytest execution. So with the example above, running each case individually they'll both pass.

If appropriate, I can add the example above as a test here.

[bblommers]

That's odd - and must have been fun to debug!

This is definitely something that would need a test case though, just to prevent us from ever re-introducing the bug by accident.

Just out of interest - can it be reproduced by using the manual mocks?

m = mock_dynamodb().
m.start().
..
m.stop()
# reloading magic
m.start()
m.stop()

If so, that would mean the fix could be verified in a single test.

[verysamuel]

@bblommers "fun to debug!" is certainly one way to put it 😅

I have added a self-contained test which reproduces the issue using the style you've suggested.

[bblommers]

Our CI does not have a region configured, so we have to specify it here explicitly

[bblommers]

We should stop the mock at the end:
mock_dynamo.stop()

Not a big problem if you only run the one test, but if you run multiple tests and one mock is still active, it wreaks havoc.

[bblommers]

Hmm.. Our CI still doesn't seem to like that.

All the tests after the new tests are now failing:

2023-11-28T21:23:54.2345124Z tests/test_dynamodb/test_dynamodb.py::test_dynamo_type_assertion_exception_not_raised_when_reloading_modules PASSED
2023-11-28T21:23:54.3234042Z tests/test_dynamodb/test_dynamodb_batch_get_item.py::test_batch_items_returns_all PASSED
2023-11-28T21:23:54.7353450Z 
2023-11-28T21:23:54.7402019Z ##[error]test_batch_items_throws_exception_when_requesting_100_items_for_single_table

botocore.errorfactory.ResourceInUseException: An error occurred (ResourceInUseException) when calling the CreateTable operation: Table already exists: users
2023-11-28T21:23:54.7414803Z tests/test_dynamodb/test_dynamodb_batch_get_item.py::test_batch_items_throws_exception_when_requesting_100_items_for_single_table FAILED