DynamoDB Checkpoint Saver failing 'list' tests

[arcticShadow]

I'm working on a dynamodb Checkpoint saver for langgraph JS. Plan to open source it - once its working properly. At the moment its still directly coded in NestJs so any interested parties not using NestJS would need to unpick that (not difficult - mainly swap out the dynamoose implementation if your not already using it with plain dynamodb sdk)

I'm having an issue resolve the last couple of tests supplied from the excellent checkpoint-validation module.

I've put my current code and test implementation here https://gist.github.com/arcticShadow/58c813c5047f7926126a8008eaec5a55

When i run my test - i'm getting a hand full of issues (28 to be exact) that all follow the same problem:

/src//dynamodb.checkpointer.adapter.test.ts > dynamodb-checkpoint-saver > dynamodb-checkpoint-saver#list > 'should return 2 tuples when thread_id matches pushed checkpoint(s), checkpoint_ns is not specified, limit is 2, before is not specified, and metadata.source matches "loop"'

AssertionError: expected 1 to deeply equal 2

− Expected
+ Received

− 2
+ 1

I've debugged quite deep into this - and I can see in the datastore - only 1 record is being persisted that matches the query conditions.

So my question then:

• Should i be expecting two matching records in the datastore (indicating my put logic is incorrect)
• Or is the list code mean to split one record into 2 entries (indicating my list logic is wrong)
• Or should the test only be looking for one record, not two (indicating the upstream tests are wrong)

@benjamincburns I've seen your involvement in the checkpointer validation library - potentially you may have some insights here.

[benjamincburns]

Apologies, it's been a little while since I wrote those tests, but I did my best to make them into as complete of a black-box validation suite as possible. That said, the list tests (which appear to be the tests that are failing for you) are written to test the list method against an assumed set of requirements. In reality the list method is used only sparingly in LangGraph (or at least, that was the case when I was part of the team), and I expect that the test suite may be overly strict in its checks there.

Also the list tests are somewhat pesky to understand, as they're combinatorial tests. In order to understand what the test failure is actually describing you'll need to look at the pre-populated checkpoint data in the test setup and walk through the test logic to determine what the expected result should be.

From memory, I believe the list method is intended to treat each input parameter (e.g. checkpoint_ns) as a wildcard when it's not specified, and all other criteria are meant to be exact matches that AND together. So, for the error you pasted above (when thread_id matches pushed checkpoint(s), checkpoint_ns is not specified, limit is 2, before is not specified, and metadata.source matches "loop"), you should be able to infer the inputs used for the failing test, and then walk through the assertions to determine the outputs.

Finally, bear in mind that for some datastores a fully complete (according to the validation suite) list method may not be practical or even possible. In that case you should probably look to the actual LangGraph source code to find where list is actually used and make sure that the tests that exercise those cases pass. If you're only seeing 28 test failures out of the 500+ (I forget the exact number) list test cases, it's quite likely that you've already covered those cases and these failures are simply indicating that you may not be "future proof" in these areas. Obviously you'd need to validate that, however.

[benjamincburns]

One other detail about the list method and tests. When I wrote the validation suite, I was under the assumption that the checkpointer list interface would be useful for application authors who are e.g. looking to query for past threads for display purposes. This may have been somewhat of a mistake on my part, as generally it's better practice for applications to treat checkpoints as a black box and do their own application-specific thread or session tracking, storing off the LangGraph thread_id (and only as necessary for e.g. forking, the checkpoint ID).

As a result of the above, it may be that the list tests are actually overspecified. I defer to current project maintainers (e.g. @hntrl and @dqbd) on this topic, however. I may be off in my current understanding of best practices.

[arcticShadow]

Thanks for the detailed response, and no apologies necessary. I'm aware its not ideal to specifically tag and call people into thread - especially when I have no connection to the project and don't know the individuals involved. So I appreciate the reply!

It's a bit of an interesting problem - And I was happy to let the dragons lie as everything i've done so far seemed to be working well. However I have since come across a new problem (I believe its unrelated) that until solved actually make this checkpointer kind of useless.

in a nut shell - DynamoDB can only store 400kb give of take per record. This implementation stuffs the entire state into one record - and can hit that limit quite quickly. Given the potential for individual message in state to be quite large - as anything can be stored in a message, i'm not sure on the best approach. I've been messing with encoding the entire checkpoint and chunking it, and trying to split the checkpoint by state key but neither is giving me a reliable approach yet. This is entirely unrelated to the test issue - Mainly adding notes here for anyone that may want to use the WIP dynamodb checkpointer linked above

Once I resolve the above, then i'll take another look at the tests.

But until then - other priorities are calling.