Feature request: Create an acceptance test suite for checkpointers

[benjamincburns]

As @hgoona mentioned in this comment, the requirements for custom checkpointers aren't abundantly obvious from the docs, or even the LangChain-provided checkpointer implementations.

It would be helpful if a black-box acceptance test suite existed for checkpointers, ideally one that could be used both by the LangGraph team, and authors of third party checkpointer implementations.

The suite would need to exercise all of the requirements of the checkpointer contract, such that if an arbitrary checkpointer passes this suite, the author and community would have high confidence that the checkpointer in question fulfills all of LangGraph's requirements for storing and retrieving persisted state.

Ideally this would also exist for stores. I'm happy to open another issue for those if the maintainers would prefer to track these separately.

[vbarda]

@benjamincburns thanks for the feedback! this is a great point and we actually do plan to do this for both Python and JS, we'll follow up here once we have more details

[benjamincburns]

@vbarda are you guys interested in external contributions for this for the JS side? I'd imagine it'd be a riskier contribution for a third party to make given that anyone external to the team would essentially be reverse-engineering the interface specification, not to mention making some assumptions around how you guys would want to structure the test suite, what tooling to use, etc.

That said, I'm finding that I need something like this for my own project, so I'm likely to be building something of this ilk soon if you guys don't do it for me. If the thing that I come up with looks like it might be more broadly useful, I'd be happy to submit it as a PR. I'd just expect that I'd need some help from the LangGraph team to massage it into a shape that fits nicely in this repository.

Any preferences you can share here on the sorts of assumptions I mention above would also be quite helpful.

[hgoona]

@vbarda I'd be keen but I'm quite green to this side of things and feel very lost in my understanding of the anatomy of these Checkpointer Savers (or whatever they are called <<< the naming of these parts are in and of itself a hard ask to form a mental model around. It really could be explained better or named better too).

Currently, @benjamincburns I'm also trying to push through to make a custom Checkpoint Saver for use with SurrealDB, regardless of the missing info (above), but not doing great at all... I'm referencing the Sqlite Saver at present.
I'd be happy to share notes if that is helpful??

[benjamincburns]

@hgoona if you DM me on the LangChain slack I'll drop my notes doc your way. It's a bit terse (mostly just mapping out types and asking/answering my own questions as bullet points) but it may help. You can find me as "Ben Burns" there with the same pfp.

I can't guarantee that I'll reply outside of my usual working hours, though (NZ timezone). I'm also planning on taking the day off tomorrow (although I may still be around as I might hack on a personal project with LangGraph).

Edit: I went ahead and uploaded them as a gist. If you want to discuss them, please reach out to me on Slack or leave comments on that gist.

[hgoona]

@benjamincburns Noted! Though, I don't have access to that Slack. Are you on the LangChain Discord server to talk further?