Datomic queries returning as little as possible

One of the value propositions of om.next is that clients have the power to decide what data they want to receive from the server. Clients that require less data can retrieve smaller payloads than clients that require more data, without modifying the API. To bring this idea home for a longer session of client requests, we’re only transmitting any new changes to the attributes a client wants. We’re able to walk the pattern the client provides for each root query, and return only data that’s changed on that path.

What we end up returning from the server is anything that datomic’s and datascript’s transact! function can take as the last argument, i.e. transaction data. Transaction data can either be data surfaced by pull or it can be datoms. Datoms are sorted by their transaction/tx number and pulled data is appened at the bottom, as pulled data refers to what's currently true.

We’ve always been unsure how scalable or performant this approach is, and we unfortunately didn’t get to try it at a large scale.

Check out the one and all functions in the eponai.server.datomic.query namespace. Their signature is [db db-history pull-pattern entity-query], where the db-history is filtered with datomic.api/since with a t of the last time the entity was called, and the entity-query is a query in our format described in a previous highlight.

Extending the API to Datascript

We briefly thought about extending the API explained above to Datascript, but we never got time to do it. Here are some notes I found in our internal Jira.

Implementing (d/since (d/history db) old-db) for datascript could mean that our reads on the client would all incrementally update.

Sean Tempesta implemented a diff for datascript's btsets: <unfortunately the link does not exist. Why didn't we download the source?> 🤦‍♂️

Transacting the result of (datoms-diff ..) to a new db is equivalent to d/since. Calculating this d/since is eager though, we need it to be lazy. In addition to d/since we need d/history, which enables one to query for [e a v tx added?] instead of just [e a v t].