Skip to content

Commit

Permalink
Edit traderx blog post into a part 1 and publish
Browse files Browse the repository at this point in the history
  • Loading branch information
@refset
refset committed Dec 9, 2024
1 parent 1d74870 commit 9bf1178
Showing 1 changed file with 7 additions and 52 deletions.
59 changes: 7 additions & 52 deletions src/pages/blog/bitemporal-traderx.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: 'Bitemporal TraderX - augmenting a sample trading system'
description: 'Adding XTDB and Clojure into a sample microservices trading application for the FINOS 2024 Tech Sprint'
category: 'database'
layout: '../../layouts/BlogPost.astro'
publishedDate: '2024-11-20'
publishedDate: '2024-12-09'
heroImage: 'bitemporal-traderx.jpg'
tags:
- 'SQL'
Expand All @@ -19,7 +19,7 @@ import { Image } from 'astro:assets'

In case you missed the news, JUXT is now a member of [FINOS](https://www.finos.org/)! The Fintech Open Source Foundation (FINOS) is an umbrella organization under the Linux Foundation, whose purpose is to accelerate collaboration and innovation in financial services through the adoption of open source software, standards and best practices.

Throughout August and September I was the main engineer on a team from JUXT who took part in FINOS' [2024 Tech Sprint](https://www.finos.org/blog/finos-tech-sprint-2024) - a multi-week hackathon centred around one the FINOS projects called '[TraderX](https://github.com/finox/traderx)'. Unlike most of the other projects that FINOS hosts, which are used in production by various member organisations, TraderX is essentially an example system (think '[Pet Store](https://en.wikipedia.org/wiki/Java_BluePrints)') that is intended purely for educational and training purposes. TraderX was initially developed by Morgan Stanley.
Throughout August and September I was the lead engineer on a team from JUXT who took part in the FINOS [2024 Tech Sprint](https://www.finos.org/blog/finos-tech-sprint-2024) - a multi-week hackathon centred around one the FINOS projects called '[TraderX](https://github.com/finox/traderx)'. Unlike most of the other projects that FINOS hosts, which are used in production by various member organisations, TraderX is essentially an example system (think '[Pet Store](https://en.wikipedia.org/wiki/Java_BluePrints)') that is intended purely for educational and training purposes. TraderX was initially developed by Morgan Stanley.

## What we achieved in the 2024 Tech Sprint

Expand All @@ -34,7 +34,7 @@ In summary, we:
- Used XTDB's bitemporal SQL features to easily model the TraderX domain with limited code changes. The underlying data model now allows for all reference and pricing data to be auditably adjusted, and for all trades to be amended or cancelled retroactively, while providing complete visibility to the end user
- Opened a [PR](https://github.com/finos/traderX/pull/237) to propose these changes back to the main TraderX repository

This work was showcased at the FINOS 'OSFF' conference in New York last month, where my colleague Jeremy (XTDB Head of Product) also presented on the importance of bitemporality more broadly - you can read/watch more about this [here](/blog/reconciliation-risk-ml-and-use-cases-for-bitemporal-systems/).
This work was showcased at the FINOS 'OSFF' conference in New York in October, where my colleague Jeremy (XTDB Head of Product) also presented on the importance of bitemporality more broadly - you can read/watch more about this [here](/blog/reconciliation-risk-ml-and-use-cases-for-bitemporal-systems/).

## Changes implemented in our PR

Expand Down Expand Up @@ -79,56 +79,11 @@ The reference-service is written in Clojure, which we chose for expedience, but

The service interfaces with XTDB via vanilla [pgJDBC](https://jdbc.postgresql.org/) and SQL, which is possible thanks to XTDB v2's compatibility with the Postgres wire protocol. This also meant that during development we could access the database using tools like VSCode's [SQLTools](https://marketplace.visualstudio.com/items?itemName=mtxr.sqltools) and [Metabase](https://www.metabase.com/) to debug our queries and data, without the need for any custom drivers or less conventional tools.

## XTDB experience report - first impressions
Overall the project was quite straightforward as all the pieces fit together nicely. However, having worked with many other databases previously, I had strong first impressions when picking up XTDB for this project and plan to share more details on that soon. Stay tuned!

This was my first time using XTDB so I had some learning to do myself. It takes a little time to get used to the temporal syntax for making ‘updates’ with valid_time (but then - you only have to learn it once).
My first reaction was to add dedicated columns for handling various application and auditing timestamps like `traded_at` and `updated_at` - but what a pleasant surprise it was to find out there’s little need for many of these use-cases - timestamping is already ubiquitous and all done under the covers.
## XTDB webinar on 11th December and the search for Design Partners

*For all intents and purposes how you interact with the database does not change, compared to traditional SQL databases* - until you need to get the historical data or introduce retrospective updates (where you now know that some data was incorrect in the past and you can update that fact specifying the correct value and period during which it was the case). This still leaves you with a perfectly auditable history - you can see that the retrospective change for a particular record was created at `system time`.
The XTDB team is running a webinar in conjunction with FINOS on 11th December, you can learn more and register for that [here](https://zoom.us/webinar/register/4617283986911/WN_3b4DvHhvQbCtt98DOvjbDQ).

Wait - what? History? Yes! You get all the change history for free! - not just what changed when - but you can inspect all those historical values. Updates don’t actually overwrite old values - they create new entries with two time lines - valid time (or application time) and system time (or what you can think of as wall-clock really).
Finally, JUXT is currently collaborating with several Design Partners using XTDB who help us discover missing features - we cannot responsibly take on too many partners (as we do need to support those features!), but if you feel like XTDB is the right fit for your problems, please do get in touch: [[email protected]](mailto://[email protected])

The way the XTDB team have extended the 'normal' update-in-place semantics of SQL is very intuitive.

### Integrating sliders with SQL

import sliders from '../../assets/blog/bitemporal-traderx.png'

<Image style="width: 100%; margin-left: auto; margin-right: auto" alt='View of TraderX showing SQL overlayed' src={sliders} />

Adding temporal 'sliders' to the UI is an easy way to expose the time-travel abilities, and makes it far more efficient to quickly observe changing data than having users click on simple calendar widgets.

Getting data from a couple of joined tables (when you want to see it ‘as of’ a certain point in time) needs to be done with some ceremony (for each table you have to specify its own `valid time` or `system time`).

It also took me a couple iterations to get correct `time points` for the ‘event’ slider (which shows ‘history’ of trades - and therefore positions).

My initial thought was to naively query for a collection of time periods, with the period start being `_valid_from` and period end being `_valid_to`, for all trades in ascending order. This however yielded unwanted results and seemed quite confusing (because `_valid_from` and `_valid_to` would be interleaved and not correlated between trades, as some trades ‘disappeared’ while others concurrently ‘changed state’ from settled to pending).

At that point I changed the ‘period dropdown’ to a slider where the points are all `_valid_from` values for all trades (and there’s no restriction on `_valid_to`). This resulted in a clear history of trades and their lifecycle - from their creation in pending state to settlement.

In addition to the main blotter slider I also added a price slider. This was much easier, essentially a basic view of price ‘as of’ some given day, using regular daily steps. The blotter query then effortlessly joins across these two underlying table sources using their respective as-of timestamps.

## My XTDB Takeaways

I have worked on several applications that incorporate reporting requirements previously, and I can safely say XTDB brings a lot of benefits over the traditional SQL database paradigm.

**Having added bitemporal storage unlocks incredible reporting capabilities** - not only over past changes, but also opening the door to speculative analysis (e.g. "what would my account PnL look like with prices from a given day in the past" / "how much could I have gained/lost if I had bought stock X on date Y").

**A first-class history close to hand is a game changer in terms of application development** - a whole range of time related complexities are taken care of by the bitemporal model and XTDB makes working with bitemporal data very easy. You could compare it to what happened when you could relegate transaction management from application logic into the database layer. There is a little bit to learn about in order to start using [SQL:2011](https://en.wikipedia.org/wiki/SQL:2011)’s temporal operators - but this is really nothing compared to benefits you get from just using it.

**XTDB provides bitemporal storage with virtually no barrier to entry** - DML (SQL statements for storing / updating entities) simply does not change existing data, while the retrieval of historical data is trivial using SQL2011 queries (leveraging ‘FOR VALID TIME’ to get a point/period in the past). The only caveat in this project was that XTDB doesn't yet fully work with JPA style persistence due to minor differences with real Postgres, but working with plain SQL or simpler abstractions (e.g. [jOOQ](/blog/15-years-of-jooq-with-lukas-eder/)) is my preference anyway.

**No more need for audit tables** or manual addition of more timestamped fields (such as ‘updated_at’) - history is transparently preserved along with the two timelines - system time and valid time.

**XTDB drastically reduces complexity** otherwise introduced by hand-rolled reporting solutions - these usually come in later as post production release requirements (dictated by reporting/auditing/legal needs).
You’re free to focus on adding competitive value to your application - unique features that will set you out from your competition - without expending effort on things which, whilst often absolutely required, do not necessarily give end users visible benefit.

## Ideas for the future

If XTDB can add complete support for JPA style annotations, then XTDB could fully replace the use of the [H2](https://www.h2database.com/html/main.html) database within the TraderX repo, without any code changes. In a future Tech Sprint we may consider attempting this, and also adding additional reporting functionality (e.g. predicted vs. actual comparisons).

JUXT is currently collaborating with several Design Partners using XTDB who help us discover missing features - we cannot responsibly take on too many partners (as we do need to support those features!), but if you feel like XTDB is the right fit for your problem, please do get in touch: [[email protected]](mailto://[email protected])

## XTDB webinar on 11th December

Finally, the XTDB team is running a webinar in conjunction with FINOS on 11th December, you can learn more and register for that [here](https://zoom.us/webinar/register/4617283986911/WN_3b4DvHhvQbCtt98DOvjbDQ).

0 comments on commit 9bf1178

Please sign in to comment.