Companies Stream

This project is a visualiser of changes made to the Companies House database of UK companies. It shows events as they happen in realtime, such as a new company registering, or a company going insolvent.

Companies House offers a streaming API, which sends events over a HTTPS connection.

Technology

• server written in TypeScript with Elysia framework and Bun runtime run in Docker container deployed on Digital Ocean.
• Redis Streams.
• WebSockets for client-server communication.
• Frontend client written in Typescript and built with Vite (no frameworks, 10kb bundle).
• Served by Caddy

How it works

A Docker compose application with 3 main components for the backend:

1. A container to listen on the Companies House streaming API and publish events to Redis ( see streamToRedis.ts).
2. A Redis instance, mostly for facilitating Pub/Sub communication of events using Streams, and also for storing most recent timepoint to avoid missing any events.
3. A container which reads events from the Redis Stream and serves them as a WebSocket endpoint on /events, where each event is sent as a WebSocket message.

The frontend is "pure" in that it doesn't use a framework like React, which keeps the JS bundle really tiny and high performance. The animations are done in CSS using SASS.

To start up the whole application, clone the repository and run docker compose up -d --build in the root. You will need an env file named .api.envcontaining streaming API key(s). To run any files without Docker, install Bun (bun install in /server and /client).

Make your own

If you are interested in using the companies house streaming API, visit developer-specs.company-information.service.gov.uk to create an account for a free API key. The API base url is https://stream.companieshouse.gov.uk with paths /companies, /filings, /officers, persons-with-significant-control, /charges, /insolvency-cases and disqualified-officers.

Here is a minimum working example in NodeJS using the split2 package:

import split2 from 'split2' // requires `npm i split2`
import { get } from "https"

const auth = process.env.STREAM_KEY + ":"
const path = "/filings"
const options = { hostname: "stream.companieshouse.gov.uk", path, auth }
get(options, (res) => {
  if (res.statusCode === 200)
    res.pipe(split2(JSON.parse)).on("data", console.log)
  else res.pipe(process.stdout)
  res.on("end", () => console.log("Stream ended."))
}).end()

For a more complete working example of listening on a stream in Javascript, see examples/splitStream.ts and then server/src/chStreamToRedis/streamToRedis.ts.

To test the streaming API from the command line with CURL, you can use the cmdline utility (after compiling):

curl --user APIKEY: -s https://stream.companieshouse.gov.uk/filings | bun examples/streamCmdLine.ts

Questions?

If you have any questions or suggestions please open an issue on the repository, and I will get back to you.

Using this code

You are welcome to use this open source code for your own projects. See the LICENSE file for more information.

Running the project with Docker

• create .env file

   SITE_ADDRESS=http://localhost:80
  

• create .api.env file with Streaming API key

  STREAM_KEY=dd559b99-7870-4792-8a38-054f203eb818
  

• create docker volume for webserver certificates

  docker volume create caddy_data
  

• build and run docker containers

  docker compose up -d --build
  

• build frontend client (and copy dist files into caddy container volume)

  cd client-pure
  bun install
  bun run build
  cd .. 
  ./copyDist.sh
  ./reloadCaddy.sh
  

• try go to http://localhost in a web browser!