How to scale order book integration for over 200 crypto exchanges
This is a more in-depth description of our previous “The WebSockets Adventure” blog post, which details our streaming order book structure. If you want to see how it works, you can go to our Order Book Streaming Documentation. I’m only covering the streamer version 2 (pure WebSockets) as you can’t get order book data though our old, streamer version 1 (Socket.Io).
As we all know, exchanges generally struggle when there is increased trading activity and they have much bigger budgets and teams than us. It might seem like a fool’s errand to try to stream and aggregate data in real-time from all of them if individually they have problems scaling to high load. We’ve been working on having both historical and streaming order book data over the last two years at CryptoCompare and we’re finally in a stage that we think is stable and scalable enough to support hundreds of thousands of order book updates per second.
To make all of this happen, we use two small but stable external tools: NSQ — A realtime distributed messaging platform and Redis Cluster — in-memory data structure store (we learned quite a lot about how Redis Cluster works when we were developing our Rate Limits with Redis-Cluster and NodeJs API service)
There are a few parts to the system, but the two main ones are the ingestion (how we get data from all the exchanges) and the distribution of data (how our customers get the data from us). The blue rectangles in the diagram below are services we’ve written in either Go or Nodejs.
The Order Book Data Ingestion
It is formed of three main services that talk to each other through NSQ queues. Each exchange runs on at least one virtual machine (we can easily shard by pairs if/when needed) and is composed of:
- The Exchange Input Service — knows how to connect to WebSockets / FIX streaming on the exchange and which messages to listen to. Acts as a proxy for other services if they need to request data from the exchange. Runs distributed to help with rate limits. Publishes raw messages to a local NSQ instance. Is mostly exchange independent.
- The Exchange Filter Service — subscribes to the raw messages published by the Exchange Input Service, knows how to convert the messages from the exchange format to our internal format. Publishes filtered/converted messages to a local NSQ instance. Schedules polling work for the Exchange Input Service. It is exchange-specific and the biggest chunk of work we need to do in order to integrate a new exchange.
- The Exchange Output Service — subscribes to filtered messages published by the Exchange Filter Service, keeps the order book state in memory, does validation (negative spread, is the sequence of messages correct, etc). Publishes validated order book level 1 and level 2 messages to a local NSQ instance and keeps the state of the order book on our Redis Data Cluster. Is mostly exchange independent.
There are a couple of auxiliary services: the Order Book to File Service— listens to validated messages and saves them in a file and the Order Book Snapshot Service — reads from the Redis Cluster every minute and saves it to a database. We have internal sequence numbers, so with the updates and the snapshots we can recreate the state of the order book at any point in time. All the data is stored unmapped under exchange asset ticker pair names.
The Order Book Data Distribution
It is formed of a few services that talk to each other through WebSockets, NSQ, Redis, and Postgres. Each streamer can serve up to 100,000 messages per second per CPU core. Like all our Nodejs services, it is globally load-balanced and has Nginx in front of it to do SSL and gzipping.
The streamers rely on four services in order to route traffic from the individual order book exchange integrations:
- Authentication — Checks API keys and auth keys to get the user permissions and roles
- Validation and mapping of the subscription — does the pair you are trying to subscribe to exist and what is it mapped? An example would be MIOTA on Bitfinex trades as IOT so you subscribe to 8~Bitfinex~MIOTA~BTC on our service and we send you remapped data from Bitfinex~IOT~BTC.
- Streaming data from NSQ — gets the NSQ connection from our lookupd cluster and starts listening to messages.
- Initial snapshot data from the Redis Cluster — gets the full state of the order book from Redis and returns it to the customer.
This was a more in-depth description of our previous “The WebSockets Adventure” blog post, which details our streaming order book structure.
Ready to see how it works? Head to our Order Book Streaming Documentation.
You might also like part one: The stack discovery, part two: The API dissection and part three: The Commercial API Journey
