Tools: Instrument mapping dashboard

This blog post, part eleven in our series, delves into CCData’s journey with instrument mapping and metadata. In the cryptocurrency exchange world, uniformity is elusive. Different platforms list identical assets under various symbols and IDs, presenting significant challenges for data aggregation and analysis. We’ll explore these complexities and introduce the Instrument Mapping Dashboard, a sophisticated solution designed to streamline data harmonization across different exchanges.

The Challenge of Diverse Exchange Listings

1. Inconsistent Asset Identifiers: A major hurdle is the variance in IDs used by exchanges for the same instrument. This discrepancy hinders the aggregation and comparison of prices and volumes across markets. For example, BTC-USD on Coinbase might be listed as XXBTXZUSD on Kraken or btcusd on Bitstamp. One of the biggest problems here is that you can have the same symbol across multiple exchanges and it’s actually representing different assets.
2. Inverted Base and Quote Assets: Exchanges often reverse base and quote assets. ETH-BTC might trade as BTC-ETH on some platforms. This issue is particularly prevalent in decentralized finance (DeFi), where pairs like asset-WETH are sometimes listed inversely as WETH-asset, requiring an inversion in trading direction for accurate comparison.
3. Varied Asset Representations: Different exchanges may list multiple representations of an asset relative to another, such as 1000SATSFDUSD on Binance. In DeFi, users typically see values with decimals instead of the smallest unit, necessitating a standard representation format.

The Initial Phase: Overcoming Early Challenges (2014)

At CCData (then CryptoCompare), our journey began in 2014. We faced the challenge of harmonizing data from numerous exchanges, each with unique asset identifiers. Our initial strategy involved manual adjustments at the integration level. While simple, this approach was labor-intensive and led to alterations in the raw data, compromising its integrity. Despite its drawbacks, this phase was crucial, providing invaluable insights into digital asset data complexities.

Developing a Centralized Solution: The First Version of the Dashboard (2017)

In response to early challenges, 2017 marked the launch of our first Instrument Mapping Dashboard. This represented a shift from manual adjustments to a centralized, database-driven approach. It enhanced efficiency and control in asset mapping, though it still operated at the integration level. This development underscored the need to maintain raw data authenticity while providing accurate mapping.

Continuous Improvement and Adaptation (2017–2021)

From 2017 to 2021, we focused on refining our processes, dedicating ourselves to our API and data quality. This period was marked by our pursuit of a system that could map assets accurately and adapt to the dynamic digital asset markets. We learned from our experiences, enhancing our dashboard’s capabilities in preparation for the next technological leap.

A New Era of Data Harmonization: The Platform Upgrade (2021)

The introduction of an upgraded platform in 2021 marked a new era for CCData. We shifted the mapping process to the API and Index level, allowing us to preserve raw data in its original form while still providing effective mapping. This API-level integration enabled dynamic data management, updated effortlessly through the dashboard. This upgrade was not just a technological advancement but a culmination of years of learning, innovation, and adaptation, reflecting our commitment to providing accurate, reliable, and accessible data.

The Instrument Mapping Dashboard is an internal tool that maps each unique combination of exchange name and instrument ID to standardized base and quote asset IDs. This ensures uniform representation of assets, regardless of their original listing format.

1. Base and Quote Asset Standardization: Consistent mapping of instruments across all exchanges.
2. Inversion and Conversion Adjustments: Application of inversions for pairs like WETH-DAI and adjustments for decimal representations.
3. API Access to Mapped and Unmapped Data: Provision of both original and standardized data formats to users.
4. Dynamic Adjustment for Symbol Changes: Adaptation to rebrands, migrations, splits, or consolidations in assets.
5. Automatic Suggestion and Expert Review: Automated mapping suggestions subjected to expert review for accuracy. Our support and content teams manually review and confirm each mapping.

This was part eleven of how and why our system works.

• Part one: The stack discovery
• Part two: The API dissection
• Part three: The Commercial API Journey
• Part four: The WebSockets Adventure
• Part five: Deploying microservices at CryptoCompare
• Part six: Exchange Integrations — Overview
• Part seven: Proxy Swarm — Redis + Queue Magic
• Part eight: Exchange Discovery Service (EDS) — Exchange Integration Sharding
• Part nine: Message Aggregation Service
• Part ten: Message Distribution / Routing
• Part eleven: Tools: Instrument mapping dashboard