Electronic Market Microstructure

HFT Requires “High-Performance Computing Systems” to Achive Low Latency July 9, 2009

Unlike the traditional definition of high-performance computing (HPC), which often states “that HPC is the use of supercomputers and computer clusters to solve advanced computation problems”, High-Performance Computing Systems (HPCS) is more encompassing. It is a super-set of HPC and has many high performance components that make up an end-to-end system that includes much more than just focusing on the compute engine, or processor; for instance it usually consists of (i) the computing capabilities, (ii) storage capabilities, (iii) external data acquisition, (iv) network communications, and (v) application performance. Also in traditional HPC, whether the compute engine is a supercomputer or distributed computers, the goal is to accelerate the calculation of the problem at hand. This is because tasks that require acceleration are so computationally intensive. This is not necessarily true in HPCS, the task at hand may be deterministic and simple in nature but the need for it to “happen” as quickly as possible is paramount. Not unlike the task of high-frequency trading.

In essence one can appreciate the difference by thinking that HPC’s objective is to maximize the through-put of a compute engine so that difficult problems can be solved as quickly as possible. Alternatively, the object of a HPCS is to maximize the through-put of a system so that transactions can be completed quickly as possible and therfore latency is low.

The objective of HPC and the minimization of latency through the processor certainly will improve the overall performance of increasing the throughput of transactions, or messages, in a system but it is not often the bottle neck. To approach frictionless throughput in a computer system one must analysis all potential sources of latency and address them so the individual improvement approaches compliment one another and produce an environmental, all encompassing improvement. I will try and create a taxonomy of candidate latency areas and what may be used to lower that latency. The taxonomy will be divided into the most logical components in terms of computing systems:

HPCS Architecture Taxonomy

1.  Comutational components
   1. Large Bus (64bit & 128bit)
   2. Multiple Core
   3. Large On-Chip Memory
   4. Clusters
   5. Grids
   6. Special Processors (GPUs, Gate Arrays, & EPROMS)
   7. Quantum Processor
2. Storage components
   1. Solid State Discs
   2. High Performance Databases
   3. Cross-CPU Shared Memory
3. External data sources
4. Network communications, and
   1. Utilizing Optimal Routing Protocols
5. Application components

Outlined above are the components I believe required to be addressed to develop a low-latency, High-Performance Computing System. I have also added subcategories under the components that represent some technologies or approaches that one might explore, and possibly, include in their low-latency architecture.

I will explore each component area in much more detail in further postings.

A Return to Commentary: What has advanced? July 5, 2009

Advances in computing have profoundly changed our society. It has provided us with the ability to capture and process massive volumes of meaningful data. We never before have had the amount of financial data available for analysis nor have had the processing power to analyze a complete market in real-time. This in conjunction with the advances in computational methods, we have been recently equipped to examine financial data in real-time and more efficiently than anytime previously.

This seris will examine the scientific and commercial relevance exposed as a result of the convergence of four advanced and diverse fields of; (i) model-driven trading, (ii) computational intelligence, (iii) the availability of high frequency market data, and (iv) the evolution of enabling technologies; such as available 64-bit processors, high performance data managers and grid computing. We will demonstrate the unique power of this technology convergence by analyzing quote depth, which is still not commercially available in historic form, in real-time and identifying important non-seasonal patterns. We will examine the BID-ASK depth of the NASDAQ cash equity market by loading and committing inhomogeneous time-series market data into cache memory. We will then applying the dataset to a continuously adaptive and biologically-inspired computational method that will conduct high speed pattern recognition. The resultant patterns will indicate market anomalies and will form stylized facts that in turn can be used to supply a paradigm for model-driven trading. Because of the technology barriers to entry and a high level of domain specific knowledge required the method described here has not been attempted by any known large non-bank entities and is truly ground breaking.