Heads in the Clouds

Once the domain of bleeding-edge technology projects, cloud computing has found a new application among market data professionals: storing the huge amounts of real-time and historical data that must be maintained and harnessed for everything from trading algorithm development and back-testing to regulatory compliance. By Max Bowie

Like their meteorological namesakes, which suck up volumes of water vapor, store it, then deposit it in huge deluges, compute clouds are now being used to capture, store and deliver data on-demand that would otherwise overwhelm existing resources.

"Broker-dealers have to store a lot of data as required by the regulatory agencies, including e-mails, and transactional and market data," says Raman Kannan, CTO of Rosenblatt Securities, in a recent interview with sibling publication Dealing with Technology (see page s4).

Rosenblatt began using cloud computing for some of its data needs in the third quarter of last year, after deploying a hosted project management tool dubbed Basecamp, which required the ability to store and access large files safely and reliably from anywhere. To achieve this, the firm enlisted Amazon.com's Elastic Compute Cloud (EC2) and Simple Storage Service (S3) clouds for data storage needs "that do not involve client-specific information or other sensitive data," Kannan says.

The firm also stores its voluminous tick history database in the cloud. "Without cloud computing, we could not competitively offer a robust transaction cost analysis solution or maintain tick history," Kannan says. "Cloud usage results in significant cost reduction and shifts capital expenditure into operational expenditure. We do not buy new servers or engage an expensive system administration consultant, but we do not suffer any reduction in service quality. It is the elasticity that is attractive to us."

But it's not just broker-dealers with regulatory obligations that have their heads in the cloud: Forbes.com incorporated real-time data from the Bats Exchange using Web services from Xignite and Amazon's EC2 offering, while Nasdaq uses Amazon's S3 cloud to host large files of historical data that subscribers download for use in its Market Replay product. According to a Nasdaq case study, when the exchange first looked at introducing the product, even the most basic data from Nasdaq amounted to 50 GB per day, which would have made the cost "prohibitive to most investors," using traditional storage and retrieval technologies, whereas Amazon provides fast retrieval, reliability and scalability within predictable cost levels.

This is helping to open up the promise of cloud computing beyond those with large budgets. Chicago-based data vendor Barchart has also been able to launch new services as a result of using cloud computing. In September, the vendor expanded its historical data to two years of historical data at one-minute intervals, and one year of tick-level data, compared to one year and one month before, after utilizing virtual server and bandwidth capacity from Amazon EC2.

"We can bring new servers online and scale up bandwidth in minutes just by placing a virtual iteration of other servers in Amazon's facility," in response to a client base that is increasingly demanding higher degrees of backup and redundancy, says Mark Haraburda, managing director of sales and business development at Barchart.

With this technology providing a solution to the problem of ever-growing volumes of data, the old adage that every cloud has a silver lining may have never been more true.

Max Bowie is editor of Inside Market Data and can be reached at

For a free four-issue trial   of IMD, please visit: www.insidemarketdata.com

-->