Algorithmic guessing: The magic of Big Data Analytics-based Portfolio Management

“It seems to be the right address, but where are the suits and ties” you say to yourself as you enter the offices of DE Shaw, an NYC hedge fund with an imposing presence in Manhattan. Everyone here seems to be a collegial nerd wearing a worn-out hoodie staring at his computer screen, deeply engrossed in crunching numbers, economic news, and code. The degrees range from computer science, physics, engineering to statistics and econometrics. In this hedge fund, once described by a prominent NYC daily as a “mysterious force on Wall Street,” almost everyone is a geek. Portfolio management, the art and science of investing money in financial markets appears to be growing increasingly scientific.

Although there always were econometricians and industrial engineers employing sophisticated analytical ideas to finance, it was only with DE Shaw and a few other funds that algorithms and big data analytics became the primary approach used by trading desks to speculate markets. The eponymous fund founded by a Columbia computer science professor was one of the first to use quantitative insights derived from data and to deploy high-frequency trading algorithms to generate small profits from big capital except for many many times a day. The algorithms used quickly capture trends from market data and use high-performance computing to make predictions which are then translated into trades. The methods are highly technical and require experts from many diverse fields. One commonly used idea is PCA or Principal Components Analysis. This approach breaks the market data into fundamental ‘modes’ or components and allows for a clearer examination of the market moves. Neural networks and deep learning are then used with these fundamental modes to arrive at predicting momentum and other market technicalities. A big data infrastructure is required to apply these statistical and machine learning algorithms to the large number of entities being traded in the market. The hardware and software need to be sophisticated since the trading decisions are to be computed and delivered to robot traders every microsecond. 

One commonly used idea is PCA or Principal Components Analysis. This approach breaks the market data into fundamental ‘modes’ or components and allows for a clearer examination of the market moves. Neural networks and deep learning are then used with these fundamental modes to arrive at predicting momentum and other market technicalities.  

DE Shaw grew from a meager USD 28 million to USD 40 billion in investments over two decades. DE Shaw however was not without competition. Renaissance Technologies or Ren Tec as it is called was founded by Jim Simons, a mathematician known for his contributions to Mathematical Physics, also deployed analytical algorithms built largely by his team of theoretical physicists, mathematicians and signal processing scientists. Although not much is known about the workings of this secretive organisation, the Medallion hedge fund, the flagship fund managed by Ren Tec and the systematic trading ideas at its core have become a force to reckon with over the years. Like DE Shaw and other algorithmic desks, the fund comprising of its team of data scientists were the first to deploy predictive algorithms in conjunction with market intuition and expert judgment to generate the highest returns in Wall Street history. Medallion’s annual returns averaged in the range of 79 per cent for the period ranging from 1994 to 2014. Simons’ personal wealth grew to billions in a very short time making him one of the richest men in the world and the most successful investor in Wall Street history.

Although not much is known about the workings of this secretive organisation, the Medallion hedge fund, the flagship fund managed by Ren Tec and the systematic trading ideas at its core have become a force to reckon with over the years. Like DE Shaw and other algorithmic desks, the fund comprising of its team of data scientists were the first to deploy predictive algorithms in conjunction with market intuition and expert judgment to generate the highest returns in Wall Street history.

Algorithms and AI could be the future of trading, working their way into not just HFT but also other kinds of arbitrage such as fundamentals-based speculation and buy-and-hold strategies. Most banks today have a rapidly increasing presence in big data analytics. Credit scoring is an application domain where datasets run into millions of entries and this is another place where big data analytics skills become indispensable as the predictive algorithms such as logistic regression and decision trees need to train themselves on very large datasets. Many warn that this ubiquitous existence of big data and machine learning could also be a cause for concern as the market may become a preserve of robots and the few people who design and build them, leaving a large majority of talented bankers unemployed. Whatever happens, one thing is for sure. Big data and AI skills are going to be in great demand in the near future

About the author

Amit Puniyani is an Associate Professor at GIM - Goa Institute of Management. He has many years of experience in the banking analytics domain, working in the quantitative risk management divisions of several top tier investment banks.

Amit began his career working at Lehman Brothers, New York moving to Standard Chartered Bank Singapore and eventually to Nomura Mumbai which had been acquired from Lehman Brothers. He also worked in senior-level positions at Credit Suisse, Mumbai and Deutsche Bank.

Note: The views expressed in this article are solely author’s own and do not reflect/represent those of Shiksha