Algorithmic guessing: The magic of Big Data Analytics-based Portfolio Management
“It seems to be the right address, but where are the suits and ties” you say to yourself as you enter the offices of DE Shaw, an NYC hedge fund with an imposing presence in Manhattan. Everyone here seems to be a collegial nerd wearing a worn-out hoodie staring at his computer screen, deeply engrossed in crunching numbers, economic news, and code. The degrees range from computer science, physics, engineering to statistics and econometrics. In this hedge fund, once described by a prominent NYC daily as a “mysterious force on Wall Street,” almost everyone is a geek. Portfolio management, the art and science of investing money in financial markets appears to be growing increasingly scientific.
Although there always were econometricians and industrial engineers employing sophisticated analytical ideas to finance, it was only with DE Shaw and a few other funds that algorithms and big data analytics became the primary approach used by trading desks to speculate markets. The eponymous fund founded by a Columbia computer science professor was one of the first to use quantitative insights derived from data and to deploy high-frequency trading algorithms to generate small profits from big capital except for many many times a day. The algorithms used quickly capture trends from market data and use high-performance computing to make predictions which are then translated into trades. The methods are highly technical and require experts from many diverse fields. One commonly used idea is PCA or Principal Components Analysis. This approach breaks the market data into fundamental ‘modes’ or components and allows for a clearer examination of the market moves. Neural networks and deep learning are then used with these fundamental modes to arrive at predicting momentum and other market technicalities. A big data infrastructure is required to apply these statistical and machine learning algorithms to the large number of entities being traded in the market. The hardware and software need to be sophisticated since the trading decisions are to be computed and delivered to robot traders every microsecond.
One commonly used idea is PCA or Principal Components Analysis. This approach breaks the market data into fundamental ‘modes’ or components and allows for a clearer examination of the market moves. Neural networks and deep learning are then used with these fundamental modes to arrive at predicting momentum and other market technicalities.
DE Shaw grew from a meager USD 28 million to USD 40 billion in investments over two decades. DE Shaw however was not without competition. Renaissance Technologies or Ren Tec as it is called was founded by Jim Simons, a mathematician known for his contributions to Mathematical Physics, also deployed analytical algorithms built largely by his team of theoretical physicists, mathematicians and signal processing scientists. Although not much is known about the workings of this secretive organisation, the Medallion hedge fund, the flagship fund managed by Ren Tec and the systematic trading ideas at its core have become a force to reckon with over the years. Like DE Shaw and other algorithmic desks, the fund comprising of its team of data scientists were the first to deploy predictive algorithms in conjunction with market intuition and expert judgment to generate the highest returns in Wall Street history. Medallion’s annual returns averaged in the range of 79 per cent for the period ranging from 1994 to 2014. Simons’ personal wealth grew to billions in a very short time making him one of the richest men in the world and the most successful investor in Wall Street history.
Although not much is known about the workings of this secretive organisation, the Medallion hedge fund, the flagship fund managed by Ren Tec and the systematic trading ideas at its core have become a force to reckon with over the years. Like DE Shaw and other algorithmic desks, the fund comprising of its team of data scientists were the first to deploy predictive algorithms in conjunction with market intuition and expert judgment to generate the highest returns in Wall Street history.
Algorithms and AI could be the future of trading, working their way into not just HFT but also other kinds of arbitrage such as fundamentals-based speculation and buy-and-hold strategies. Most banks today have a rapidly increasing presence in big data analytics. Credit scoring is an application domain where datasets run into millions of entries and this is another place where big data analytics skills become indispensable as the predictive algorithms such as logistic regression and decision trees need to train themselves on very large datasets. Many warn that this ubiquitous existence of big data and machine learning could also be a cause for concern as the market may become a preserve of robots and the few people who design and build them, leaving a large majority of talented bankers unemployed. Whatever happens, one thing is for sure. Big data and AI skills are going to be in great demand in the near future
About the author
Amit Puniyani is an Associate Professor at GIM - Goa Institute of Management. He has many years of experience in the banking analytics domain, working in the quantitative risk management divisions of several top tier investment banks.
Amit began his career working at Lehman Brothers, New York moving to Standard Chartered Bank Singapore and eventually to Nomura Mumbai which had been acquired from Lehman Brothers. He also worked in senior-level positions at Credit Suisse, Mumbai and Deutsche Bank.
Note: The views expressed in this article are solely author’s own and do not reflect/represent those of Shiksha
News & Updates
Contact Details
Address
Goa Institute of Management,
Poriem, Sattari
Goa - Other ( Goa)
Get details of Similar Colleges
Student Forum
Answered 5 months ago
Hello,
Here is the direct comparison to help you choose:
Grt Lakes is the clear winner* for an Executive MBA (via their 1-year PGPM for candidates with 2+ years of experience).
Why Great Lakes Wins:
Flagship Status:The 1-year programme is Great Lakes' flagship course. The college pours all its corporat
Answered 5 months ago
No mention of whether candidates who are in final year can apply for PGDM at Goa Institute of Management was made on website. Thus, candidates should fill enquiry form and ask this question from an executive or supervisor for PGDM admissions. Most likely candidates will be given a deadline for submi
S
Contributor-Level 10
Answered 5 months ago
The university provides scholarships on the basis of merit, financial conditions and other categories. Merit scholarship is offered to students having a good academic record. Some offered scholarships are Khelo India Sports Delight scholarship, Beti Padhao Scholarship, etc.
B
Contributor-Level 10
Answered 5 months ago
Goa Institute of Management does ot take direct admission into PGDM courses. Candidates are required to appear in either CAT or XAT before applying for admission into this institute.
However, if a candidate is able to secure admission through management quota, direct admission if offered based on las
S
Contributor-Level 10
Answered 5 months ago
To get tuition fee waiver, students must apply for the available scholarships. Some of the available scholarships are given below:
| GIM Goa Scholarships | ||
|---|---|---|
| Academic Competence and Excellence Scholarship Scheme (ACCESS) for PGDM students | GIM - MBBS Scholarship (GMS) for HCM students | Academic Competence and Excellence Scholarship Scheme (ACCESS) for BDA students |
| Academic Competence and Excellence Scholarship Scheme (ACCESS) for BFIS students | - | - |
J
Contributor-Level 10
Answered 6 months ago
GIM Goa has all the qualities of tier 1 and tier 2 colleges. Goa Institute of Management GIM Goa ranking is 43rd by NIRF 2025 under the Management category. The college offers good placement support to students.
As per the student's review, "The infrastructure of the college is exceptional. You al
A
Contributor-Level 10
Answered 6 months ago
Goa institute of management popular programme is PGDM. GIM Goa Research scholars are equipped with in-depth functional skills & analytical knowledge to succeed in both industry & academia. PGDM & HCM students are placed in top companies such as Deloitte, Amazon, TCS, Accenture, & others.
L
Contributor-Level 10







Which one is better for Executive MBA, Great Lakes Institute of Management or GIM - Goa Institute of Management?