Data Scientists - The skills needed and opportunities up for grabs for this generation

Data science, machine learning, data mining, artificial intelligence – these are some very popular terms in use these days. Do they mean the same thing, or do they mean something totally different from each other? The answer is, yes and no.  Without going into too much detail, these days these terms are often used interchangeably but there are some differences in the textbook definitions of these terms. Data science and artificial intelligence are quite broad terms that incorporate several areas and there is a significant overlap between the two.  Machine learning refers to a set of algorithms that improve at performing a task as the algorithm seems more and more samples that are relevant to the task. Data mining refers to a set of algorithms that detect patterns in big data sets. These patterns are obviously not something that humans can detect. Machine learning algorithms also detect patterns in the data, but machine learning algorithms also become better and better at detecting these patterns as they process more and more samples that are relevant to the problem.

If you find this confusing, that’s completely normal. Let me try to make this a bit easier. Think about Data Science, machine learning, AI and data mining as a set of algorithms that a good Data Scientist should be familiar with. Being familiar with these algorithms requires a good handle on the mathematics, statistics, and computer science concepts that all these algorithms employ. A lot of the concepts are common across the fields of Data science, machine learning, artificial intelligence. Therefore, a good data scientist should learn these basic concepts which are applied in several algorithms. For example, there is an algorithm called “gradient descent”. Gradient descent uses calculus and linear algebra to solve unbounded optimization problems. Gradient descent is used by several machine learning algorithms such as deep neural networks which are some of the most powerful learning algorithms of contemporary times. A data scientist will therefore have to understand the mathematics used in the gradient descent algorithm to understand how it works. This is not an easy ask. A data scientist needs to be a part mathematician, part statistician and part computer scientist. While it is certainly not easy to train as a data scientist, it also the reason why data scientists are some of the most highly paid professionals in the tech industry.

The opportunities for data scientists should grow in step with the growth in the digital footprint of humanity. We are now generating data at unprecedented rates. This data is the fuel that powers the data science industry. As the supply of this fuel increases the stronger will be the growth in the data science industry. While this relation may saturate in the future, it is unlikely to happen in the next 5-10 years. In fact, we are more likely to see an acceleration in the applications of data science in our day to day lives. Applications such as driverless cars, AI drones, AI robots make use of these algorithms to alter the way we have been doing things. Imagine a robotic dog which can patrol a perimeter and detect threats using computer vision which is powered by a deep convolutional neural network trained using gradient descent. This also means that the next google or apple or Facebook is likely to be founded in this industry. As for a data scientist, this means opportunities to build the applications that will have a profound effect on the way we live. As for data scientists with entrepreneurial ambitions, sky is not the limit.

About the author

Dr Aditya Narvekar, Deputy Director and Assistant Professor, Bachelor of Data Science at SP Jain School of Global Management

Note: The views expressed in this article are solely author’s own and do not reflect/represent those of Shiksha