Summary of a seminar at IIM-B
The good thing about professors is that they have the ability to pick their favourite subject and talk about it for hours. Okay, some of you may not agree that this is a good thing. Some of you may not even agree that they are good at communicating their ideas. I guess Prof. Dinesh Kumar is different.
At IIM Bangalore yesterday, the session was about analytics and opportunities for start-ups in analytics. In the keynote address, Prof. Dinesh Kumar laid out the basic definitions by way of interesting examples, thereby clearing up misconceptions that people might have about analytics. He then delved into the depths of the topic. He took a holistic view. His examples were drawn from a variety of domains: airline industry, healthcare, restaurants, sports, retail.
Examples of Big Data going wrong
My earlier articles on big data presented quite a rosy picture. Big data is supposed to do this, supposed to do that and eventually make this world a better place. Any new technology has its period of hype and expectations. There are always teething troubles. Basic things don't work as expected. Technology needs time to mature.
Strangely, very few even realize that there are problems with big data. Data scientists and engineers dealing with big data know that there are improvements to be made but they are not fully aware how ineffectively big data analytics work today. They give a solution based on certain assumptions but funnily they don't actually know what a particular user needs. My own personal experience will serve to vindicate these points.
A Look at Big Data Applications
In an earlier article, I gave an introduction to big data and attempted to define it. So what do we really do with all this data, many terabytes that we keep generating, collecting and storing every second? To understand big data, we need to look at applications that use this data. Big data brings value to governments, businesses and consumers. At least, that's the promise.
Big data can help organizations be more efficient and effective. They can find particular needs of individuals and target marketing campaigns rather than take a basket approach. They can observe a user's past behaviour and learn about preferences. Unlike in the past when advertising was a one-to-many approach, with big data it can become one-to-one. Traditionally, one-to-one marketing was rarely possible and cost prohibitive. With users having a regular presence online, businesses can do all their marketing online on a one-to-one basis. Being online does not necessarily translate into targeted marketing. Spam e-mails sent out to thousands of online users is an example of old-style marketing. With big data, online users will get mails and advertisements on products that might interest them.
Size does matter but it's not the only one
I was at a recent IBM Big Data Meetup in Bangalore and someone asked the question, "When does data become big data, at what size?" The question was suggestive and the IBMer was forced to give a number, "Anything more than a terabyte of data is big data." He was clearly reluctant to give a number because big data is hard to define by size alone.
Since the time of Gutenberg's printing press, the amount of data has been on the rise. When thoughts and ideas are published, there is no question of forgetting them. Individuals may forget but the system remembers. After the printing press, the next wave of information exchange came with radio and television broadcasting. Information was consumed in much greater extent than the days of only books and newspapers. Even illiterates could watch television and a picture paints a thousand words. Then something else happened.