The Big Data Paradox

By Vivek Tyagi Big Data will have a profound impact on scientific research, manufacturing and a wide spectrum of activities that impact our daily lives. But only if we can keep up with it. Call it the Big Data Paradox. New software technologies like Hadoop and predictive analytics are paving the way for us to tackle problems that have been considered too complex or difficult to solve. For instance, with the help of high-end technologies such as smart phones, wireless printers, surveillance cameras and backend connectivity, a metro city like Bangalore monitors traffic real time and enforces rules. Data analytics further helps in taking informed decisions on the strategic spots for sign boards and markings across the city, helping reduce congestion and making it safer to travel. [caption id=“attachment_2179465” align=“alignleft” width=“380” class=" “]  Thinkstock[/caption] Another big phenomenon much talked about in India are the Smart Cities. As part of the Government’s ambitious plan to setup Smart Cities in Bangalore, the Electronics City -IT corridor will soon have smart features like mobile enabled technology to finding parking space in commercial centers and offices, waste and water management solutions, energy saving technologies etc. Also, Big Data analytics in healthcare has helped find insights from the surging data sets and improve outcomes while reducing cost. For example the Narayan Health group has established a ‘Health City’ in Bangalore. The facility comprises of a 2000 to 5000-bed conglomeration of multiple specialty hospitals in a single campus. The health group is utilizing big data technology to capture all of the information and mine insights based on the patient data to get a complete view and drive efficiency. However, be it traffic management, smart cities or healthcare, we need new hardware, software, networking and data center designs to be able to handle and adequately analyze the staggering amount of data being generated to use for analytical data mining. Hyperscale data centers, software-defined networking, and new storage technologies represent the first steps in what will be a tremendous cycle of innovation over the next decade. The deeper analysis will generate data sets that will require even more elaborate computing and storage systems so researchers can start to test particular hypotheses in days or months rather than years. Harnessing the volume and velocity of rivers of data like this will require us to completely rethink servers, data centers and storage systems. In many data centers today, less than 50 percent of the computing cycles of computers are used for productive work because the data simply can’t be delivered fast enough to the processor. Traditional storage solutions based on hard drives can store large amounts of data, but they fall down in terms of speed, leading to waste and inefficiency. Latency, or how long it takes hard drive systems to finally deliver data to a processor, can cost companies millions or even billions of dollars a year in terms of business lost due to slow transactions. Hard drives also use a substantial amount of power. Each hard drive comes with a motor inside. Not only do these motors consume power, they generate heat, which in turn increases the demand for air conditioning, which in turn increases power consumption. We can’t stop innovation simply because we can’t get enough electricity. Solid-state technology, or flash, can increase computing density by 5x, lower power consumption by 80 percent, increase reliability by 4x and increase performance by up to 50x. By increasing speed and performance while reducing power at the same time, solid-state flash technology provides a path forward for huge data sets to be synthesized into actionable intelligence Big Data is really one of the magical concepts of our era. Its ability to give us greater insight and understanding of the world around us increases our ability to create a better society. A survey done by Pew Research Center shows that the global population is expected to increase by 38 percent, from 6.9 billion in 2010 to 9.6 billion in 2050. Therefore, there is a dire need for cities to get smarter to handle this large-scale urbanization and adopt new ways to manage complexity, increase efficiency, reduce expenses, and improve the quality of life. This is going to require a tremendous amount of effort behind the scenes to create solutions that can help wield these Big Data stores in a compact, cost effective, reliable and environmentally conscious way. (The author is director, business development at SanDisk India)