IT departments sidelined as finance and marketing harness big data

The pressing need to analyze huge data sets containing high volumes of unstructured information is forcing some corporates to re-evaluate how they manage their data assets, with individual business departments like marketing and finance bypassing the IT department to take the job on themselves or outsourcing it to cloud providers.

Oracle launched a specialised big data appliance this week, a dedicated piece of hardware running a mixture of its own proprietary and open source software designed to speed up the process of filtering, sorting and indexing structured and unstructured information from diverse sources before loading it into a data warehouse.

Big data can include social media content, email, HTML pages, instant messaging (IM) logs, blogs, digital images, video files, surveillance footage, e-commerce transactions, call records and medical records, as well as datasets created by academic, scientific and research projects, which process large volumes of information.

The companies analysing these huge volumes generally do so either for risk protection or business intelligence purposes, and Oracle is just the latest in a long line of IT vendors – including IBM, Dell, EMC and Microsoft – looking at ways to tap into enterprise fears of being left behind by competitors who are able to mine and interpret market data faster and more accurately than they can.

"People are either asking what is all this stuff and what is the risk to us, or what market opportunities are we missing because we can't see what people are saying about us on Twitter or whatever and we can't figure out what it [all this data] is.

But the thing that has really changed is the variety in the kind of data available, and that is where IT managers run up against absolutely hard and fast limitations on what they can do," Debra Logan, vice president and distinguished analyst at research firm Gartner told Computing.

"So you need massively parallel computing systems to break that down – more power, speed and memory – to break up unstructured data and index big data sets, which is what people have always struggled to do."