Solving business data overload

Com­pa­nies are drown­ing in data and most have only a par­tial view into their own oper­a­tions. But inte­grat­ing infor­ma­tion into clear­ly defined for­mats of files and lay­out gives busi­ness­es the abil­i­ty to under­stand their process­es and make informed deci­sions.

Organ­i­sa­tions are real­is­ing the urgency of using inte­gra­tion and access soft­ware – defined by ana­lyst house IDC as enabling the access, blend­ing and move­ment of data among mul­ti­ple sources – and by next year glob­al spend­ing is set to hit $6 bil­lion (£3.9 bil­lion).

“Com­pa­nies have many sources and dif­fer­ent fun­nels of infor­ma­tion list­ed in mul­ti­ple for­mats,” says Alys Wood­ward, IDC research direc­tor, adding that this has to be addressed.

Busi­ness­es have to cre­ate an inte­grat­ed view, oth­er­wise it’s not clear how well they are per­form­ing, how to mit­i­gate risks and how to enable peo­ple to col­lab­o­rate

Philip Howard, research direc­tor at Bloor, adds: “Data inte­gra­tion is like a road that con­nects two cities. If you need two appli­ca­tions to work, they must be able to com­mu­ni­cate.”

In one clas­sic form of inte­gra­tion, a com­pa­ny may pull togeth­er infor­ma­tion on cus­tomers, a pipeline of sales, prod­uct types and stock avail­abil­i­ty. If infor­ma­tion is kept in stan­dard for­mats, col­la­tion is eas­i­er, but the grow­ing com­plex­i­ty of data is quick­ly putting com­pa­nies on the back foot.

“Busi­ness­es have to cre­ate an inte­grat­ed view, oth­er­wise it’s not clear how well they are per­form­ing, how to mit­i­gate risks and how to enable peo­ple to col­lab­o­rate,” explains Ted Fried­man, a vice pres­i­dent at Gart­ner.

Many firms begin on the inte­gra­tion path by writ­ing their own code. As the scale of the task becomes appar­ent, they turn to more advanced tools that graph­i­cal­ly illus­trate and auto­mate data.

Reg­u­lat­ed indus­tries, such as finan­cial ser­vices and health, tend to be ahead of the curve in using data inte­gra­tion tools, Ms Wood­ward argues, “because of the require­ment to demon­strate what they are doing”. Tele­coms and retail, fac­ing vast vol­umes of data, have also tak­en large strides.

One com­pa­ny to have pulled togeth­er its many sources of infor­ma­tion is large US hos­pi­tal alliance Pre­mier. Kei­th Figli­oli, senior vice pres­i­dent, says that giv­en the pre­vail­ing shift to pay­ment based on qual­i­ty and effi­cien­cy, its 3,400 hos­pi­tal mem­bers have to analyse com­plex finan­cial, oper­a­tional and clin­i­cal data quick­ly.

The alliance devel­oped a sin­gle plat­form for near-real-time infor­ma­tion, pro­cess­ing 3,000 data trans­ac­tions a sec­ond. “We now have the abil­i­ty to take data from any of our providers’ trans­ac­tion­al source sys­tems and use it to pro­vide tru­ly inte­grat­ed ana­lyt­ics,” Mr Figli­oli says.

In the UK, car man­u­fac­tur­er Hon­da had a seri­ous chal­lenge as its 200 deal­ers used myr­i­ad man­age­ment sys­tems and file for­mats. It drew togeth­er and for­mat­ted the infor­ma­tion so that it could bet­ter under­stand cus­tomers and sales, and pre­dict part require­ments.

Many busi­ness­es tend to favour a tool called ETL (extract, trans­form and load). It enables the process of selec­tive­ly pulling from var­i­ous data­bas­es, trans­form­ing the data into a com­mon for­mat and merg­ing it. The oper­a­tion is usu­al­ly run in batch­es.

More recent sys­tems also help com­pa­nies man­age the dif­fer­ent inter­faces between appli­ca­tions and visu­al­ly map inte­gra­tion from a source data­base to the tar­get. Equal­ly, new tech­nol­o­gy can take uncon­nect­ed data and learn rela­tion­ships.

As the tech­nol­o­gy devel­ops, so does the sever­i­ty of the prob­lem it has to tack­le. The some­what encour­aged growth in unstruc­tured infor­ma­tion, as com­pa­nies build huge ‘”lakes” of data, leaves them with a repos­i­to­ry of files that are rarely for­mat­ted or link­able.

This leads to seri­ous accu­ra­cy con­cerns, accord­ing to Gartner’s Mr Fried­man. “Com­pa­nies need to look beyond the plumb­ing and think about the qual­i­ty of what trav­els through those pipes,” he says.

Focused invest­ment in infor­ma­tion gov­er­nance and qual­i­ty is the answer. “Hav­ing more tech­nol­o­gy to help the process around data gov­er­nance and mas­ter data man­age­ment is the key,” says Mr Figli­oli. The “messy” data in the health­care indus­try, he explains, is prompt­ing large gov­er­nance invest­ment.

Anoth­er chal­lenge is the strug­gle to analyse the data quick­ly. There is a grow­ing appetite for real-time infor­ma­tion, par­tic­u­lar­ly when a cus­tomer is on the phone or a web­site needs to cre­ate a rec­om­men­da­tion. Cor­rect­ly assem­bled data, com­bined with pre­dic­tive ana­lyt­ics, are cru­cial.

The sit­u­a­tion is fur­ther com­pli­cat­ed by the growth in cloud com­put­ing, which has led to data being held out­side com­pa­ny walls. IDC’s Ms Wood­ward warns: “Com­pa­nies need to under­stand it may cost them to access the data lay­er from the cloud provider; or worse, they may not be able to access it at all.”

The expan­sion in con­nect­ed home, busi­ness and city devices as part of the inter­net of things is lead­ing to a large amount of extra data being locat­ed in many places. “It will huge­ly ramp up the sort of tech­nol­o­gy required,” says Bloor’s Mr Howard, “and many of the con­nect­ed devices have their own for­mat for infor­ma­tion.”

As the chal­lenges increase, com­pa­nies must invest in both expe­ri­enced man­age­ment and advanced tech­nol­o­gy if they are to pull togeth­er all their data for quick­ly action­able deci­sions.