Unblocking the supply chain to make data useful

Data’s com­mon descrip­tion as the ‘new oil’ was ini­tial­ly meant to accen­tu­ate its val­ue as a new com­mod­i­ty, but it now risks under­es­ti­mat­ing its impor­tance in every aspect of how we work and live. Unlike oil, data is con­stant­ly replen­ished and amid the slow but inevitable decline of the petro­le­um econ­o­my, every suc­cess­ful organ­i­sa­tion is now a data com­pa­ny. It’s impos­si­ble to be a mod­ern enter­prise with­out data-dri­ven oper­a­tions and cul­ture.

Data helps stream­line busi­ness process­es, mak­ing organ­i­sa­tions work bet­ter and help­ing com­pa­nies under­stand and there­fore serve cus­tomers bet­ter. And with arti­fi­cial intel­li­gence and machine learn­ing, they can make bet­ter deci­sions more effi­cient­ly. But data doesn’t just bring val­ue in a core busi­ness sense. From devel­op­ing drugs to irri­gat­ing farm­land or accel­er­at­ing inno­va­tion in health, sci­ence and ener­gy, it is advanc­ing soci­ety.

Many of these advances can be attrib­uted to the cloud, which is not only where new inno­va­tion is hap­pen­ing but also why it’s pos­si­ble. The cloud pro­vides, for the first time in the his­to­ry of busi­ness and tech­nol­o­gy, a lim­it­less­ly pow­er­ful, easy to stand up and scale, and cost effec­tive way of cap­tur­ing, stor­ing, pro­cess­ing and get­ting val­ue from data.

A clog in the system

There is a prob­lem, how­ev­er. Every ana­lyt­ics, AI or machine learn­ing use case relies on a sup­ply chain; a pipeline of use­ful ana­lyt­ics-ready data. And the pipeline is very clogged up.
“We have more data than ever and it’s grow­ing log­a­rith­mi­cal­ly, but it does­n’t start off in a state that’s use­ful for ana­lyt­ics, AI and machine learn­ing,” says Matthew Scul­lion, CEO and founder of Matil­lion, a cloud-native data inte­gra­tion com­pa­ny.

“Data is like iron ore and steel. Just like you need steel to make a bridge, fac­to­ry or car, to gain ana­lyt­i­cal insight you need ana­lyt­ics-ready data. Data starts off like ore and needs refin­ing into steel by join­ing it togeth­er, clean­ing it up, mak­ing sure it’s the right data and then embell­ish­ing it with met­rics.

“That refine­ment process needs to hap­pen before we can make every aspect of how we work, live and play bet­ter. But today the world’s abil­i­ty to make data use­ful is con­strained because the refine­ment process is done in a prim­i­tive way: by peo­ple who write code. Writ­ing code is the slow­est and hard­est to main­tain way of doing it, but most cru­cial­ly it relies on a small num­ber of high­ly skilled peo­ple, which the world does­n’t have enough of.”

A perfect storm

The prob­lem is felt by every com­pa­ny try­ing to inno­vate with data, but it is most acute in large organ­i­sa­tions, which have the biggest piles of data. A study of Matillion’s 200 largest cus­tomers – bil­lion-dol­lar rev­enue and above com­pa­nies – found they had, on aver­age, more than 1,000 dif­fer­ent com­put­er sys­tems from which they were extract­ing data to put to work in ana­lyt­ics, AI and machine learn­ing. Some sys­tems date back to the 70s and 80s, while some are con­tem­po­rary SaaS and cloud-based sys­tems. Some are com­mer­cial off-the-shelf sys­tems bought from well-known ven­dors, oth­ers are built bespoke.

Enter­pris­es have the most to win or lose by get­ting this right. If a large com­pa­ny improves a busi­ness process, under­stands the cus­tomer bet­ter and enhances a prod­uct – or fails to do any of these things – there are tens of mil­lions of pounds at stake. Yet large enter­pris­es also suf­fer from the lim­it­ed pool of high­ly skilled engi­neers in the most pro­nounced way. It is not uncom­mon for more than half of the employ­ees in a tech start­up to have the cod­ing skills to make data use­ful. But in a large bank or man­u­fac­tur­ing firm, it’s more like 0.5% of the work­force.

“Large organ­i­sa­tions have the biggest pile of het­ero­ge­neous data and the most to win or lose, but the least capa­bil­i­ty per head of capi­ta in their work­force,” says Scul­lion. “We felt this prob­lem per­son­al­ly because we used to build fin­ished ana­lyt­ics solu­tions for these com­pa­nies, and the biggest part of the job in any ana­lyt­ics, AI or machine learn­ing use case — prob­a­bly 70% of the work — is in mak­ing the data use­ful. So we devel­oped a solu­tion to solve it.”

Setting the pipeline free

By refin­ing data in a way that doesn’t require skilled engi­neers to write code, Matil­lion enables organ­i­sa­tions to make data use­ful more quick­ly and in greater quan­ti­ties. It also means the task of mak­ing data use­ful is no longer the purview of a small num­ber of cost­ly peo­ple doing it in a prim­i­tive way. It can be done by far more peo­ple on the data engi­neer­ing team and in oth­er data dis­ci­plines like data sci­ence, which allows com­pa­nies to unblock the pipeline of ana­lyt­ics-ready data and accel­er­ate ana­lyt­ics, AI and machine learn­ing projects.

“The end of the arc is get­ting to a point where all aspects of mak­ing data use­ful – how you load, trans­form, syn­chro­nise and then orches­trate that process at enter­prise scale – are deliv­ered for you in a sin­gle place — a data oper­at­ing sys­tem — that does­n’t require high-end cod­ing skills to use and works across clouds and cloud data plat­forms,” Scul­lion adds.

“A com­pa­ny’s abil­i­ty to make data use­ful is direct­ly cor­re­lat­ed to its core out­comes. Being able to inno­vate with data at an accel­er­at­ed rate is a com­pet­i­tive imper­a­tive. You either get good at this stuff, and real quick, or you’re not going to sur­vive much longer. Matil­lion is build­ing a plat­form to enable that, but it’s up to busi­ness­es to use that tech­nol­o­gy to become a data busi­ness and put data to work faster than their com­pe­ti­tion.”

Learn more about Matil­lion at matillion.com/demo

Pro­mot­ed by Matil­lion