What is a data lakehouse and do you need one?

The inex­orable rise of data has changed our world and the way we do busi­ness. Data-dri­ven insights, enabled by the sheer vol­ume of infor­ma­tion gen­er­at­ed by every­day users, are rev­o­lu­tion­is­ing cor­po­rate deci­sion-mak­ing. But work­ing out how best to han­dle all this mate­r­i­al remains a chal­lenge for senior exec­u­tives.

The lat­est term to have become a buzz­word for busi­ness lead­ers is the ‘data lake­house’. But what does it mean, exact­ly, and how can a com­pa­ny deter­mine whether it needs one?

The emer­gence of the data lake­house is the result of a range of devel­op­ments in the field. As organ­i­sa­tions used more and more data in their every­day oper­a­tions, they start­ed plac­ing it into data ware­hous­es – a cen­tralised man­age­ment sys­tem that stores the mate­r­i­al in such a way that makes it easy to inter­ro­gate. But doing that soon became too restric­tive for some com­pa­nies, which instead pooled their data into so-called lakes: vast, unor­gan­ised gath­er­ings of mate­r­i­al in its native for­mat, wait­ing to be analysed.

“These approach­es have been used for years, dri­ven by organ­i­sa­tions’ increas­ing reliance on data insights to increase prof­itabil­i­ty, uncov­er new oppor­tu­ni­ties and detect issues,” explains Jitesh Ghai, chief prod­uct offi­cer at US soft­ware firm Infor­mat­i­ca. 

Yet both approach­es have run into a scal­a­bil­i­ty prob­lem. The increas­ing vol­ume of data sources that fed into the smooth run­ning of a busi­ness began to test the lim­its of firms’ tech­ni­cal capa­bil­i­ties. The weak­ness of data lakes became so sig­nif­i­cant that they were rechris­tened data swamps by some in the indus­try – a tes­ta­ment to the mud­dy chal­lenge of dredg­ing up the insights lurk­ing with­in them

The data lake­house is the log­i­cal solu­tion to this prob­lem. Ghai calls it an attempt to gain con­trol of free data, enabling users to dump unstruc­tured mate­r­i­al into their sys­tems. It’s an advance­ment on ware­hous­es, which require the data put into them to be care­ful­ly processed and struc­tured before­hand. 

“Lake­hous­es merge ele­ments of data ware­hous­es and data lakes in one plat­form,” he says. “This mod­el promis­es the best of both worlds by blend­ing tech­nolo­gies for ana­lyt­ics and deci­sion-mak­ing with those for data sci­ence and explo­ration.”

Leila Sei­th Has­san, head of data sci­ence and ana­lyt­ics at Dig­i­tas UK, observes that a lake­house com­bines the high-speed per­for­mance of a ware­house on a scale that’s enabled by lake tech­nol­o­gy. “If you think about how the world has evolved in the past 20 years and the explo­sion in the vol­ume of data that has become avail­able, data ware­hous­es have had some lim­i­ta­tions about what you could do with them at speed,” she says. 

A lake­house enables unstruc­tured data to be analysed more quick­ly – poten­tial­ly cre­at­ing link­ages that would pre­vi­ous­ly not have been seen. It not only pro­vides a per­for­mance improve­ment; it also avoids some of the data man­age­ment pit­falls that have blight­ed busi­ness­es’ ways of work­ing in the past. 

One of the key prob­lems fac­ing com­pa­nies in the pre-lake­house world was that their data was often con­tained in sev­er­al dif­fer­ent­ly struc­tured sys­tems. In order to analyse this mate­r­i­al, they would first have to trans­fer it into a sin­gle loca­tion and restruc­ture it – process­es that could make the under­ly­ing data unre­li­able. 

“In a lake­house, pro­cess­ing becomes much faster, as the sys­tem helps to organ­ise big data more effec­tive­ly,” Ghai says. “It also uni­fies data, bring­ing it all under one sys­tem, which elim­i­nates redun­dan­cies and allows for bet­ter data man­age­ment.” 

A lake­house con­tains some struc­ture, but not so much that makes it impos­si­ble for users to glean new insights from organ­ic hap­pen­stance, Sei­th Has­san adds. “It gives ana­lysts and non-tech­ni­cal data prac­ti­tion­ers the abil­i­ty to access data and do things such as using it for deci­sion-mak­ing with­out hav­ing to extract it and put it some­where else first.”

Has a lake­house become a busi­ness essen­tial and, if so, is there an ide­al time to imple­ment such a sys­tem? Com­pa­nies need to con­duct a cost-ben­e­fit analy­sis of what imple­ment­ing a data lake­house could do for their oper­a­tions and decide care­ful­ly whether it’s worth their while

“The good thing is that the way to answer this ques­tion isn’t new: work out what your require­ments and use cas­es are for it,” says Sei­th Has­san, who can fore­see a world in which most organ­i­sa­tions would ben­e­fit from using a lake­house. But she acknowl­edges that “the tech­nol­o­gy will cost quite a bit of mon­ey and resources to get it up and run­ning. If it doesn’t do some­thing for you now and poten­tial­ly in the future, you shouldn’t move for­ward with it.” 

James Cor­co­ran is senior vice-pres­i­dent of engi­neer­ing at stream­ing ana­lyt­ics plat­form KX. His com­pa­ny adopt­ed a lake­house after recog­nis­ing that shoe­horn­ing dif­fer­ent data from a mul­ti­tude of sources into a sin­gle mod­el wasn’t work­ing. 

He sug­gests that busi­ness­es con­sid­er invest­ing in a lake­house as soon as they start expe­ri­enc­ing intractable prob­lems with organ­i­sa­tion­al silos that pre­vent data from being joined togeth­er log­i­cal­ly. 

But Cor­co­ran rec­om­mends hold­ing back until the ulti­mate goal of such an invest­ment is clear. “Maybe you’re devel­op­ing or launch­ing a new prod­uct, or you’re strug­gling to get the insights you need to be com­pet­i­tive with an exist­ing one,” he says. “Tying it to some sort of busi­ness out­come is real­ly, real­ly key.” 

The process of plan­ning, devel­op­ing and deploy­ing a data lake­house is nei­ther straight­for­ward nor cheap, but the poten­tial rewards are great. Data lake­hous­es can unlock huge promise for the future, accord­ing to Cor­co­ran. 

They will enable a busi­ness to “update the heart of its strat­e­gy, its data and its deci­sion-mak­ing process”. he says. “You’ll be able to move a lot faster than your com­peti­tors.”