Democratisation of data calls for a new philosophy

With data increas­ing­ly viewed as the lifeblood of any mod­ern organ­i­sa­tion, the idea of democ­ra­tis­ing that data is tak­ing hold among busi­ness lead­ers.

Put sim­ply, data democ­ra­ti­sa­tion places the pow­er of data into employ­ees’ hands, rather than keep­ing it hid­den from view or restrict­ed to a select few. When access to data is lim­it­ed, its poten­tial is great­ly dimin­ished. But democ­ra­ti­sa­tion can pro­vide fast and valu­able data-dri­ven insights, often on the front­line, where it is need­ed most.

How­ev­er, more than sim­ply mak­ing more data avail­able to more peo­ple, the con­cept often requires a rethink about how organ­i­sa­tions man­age, dis­trib­ute and con­sume data. It can also involve wide­spread cul­tur­al change across the busi­ness.

Accord­ing to Gart­ner, by 2023, data lit­er­a­cy will become an explic­it and nec­es­sary dri­ver of busi­ness val­ue, demon­strat­ed by its for­mal inclu­sion in over 80% of data and ana­lyt­ics strate­gies and change man­age­ment pro­grams*. How­ev­er, there remain some road­blocks on the path to data lit­er­a­cy.

One is the obvi­ous influx of data that organ­i­sa­tions are fac­ing. In 2020, 64.2 zettabytes of data was cre­at­ed or repli­cat­ed. More­over, the amount of dig­i­tal data cre­at­ed over the next five years will be greater than twice the amount of data cre­at­ed since the advent of dig­i­tal stor­age.

Access­ing and analysing this data is cen­tral to an organisation’s inno­va­tion and agili­ty – both crit­i­cal in the cur­rent dis­rup­tive land­scape. But along­side the sheer vol­umes of data, many enter­pris­es must con­tend with dis­parate pock­ets of data, siloed across the organ­i­sa­tion in appli­ca­tions and sys­tems.

At the same time, cur­rent tech­nol­o­gy lim­i­ta­tions are pre­vent­ing easy and cost-effec­tive access to that data.

In the era of big data, lega­cy sys­tems had to be pro­vi­sioned and con­fig­ured, and the infor­ma­tion entered into data­bas­es. This could take months and inhibits access to vital and time-sen­si­tive data. And because of this cost and com­plex­i­ty asso­ci­at­ed with lega­cy tech­nol­o­gy, few organ­i­sa­tions can afford true democ­ra­ti­sa­tion of infor­ma­tion.

These are all gates that stand in the way of data democ­ra­ti­sa­tion, says Thomas Hazel, CTO & founder at ChaosSearch.

“These gates – whether time, cost or com­plex­i­ty – are pre­vent­ing com­pa­nies from being data lit­er­ate,” says Hazel. “You can have hun­dreds of peo­ple in your organ­i­sa­tion, and it takes weeks, months, and even years to stand up infra­struc­ture to access the data. Data is the lifeblood of organ­i­sa­tions, and when it takes weeks and months to get access to it, it’s a prob­lem.

“You need to have a dif­fer­ent phi­los­o­phy as to how data is con­sumed, stored, man­aged and accessed.”

The ChaosSearch approach to mov­ing, stor­ing, organ­is­ing and pro­vid­ing access to data quick­ly and effi­cient­ly is based on a data lake phi­los­o­phy.

Hazel says the con­cept of a data lake is no longer bound by the time, com­plex­i­ty and cost restraints asso­ci­at­ed with some big data tech­nolo­gies from a few years ago. Cloud stor­age is the enabler of this lake phi­los­o­phy.

Now, cloud object stor­age – as pio­neered by Ama­zon Web Ser­vices (AWS) and serv­ing as the foun­da­tion of all the cloud providers – is the sim­plest and most secure way to store data that can scale infi­nite­ly but cost effec­tive­ly.

“A lake or cloud stor­age makes it so easy to stream data in. There’s no com­plex­i­ty of stand­ing any­thing up. There’s no schema to struc­ture it in a for­mat, it can just be con­sumed. You just set it up and for­get it,” says Hazel.

At the same time, he notes that many enter­pris­es want a cen­tralised way to iden­ti­fy what data they have and get access to it. The prob­lem is, there aren’t solu­tions that can take
advan­tage of a lake phi­los­o­phy.

ChaosSearch is a new way to rep­re­sent infor­ma­tion. Sup­port­ing mul­ti-mod­el data access meth­ods, ChaosSearch makes data simul­ta­ne­ous­ly avail­able through Elas­tic, SQL and, in future, machine learn­ing APIs. The ChaosSearch Data Lake Plat­form can con­nect to and index all data with­in a customer’s own cloud stor­age envi­ron­ment – mak­ing it ful­ly search­able and imme­di­ate­ly avail­able for analy­sis with exist­ing data tools.

“We have built the tech­nol­o­gy to remove those gates to democ­ra­tise that infor­ma­tion because those old archi­tec­tures, by def­i­n­i­tion, can’t,” explains Hazel.

“This com­bi­na­tion of inno­va­tion with a new archi­tec­ture and a new phi­los­o­phy means what used to take months, or maybe years, to build out at petabyte scale, now takes weeks or even maybe a day to stand up. That to me is a huge democ­ra­ti­sa­tion move.”

It is also impor­tant to acknowl­edge that the dif­fer­ence between the old and new way of approach­ing data lakes is that now they can be acti­vat­ed for gen­er­at­ing every­day busi­ness val­ue.

“A lot of cloud stor­age plat­forms are archiv­ing data for secu­ri­ty or com­pli­ance. If their siloed data­bas­es fall down, they send it to the lake. We had to tell cus­tomers that they could acti­vate it as their pri­ma­ry ana­lyt­i­cal source,” says Hazel.

“We tell our cus­tomers: ‘Those use cas­es you’re run­ning for log ana­lyt­ics or for secu­ri­ty, or you’re using maybe Snowflake for busi­ness analy­sis, what if you had one plat­form with a lake phi­los­o­phy that we acti­vate to per­form those use cas­es with­out chang­ing a thing?’”

Addi­tion­al­ly, because it plugs direct­ly into enter­pris­es’ exist­ing cloud object stor­age, they can have role-based access con­trol for dif­fer­ent depart­ments and users, with access rights grant­ed imme­di­ate­ly. There is no need for a data engi­neer to wran­gle the data, a team of engi­neers to for­mat it, or a data­base admin­is­tra­tor to define schema and rela­tion­al tables and pro­vide back con­trols.

“You could self-serve, and once you can self-serve the data is democ­ra­tised,” says Hazel. “You have to rethink how you man­age infor­ma­tion, as well as rein­vent how to access the infor­ma­tion.”

By 2022, IDC says 90% of cor­po­rate strate­gies will explic­it­ly men­tion infor­ma­tion as a crit­i­cal enter­prise asset and ana­lyt­ics as an essen­tial com­pe­ten­cy. Remov­ing any gates to data lit­er­a­cy will be imper­a­tive to an organisation’s agili­ty and growth.

“If your busi­ness isn’t adopt­ing this new phi­los­o­phy, you’re in trou­ble,” says Hazel. “If you can’t access infor­ma­tion, you’re going to be left out. The time is now. You can start send­ing your data to your lake, with­out hav­ing to make a choice, it’s just a phi­los­o­phy choice.”

• Gart­ner, 10 Ways CDOs Can Suc­ceed in Forg­ing a Data-Dri­ven Orga­ni­za­tion, Mike Rollings, Alan D. Dun­can, Valerie Logan. Refreshed 15 Octo­ber 2020. Pub­lished 22 May 2019. GARTNER is a reg­is­tered trade­mark and ser­vice mark of Gart­ner, Inc. and/or its affil­i­ates in the US and inter­na­tion­al­ly and is used here­in with per­mis­sion

For more infor­ma­tion please vis­it chaossearch.io

Spon­sored by

ChaosSearch