Simplifying the data stack: why businesses need a real-time upgrade

Data can be key to solv­ing busi­ness prob­lems, but only if deci­sion-mak­ers have the abil­i­ty to access it while it’s still rel­e­vant. Find­ing out what action should be tak­en now is more impor­tant than deter­min­ing what should have been tak­en yes­ter­day. CDOs are under pres­sure to man­age vast amounts of data at an unprece­dent­ed pace: deliv­er­ing data and insights in real-time is impor­tant across every type of busi­ness and in every aspect of those busi­ness­es.

Prod­uct teams that get usage insights in sec­onds instead of days can increase user adop­tion and reduce churn. Stock and cryp­tocur­ren­cy ana­lysts who can stream and analyse trad­ing data instan­ta­neous­ly have an advan­tage over their mar­ket peers in iden­ti­fy­ing the most prof­itable trades. Ecom­merce stores that run ana­lyt­ics with mil­lisec­ond laten­cy can instant­ly per­son­alise the shop­ping expe­ri­ence deliv­ered through a seam­less UX, boost­ing con­ver­sion rates and rev­enues. Regard­less of indus­try or prod­uct, real-time ana­lyt­ics improves the KPIs that data lead­ers — and their stake­hold­ers — care about.

But get­ting to the stage where prod­uct devel­op­ment teams can make use of data at scale is eas­i­er said than done. Busi­ness­es process huge vol­umes of data each day from numer­ous sources, stored in mul­ti­ple ways by mul­ti­ple peo­ple across the organ­i­sa­tion. Data ware­hous­es are well equipped to store uni­fied data that sup­ports spe­cif­ic ana­lyt­ics needs, but hand­offs and delays are often hard to avoid while data is being batched and processed. Even minute delays may have rip­ple effects that impact user expe­ri­ence, and act­ing on ana­lyt­i­cal queries in real-time is not ten­able, with the val­ue of the insights they pro­vide drop­ping as the min­utes tick by.

Data lead­ers must rise to the chal­lenge of build­ing or rebuild­ing data infra­struc­ture that ensures every­one can get the data they need when they need it. In the­o­ry, clos­ing the gap between ana­lyt­ics and action can be achieved inter­nal­ly. The instinct may be to throw data engi­neers, time, and mon­ey at the prob­lem in the hopes of improv­ing laten­cy or con­cur­ren­cy met­rics. But this also means divert­ing sub­stan­tial resources away from the actu­al busi­ness of the busi­ness — and away from improv­ing the prod­uct.

Jorge Gomez San­cha, co-founder and CEO of Tiny­bird, knows that engi­neers are often used to man­ag­ing their own data infra­struc­ture, which brings with it a score of things to think about, from basic serv­er con­fig­u­ra­tion to the intri­ca­cies of devel­op­ing secure, low-laten­cy, high-con­cur­ren­cy end­points.

“‘What’s the right CPU lev­el, the right num­ber of CPU cores, or amount of mem­o­ry? Should I use repli­ca­tion or shard­ing? How do I keep track of whether every­thing is up and run­ning?’ They’re think­ing ‘we need a data­base that we then build things on top’ ver­sus ‘we need tools to solve our prob­lems with data’,” says San­cha. “Your devel­op­ers should­n’t even need to think about any of that, they should be think­ing ‘What’s the next busi­ness prob­lem I can solve and pro­vide val­ue?’” San­cha believes that data lead­ers should focus on enabling devel­op­ers to apply ana­lyt­ics with­in the prod­ucts they build in as close to real-time as pos­si­ble.

In many cas­es, the prob­lems data lead­ers need to solve in order to give devel­op­ers access to real-time ana­lyt­ics are not unique. Almost every CDO or head of data is con­fronting or has con­front­ed the chal­lenge of build­ing real-time data archi­tec­ture. But why is it so chal­leng­ing to solve? Data ware­hous­es weren’t built for real-time, but data teams have spent the last decade invest­ing in infra­struc­ture and tool­ing that sur­rounds and sup­ports the data ware­house. To solve for real-time use cas­es, data lead­ers will need to branch out.

But why start from scratch when tools like Tiny­bird are already solv­ing the prob­lem? Increased per­for­mance and pre­ci­sion are core goals for any CDO, and stream­lin­ing these process­es requires the right set-up. Data teams that use a server­less approach with best-of-breed soft­ware can inte­grate with exist­ing streams, data­bas­es, and ware­hous­es, process them through an opti­mised and sim­pli­fied data stack, and then pro­vide near-instan­ta­neous access on the oth­er side — often in a way that com­ple­ments the tra­di­tion­al data ware­house. These server­less tools can plug into dash­boards, trig­ger alerts, pow­er automa­tion, or feed into what­ev­er oth­er data prod­ucts the busi­ness uses.

San­cha points to Key­rock, a Tiny­bird cus­tomer oper­at­ing in the cryp­tocur­ren­cy mar­ket, as a prime exam­ple: “they ingest data from mar­kets all across the world, in cryp­to but also in oth­er assets, and then they are con­stant­ly mak­ing bets as to where things are going and cre­at­ing trans­ac­tions.” Inevitably this involves a mas­sive amount of data, and Key­rock was bat­tling a host of laten­cy, fresh­ness and con­cur­ren­cy issues. Data took many sec­onds or even min­utes to process and hand off, and even went miss­ing entire­ly, mak­ing it chal­leng­ing to analyse and act on that data in real-time. In such cas­es, attempt­ing to fix the issue inter­nal­ly was cost­ly and time-con­sum­ing: using Tiny­bird proved a far more effi­cient solu­tion.

Data teams that add real-time archi­tec­ture to their stack often dis­cov­er oppor­tu­ni­ties to apply the tech to new, tan­gen­tial use cas­es that have sat unre­solved. San­cha remarks on a client who had ini­tial­ly tran­si­tioned to a real-time archi­tec­ture to reduce the lag time in their ana­lyt­ics plat­form. This shed light on a new solu­tion to an old prob­lem — could real-time ana­lyt­ics be the key to iden­ti­fy­ing and pre­vent­ing denial-of-ser­vice attacks on their ser­vices? Imple­ment­ing instant log­ging and ana­lyt­ics enabled them to turn hours of work into an auto­mat­ed process that could effec­tive­ly inter­cept and respond to attacks with­in min­utes. Once the val­ue of real-time data becomes evi­dent in one area, dif­fer­ent teams across the busi­ness will find that they rapid­ly devel­op new ways to apply real-time across every­thing they do.

Data lead­ers need to give devel­op­ers the tools to make action­able use of the data that’s pour­ing into their data­bas­es at a moment’s notice, and wait­ing min­utes to under­stand what’s hap­pen­ing while their com­peti­tors wait sec­onds or mil­lisec­onds is not a sus­tain­able option. They need real-time data archi­tec­tures with an empha­sis on sim­plic­i­ty and per­for­mance. Hav­ing the data to make crit­i­cal busi­ness deci­sions isn’t enough; devel­op­ers and data teams need to be able to build data prod­ucts that can have a real-time impact at any scale. CDOs and heads of data need to act now to realise this, as the size of their data is only accel­er­at­ing.

For more infor­ma­tion please vis­it tinybird.co

Pro­mot­ed by Tiny­bird