Metrics maze: marketers struggle to measure effectiveness

The Data & Mar­ket­ing Asso­ci­a­tion (DMA) turned up a dis­turb­ing insight in 2021. After three years of con­sis­tent per­for­mance in the mar­ket­ing sec­tor, effec­tive­ness sud­den­ly slumped. What hap­pened? 

The data came through the DMA’s awards, which have received more than 1,000 entries since 2017. The asso­ci­a­tion found that in 2021, effec­tive­ness slumped by a mas­sive 23%.

Tim Bond, direc­tor of insight at the DMA, says “the pan­dem­ic was always going to be a curve­ball and we weren’t sure what would hap­pen. But last year we were encour­aged by the increase we saw. This year [the drop] is poten­tial­ly of con­cern”.

He offers poten­tial expla­na­tions for the sud­den and steep drop. “Dur­ing the pan­dem­ic, when brands were hyper-con­scious on reduced bud­get, there were added mea­sures put in place to make sure they were get­ting bang for their buck. When that spend returned, some of those hygiene fac­tors may have dropped.”

Paul Sin­clair is mar­ket­ing direc­tor at Zen Inter­net. He’s not sur­prised by the data. But although it is not new, it’s good to have sta­tis­tics to demon­strate the sector’s fears. 

“Part of it is the pro­lif­er­a­tion of data, more and more dig­i­tal chan­nels, TV and radio becom­ing digi­tised. There’s more data to be looked at and analysed and many mar­keters or lead­ers have tak­en their eye off the ball for the ones that real­ly mat­ter.”

Mythical marketing metrics

So could the prob­lem be traced to the meth­ods com­pa­nies use to mea­sure their effec­tive­ness? 

The DMA research high­lights four cat­e­gories to deter­mine mar­ket­ing effec­tive­ness: busi­ness effects (prof­it, mar­ket share); brand effects (aware­ness, con­sid­er­a­tion); response effects (leads, Book­ings); and cam­paign deliv­ery effects (reach, impres­sions).

The research found that mar­keters enter­ing the awards used a total of 170 dif­fer­ent met­rics, with the use of cam­paign deliv­ery met­rics mak­ing up 41% of entries and 59% devot­ed to busi­ness, brand or response effects. In fact, only 6% of effec­tive­ness mea­sures seen in the research relat­ed to busi­ness effects. 

Cam­paign deliv­ery effects are often termed van­i­ty met­rics, easy num­bers to obtain and often cit­ed in mil­lions of some­thing – views, clicks, audi­ence – that grab the atten­tion but offer lit­tle in terms of under­stand­ing busi­ness out­comes. So why are they even used?

Van­i­ty met­rics have a place, says Jamie Irv­ing, glob­al head of dig­i­tal mar­ket­ing at Boden, but he not­ed “it’s about qual­i­ty, not quan­ti­ty after all”. They should be sec­ondary or wrapped into a sin­gle KPI to be effec­tive, Irv­ing says. 

“Ulti­mate­ly, the most impor­tant met­ric is the one which will shift busi­ness out­comes.  If we view van­i­ty as guardrails and acknowl­edge the impact they’ll have on the met­ric you need to hit, then you are in con­trol of the per­for­mance you need to deliv­er. That should help main­tain effec­tive­ness.”

Cam­paign effects have a cer­tain imme­di­a­cy: deploy an ad and see the num­ber of clicks only an hour lat­er. That’s seduc­tive, but not nec­es­sar­i­ly help­ful, and cer­tain­ly not on its own. The DMA’s report shows that medi­um-term activ­i­ty – cam­paigns between four and 12 months in dura­tion – were squeezed in 2021, even though they’re gen­er­al­ly seen as opti­mal for dri­ving ROI. With the desire to pin imme­di­ate prof­it to mar­ket­ing spend, that change could have a neg­a­tive impact in ROI mul­ti­pli­ers. 

Cam­paign effects can be use­ful for media plan­ning, but for any­thing more pro­found “they’re not fit for pur­pose,” Bond says. It’s best to take a blend­ed approach, pri­ori­tis­ing busi­ness, brand and response effects while match­ing them to busi­ness real­i­ties such as cat­e­go­ry and bud­get. This helps to real­ly under­stand effec­tive­ness. 

“At the most basic lev­el, log­ic dic­tates that if we’re choos­ing to take an action there will be a return, whether it’s a shift in aware­ness, con­sid­er­a­tion or sales. There­fore there are only real­ly three met­rics that could mat­ter,” Irv­ing says. “What changes is the timescale we’re look­ing at and the need for prox­ies to have an ‘ear­ly read’. It’s the ear­ly read that’ll be the killer if not every­one is aligned to the true met­rics.” 

Long-term commitment to brand building

Sin­clair acknowl­edges oth­ers may feel pres­sure from on high, which can lead mar­keters to grasp at the quick fix straws of cam­paign deliv­ery data. “We [have] no short-term share­hold­er demands to meet. [But] I often report back to the board quar­ter­ly with met­rics that give us con­fi­dence – the type of cus­tomers we’re acquir­ing, under­stand­ing if we’re keep­ing exist­ing cus­tomers hap­py.”

One of the key impli­ca­tions to come out of the DMA’s research is that there should be a renewed focus on brand build­ing. Plac­ing less weight on imme­di­ate out­comes, this impacts ROI by stim­u­lat­ing future demand. It can also insu­late com­pa­nies from the price pro­mo­tion race to the bot­tom, by encour­ag­ing con­sid­er­a­tion based on brand val­ues rather than cost. 

How­ev­er, as brand effects tend to be more com­plex and cost­ly to mea­sure, the research sug­gests com­pa­nies will have to com­mit more resources to the area if they’re to see long-term results. 

What is cer­tain is that 170 met­rics is too many. Mar­ket­ing depart­ments are going to have to ratio­nalise their KPIs to find those that give a true pic­ture of the state of their busi­ness, and reduce their depen­den­cy on the quick, reas­sur­ing hit of sky-high but use­less cam­paign effects. 

Sin­clair sug­gests that mar­keters stop using van­i­ty met­rics as a proxy for a sci­en­tif­ic approach to mea­sure­ment and actu­al­ly under­stand what it means to gain real insight. “We’re com­mer­cial peo­ple with­in the organ­i­sa­tion so we need to get com­fort­able with data sets, how they link to pric­ing and so on. I’m joined at the hip with the CFO.”

There is already some­thing of a mar­ket cor­rec­tion. Famous­ly, chal­lenger bank Star­ling pulled its ads from Face­book and Insta­gram over pri­va­cy con­cerns, but sev­er­al weeks lat­er stat­ed that it “hasn’t caused a notice­able decline”. 

Irv­ing believes recent moves to retire the third-par­ty cook­ie could be a step in the right direc­tion for refo­cus­ing that con­ver­sa­tion. “By default, busi­ness met­rics [will] come more to the fore than they may have done in recent years and the data we need to use to make deci­sions [will] become more reli­able.”

For mar­keters and their col­leagues in finance, sales and even the C‑Suite, it may feel like there’s a need to brush up on the basics. To this end, in part­ner­ship with the DMA Media Coun­cil, the DMA has recent­ly launched a Mar­ket­ing Frame­work 101. This will pro­vide best-prac­tice guid­ance on how to mea­sure busi­ness out­comes and under­stand the impacts of mar­ket­ing across dif­fer­ent timescales. 

The frame­work will also include a glos­sary to define terms around mea­sure­ment so that every­one is at least using the same ter­mi­nol­o­gy. Siloed media plan­ning cre­ates siloed media mea­sure­ment and dis­torts mar­ket­ing effec­tive­ness. The report sug­gests that it leaves mar­keters try­ing to jus­ti­fy mar­ket­ing spend “with one hand tied behind their backs”. 

The glos­sary, the DMA hopes, will help organ­i­sa­tions under­stand what is meant by var­i­ous terms across the media mea­sure­ment process and ‘de-silo’ the process of mea­sure­ment. Armed with a more rig­or­ous approach to effec­tive­ness, per­haps in future ini­tia­tives that seek to prove their suc­cess­es on the awards cir­cuit will then tru­ly be wor­thy of a gong.