How to measure the ROI of agile projects

Cal­cu­lat­ing the return on invest­ment (ROI) of a project would seem to be a straight­for­ward thing to do – you take the income or prof­it from the project, whether actu­al or pro­ject­ed, and divide it by the costs. But some project man­age­ment experts go fur­ther than that.

Agile projects is an approach which is bro­ken down into more man­age­able sec­tions and helps deter­mine the over­all val­ue through­out the project, in addi­tion to tra­di­tion­al meth­ods which cal­cu­late finan­cial gain at the end.

Dr David Rico, an agile tech­ni­cal leader who has han­dled almost $2bn worth of agile projects — includ­ing for the US gov­ern­ment — car­ried out a meta study in 2009 of 29 projects. Rico found that the ROI for agile meth­ods was four times more expen­sive than tra­di­tion­al meth­ods, but two times low­er than inex­pen­sive ones, mak­ing it tricky to deter­mine the best agile and tra­di­tion­al meth­ods. 

How­ev­er, Rico point­ed out a key flaw in the sim­ple view of ROI, that it is dif­fi­cult to com­pare “tra­di­tion­al meth­ods opti­mised for pro­duc­tiv­i­ty and qual­i­ty, to agile meth­ods opti­mised for cus­tomer sat­is­fac­tion, project suc­cess and risk reduc­tion.”

Sim­ply put, both agile and tra­di­tion­al meth­ods can deliv­er com­pa­ra­ble ROI but we should not nec­es­sar­i­ly view ROI sole­ly in terms of finan­cial return.

Indeed, this is a core prin­ci­ple of the agile’s found­ing doc­u­ment — the Agile Man­i­festo — which emerged from a meet­ing of like-mind­ed soft­ware devel­op­ers back in 2001 who were look­ing for an alter­na­tive to “doc­u­men­ta­tion dri­ven and heavy­weight soft­ware devel­op­ment process­es.”

The first of its 12 key prin­ci­ples reads: “Our high­est pri­or­i­ty is to sat­is­fy the cus­tomer
through ear­ly and con­tin­u­ous deliv­ery of valu­able soft­ware.”

It is per­haps val­ue rather than finan­cial return that should be tak­en from agile.

Dr Nicholas Dacre is asso­ciate pro­fes­sor of project man­age­ment at the Uni­ver­si­ty of Southamp­ton and direc­tor of the Advanced Project Man­age­ment (APM) research cen­tre. Dur­ing the coro­n­avirus pan­dem­ic, he worked with a UK gov­ern­ment depart­ment on a major project and the speed of pos­i­tive change was deemed most impor­tant.

“Covid-19 spurred on new ways of think­ing in terms of project man­age­ment,” he says. “Based on decades of water­fall project man­age­ment approach­es, it was deemed that the speed and effi­cien­cy of tra­di­tion­al project man­age­ment at the time of a glob­al cri­sis was com­plete­ly unsuit­able. One of the advan­tages of using an agile approach was to reduce the speed at which they could affect pos­i­tive change.

“If that depart­ment hadn’t tran­si­tioned from tra­di­tion­al project man­age­ment approach­es to more agile process­es and ways of think­ing, ulti­mate­ly the project would not have been deliv­ered on time to have any pos­i­tive impact.”

Andrew Tay­lor, chief tech­nol­o­gy offi­cer at agile con­sul­tan­cy Q5 Part­ners, says there is no mag­ic for­mu­la for mea­sur­ing ROI. For some com­pa­nies the mea­sure­ment could be prof­it, and for oth­ers it could be the speed of deliv­er­ing a prod­uct or new fea­tures.

“Fun­da­men­tal­ly, we talk about the val­ue you are try­ing to deliv­er in your con­text,” Tay­lor says. “What are the out­comes? If I am a CEO look­ing from the top down, I might want to talk about the impact on the bot­tom line, how quick­ly we can release a new prod­uct or cut costs in a par­tic­u­lar area.

“The tra­di­tion­al mea­sures of the suc­cess of a project of just being about ‘on time, on scope and on bud­get’ are dead. I am not sole­ly inter­est­ed in that any­more, those are just out­puts of work. I am focused pri­mar­i­ly on the val­ue and out­comes that are being realised and far less about project deliv­ery dead­lines,” he says.

How is it achieved?

Ari Ghosh, head of Q5’s agile coach­ing prac­tice, says, “one impor­tant mea­sure we talk about are the OKRs – the objec­tives and key results.

“The objec­tive could be, say, that you want to be the most suc­cess­ful fran­chise oper­a­tion in the world; a key result, and there tend to be three to five of these, could be 15% growth in fran­chise loca­tions in the EMEA region in a year.

“You track those and see whether ROI is com­ing in, build­ing towards the over­all out­come you are look­ing to deliv­er,” Ghosh says.

OKRs were pop­u­larised by Intel’s Andy Grove in the 1970s.

In the sem­i­nal book Mea­sure What Mat­ters by John Doerr — who worked at Intel with Grove and intro­duced OKRs to Google — the search giant’s co-founder Lar­ry Page says: “OKRs have helped lead us to 10x growth, many times over. They’ve helped make our crazi­ly bold mis­sion of ‘organ­is­ing the world’s infor­ma­tion’ per­haps even achiev­able. They’ve kept me and the rest of the com­pa­ny on time and on track when it mat­tered the most.”

Q5’s Tay­lor says one approach is to cre­ate a whole hier­ar­chy of “mini OKRs” and make indi­vid­ual teams account­able for their per­for­mance and their out­comes. Then resources would be real­lo­cat­ed between teams over time, and mea­sur­ing ROI and react­ing to vari­ances can hap­pen much faster.

“Instead of hav­ing a quar­ter­ly or annu­al report where you say you have missed a tar­get by 10–20%, you are con­stant­ly track­ing and mak­ing micro-adjust­ments. You cre­ate vis­i­bil­i­ty and track the val­ue mea­sure you have set, and can lit­er­al­ly see how that per­co­lates up from the organ­i­sa­tion, and you can then align and hold indi­vid­ual teams to be more account­able.

“Investor expec­ta­tions are chang­ing. It is not about the annu­al report any­more. They want feed­back much ear­li­er and want to see what is hap­pen­ing on a month­ly basis,” Ghosh adds.

Though, set­ting OKRs should not be just about dri­ving change.

Tay­lor, who worked with a large tele­coms organ­i­sa­tion, says: “One of their strate­gic objec­tives is obvi­ous­ly to keep the net­work run­ning. There is a nat­ur­al incli­na­tion to set strate­gic objec­tives and mea­sure out­comes focused on change and the shiny new thing. But when it works most pow­er­ful­ly it encom­pass­es all of the valu­able work of the organ­i­sa­tion and that includes sus­tain­ing the busi­ness.”

ROI also has to be seen against a back­drop of a high lev­el of megapro­jects which have failed to meet their objec­tives, says Dacre. APM research among more than 850 project man­age­ment pro­fes­sion­als shows that just 22% of projects are whol­ly suc­cess­ful yet 76% of project pro­fes­sion­als state that agile project man­age­ment has had a pos­i­tive effect.

Some argue that the need to have an agile team leader or coach reduces the ROI on projects but Dacre dis­agrees. 

“A sur­pris­ing­ly large num­ber of organ­i­sa­tions have less than five years’ expe­ri­ence in agile process­es,” he says. “If you are cal­cu­lat­ing from a six-month per­spec­tive, there will be dis­rup­tion in terms of upskilling and adopt­ing new process­es. I would argue that in the medi­um-to long-term you have your ben­e­fits and your ROI. You add val­ue to the project at a quick­er pace.”

Eval­u­at­ing ROI is chal­leng­ing for any project, whether tack­led by tra­di­tion­al or agile meth­ods. The answer is not to focus too close­ly on finan­cial returns, although agile can clear­ly com­pete and beat tra­di­tion­al meth­ods in many projects.

To achieve suc­cess, organ­i­sa­tions must iden­ti­fy the val­ue that mat­ters most to them and adjust accord­ing­ly. In a post-Covid era, this is more impor­tant than ever.

Dr Nicholas Dacre, asso­ciate pro­fes­sor of project man­age­ment at the Uni­ver­si­ty of Southamp­ton, says that the agile method­ol­o­gy has sev­en key ben­e­fits and these should be con­sid­ered when eval­u­at­ing the poten­tial ROI of a project.

1. Has a notable effect on the qual­i­ty of the out­come of a project.
2. Enhances the abil­i­ty to man­age pri­or­i­ties as exter­nal fac­tors change.
3. Short­er deliv­ery times.
4. Improved cus­tomer sat­is­fac­tion as ROI is seen much soon­er when deliv­er­ing a min­i­mum viable prod­uct.
5. Reduces risk in chang­ing envi­ron­ments.
6. Increase project vis­i­bil­i­ty as stake­hold­ers feel they are part of a project.
7. Improve team morale and pro­duc­tiv­i­ty.

---