Fighting (fake) fire with fire: can deepfakes catch financial scams?

Most of us have seen a deep­fake video at one time or anoth­er, be it Don­ald Trump appear­ing on Bet­ter Call Saul or Tom Cruise per­form­ing mag­ic tricks on Tik­Tok. The media cov­er­age is often neg­a­tive: from sex­ploita­tion to cor­po­rate black­mail, we’re con­stant­ly told that deep­fakes will enable fraud and decep­tion on a mas­sive scale.

At Amer­i­can Express, how­ev­er, the tech­nol­o­gy behind deep­fakes is serv­ing the oppo­site end: fight­ing fraud. By using hyper-real­is­tic data to help train the company’s own detec­tion sys­tems, the company’s researchers believe they can warn cus­tomers more accu­rate­ly and min­imise the num­ber of unnec­es­sary card stop­pages.

It’s cer­tain­ly a bold strat­e­gy, not to men­tion a time­ly one. Glob­al pay­ment card fraud loss­es reached $28.65 bil­lion in 2019, accord­ing to the Nil­son Report. It’s almost cer­tain this fig­ure has risen dur­ing the Covid-19 pan­dem­ic; var­i­ous finan­cial agen­cies have report­ed an uptick in fraud dur­ing recent months, dri­ven by a rise in online shop­ping.

The Machine Learning advantage

We’ve wit­nessed a dra­mat­ic evo­lu­tion in fraud tac­tics over recent years, dri­ven by dig­i­tal tech­nol­o­gy. From phish­ing scams to bot­nets that can run card test­ing schemes (where a fraud­ster “tests” a cred­it card num­ber that they may have pur­chased on the dark web, ran­dom­ly gen­er­at­ed, or acquired via phish­ing or spy­ware soft­ware) on an indus­tri­al scale, the fraud­ster has nev­er had more weapons in their arse­nal. The advent of deep­fakes opens up a poten­tial new front, enabling con artists to dupe vic­tims into hand­ing over their details by sim­u­lat­ing the voic­es of rel­a­tives or com­pa­ny boss­es.

In their attempts to stem the ris­ing tide, many card com­pa­nies are now using machine learn­ing, a form of arti­fi­cial intel­li­gence in which com­put­er sys­tems improve auto­mat­i­cal­ly by adapt­ing to the data they receive. Engi­neers feed reams of trans­ac­tion data into the ML algo­rithm; using this data, the algo­rithm iden­ti­fies pat­terns around fraud­u­lent trans­ac­tions — their size, their loca­tion, the time of day they take place — and sub­mits this data to the fraud pre­ven­tion team.

ML mod­els offer three dis­tinct advan­tages over con­ven­tion­al, rules-based pre­ven­tion strate­gies. First, they can incor­po­rate a mul­ti­tude of fac­tors. Sec­ond, they can adapt to chang­ing behav­iour pat­terns. And final­ly, they cre­ate few­er false pos­i­tives, avoid­ing the need for those block­ages that cause cus­tomers so much frus­tra­tion. How­ev­er, there is one cru­cial caveat: they rely on real­is­tic, high-qual­i­ty data inputs to iden­ti­fy pat­terns accu­rate­ly.

This is where deep­fake tech­nol­o­gy comes in. The tech­nol­o­gy is itself a form of ML, which relies on a pair of algo­rithms known as gen­er­a­tive adver­sar­i­al net­works (GANs). The two algo­rithms are, essen­tial­ly, try­ing to out­smart one anoth­er: one algo­rithm, the gen­er­a­tor, cre­ates the con­tent, while its rival, the dis­crim­i­na­tor, looks for flaws. Accu­ra­cy and rigour are baked into the sys­tem.

Accord­ing to Dmit­ry Efi­mov, Amer­i­can Express’s VP of ML research, GAN tech­nol­o­gy enables the company’s data sci­en­tists to react rapid­ly to new types of fraud. By sim­u­lat­ing spend­ing pat­terns from gen­uine trans­ac­tion­al data, the data sci­ence team can cre­ate vast amounts of records for their ML mod­els with­out rely­ing on real-life infor­ma­tion.

“GAN is use­ful when we’re not able to train a mod­el because of lack of data. An imme­di­ate use case is our fraud mod­el, because fraud pat­terns can change rapid­ly. Ear­ly detec­tion is the key pre­ven­tion. 

“If we detect a new pat­tern of fraud we’ve nev­er seen before, we want to be able to pro­tect our cus­tomers from it, so we want to train our mod­el to detect these new pat­terns. But to train the mod­el you need lots of data, and we may have only seen this new fraud pat­tern a cou­ple of times. 

“That’s why we start­ed to explore whether GAN could help us solve that prob­lem, by allow­ing us to use sim­u­lat­ed data of that fraud pat­tern in order to train the mod­els and improve mod­el per­for­mance.”

At this stage, how­ev­er, the project is still only at the research and exper­i­men­ta­tion stage. While the GAN data has proved extreme­ly use­ful when the researchers haven’t had vast swathes of his­tor­i­cal spend data to work with (as is the case when deal­ing with new cus­tomers), Efi­mov admits that all the exper­i­ments car­ried out so far show that “GAN-sim­u­lat­ed data did not always improve the final [ML] mod­els.”

It remains to be seen whether GAN-based data will become a stan­dard tool for fraud detec­tion across the finance indus­try. Some com­men­ta­tors are scep­ti­cal, sug­gest­ing there is a lim­it to the accu­ra­cy that these sim­u­lat­ed records can pro­vide.

Cautious optimism

“These arms race dynam­ics are very chal­leng­ing,” says Hen­ry Ajder, a free­lance advi­sor on deep­fakes and emerg­ing tech­nolo­gies.. “Whether it’s detect­ing deep­fakes or sus­pi­cious bank trans­ac­tions, attack­ers are con­stant­ly work­ing to find new exploits, with defend­ers often play­ing catch up.”

Gen­er­at­ing syn­thet­ic data to train detec­tion sys­tems might give them an edge in the short term, Ajder adds, but there’s no guar­an­tee how long that advan­tage will last. Still, he thinks there could be ben­e­fits.

“Think of anti-virus soft­ware. No com­pa­ny claims its soft­ware will catch every virus, although it does raise the bar­ri­er of entry by catch­ing the major­i­ty of exam­ples that aren’t on the cut­ting edge.”

Shifting patterns

For the GANs to be tru­ly use­ful, the Amer­i­can Express researchers will need to con­sid­er a full range of fraud sce­nar­ios in their data inputs. The sheer vari­ety of sit­u­a­tions that could lead to fraud can be extreme­ly dif­fi­cult to repli­cate. As well as con­ven­tion­al card theft, for exam­ple, the inputs must include cas­es in which the tar­get has been tricked into mak­ing the trans­ac­tion them­selves. 

“Humans are unpre­dictable,” says cyber­se­cu­ri­ty con­sul­tant Dr Edewede Ori­woh, who pre­vi­ous­ly worked at the Uni­ver­si­ty of Bed­ford­shire. “Fixed, repeat­ed pat­terns of behav­iour, even when it comes to spend­ing mon­ey, may not always appear. Pat­terns may change dras­ti­cal­ly depend­ing on an indi­vid­u­al’s mood or recent events.”

Amer­i­can Express will need high­ly var­ied meth­ods “to ensure their algo­rith­mic mod­el does not flag too many false pos­i­tives”, Ori­woh adds.

The scale of the chal­lenge, then, is con­sid­er­able. How­ev­er, some are opti­mistic about the project, view­ing deep­fakes as a gen­uine solu­tion to fraud in the long term.

“If you’re talk­ing about recre­at­ing human faces or voic­es, there are still some tell­tale signs with deep­fakes,” says Leroy Ter­re­longe, a senior cyber risk ana­lyst at Moody’s Investors Ser­vice. “But when you’re deal­ing with a doc­u­ment that’s essen­tial­ly just num­bers and text, I don’t see what the bar­ri­er is.”

Amer­i­can Express has data that poten­tial­ly goes back all the way to the start of the com­pa­ny, Ter­re­longe not­ed. ML sys­tems are pow­er­ful because they can recog­nise pat­terns far more quick­ly than humans. 

“This seems like a very fea­si­ble use case for deep­fakes.”