How to avoid AI accidents: a guide for the public sector

Algo­rithms have the poten­tial to wreck lives. Take, for instance, the fias­co sur­round­ing 2020’s A‑level results in Eng­land and Wales, when many thou­sands of stu­dents who’d been unable to sit their exams rebelled against the unfair grades they’d been assigned by a flawed algo­rithm.

Yet algo­rithm-based arti­fi­cial intel­li­gence (AI) sys­tems are pow­er­ful tools that could rad­i­cal­ly improve the work of many pub­lic bod­ies. They offer new pos­si­bil­i­ties for the deliv­ery of many ser­vices, advances in health­care research, effi­cien­cies in the labour mar­ket and the per­son­al­i­sa­tion of online ser­vices.

Accord­ing to the Ada Lovelace Insti­tute, an inde­pen­dent research group that mon­i­tors the use of data and AI, algo­rith­mic deci­sion-mak­ing sys­tems are being deployed at an unprece­dent­ed speed in both the busi­ness world and the pub­lic sec­tor. They are becom­ing ubiq­ui­tous, embed­ded in every­day prod­ucts and ser­vices.

But their ‘black box’ nature – the opac­i­ty with which they are designed and used – indi­cates an absence of human con­trol and respon­si­bil­i­ty, ring­ing alarm bells. This lack of trans­paren­cy under­mines trust in algo­rithm-based deci­sion-mak­ing and the organ­i­sa­tions that use such process­es. 

Of par­tic­u­lar con­cern to the Ada Lovelace Insti­tute is “the expan­sion of algo­rith­mic deci­sion-mak­ing sys­tems across the pub­lic sec­tor, from AI ‘stream­ing tools’ used in pre­dic­tive ana­lyt­ics in polic­ing to risk-scor­ing sys­tems to sup­port wel­fare and social care deci­sions”.

There is a wide­spread lack of under­stand­ing about where and how such tech­nol­o­gy is being used, notes its asso­ciate direc­tor of pub­lic and social pol­i­cy, Imo­gen Park­er. Pub­lic bod­ies need to address this mat­ter urgent­ly if they are to engen­der trust in their deci­sion-mak­ing process­es.

Although the insti­tute knows that “data-dri­ven tools are being used to match peo­ple to the right ser­vices, assess visa appli­ca­tions, pre­dict risk and even esca­late fam­i­lies into children’s social care, we have a pauci­ty of infor­ma­tion about the pub­lic sector’s use of algo­rithms”, she says.

The gov­ern­ment is at least pilot­ing a trans­paren­cy reg­is­ter for algo­rithms and con­sult­ing on whether this should be made manda­to­ry. It is also com­mit­ted to pub­lish­ing a white paper on AI reg­u­la­tion ear­ly next year.

“Greater trans­paren­cy is only the first step towards full account­abil­i­ty,” Park­er stress­es. “The ulti­mate goal must be to cre­ate sys­tems that work for peo­ple. We need mech­a­nisms that would involve peo­ple who are affect­ed by these sys­tems in devel­op­ing and deploy­ing them; in assess­ing their risks and impacts; and in enforc­ing sanc­tions as robust reg­u­la­tors who can pass judg­ment where need­ed.”

Stian West­lake, CEO of the Roy­al Sta­tis­ti­cal Soci­ety, agrees that organ­i­sa­tions using com­plex algo­rithms need to engen­der trust among those whom their deci­sions will affect. They can start doing this by being clear about their intend­ed appli­ca­tions for the tech­nol­o­gy.

“Trust can be harmed when opaque algo­rithms are used with­out good-qual­i­ty data behind them,” he adds, stress­ing that local author­i­ties and oth­er pub­lic bod­ies should care­ful­ly assess the datasets they plan to use. 

“Gov­ern­ment data can be biased or sim­ply wrong,” West­lake notes. “It is also a good idea, as the Office for Sta­tis­tics Reg­u­la­tion has sug­gest­ed, to test the accept­abil­i­ty of the algo­rithm with affect­ed groups. “Pub­lic bod­ies should be able to explain how con­clu­sions are reached in indi­vid­ual cas­es, not just fall back on ‘com­put­er says no’.”

Park­er adds: “We’re see­ing a trust deficit in how data is being used. We’ve had the A‑level protests and the suc­cess­ful legal chal­lenge against the police’s use of facial-recog­ni­tion tech­nol­o­gy in Eng­land and Wales. And, more recent­ly, more than 3 mil­lion peo­ple opt­ed out of shar­ing their med­ical infor­ma­tion as part of the government’s Gen­er­al Prac­tice Data for Plan­ning and Research frame­work.”

The out­cry over the A‑level grad­ing deba­cle, and the government’s U‑turn in response to it, indi­cates that there can be no true algo­rith­mic account­abil­i­ty with­out a “crit­i­cal audi­ence” that can com­mu­ni­cate effec­tive­ly with pol­i­cy-mak­ers. So says Dr Daan Kolk­man, research fel­low at Eind­hoven Uni­ver­si­ty of Tech­nol­o­gy and the Jheron­imus Acad­e­my of Data Sci­ence in ’s‑Hertogenbosch. 

A for­mal crit­i­cal audi­ence could take the shape of an inde­pen­dent, pub­licly fund­ed watch­dog that can draw on tech­ni­cal exper­tise to inves­ti­gate sys­tems, report in the pub­lic domain and, if nec­es­sary, pro­vide redress to par­ties it deems unfair­ly dis­ad­van­taged by poor­ly designed algo­rithms.

Kolk­man stress­es that algo­rithms are only as good as the data they are fed and are inher­ent­ly biased because peo­ple are. AI sys­tems are trained on his­toric data pat­terns, many of which incor­po­rate social inequal­i­ties and are there­fore at risk of per­pet­u­at­ing such inequal­i­ties. He quotes renowned British sta­tis­ti­cian George Box’s cau­tion­ary words in say­ing: “All mod­els are wrong, although some are use­ful.”

This is a wide­ly acknowl­edged prob­lem, which pro­gram­mers have sought to address by mak­ing their algo­rithms more explain­able. But, as Park­er notes, “you can work to cor­rect some tech­ni­cal bias­es, but there will always be bias­es. The key ques­tions are whether those bias­es are legal­ly accept­able and whether devel­op­ers and pub­lic-sec­tor bod­ies are antic­i­pat­ing and mit­i­gat­ing them where they can.”

She adds that “bias can arise where the data is of a high qual­i­ty and ‘accu­rate’, because an algo­rithm encodes struc­tur­al inequal­i­ties in soci­ety into the future and ampli­fies them. A good exam­ple of this is a hir­ing algo­rithm that gives high scores white men seek­ing senior roles because it has learnt that they are his­tor­i­cal­ly the suc­cess­ful can­di­dates for such jobs.”