What to do when the servers go down

Nation­wide cus­tomers faced repeat­ed dis­rup­tion to their bank­ing ser­vices around Christ­mas, unable to access their funds or pay their bills. While the build­ing soci­ety described the out­age as a “tech­ni­cal issue”, some experts iden­ti­fied a serv­er fail­ure. 

The dis­rup­tion demon­strat­ed just how much dam­age a tech crash can cause and the salient impor­tance of servers for smooth busi­ness oper­a­tions, with mil­lions of users unable to receive or make pay­ments.

What­ev­er the caus­es of a serv­er crash – rang­ing from sim­ple hard­ware fail­ure and pow­er out­ages to soft­ware glitch­es, cyber­at­tacks and nat­ur­al dis­as­ters – the con­se­quences can be cat­a­stroph­ic. Busi­ness­es, big and small, rely on con­nec­tiv­i­ty; in a dig­i­tal age, it’s the lifeblood of com­merce. Con­se­quent­ly, organ­i­sa­tions have become increas­ing­ly reliant on servers.

As screens go blank, dig­i­tal and human con­nec­tions are cut. The afflict­ed organ­i­sa­tion los­es pro­duc­tiv­i­ty, orders and prof­its, while cus­tomers are affect­ed, caus­ing rep­u­ta­tion­al dam­age and pos­si­ble loss of future busi­ness. In addi­tion, if pri­vate data is lost, reg­u­la­to­ry fines and penal­ties can result, as well as class-action law­suits.

Servers sup­port essen­tial con­nec­tions, which facil­i­tate busi­ness oper­a­tions includ­ing inter­ac­tion with staff and cus­tomers. Their impor­tance means more and more com­pa­nies use a net­work of cloud servers.

These servers now play a vital role in busi­ness tech­nol­o­gy. They pro­vide a cen­tral repos­i­to­ry, receiv­ing, stor­ing, retriev­ing and send­ing data, ensur­ing all team mem­bers have time­ly access to the infor­ma­tion they need. 

Web, email and file servers, to name just a few, are essen­tial for employ­ees, teams and sys­tems to per­form the tasks that make up their jobs. The pan­dem­ic and result­ing shift to remote and hybrid work­ing have nec­es­sar­i­ly accel­er­at­ed data-cen­tric, cloud-based dig­i­tal­i­sa­tion, so busi­ness­es have become increas­ing­ly depen­dent on the unin­ter­rupt­ed oper­a­tion of their servers. 

Achilles’ heel

But have servers – a com­put­er with advanced hard­ware run­ning a serv­er pro­gram – become an Achilles’ heel? Essen­tial to busi­ness oper­a­tions, what hap­pens when servers crash? How can an organ­i­sa­tion recov­er quick­ly and get back to work? 

Azeem Javed is a con­sul­tant at Cre­ative Net­works, man­aged IT and tele­coms spe­cial­ists. He says that encap­su­lat­ing back­up as part of a busi­ness con­ti­nu­ity and dis­as­ter recov­ery (BCDR) strat­e­gy “is crit­i­cal for all busi­ness­es, ensur­ing con­ti­nu­ity of their oper­a­tions and suit­able recov­ery. This should extend to all aspects of busi­ness.”

Con­tin­gency plan­ning and a sys­tem back­up strat­e­gy – installing local­ly based or remote back­up servers or back­up to an exter­nal hard dri­ve and dis­as­ter recov­ery soft­ware – can cer­tain­ly help the chief tech­nol­o­gy offi­cer sleep more sound­ly at night. If a busi­ness has alter­nate back­ups for its files, it can quick­ly bounce back and resume oper­a­tions.

A full back­up is a com­plete copy of an organ­i­sa­tion’s data assets. This process requires all files to be backed up into a sin­gle ver­sion. How­ev­er, the dataset should be copied in its entire­ty and stored in a sep­a­rate loca­tion, away from the serv­er.

Such an off­site back­up, which can be accessed, restored or admin­is­tered from a dif­fer­ent loca­tion, guar­an­tees high-lev­el secu­ri­ty and peace of mind as it allows data stor­age off­site and online.

“If your data is mis­sion-crit­i­cal to your busi­ness, back­up servers are absolute­ly vital to ensure seam­less busi­ness con­ti­nu­ity and to avoid data loss,” says Jake Mad­ders, a direc­tor at Hyve, a man­aged cloud host­ing provider. 

“We now live in an ‘always-on’ world, where just one hour of down­time can cost any­thing from thou­sands to hun­dreds of thou­sands of pounds, depend­ing on the size of a com­pa­ny. Time is mon­ey.”

Worst-case scenario

Irre­spec­tive of the loca­tion of the serv­er, a BCDR plan is essen­tial for the worst-case sce­nario.

“The pan­dem­ic has forced com­pa­nies to realise that being pre­pared for even the most unlike­ly sit­u­a­tion can no longer be treat­ed as an option­al part of busi­ness plan­ning,” says Mad­ders. “While it might seem dif­fi­cult to mea­sure the return on invest­ment of a dis­as­ter recov­ery solu­tion, because it’s a pre­cau­tion­ary fea­ture that ide­al­ly would nev­er need to be used, it shouldn’t be seen as a ‘lux­u­ry’ add-on ser­vice sole­ly for larg­er com­pa­nies. It should be a fun­da­men­tal part of every business’s IT strat­e­gy.”

A dis­as­ter recov­ery plan is a doc­u­ment­ed, struc­tured approach that describes how an organ­i­sa­tion can quick­ly resume work after an unplanned inci­dent. It is an essen­tial part of a busi­ness con­ti­nu­ity plan and should be applied to the aspects of the oper­a­tion that depend on a func­tion­ing IT infra­struc­ture. 

The step-by-step plan con­sists of pre­cau­tions to min­imise the effects of a dis­as­ter so the organ­i­sa­tion can con­tin­ue to oper­ate or quick­ly resume mis­sion-crit­i­cal func­tions. Typ­i­cal­ly, dis­as­ter recov­ery plan­ning involves an analy­sis of busi­ness process­es and con­ti­nu­ity needs. 

Before gen­er­at­ing a detailed plan, an organ­i­sa­tion should per­form a busi­ness impact analy­sis and risk analy­sis, and estab­lish recov­ery objec­tives.

All strate­gies should align with the organ­i­sa­tion’s goals. Once a busi­ness con­ti­nu­ity strat­e­gy has been devel­oped and approved, it can be trans­lat­ed into a dis­as­ter recov­ery plan, with an inci­dent response team and list of impor­tant con­tacts.

The plan should be reviewed by man­age­ment, test­ed, audit­ed and reg­u­lar­ly updat­ed. It should be sub­stan­ti­at­ed through test­ing, which iden­ti­fies defi­cien­cies and pro­vides oppor­tu­ni­ties to fix prob­lems before a crash occurs. Addi­tion­al­ly, it’s impor­tant for busi­ness­es to mon­i­tor and pro­tect their servers with soft­ware that can flag up poten­tial prob­lems.

And before call­ing in the tech experts, there are a few basic house­keep­ing tips that can low­er the pos­si­bil­i­ty of servers crash­ing in the first place. Pre­ven­tion mea­sures include keep­ing the serv­er room iso­lat­ed and cool with air con­di­tion­ing. It should also be clean because dust can cause over­heat­ing.

In-house tech staff may be able to trou­bleshoot a serv­er fail­ure, but more com­plex issues could require out­side help. This means that ade­quate train­ing of tech staffers in how to deal with a failed serv­er in the first instance is a good invest­ment, as is main­tain­ing a work­ing rela­tion­ship with an exter­nal IT spe­cial­ist. 

Of course, if the serv­er is in a remote data cen­tre, the organ­i­sa­tion is at the mer­cy of the good prac­tice of an out­side agency and reliant on their speedy action to get sys­tems back up and run­ning – so choose your provider care­ful­ly.