How pen and paper comes to the rescue in an IT crisis

When the CrowdStrike software bug bricked 8.5 million computers around the world on 19 July, some of the first people to notice the effects were air travellers.

Anthony Bosman, an academic at Andrews University in Michigan was trying to board his flight from Michigan to Florida when he realised he couldn’t download a mobile boarding pass to his smartphone.

So he went to check in at the airport, in person, and watched in amazement as an airline employee looked up his name on a paper list and then wrote out his boarding pass – by hand.

“It felt like a blast from the past,” he recalls. “The ticket agent, I remember how she commented that her hand was tired from having to write so many of them.” His flight took off as planned.

Multiple other passengers, including many in India, reported having the same experience that day.

The CrowdStrike bug also hit banks, telecoms firms, health services and online retailers.

This week a senior executive at the firm appeared before a US congressional committee and said he was "deeply sorry" for the chaos caused.

For a brief moment in July, some organisations had to forget about their computer-based processes and do things the old-fashioned way.

If you look through articles about past cyber-attacks and IT failures on the BBC News website, you’ll find countless examples of organisations that have had to “go back to pen and paper” in the face of disruption.

British GPs, staff at foreign exchange firm Travelex, medics at Rouen hospital in France and employees of Lincolnshire County Council have all experienced this.

It sounds an almost pitiful predicament. And yet, while it certainly isn’t desirable, some cyber-experts are now advising companies to plan for switching to paper-based processes in the event of IT failure.

Rather than an ad hoc workaround, pen and paper systems could be something staff practise using from time to time so that they can switch away from their computers seamlessly if required.

One company that knows the value of paper is Norsk Hydro, a Norwegian aluminium and renewable energy firm.

In 2019, hackers targeted Hydro with ransomware that locked staff out of more than 20,000 computers. Bosses at Hydro decided they would not pay a ransom fee to restore access, meaning that 35,000 staff working across 40 countries had to find other ways of doing their jobs, temporarily.

They dug old binders out of basements with instructions on how to produce particular aluminium products, for instance, recalls Halvor Molland, a spokesman for Hydro. At some locations, by sheer chance, staff had printed out order requests just before the cyber-attack hit.

“Their creativity… was tremendous,” says Mr Molland. While computers with customer information and company data were locked out, factory equipment was mercifully unaffected by the ransomware. At some facilities, staff bought computers and printers from local retailers so they could print off information for factory workers. And vintage office kit came in handy. “We actually had to dust off some old telefaxes,” remembers Mr Molland.

Although production fell by up to 50% at certain plants, these workarounds kept the business going. “You need to do what you need to do,” as Mr Molland puts it. Reflecting, he suggests that companies might want to keep printed copies of key information such as internal telephone numbers or checklists so that some work can continue even in the event of a massive cyber-attack.

“People have realised the importance of having these manual methods because of the severity of some of the recent cyber-attacks and IT outages,” says Chris Butler, resilience director at disaster recovery and business continuity firm Databarracks.

He mentions one customer his company works with – an industrial distribution firm – that has put together "disaster recovery packs” and sent them to all of its branches. The packs include paper forms and a fax machine – a contingency in case their digital ordering system becomes unavailable. “If that goes down, their only alternative, they realised, was to have these forms.”

Mr Butler suggests that companies have a training day where employees practise using flipcharts and whiteboards instead of computers, to see if they can still do their jobs effectively that way.

Some organisations recommend using paper for security reasons. Parts of the US court system require certain documents to be filed on either paper, for example, or a secure device such as an encrypted USB drive.

Obviously there are limits to paper-based processes. Mr Butler notes that if bankers, for example, lose access to their trading terminals during an IT incident, they can’t easily switch to paper-based alternatives.

The biggest problem with pen and paper systems is that they don’t scale well, says Gareth Mott, from the Royal United Services Institute. It’s slower than using a computer for many tasks ,and it’s hard or perhaps impossible to coordinate thousands of employees using such methods across multiple office locations.

But practising workarounds really can help, adds Dr Mott. He and colleagues have researched how “war-gaming” and IT failure roleplay exercises can impact employees’ responses to real-life cyber-attacks. “We found that the companies that had done that, sometimes a few weeks before they had a live incident, really benefitted,” he says.

It’s not just pen and paper that could come in handy. Dr Mott is aware of one firm that bought “crates worth of Chromebooks” for staff in the wake of a cyber-incident, so that they could work without needing access to the company network.

Some companies might have dormant WhatsApp or Signal messaging groups that they can ask employees to use for internal communications, if access to the company email servers goes down, for instance.

Both Dr Mott and Mr Butler stress the importance of off-site or otherwise segregated data backups so that, in the event of a ransomware attack, all that vital information is not necessarily lost.

Cathy Miron is chief executive of eSilo, a data backup firm based in Florida. There are hundreds of such companies around the world, including Databarracks, that provide secure data backup services.

Ms Miron’s company offers off-site, cloud-based data storage on a separate network to that of their customers; and on-site, custom-built servers as well. “We have had a 100% ransomware recovery rate thus far,” she says.

For all the sophistication of contemporary computer systems, it’s the simple, improvised workarounds that can save companies when a crisis hits. Mrs Miron mentions one customer who, at the time of writing, was using a Verizon mi-fi, or mobile broadband wireless router, system to access backup data because their main computer network had been completely shut down following a cyber-incident.

“You should expect it, at some point in time, to be a victim of a cyber-attack,” emphasises Mr Molland. “What do you do in the meantime? How do you keep the wheels turning?”