On the 27th of June, 2017, a devastating variant of Petya ransomware, nicknamed NotPetya, was used to launch a global cyber attack.

It’s widely believed that the attack was a politically-motivated one against Ukraine, as it occurred on the eve of the Ukrainian holiday, Constitution Day, and Ukrainian organisations were the first to confirm they had been hit.

That same day, virus protection specialist Kaspersky Labs reported infections in the UK, France, Germany, Italy, Poland and the US, although the majority of infections had targeted Russia and Ukraine, where more than 80 companies were initially attacked, including the National Bank of Ukraine.

Regardless of who or what was the primary target, there were numerous casualties across the globe. One notable example was A. P. Moller-Maersk Group, a Danish business conglomerate with activities in the transport, logistics and energy sectors.

On the 28th of June, 2017, the company released a statement confirming it had been one of the cyber attack’s many victims.

The statement read: “IT systems are down across multiple sites and select business units. We have contained the issue and are working on a technical recovery plan with key IT-partners and global cybersecurity agencies. We have shut down a number of systems to help contain the issue.

“We continue to assess and manage the situation to minimize the impact on our operations, customers and partners from the current situation. Business continuity plans are being implemented and prioritized. The aggregate impact on our business is being assessed.”

At this stage, A. P. Moller-Maersk Group couldn’t possibly know that the damage caused by NotPetya would keep its IT infrastructure out of action for a full week, in a cyber attack that would cost it between $250 million and $300 million, and test its business continuity plan to the very limit.

Mærsk Oil

When A. P. Moller-Maersk Group was struck by NotPetya, its subsidiary, Maersk Oil, was producing about 220,000 barrels of oil every day and employed roughly 2,800 people – 688 of whom were based at its Aberdeen office in Scotland.

One such Aberdeen-based employee was John Duncan. Now the senior advisor of emergency response and security at Total Exploration and Production UK, which acquired Maersk Oil in 2018, Duncan was the man responsible for Maersk Oil’s business continuity plan.

“One of the first things I was asked to do when I was taken on at Maersk Oil was to get our business continuity plan up to speed,” Duncan explains.

The plan in place when he was hired was at least five years old and hadn’t been looked at again since its creation. “It was a top priority,” he notes.

Through a series of business impact assessment workshops, the firm began putting an updated and “fairly standard” business continuity plan in place.

“We had each of the departments identify two people – a champion and a deputy,” Duncan explains. “They were the key people who would take the information from the department, meet with me and we would draw up a business continuity plan. That was based on key strategic risks that the strategic team identified as being what we needed to concentrate on.”

These risks included loss of IT, as well as loss of office. “We had 550 people working in our office,” says Duncan. “What happens if we can’t get to the office for whatever reason? Think of the Art School that caught fire in Glasgow. How long have businesses been impacted for there?

“We also look at disease – not just pandemic, but all disease. Loss of key suppliers. What about key staff? Are there key staff, whom, if they were lost, would impact on the company? Environmental incidents. We have emergency response vessels that have to be beside oil rigs. What happens if they go on strike or we lose them for whatever reason?

“Helicopters – they can be grounded, be it by an ash cloud or technical issues. How would we get 50,000 oil people offshore in the North Sea? That’s a major impact. You have to think about it.

“We’re pretty good in the oil & gas industry at emergency response, because you’re always going back to Piper Alpha, helicopter crashes, major oil spills. We exercise and deal with real incidents yearly. So we’re really good at emergency response. We were crap at business continuity.”

Maersk Oil’s emergency response consisted of its crisis management team, its emergency response team and, at the front end, would be the offshore installation managers (OIMs) fighting the fires and keeping people safe.

“From a business continuity point of view, what we were looking at was having the same structure,” Duncan explains. “We’d have our crisis management team to have that strategic direction. We’d then have the business recovery team who are the incident management team. That has the champion from each department within that team. And we’d have the computer incident response team, which is our IT team fighting the fires.”

Creating that worked “quite well”, according to Duncan. And, with the plan in place, the company began conducting exercises, running workshops and holding team talks. “Everyone understood the plan once we put it in place, which was key,” Duncan adds.

On top of this, the company also owned a disaster recovery site. “We spent thousands and thousands of pounds on that site, which has independent power. And we had space for 50 workers there, and all of our IT could be piped into the building for us to work as normal, which was good.”

The attack

Maersk Oil was just a small part of a huge group –  A. P. Moller-Maersk Group. “There is Maersk shipping, Maersk containers, Maersk everything,” Duncan explains. “It’s a huge, global company. Maersk Oil was just a small part of it.”

Once every three years, oil companies have to prove to the industry regulator that they can deal with a major oil spill by successfully completing a Secretary of State’s Representative Maritime Salvage & Intervention (UK) (SOSREP) exercise.

“The regulator has quite substantial powers if you fail that,” Duncan says. “If we fail that we can be shut down and not be able to produce oil. Maersk Oil was probably producing $1.2 million of oil per day. That’s a huge impact if you don’t pass this exercise.”

On the 27th of June, 2017, Duncan was spearheading a SOSREP exercise.

Then, at lunchtime, this happened:

NotPetya had struck at another A. P. Moller-Maersk Group asset. “It wasn’t Maersk Oil at all,” he explains. “But that came on and, all of a sudden, a little bit of panic occurred.”

All workers at Maersk Oil and, indeed, the entire A. P. Moller-Maersk Group were ordered to hastily switch off their computers. They would not be able to reboot for a week.

Duncan says: “Okay, what do you do? You’re sitting there for a company and you’ve lost your computer for seven days. What are you going to do with your staff? How are you going to work?”

The response

With the attack taking place during the company’s SOSREP exercise, it just so happened that Duncan was already in a briefing with the crisis management team, and he immediately advised senior management to activate the business continuity plan.

“I suggested that the health, safety and environment director go and get everyone and that he should Chair the committee because he was the ideal person to do so,” he says. “Within 30 minutes we had our business recovery team up and running, beginning to work out what the issues were.

“We’d exercised the plan and we knew what we were doing. We got all the department people together, so that was great. We needed to communicate with our staff. We didn’t have any comms now because everything was switched off – the computers are switched off, the network is switched off, our mobile phones are switched off. How do we communicate with our staff?”

Maersk Oil didn’t actually make use of its disaster recovery site. Instead, it turned to a third-party provider for help – its key suppliers. “They were big companies and they offered us space, which was great,” Duncan says.

“Support from contracting companies was really, really crucial. We had some key actions we needed to take as an oil company and, by them allowing us to go to their premises, they basically allowed us to log on to their computers to access the internet, and that was a big boon.

“I’m glad we didn’t outsource IT because that would have caused us a major issue. We had a very experienced IT team who were based in Aberdeen, as well as in Copenhagen where the headquarters was.

“The IT staff were great. They were working more than 12 hours per day to try to resolve this and get us up and running. I can’t praise them enough.”

Duncan also has praise for senior management. “The leadership team understood us,” he insists. “Because of the setup, because of the exercises, they understood what the process was so they were fine with the business recovery team being in place and making decisions. There was a lot of trust from the leadership team, and communication was key for that.”

The company also had some critical payments to take care of that week – two invoices totalling £28 million needed to be paid on the Wednesday. “We couldn’t do it,” Duncan says. “And those key payments could have had a major impact on a key project we were working on. It was a major, national infrastructure project. One of our key actions was to try to pay this before we got cut off.”

Ultimately, communication is key to absolutely everything the company needed to do in this time of crisis. The business recovery team met twice per day – in the morning and the afternoon.

They took information from the departments, discovering the key functions each of them had to perform on a daily basis and fed that information to the business continuity team. Priorities for the day were then discussed and debated before the group’s Chair decided what the priorities for the company were.

“Town halls also went down really well,” Duncan recalls. “The fact that we were updating staff all the time. How many times do hear of things going on in your company and nobody tells you about it? We made sure that we had town halls, which means you get all your staff together in a big room with the managing director and a key member of the IT department who could speak in English.

“This is particularly helpful because when they speak about IT it can sometimes go right over people’s heads. We took it down to the basics so everyone could understand what the impact was. We got really good feedback on the town halls,” he adds.

In-house comms also went back to basics. It had to. With no internet and no phones, how did the management team put its messaging across to staff? Flip charts.

“Entrance and exit doors,” says Duncan. “The key messages were put up and they were continually updated by the internal comms staff. That certainly worked.”

The crisis management team and business recovery team comms were generally excellent, according to Duncan. Having a director in place as the Chair of the business recovery team meant that, after every meeting, he would go upstairs to the crisis management team, brief the other directors, then return with any strategic direction.

“First thing in the morning, we’d have a conference call between myself, the business recovery team director and IT, with headquarters,” Duncan says. “We got information on what they were doing. Maersk Oil had offices in Norway, Denmark, Kazakhstan and Aberdeen, so they had to prioritise how they were going to get everyone back up and running. We were able to get, straight from the horse’s mouth, exactly what was happening in the corporate world, so we could then brief our business recovery team and our crisis management team. That went quite well.”

Duncan also recommends a touch of the old school when it comes to business continuity plans. “How many people have got rid of faxes? Bring them back! Faxes were key for us. Now, remember, fax machines should sit on an independent landline and not be part of your main network.

“When we’re offshore and we need to lift anything there has to be a plan. And we can be doing lifting plans on a daily basis. Without that plan, you can’t do anything. That could have a major impact on us producing oil. The lifting plans were drawn out and they were faxed to the office in Aberdeen where our lifting expert looked at it and approved the plans. Faxes were very beneficial.

“We just had one problem with a fax machine at one of our offshore assets. Because we hardly used the fax all the ink had dried up and it was useless. My recommendation, if you do have a fax machine and don’t use it, is just keep it in the box so that it will work whenever you do come to use it. That’s something we learned.”

While the company’s business continuity was tested to its very limit, so was the patience of the workforce.

“In the first two days, you can only clean your desk so many times,” says Duncan. “I think I did it three times. You can only read so many documents. This was seven days worth.

“And there was inconsistency. Staff at headquarters were being allowed to go home. Information we were getting in Aberdeen was that we all had to stay at work so that caused a little bit of impatience from the workforce.

“Staff in our Aberdeen office were coming up with ideas such as ‘why don’t we go down to the beach and do a beach clean up?’ Why don’t we offer our services to a charity?’ Anything, just to get away from the office for a little time because they couldn’t do anything. We all rely on our computers. If you lost your computer for seven days how would you work?”

When the IT team finally felt it had resolved the issue and it was agreed that employees’ laptops could be turned on again, the computers could only be rebooted under the supervision of an IT specialist.

“In the background in Copenhagen and in Aberdeen we had Microsoft working with us to provide patches, providing us with software updates to make sure we didn’t get hit,” Duncan says. “There was an awful lot of work going on in the background but we were still a little bit cagey about switching on the computers. So you would have an IT person standing behind you as you turned the laptop on just to make sure it went through the boot-up process and the anti-virus software kicked in to ensure we didn’t get that virus.”

And slowly but surely, one by one, Maersk Oil’s 2,800 employees were brought back online. Duncan held debriefs at the end of that Monday, reviewed the business continuity plan and has since updated it. Maersk Oil has examined its strategic risks and now performs this review annually to ensure its procedures are fit for purpose.   

What could be improved?

1. Communications

“We were trying to tell everyone to shut their computers down. 550 people and we didn’t have a tannoy. How do you tell 550 people to shut down their computers immediately? Not log off. Shut down everything immediately. We were absolutely terrified that we’d get these computers locked so we wanted them all shut down. There was an inconsistency in the messenging. People thought it was a joke.

“They were still working on their computers and IT staff were running around. I had actually been thinking that we would need a tannoy for security reasons anyway, but I was looking at getting a tannoy system that would work through the phones, but that wouldn’t have helped either. If you’re going to get a tannoy, make sure it’s standalone.”

2. ‘Business as usual’ message

“The corporate message was ‘business as usual’ but it wasn’t bloody wasn’t business as usual. We didn’t have any computers! We couldn’t do what we usually did. We needed clear direction from senior management. That was certainly missing from a communications point of view. What did they expect of us?”

3. Contact lists

“We kept telling staff about having contact lists and it was great they all had contact lists, but guess where they were…on the computer. This made it difficult for me to contact all the staff who were not in the office. Some people had paper contact lists but they didn’t include everyone.”

4. Payments

“Our finance team have worked together to ensure that, if we do lose our IT systems in future, we will always have visibility on the really key payments we needed to make that week. So we’ll always be well informed as to exactly what we have to do.”

5. Expectation to return to normality quickly

“Corporate were a pain in the rear end. Honestly, they didn’t realise the pressure the staff were under. Take the finance department, for example; corporate were telling them to continue to get all the data together so they could update systems whenever they came back on but that was going to take a long time.

“There was a lack of consistent messaging from headquarters, which didn’t help. As I say, they let people go home and we kept people at work. We could have reduced staff but it was a strategic decision. They were pressing for the deadlines. There was no flexibility shown.

“The finance staff were kind of getting fed up with the pressure when they were still trying to pay bills and keep everyone happy.”

6. Non-network laptops and data storage

“All our data is stored on-network. All our procedures, all our processes and we access that either through a laptop or a fixed computer on a desk. We couldn’t use any of them at all. One of the things we looked at was getting a standalone laptop, non-networked, and a USB stick with key documents, contact details and, if you need to activate it, you just plug it in and away you go.

“We keep that laptop up-to-date but it cannot be networked. Some people just keep a laptop in a box and put it away just in case and some people actively use them and keep them updated. But the key is to keep them up-to-date because we’ve not had an attack in two years. We need to make sure all the data is updated much more regularly than that.”

7. Maintain the BRT beyond the initial re-start

“One thing I learned from the business recovery team when we got up and running on the following Monday – I stood the team down. I think that was a mistake. I should have kept it going right until the end of the week, even just for a two or three-minute meeting to check everything was going okay, just to make sure.”