We put the IT in city®

CitySmart Blog

Tuesday, January 30, 2018
Cale Collins, Network Infrastructure Consultant

Cale CollinsOn December 17, 2017, the Hartsfield–Jackson Atlanta International Airport experienced a power outage that lasted for about 11 hours. The outage was disastrous on all levels because:

  • An electrical fire destroyed both the main power system and the backup system that were located right next to each other.
  • The outage lasted far longer than airport security experts said should happen.
  • Passengers had very little idea what was going on most of the time.
  • Airlines lost millions of dollars in revenue (with Delta alone losing up to $50 million).
  • As the world’s busiest airport, flights were massively disrupted around the world.

The shocker? Hartsfield–Jackson Atlanta International Airport did not have a clear plan for a power outage that took out the entire airport.

Your city may not be the world’s busiest airport but you can learn some important lessons about your disaster recovery plan from this actual disaster.

1. Create a true disaster recovery plan that accounts for a complete disaster.

You’re not building a “mild inconvenience recovery plan.” Disaster recovery needs to mean what it says. What happens when a real disaster hits like a massive power outage, a tornado, a flood, or a fire? Then, work backward from there. For example:

  • Who’s here?
  • What are the priorities?
  • How will you get your technology up and running after a disaster?
  • What data will you restore, and in what order?
  • What contingency plans will you create while specific data and information is not accessible?

2. Ensure that you have an offsite data backup component as a part of your disaster recovery plan.

If a disaster strikes, then your backups cannot exist in the same physical location as the information you’re backing up—even if they are right next door. You will need a distant offsite component as a part of your disaster recovery plan to ensure that your information is protected. Ideally, that offsite backup is stored in a geographical location far from your city. During a disaster, your data is safe—and you’re even able to access it while you wait for new equipment to arrive. By having an offsite data backup component, you also make sure you don’t have a single point of failure.

3. Test your disaster recovery plan.

The City of Atlanta and the Hartsfield–Jackson Atlanta International Airport admitted later that they had plans for partial outages but not a plan for a full outage because it was a “one in a million chance.” However, that’s the entire point of a disaster recovery plan. Be prepared for the worst that can happen. If you can handle the absolute worst-case scenario, then you can handle less serious scenarios.

The only way to know that you will be able to handle that worst-case scenario is to test your plan. And yes, test your plan regularly. Is your critical data actually getting backed up? Are you able to recover your data and use it in an operable fashion if a disaster hits? It’s not uncommon to find cities that never test data backups and find out too late that they do not work. By testing on a regular basis, you ensure that your disaster recovery solution works. You are not hoping that it does—you know that it does.

4. Include communication as part of your plan.

While not a technical component, communication is essential and should form part of your disaster recovery plan—both communicating to citizens and communicating internally to your staff. The Hartsfield–Jackson Atlanta International Airport communicated poorly to people and the media—leading to a lot of uncertainty, fear, and conjecture.

In case of a disaster, who will communicate to the public? Who will communicate to city staff? What happens if someone is unable to fulfill their duties? Who takes over? Communicating basic information such as the nature of the problem, how long it will take to get resolved, and what contingency plans are activated in the meantime will help you manage uncertainty. Otherwise, people panicking and barraging you with questions just adds more problems to your plate.

Learning from the Hartsfield–Jackson Atlanta International Airport’s power outage can save you some unnecessary trouble in case a disaster hits your city. Citizens depend upon you to safeguard important information and keep city operations running no matter what happens. They will depend upon 911, public safety, and city hall after a flood, tornado, or other catastrophe. If you plan and test for the worst, then you have the confidence of knowing you will be able to handle any disaster.

Need help with your disaster recovery plan? Reach out to us today.