We put the IT in city®

CitySmart Blog

Tuesday, April 04, 2017
Jabari Massey, Network Infrastructure Consultant

Jabari MasseyOn the surface, a coastal city did some correct things to back up its data. The city had a few servers in a physically secure basement room that were well-maintained by IT staff. One of the servers backed up important data. In case a server failed, the backup server would run until the city could replace the original server.

A long time had passed since the city last experienced a hurricane. When a hurricane finally seemed eminent, the city was ordered to evacuate until the massive storm passed. The city manager and IT staff didn’t think much about the servers other than placing them upon concrete blocks in case of flooding. As long as the city implemented its emergency action plan and evacuated everyone safely, the city manager assumed its information technology would remain safe.

After the hurricane passed, city staff returned to find that no massive devastation occurred but they did experience heavy flooding. The IT staff had placed the servers upon concrete blocks as a precautionary measure, but they learned an incredibly hard lesson in hindsight.

Located in a basement room, the servers sat below sea level. Although the rest of city hall experienced moderate flood damage in places, the basement had filled up to dangerously high levels. All of the servers—including the backup server—were rendered unusable by the flooding.

With a sinking feeling, the city manager realized all critical data—including financial, public safety, document management, email, and website data—was gone. The only backup server got destroyed along with the others. It might be easy for the city manager to point some blame in the direction of the IT staff, but it was well-known that he had refused requests to explore other data backup options because of “budget concerns.”

Now, the mayor, city council, the media, and public would be asking questions.

Preventing This Disaster

Sure, the city manager and IT staff made a bad decision to place servers in a basement room below sea level. But their errors go deeper than this poor choice of physical location for the servers.

Let’s look at the errors in the story above.

Error #1: Locating servers in a flood-prone area of your building.

Getting the most obvious error out of the way, it’s clear that the servers needed to reside on an upper floor. In addition, the server room needed to be in a room that mitigates flood risks through preventative measures such as water leak sensors or eliminating areas where water can enter.

Error #2: Lack of offsite data backup.

While locating the servers on a higher floor may have prevented this immediate flooding disaster, it’s still not a full disaster recovery plan. Anything can happen to your technology onsite. To guarantee full recovery of your data after a disaster, you need an offsite data backup component to your emergency plan.

We recommend storing your data offsite in geographically dispersed locations (such as in data centers both on the East and West coasts). Then, even if the worst disaster wipes out your buildings, you will be able to recover and access your data.

Error #3: Lack of technology planning.

The lack of offsite data backup also signifies a larger issue—a lack of planning. The city had developed an emergency plan and used it in the case of the hurricane. But when was the plan developed? When was it last updated? Did it include technology-related scenarios? What was the plan to protect data in case of a disaster?

First, the city needed to update its emergency plan and include technology. That would have addressed technology-related gaps in the city’s data backup, disaster recovery, and business continuity plans. Second, the city needed regular technology planning meetings (at least once a quarter) and ongoing monitoring to ensure that data backups were tested and working. This regular monitoring and planning would help the city adapt to changes (such as new technology, more staff, building changes. etc.) and ensure that the risk of data loss is minimal.


Flooding is one of the most common disasters. It can happen anywhere in the country and devastate a city. Because citizens will rely on your city after severe flooding, you must be operational as fast as possible. That means having access to your data—your website, your documents, and your applications that are essential to operations.

By developing a disaster recovery plan that includes an offsite data backup component, you will lessen the risk of permanent data loss and angry “Why?” or “How?” questions after the fact from council, the public, and others.

Concerned about your data backup and disaster recovery? Reach out to us today.