We put the IT in city®

CitySmart Blog

Wednesday, June 6, 2018
Adrian McWethy, Network Infrastructure Consultant
Adrian McWethy

We know we’ve mentioned the Atlanta ransomware attack many times, but we must mention it again as inspiration for today’s post because of a quote from a recent Atlanta Journal-Constitution article about the city losing years of dashcom footage. Buried toward the bottom of the article was this paragraph:

“Employees have to back up documents,” [Chief Erika Shields] said. “Even if it’s not related to a criminal investigation, if it is of some value to you, you have got to be backing this stuff up. I think it was a painful but useful lesson in IT security for all of us.”

Employees have to back up documents? This is so clearly not a data backup and disaster recovery best practice that it’s startling. Data backup and disaster recovery is not the responsibility of employees. Otherwise, you set yourself up for failure.

However, we are involved in IT every day. We live and breathe data backup and disaster recovery, but many non-technical city staff do not. Therefore, it’s reasonable to assume that their perception and our perception of data backup are different.

Here are a few common assumptions about disaster recovery that are incorrect, dangerous, and risky. Ask yourself if your city makes these assumptions.

It’s not disaster recovery when you rely on employees to back up information.

Let’s start with the Atlanta example. Data backup and disaster recovery is both the responsibility of your city’s decision makers (approving the means) and your IT team or vendor (implementing and managing the solution). Employees are not IT experts. They have jobs and skills that do not include the requirement of ensuring that your data will be recovered in the event of a disaster. They are distracted, busy, untrained, and inexperienced when it comes to IT-related responsibilities.

Ideally, an IT vendor should deploy, configure, manage, monitor, and regularly test your onsite and offsite data backup and disaster recovery solution. Only IT professionals can ensure that you’re properly backing up information and complying with policies and the law—as well as reducing liability while knowing you’re able to recover from a disaster.

It’s not disaster recovery when you manually back up data with external hard drives or flash drives.

Let’s say you’ve given someone the “role” of backing up your data by telling a non-technical employee to back up servers and/or computers with an external device. Then, you might store that external hard drive, flash drive, or other storage device in a vault, an employee’s home, or a room in city hall.

This is dangerous and risky on many, many levels. First, you’re relying on a person to conduct a manual process every day or week who may forget, get sick, or go on vacation. Second, hard drives and flash drives may not capture all the information you need in a recoverable way as your data and systems evolve over time. Often, we find that the dataset captured by storage devices used in manual processes is incomplete or even corrupted—meaning the data is not really backed up. Third, you may be in some dodgy legal and compliance territory by the way you’re handling the backups. And finally, you don’t have a proper offsite data backup component if you’re storing the external devices too near the original data (as seen by the following point).

It’s not disaster recovery when your “offsite” data backup is really just more onsite backup.

We wrote about the issue of the real definition of offsite data backup a few years ago and will summarize some scenarios that do not constitute offsite data backup:

  • Scenario 1: A city stores its data backups “offsite” at the fire station down the street.
  • Scenario 2: A city stores its data backups “offsite” on a flash drive at the mayor’s house.
  • Scenario 3: A city’s IT provider stores its backups at their house.
  • Scenario 4: A city’s data backups are stored at a building about six miles away from City Hall.

All these scenarios are not offsite data backup. Why? They are too close to the original location of the data. (Plus, scenarios 2 and 3 repeat risks from the above point about hard drives and flash drives.) True offsite data backup is geographically distant from your city and completely separate from your onsite data.

It’s not disaster recovery when you don’t test your data backup and disaster recovery solution.

For us, the saddest data backup and disaster recovery scenarios are at cities where they do many things right...but fail to test their solution. After making significant technology investments, a disaster finally happens and they are unable to restore their data. Why? They did not regularly test.

Testing flushes out problems before an actual disaster that may include:

  • The solution not backing up all critical data
  • Corrupted data
  • Problems restoring data, despite its capture during the backup process
  • Time to recovery issues

Without testing, these surprises threaten a successful data restoration—impacting city operations and citizen services. You also risk permanent data loss.

Remember, what you think may be data backup and disaster recovery may not actually fit the definition. Worse, your current solution may fail you when you need it most.

Ready to reassess your data backup and disaster recovery solution? Reach out to us today.