We put the IT in city®

CitySmart Blog

Tuesday, April 4, 2017
Jabari Massey, Network Infrastructure Consultant
Jabari Massey

On the surface, a coastal city did some correct things to back up its data. The city had a few servers in a physically secure basement room that were well-maintained by IT staff. One of the servers backed up important data. In case a server failed, the backup server would run until the city could replace the original server.

A long time had passed since the city last experienced a hurricane. When a hurricane finally seemed eminent, the city was ordered to evacuate until the massive storm passed. The city manager and IT staff didn’t think much about the servers other than placing them upon concrete blocks in case of flooding. As long as the city implemented its emergency action plan and evacuated everyone safely, the city manager assumed its information technology would remain safe.

After the hurricane passed, city staff returned to find that no massive devastation occurred but they did experience heavy flooding. The IT staff had placed the servers upon concrete blocks as a precautionary measure, but they learned an incredibly hard lesson in hindsight.

Located in a basement room, the servers sat below sea level. Although the rest of city hall experienced moderate flood damage in places, the basement had filled up to dangerously high levels. All of the servers—including the backup server—were rendered unusable by the flooding.

With a sinking feeling, the city manager realized all critical data—including financial, public safety, document management, email, and website data—was gone. The only backup server got destroyed along with the others. It might be easy for the city manager to point some blame in the direction of the IT staff, but it was well-known that he had refused requests to explore other data backup options because of “budget concerns.”

Now, the mayor, city council, the media, and public would be asking questions.

Preventing This Disaster

Sure, the city manager and IT staff made a bad decision to place servers in a basement room below sea level. But their errors go deeper than this poor choice of physical location for the servers.

Let’s look at the errors in the story above.

Error #1: Locating servers in a flood-prone area of your building.

Getting the most obvious error out of the way, it’s clear that the servers needed to reside on an upper floor. In addition, the server room needed to be in a room that mitigates flood risks through preventative measures such as water leak sensors or eliminating areas where water can enter.

Error #2: Lack of offsite data backup.

While locating the servers on a higher floor may have prevented this immediate flooding disaster, it’s still not a full disaster recovery plan. Anything can happen to your technology onsite. To guarantee full recovery of your data after a disaster, you need an offsite data backup component to your emergency plan.

We recommend storing your data offsite in geographically dispersed locations (such as in data centers both on the East and West coasts). Then, even if the worst disaster wipes out your buildings, you will be able to recover and access your data.

Error #3: Lack of technology planning.

The lack of offsite data backup also signifies a larger issue—a lack of planning. The city had developed an emergency plan and used it in the case of the hurricane. But when was the plan developed? When was it last updated? Did it include technology-related scenarios? What was the plan to protect data in case of a disaster?

First, the city needed to update its emergency plan and include technology. That would have addressed technology-related gaps in the city’s data backup, disaster recovery, and business continuity plans. Second, the city needed regular technology planning meetings (at least once a quarter) and ongoing monitoring to ensure that data backups were tested and working. This regular monitoring and planning would help the city adapt to changes (such as new technology, more staff, building changes. etc.) and ensure that the risk of data loss is minimal.

Flooding is one of the most common disasters. It can happen anywhere in the country and devastate a city. Because citizens will rely on your city after severe flooding, you must be operational as fast as possible. That means having access to your data—your website, your documents, and your applications that are essential to operations.

By developing a disaster recovery plan that includes an offsite data backup component, you will lessen the risk of permanent data loss and angry “Why?” or “How?” questions after the fact from council, the public, and others.

Concerned about your data backup and disaster recovery? Reach out to us today.

Tuesday, March 28, 2017
Ryan Warrick, Network Infrastructure Consultant
Ryan Warrick

In recent posts, we’ve talked about disasters at cities that result in permanent data loss, incredible damage to city operations, and city department heads wondering if their job is now at risk—all sadly because of preventable risk. The stories we use to illustrate these disasters—and the lessons learned—are based on a combination of many, many scenarios we’ve witnessed at cities throughout the years.

However, we recently saw a story that’s quite specific to one city and a very public, front page news illustration of some important IT-related lessons. Let’s look at what happened to the City of Miami Beach, Florida in December 2016.

Third Parties Steal $3.6 Million—and No One Notices for Six Months

In a nutshell, unknown third parties stole the account and routing numbers from the city’s banking account. According to the Miami Herald, the criminals “[rerouted] automatic payments intended to pay vendors and other government bills.” The criminals did it for six months and stole $3.6 million before staff in the finance department noticed.

We carefully reviewed the Miami Herald article and the city manager’s report. While this crime is a form of cybersecurity, the situation also includes lessons about IT-related processes and controls that are incredibly important to cities. A few bad practices stick out from our analysis of the report that cities need to avoid.

1. Completely ignoring basic, elementary best practices.

The city of Miami Beach was offered free fraud control tools when they set up the account in 2012—the same kind of fraud control tools that many individual banking customers enjoy. Did the city take advantage of these tools? No. Maybe they had a reason at the time such as wanting to implement their own fraud controls. If so, that never happened.

Cities need to stay aware of and implement important best practices that help mitigate information security risks. In this case, both finance and IT staff needed to say “yes” to such an obvious best practice back in 2012.

2. Using easy-to-steal information as authentication for financial transactions.

Think about how many people in a city can take a quick peek at a check. If third parties could steal city money through only this information, then the city had a security vulnerability that was wide open for people to exploit.

We find that cities also have similar weaknesses in areas such as passwords, unencrypted wireless devices, and website hosting that makes it easy for hackers to exploit security vulnerabilities.

3. Apparent lack of financial data oversight.

In a recent post about data processing, we said, “Experienced IT professionals should monitor everything related to your data processing such as transactions and processing, errors and incorrect information, overrides, unauthorized use of the application (especially when it appears that someone is altering data or ignoring/tampering with processes), reconciliations, and application performance (such as after a power outage or server failure).”

Obviously, finance department staff have an even more important role in monitoring this information too. While online banking is great, it’s unwise for even an individual consumer to not regularly review banking transactions. Great risk was introduced by not reviewing for six months and hoping that everything was okay. Cities need to become more proactive at monitoring and reviewing important aspects of their operations where data changes constantly—from accounts payable to information technology.

4. Lack of modernization.

Many cities often hear the word “modernize” and think of it as “unnecessarily wasting money or time on something new and fancy that we don’t need.” Sure, some solutions might fit that definition. But technology modernization is important especially when your old technologies and processes lead to security vulnerabilities, inefficient operations, and significant liability.

In the case of Miami Beach, the city manager’s report includes many “sudden” modernizations in one fell swoop such as ACH fraud controls and using UPIC (Universal Promotional Identification Code) to avoid sharing confidential banking information. The city manager even notes in the report that “the ACH Fraud Control program already prevented an unauthorized ACH transfer.”

I know we beat this drum a lot. But why do cities wait? Why do cities put off modernizing their technology and processes until a massive crisis hits? We see this “putting off” logic holds true at many cities for data backup, disaster recovery, website hosting, records and document management, email, and aging hardware. In all of these cases, lack of modernization increases the risk of a significant city incident or disaster.

Learn from cities like Miami Beach. Are you sure that fraudsters aren’t currently stealing money from you? Is your technology modernized in such a way that you aren’t headed for a major disaster like permanent data loss?

If you are worried about addressing critical technology aspects of your city before a disaster happens, reach out to us today.

Tuesday, March 14, 2017
John Miller, Senior Consultant
John Miller

A small city with two servers also stored many paper documents containing critical information. The city backed up its servers with tape-based data backup which a city employee would take home every week or so to store “offsite” at their house. Many of the paper documents were not replicated electronically, and so these paper documents were the only versions in existence.

One night, a fire began that destroyed nearly all the building before firefighters arrived at the blaze. Fire alarms went off but no fire suppression occurred until the fire department showed up.

Assessing the damage the next morning, the city discovered that its paper documents and servers were completely destroyed. With the paper a total loss, the city decided to recover the server data from the tape backups. However, after a two-day attempt at trying to restore the data, the city could only retrieve about 10% of it. Many of the tape backups hadn’t been tested and the city didn’t realize that the backups weren’t running properly for a long time.

As a result, operations ground to a halt and the city found itself in dire trouble. They lost their accounting and billing systems along with many public records and documents. So many critical operational records were lost related to accounting, taxpayers, and businesses. The public outcry had only yet to begin after the admission of data lost—and why the city had not properly backed that data up.

Preventing This Disaster

A fire can happen to any city at any time. Is your city prepared? For such a common disaster, we find that many cities do not have disaster recovery plans that account for a simple yet deadly fire.

Let’s look at the errors in the story above.

Error #1: Using paper as the only copy of important documents.

In today’s electronic information age, relying only on paper for important documents is way too risky. A simple fire can wipe out paper in a matter of minutes. Paper also fails in a flood, tornado, or other natural disaster. Any paper-based documents that are critical to your city need to be scanned electronically and backed up offsite to ensure they are not lost.

Error #2: Poor offsite data backup plan in place.

Relying on a city employee to take tapes offsite every week to their house is not a sure-fire plan. First, these tapes were not tested on a regular basis. When the city actually needed to restore data, most of the tapes failed. Second, too many security and liability risks exist when a city relies on an employee to manually collect backup tapes and store them in a private home. What happens if the employee is negligent or disgruntled? What if they forget one week to take the backups home?

Error #3: Lack of appropriate fire suppression for a server room.

Any room that stores servers needs best-of-breed fire suppression. Fire alarms alone are inadequate. Most data centers feature fire suppression technology that helps eliminate or reduce the severity of a fire. If your city decides to host its own servers, then you need to explore fire suppression options beyond an alarm.

Error #4: Lack of an overall disaster recovery plan.

The city clearly did not think through the consequences of a disaster. Otherwise, it would have identified critical information—such as its paper documents—and planned for a worst-case scenario such as a fire. This plan would include:

  • Identifying which data is most critical and cannot be lost.
  • Estimating the maximum amount of acceptable downtime before restoring city operations.
  • Detailing how the city will get up and running after a disaster.
  • Outlining contingency plans while the data is being restored.
  • Ensuring that any data backups are tested regularly.

While large disasters like tornados can seem more improbable and less likely, cities need to keep in mind that disasters also include more common scenarios like fires. A fire can wipe out critical information quickly. Your disaster recovery plan needs to account for both paper-based and electronic information—ensuring that you can recover your most critical information soon after a fire or other common disaster.

Questions about your city’s ability to protect and recover your most important information after a fire? Reach out to us today.

Wednesday, March 8, 2017
Dave Mims, CEO

Dave MimsWe know. It’s the federal government. Yet, cybersecurity legislative trends show that security risks within government—whether it’s federal, state, or local—are being addressed because they affect national security and the privacy of citizens. There’s an incentive for Congress to help your city shore up its cybersecurity.

The federal bill is called the State Cyber Resiliency Act and it’s in the proposal stage. As a bipartisan bill, it has a higher chance of making it through the House and Senate depending on Congressional priorities. Matt Zone, President of the National League of Cities is quoted as saying:

“Cities manage substantial amounts of sensitive data, including data on vital infrastructure and public safety systems. It should come as no surprise that cities are increasingly targets for cyberattacks from sophisticated hackers. Cities need federal support to provide local governments with the tools and resources needed to protect their citizens and serve them best."

The idea is that FEMA will administer grants for state, local, and tribal governments. Particulars about the grants are not clear at the moment as the text of the bill has not yet been submitted.

We’ve been concerned about city cybersecurity for a long time, and it’s reassuring to us that lawmakers want to help cities address this issue. An article from FCW points out some drivers behind this bill:

  • “[State, local, and tribal governments] typically devote less than two percent of their IT budget to cybersecurity.”
  • “…in 2015, 50 percent of state and local governments had six or more cyber breaches within the last two years.”

We’ll be tracking this bill (S.516) after its introduction last week. Stay tuned!

Wednesday, March 8, 2017
Dave Mims, CEO

Dave MimsSB 138, introduced in the Arkansas State Legislature on January 17, 2017, was passed in the Arkansas Senate on March 6 and now proceeds to the House. Why is SB 138 so important? And why are we, a municipal-focused technology company, pointing it out?

The bill states that an Arkansas municipal charter can get revoked (yes, revoked!) if the Legislative Joint Auditing Committee finds two incidents of non-compliance with accounting procedures in a three-year period. Revoking a charter is serious, rare, and extreme. That’s pretty much the end of your municipality.

The Arkansas Legislative Audit (ALA) includes both general controls and application controls around information systems. For municipalities, accounting systems are often the most important information system they oversee.

According to the ALA:

  • General Controls are mechanisms established to provide reasonable assurance that the information technology in use by an entity operates as intended to produce properly authorized, reliable data and that the entity is in compliance with applicable laws and regulations.”
  • Application Controls relate to the transactions and data for each computer-based automation system; they are, therefore, specific to each application. Application controls are designed to ensure the completeness and accuracy of accounting records and the validity of entries made.”

While this bill has yet to pass the Arkansas House and get signed into law, its appearance and passage by the Arkansas Senate is a sign that municipalities are being held more—not less—accountable for information security, compliance, and best practices related to information technology.

Even if you’re not an Arkansas municipality, you should still get ahead of the curve. Federal and state laws that urge stronger technology-related compliance and best practices seem inevitable.

In the meantime, you can track the Arkansas bill and read up on the different components of what the ALA examines in its audit.

Concerned about the state of your information security or compliance with the law? Reach out to us today.

Tuesday, February 28, 2017
Brian Ocfemia, Technical Account Manager
Brian Ocfemia

A city had relied on an old, aging email server for 10 years. Purchased in 2007, the email server often froze up and hit storage limits constantly. With the excuse of “budget,” the city did not want to invest in a new server despite these issues.

As a result, employees were often forced to delete emails in order to free up space. A city policy said the employees needed to keep “important” emails. However, it was unclear what “important” meant and the policy only loosely defined how the employees should retain them. Some employees used flash drives, some used external hard drives, and some even transferred files onto personal laptops.

One day, an outside investigation began that concerned a city department. Allegedly, funds may have been stolen and investigators wanted to get to the bottom of what happened. Suddenly, all eyes were on the city as word got out to the media.

The media made several FOIA requests to see emails related to the city department under investigation. Once the city clerk began trying to carry out the requests, she hit a wall. Not sure who kept what, she began to fear that key emails were deleted. Sending out requests to city employees in that department, the city clerk received uncertain replies about who had the specific emails.

Within days, she realized the city may not have been able to fulfill the FOIA request—even with a delay. The crushing realization settled in that emails the city was required to keep by law may have disappeared. Once the media suspected this happened, they began reporting on the city in a negative light—casting suspicion over the city in the local paper. The stories spread to various other papers around the state. Investigators also noted the serious nature of these missing emails and began to talk of misdemeanors, fines, penalties, lawsuits, and even possible prosecution for employees who possibly destroyed records.

Preventing This Disaster

Even for FOIA-related circumstances less serious than this situation, cities can feel painful repercussions when retrieving emails that are public records. Delays, excessive hours consumed searching for emails, storage limitations, and uncertainty about locating emails all increase your risk of liability. Let’s look at some errors in our story that the city committed.

Error #1: Relying on an old, aging email server.

The city thought it maximized its original email server investment. But holding onto an aging server presents too many problems that impact the accessibility and security of the information you store on it.

  • Cost: It’s expensive to maintain the hardware and software on a server that breaks down a lot, fails to operate at full capacity, and often isn’t supported by the hardware and software vendors any longer.
  • Threat of Server Failure: Whether you have data backup or not, a server failure is disruptive to your operations. Eventually, you will have to buy a new (unbudgeted) server if it fails.
  • Risk of a Data Breach: Older servers are less secure because vendors often stop providing security patches and updates after a specific period of time.

Error #2: Ignoring email storage limits.

Hitting email storage limits is no excuse for not following state retention laws. Today, many cloud email options exist that provide more than enough email storage space for an affordable price. Employees should never have to worry about deleting important emails or storing them in a separate location just because of email storage caps.

Error #3: Relying on employees to manually archive and retain emails.

This city lacked policies and procedures to ensure proper records retention—and they passed along their lack of problem solving to employees. It’s not a good idea to rely on employees to manually store emails in a consistent, legal way. Most employees have the best intentions—but they get busy, forgetful, or overwhelmed by their roles and responsibilities. They are not necessarily going to retain those emails in the most secure, consistent way.

Error #4: Following a weakly enforced policy not aligned with records retentions laws.

State records retentions laws specifically note how emails (and other public records) must be archived, retained, accessed, and deleted. Modern email servers can automate much of this process to align with laws. This city clearly needed to leverage technology more to help them automate the records retention process. Too many steps were reliant on manual, uncertain processes.

While it’s less likely that a scandal or investigation will happen at your city, it’s not impossible. On whatever level you respond to FOIA requests, it’s your legal duty to provide the information requested. If you can’t, then you’re asking for trouble.

Questions about your ability to respond to a FOIA request? Reach out to us today.

Tuesday, February 21, 2017
Nathan Eisner, COO
Nathan Eisner

When is offsite data backup not offsite data backup? The following story offers an example—and a warning—to cities.

A city was already backing up its data onsite using an extra server. If the server failed at city hall, the other one would take over to restore the city’s data. However, some department heads urged the city to also consider an offsite data backup plan in case of a major disaster. The city manager researched some options and brought in a few IT experts to talk about possible solutions.

After some outside IT experts reinforced and reiterated the idea of creating both an onsite and offsite data backup plan, the city took a shortcut. The city manager didn’t like the idea of sending data off to a data center. He viewed it as unnecessarily expensive. Plus, he wanted control—to “see” the data when he wished. And so the city nixed the idea of offsite data backup located far away from the city.

As a result, the city worked around these parameters to build an “offsite” data backup plan. Working with their local IT vendor, the city set up a backup server in a building they owned located just down the block from city hall. The city manager argued that this building was separate from the city hall building and, thus, “offsite.” If something destroyed city hall, this server would contain all their data. Problem solved.

Or was it?

One day, a huge EF3 tornado descended upon the city. With winds upward of 150 miles per hour, the tornado destroyed many buildings in a swath of downtown. As the city assessed the damage, they discovered that the tornado destroyed not only city hall but also all buildings on that block—including the “offsite” building that stored the city’s backed up data.

With its data permanently lost, the city found itself at a crippling disadvantage at the very moment when citizens needed city hall and public safety operating at full capacity as soon as possible after the disaster. And even beyond the disaster, the city would have to deal with permanent data loss affecting its operations for a long, long time.

Preventing This Disaster

Does this scenario seem unlikely? That’s what all cities, businesses, organizations, and people often think...until after the disaster strikes. With increasing numbers of tornadoes each year in the United States that grow bigger and more devastating, it’s not unlikely that your city may face this threat—or any other similar threat.

Let’s look at the errors in our story and how your city can avoid them.

Error #1: The city’s definition of “offsite” is not really offsite.

Offsite does not mean down the block. It does not even mean two blocks away. True offsite data backup means many many miles away. When your data is stored in a geographic location far away from your city, it’s likelier to be protected from a localized disaster such as a tornado.

We often recommend that you send offsite data to at least two data centers (for example, one on the East Coast and one on the West Coast). It takes some time to set up the technology and the automated data transference to these data centers. But once set up, the offsite data backup runs without the city having to do much of anything. And if a city block is destroyed, your data is safe and accessible from multiple data centers. Your city can start operating within hours of the disaster while you are in the process of ordering new servers.

Error #2: An improper risk assessment focused too much on cost instead of the cost of a disaster.

Sure, it might be cheaper to set up another server in a building down the block. It’s also cheaper to buy health insurance with high deductibles that don’t cover serious medical conditions. In each case, the costs are astronomical when a disaster hits. Cheaper isn’t better and it’s a poor tool to judge a data backup solution’s ability to mitigate risk.

What’s the cost of losing your data? How will your community be impacted if all city records are lost? That’s the cost you should assess. From there, you can make a better case for investing in a disaster recovery solution that mitigates risks by storing data in a geographical location far from your city.

Error #3: A need to “see” the data and keep it close.

An ability to “see” and be near where your data is stored doesn’t mean it’s more secure. A server inside your city can lack the most basic security protection and be more open to hackers than your offsite data backup locked down with the highest security standards in a data center far away. Focus on security and an ability to recover from a disaster, not proximity to your data.

Error #4: A lack of a disaster recovery plan.

Clearly, this city did not think through the consequences of a disaster. They didn’t think through scenarios such as a tornado that can affect a wide area. Not prepared for a probable worst-case scenario, the city found itself completely without its data or a plan if it lost its data. Instead, it assumed that a disaster destroying both buildings was so unlikely that they didn’t have to worry.

For cities, a disaster recovery plan needs to include proper offsite data backup. We recommend that any offsite data backup plan considers:

  • A minimum of daily backups sent offsite.
  • Sending those backups to a data center in a distant geographic location.
  • A minimum of quarterly testing to ensure that your data backups are working.

Questions about your offsite data backup and disaster recovery plan? Reach out to us today.

Tuesday, February 14, 2017
Mike Smith, Network Infrastructure Consultant
Mike Smith

A city wanted wireless access for guests and employees. Easy, right? The city manager told a trusted non-technical employee to “make it happen.” Going to the nearest popular retail electronics store, the employee picked up a wireless router that seemed to do the trick. The wireless router box said it covers 12 devices, so the employee picked up two routers to cover the city’s 20 computers.

Back at city hall, the employee tinkered around until they set up both wireless routers—one on the first floor and another on the second floor. Following the instructions to set it up, the employee got it working. People could now hop on a wireless network with their laptops, smartphones, and tablets.

For a few weeks, employees enjoyed the perks of wireless. So easy! They didn’t even need their on-call IT vendor to help set it up. City council loved the internet access at meetings. Employees could now access their desktop and documents while meeting in a conference room. Guests could now access the internet. How wonderful.

One day, a representative from the state’s bureau of investigation informed the city of a data breach. An unknown person hacked into the city’s server using a stolen password and collected sensitive information about taxpayers. That information appeared on an online black market for sale. Not only must the city now inform taxpayers that they are at risk for identity theft but the city may also need to pay for identity theft protection services for hundreds of taxpayers.

This event hit the city administration like a bolt of lightning. They thought through the repercussions. Loss of citizen trust. Bad media exposure. Money lost. What caused the data breach? When they performed an IT audit to figure out what happened, the answer became obvious.

The city’s unsecured wireless router—the one their trusted employee set up “so easily.”

Preventing This Disaster

A recent study from Kaspersky Lab confirms that this situation is all too common. They estimate that about one in four Wi-Fi hotspots lack even the most basic security. We find that cities often don’t realize the gaping security holes their wireless routers pose.

Let’s look at the errors committed in our story.

Error #1: Buying a consumer-grade wireless router.

A city is not someone’s house. It’s a government entity that conducts important business, serves citizens, and carries out the law. You need business-class equipment that includes enterprise-level wireless routers. These kinds of routers are better equipped to handle the demands and complexity of your city. They will provide better coverage, security, and scalability as your city grows.

Error #2: Tasking a non-technical employee to configure the router.

No matter what the back of the box claims on the consumer-grade wireless router, you need an IT professional to configure this equipment. Just setting it up out of the box is not good enough and you risk leaving open gaping security holes. Configuration involves a complex array of settings that only IT professionals thoroughly understand. They will make sure your wireless router is set up securely (such as making sure you encrypt information) and restricts who can access your wireless network (such as from a “guest” network).

For example, we see too many instances of a Wi-Fi hotspot secured with a default administration password (such as “admin”). With such a weak password, even an amateur hacker can access your most sensitive city information.

Error #3: No ongoing monitoring and maintenance of the wireless router.

In our story, the city doesn’t use proactive IT support. If they depend on reactive IT support, then security breaches could take place and the city wouldn’t know for weeks or months. With proactive support, IT professionals will monitor your network environment and make sure it’s patched, secure, upgraded, and healthy.

Are your city’s wireless routers secured? They are one of the most common hacker targets because 25% of hotspots have pretty much zero security. Unfortunately, that 25% applies to cities.

If you haven’t assessed and addressed your wireless security, then it’s just a matter of time before you’re hit with a data breach. Deal with this problem as soon as possible.

Need help assessing your wireless security? Reach out to us today.

Tuesday, February 7, 2017
Brandon Bell, Network Infrastructure Consultant
Brandon Bell

Imagine a small city with a small public safety department. Budgets are always tight and so they have used the same server they purchased back in 2003. Plus, both the police chief and the one-person IT vendor who they call on an hourly as-needed basis know this server well. They are used to it like the feeling a person gets when they sit in their favorite comfy chair.

However, extended support from the hardware vendor ended years ago. That means the operating system no longer gets security patches and bug fixes on a regular basis. The as-needed IT person checks the server every now and then for issues and makes sure nothing really bad happens to it.

Unfortunately, that became a harder job as time went on. Even in good times, the police officers all complained how their computers (which access the server) are so slow. The server froze a lot and the police chief often reset it. When the problems got really bad, they called the IT person who would inevitably fiddle around with the server until it started working again. The billable hours for this IT person kept increasing month by month, but the police chief thought, “It’s probably still cheaper than getting a new server.”

One day, the server just...stopped working. The police chief called the IT person and assumed the usual fiddling would get it back up. Well, the IT person fiddled...and fiddled...and fiddled. Nothing. The server became as useless as a stone.

“Not to worry,” said the police chief. “We back up to an external hard drive every day. Or at least mostly every day.” The IT person tried to recover the server’s data but found that the files were incomplete and some were corrupted. The backup wouldn’t restore.

As the IT person told the police chief that the data was lost, for good, a sinking feeling entered his stomach. Now, his job—and the public’s safety—was completely at risk. Lost evidence and records, risks to active investigations, how to respond to citizen and press requests, and thinking about what would happen if a lawyer calls were only a few of the things that came to his mind as he envisioned the horror of the next few weeks and months.

Preventing This Disaster

The police chief’s approach to using and maintaining a server offers up several lessons to help you avoid this nightmare. Use this story and the following error checklist to see if you’re headed for a disaster related to server failure.

Error #1: Using hardware over five years old.

You might skirt by in life using a 2003 car. But your city flirts with significant danger by using a 2003 server. In this story, the public safety server is so old that the vendor doesn’t even support it anymore. That means it can’t be professionally fixed, secured, or updated. It’s not a matter “if” it will break down, but “when.” And “when” can be any day if it’s over five years old. Your city needs to budget for and replace server hardware every 3-5 years.

Error #2: Relying on an as-needed, reactive IT support person to barely maintain the server.

Just enough to get by. In this story, that’s the attitude the public safety department takes toward the server that holds its most important data. At home, do you handle an ant infestation just enough to get by? “Hey, there’s only a dozen ants crawling in my bed tonight. That’s good enough.” Of course not. Through many methods from cleanliness to spraying, you proactively prevent ants from entering your home.

By just band-aiding the server when it acts up, the public safety department is always barely warding off an inevitable disaster (and racking up unpredictable billable hours). Instead, all servers need to be managed, monitored, patched, and later upgraded when they reach end-of-life. Proactive IT maintenance will also alert you if a server is showing signs of a likelihood to fail in the future—preventing a disaster before it happens.

Error #3: Ignoring red flags such as slow computers and freezing.

Why do you use technology in the first place? To help you perform your job better. If a car can’t get you to work, it’s not much use. If a server interferes rather than helps with work, then it’s not much use. Slow computers, frequent memory and storage limits, and an inability to use modern applications are all signs that your equipment needs replacing before it fails.

Error #4: Failing to test data backups.

In the worst-case scenario, the server fails and your data is lost. Data backups can have problems and there are many reasons why data backups encounter possible issues. The city in our story did not test their data backups and assumed they were working. Even if a city does cling to an old server that’s soon to fail, they need to back up and test the backup on a regular basis to ensure that they can recover the data in case of a failure.

For a variety of reasons, sticking with an old server until it dies is not wise. Information security risks, slowed productivity, wasted billable hours, and lost data are only a few of the pitfalls. Modernize your technology and switch to a proactive IT support vendor to ensure that your servers don’t just fail one day and cripple your city.

Tuesday, January 31, 2017
Jabari Massey, Network Infrastructure Consultant
Jabari Massey

Imagine that a city employee who works in the finance department opens their email in the morning. As they check their email, they see one message that seems to come from the city manager. Without thinking, the employee clicks on a zip file attachment assuming that it’s an important set of documents related to a meeting that day.

This employee is not technically savvy, so they are not too alarmed when they see something downloading onto their computer. A window pops up that says to accept something. The employee clicks “yes.”

Within seconds, a chill goes down their spine. Something is wrong. Multiple pop-up windows appear on the person’s computer screen and a new program seems to be running in the background. The employee tells their supervisor, and the supervisor places a call to their reactive IT support vendor who says they might be able to stop by tomorrow.

A day passes while the employee manages to continue doing work that involves accessing software on the city’s financial server. But the employee’s computer continues to slow to a crawl until they can’t use it anymore. The city manager persuades their IT vendor to send someone over today instead of tomorrow.

A junior IT support person arrives and pokes around on the employee’s computer. “Yep, there’s a problem,” they confirm. Figuring it’s a virus, they restart the computer and go into “safe mode” to try to eliminate the virus. Plugging into the financial server to make sure it’s working properly, the junior IT support person now gets a chill down their spine.

They cannot access any data on the financial server because it’s also infected with the virus.

Panic ensues. The junior IT support person calls a senior IT support person. By then, it’s too late. Both the server and the employee’s computer had not been patched in a while, and so many recent security patches had not been applied. Plus, the city runs a free version of some antivirus software that’s only updated when the IT vendor sends someone on site.

“Thank goodness there’s a data backup of the server,” says the city manager. But when the IT support vendor tries to restore the financial data from the backup...that backup doesn’t work. At all. “But we’ve been backing it up manually at least once a week,” says the city manager.

“Have you tested the backup?” asks the senior IT support person.

“No,” says the city manager. Everyone now realizes a nightmare scenario became real. The city’s financial data is lost. Permanently.

Preventing This Disaster

Some variation of this story is all too common for many cities. The good news? Cities can easily prevent a devastating virus attack by addressing some of the errors committed in this story.

Error #1: Lack of business class antivirus software.

Notice the reference in the story to free antivirus software? Many cities try to save money by installing a free, consumer-grade version of antivirus software on computers. This is a mistake because consumer-grade antivirus software is not sophisticated enough to protect city data at the server level. That usually leaves servers unprotected and computers reliant on employees making the updates.

Error #2: Reactive IT support not maintaining and monitoring servers and computers.

The IT support people in our story weren’t getting paid to do ongoing, proactive IT support. Thus, they only updated the antivirus software when the city called on them for an onsite visit. Plus, it appeared that they did not have a process in place for regularly updating the antivirus software and testing the city’s data backups. Experienced IT professionals need to regularly audit antivirus software to confirm that it’s installed on every machine and that virus definitions (which help detect nearly all known viruses) are up to date.

Error #3: An employee clicked on an email attachment.

You might have thought we’d mention this error first. However, your employees cannot be the front line for preventing viruses. We all occasionally make mistakes by clicking on a malicious email attachment or website. That’s why you need a strong foundation in place—business class antivirus software, regularly tested data backups, and proactive IT support—to stop as many viruses as possible from activating. And even if an employee clicks on something malicious, you need to be able to recover from a virus that has been activated.

Because a virus can still get through strong defenses, employee training is a must. Train your city staff about common sources of viruses such as email attachments, websites, online software, and games. With training, you can make your employees more aware about online threats that are easy to avoid if they know how to spot them.

Concerned about a virus crippling your city? Reach out to us today.

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 |