We put the IT in city®

CitySmart Blog

Monday, May 15, 2017
Dave Mims, CEO

Dave MimsEven if your city is not located in Arkansas, it’s still worth noting that the state’s Senate Bill 138 was signed into law by Governor Asa Hutchinson on March 29, 2017. For a quick recap on the law, read our March 8 blog post where we summarized and tracked the law while it was going through the state’s House and Senate.

The passing of this bill is important to cities for a few reasons.

1. Arkansas cities can now lose their charter from non-compliance with IT-related accounting practices.

Arkansas already has a Municipal Accounting Law (§ 14-59) that requires compliance with accounting best practices and includes penalties for non-compliance. But now, Senate Bill 138 adds some teeth to the law by clarifying that not following specific IT-related accounting best practices also constitutes “malfeasance.”

Three key points of the law include:

§ 14-59-117 (a) (1) If the Division of Legislative Audit determines that a municipal treasurer is not substantially complying with this chapter, the division shall report the findings to the Legislative Joint Auditing Committee.
§ 14-62-102 (a)(1) If the Legislative Joint Auditing Committee concludes the process under § 14-59-117 on a municipal corporation, and in the immediately subsequent three-year period the Legislative Joint Auditing Committee concludes the process a second time, the Legislative Joint Auditing Committee may notify the Attorney General and the Governor of its actions.
§ 14-62-102 (b) Upon a finding that the conditions under subsection (a) of this section have been met, the circuit court of the Sixth Judicial Circuit shall revoke the charter of a municipal corporation under this section...

Losing one’s city charter is serious. And now ingrained within Arkansas law, a city must make it part of its accounting best practices to take information technology seriously.

2. While the law applies to application controls, it’s wise to follow all IT best practices recommended by the Arkansas Legislative Audit.

Specifically, the new law applies to application controls listed in the Arkansas Legislative Audit best practices. According to the Arkansas Legislative Audit, application controls “relate to the transactions and data for each computer-based automation system; they are, therefore, specific to each application. Application controls are designed to ensure the completeness and accuracy of accounting records and the validity of entries made.”

Application controls include areas such as data input, data processing, data output, and application-level general controls. However, it will help a city if they follow all the IT best practices listed in their document—including areas such as information systems management, contract / vendor management, network security, wireless networking security, physical access security, logical access security, and disaster recovery / business continuity.

That’s because general IT best practices create the foundation for your application systems technology. Without following general IT best practices, you are likely to create too much risk with your applications. Indirectly, you may find yourself in non-compliance with application controls if you don’t plan, invest, and proactively manage your general information technology.

3. Other states should see Arkansas as a sign of what’s to come—and prepare.

The trend for technology-related security, privacy, and best practices legislation is more, not less. Information technology now holds the crucial role of keeping citizen data private and ensuring that government remains operational even during or after a disaster.

Because government entities—including cities—often don’t spend money on implementing IT best practices even when the danger signs are obvious, laws are getting increasingly passed to ensure accountability and compliance. After Arkansas, it’s likely that other states will pass similar or parallel forms of legislation that hold local governments accountable.

In other words, if you’re not a city in Arkansas then that doesn’t mean you should rest on your laurels. Hold yourself accountable to your citizens and city operations proactively—before your state passes stricter laws like in Arkansas.

Concerned about the state of your information security or compliance with the law? Reach out to us today.

Tuesday, May 9, 2017
Victoria Boyko, Software Development Consultant

Victoria BoykoYes, you read our headline correctly. The Associated Press recently reported on the city of Springfield, Florida’s old website taken over by a Japanese pornographer. If citizens checked the old website URL for a period of time, then they would have viewed the homepage of a pornographic website.

The article noted:

Springfield switched its website to a .gov domain about three years ago. The city's information technology department is seeking to buy back the old domain and any domains [sic] names similar to the city's current website, springfield.fl.gov.

Before you laugh or judge, consider your city. It’s likely you already have or will do a website redesign in the future. And it’s likely you are switching or have switched from an old URL (such as a .org URL) to a new URL (such as a .gov URL).

In other words, you could make the same mistake.

Whenever you redesign and/or assign new URLs to your website, we recommend having website professionals manage all of the parts and pieces as you go from one URL to another. But two basic website best practices could have prevented Springfield, Florida’s public embarrassment.

1. Keep ownership of all old city website URLs.

Do not give up ownership of these URLs. You will need them.

First, it’s good practice to own these URLs so that other people don’t buy them, use them for websites that have nothing to do with your city, and then unintentionally (or intentionally) confuse citizens. Think of it like a celebrity or a well-known brand buying up URLs that might contains names that people would sensibly search for. It’s a way to make sure that common, incorrect website URL searches all go to your city’s website.

Second, you will need your old URLs to redirect people to your new website. It’s like a store that needs to tell people at the old location that there is a new location.

2. Use 301 redirects.

A “301 redirect” is a technical website term. It means when a person goes to your old URL they are automatically redirected to your new URL. Hubspot uses a great analogy of mail forwarding. When you move, you set up mail forwarding so that your mail goes to your new address. 301 redirects are an online version of that concept.

301 redirects are essential for a few reasons:

  • Citizens will get automatically directed to your new website when they type in your old website. Many of your citizens will not use Google to find your city’s website. They will type in or cut and paste a familiar URL—your old website. Old habits die hard, and many citizens won’t know you changed your URL. You need to make sure that when they type in that old URL they get automatically directed to your new website.
  • Other websites that link to your city’s website need to work. If 100 different websites link to your city’s website, what happens when you change your URL? All of those links won’t work—unless you use 301 redirects. This simple website tactic is the difference between people continuing to visit your website from those 100 external websites versus seeming to go dark. Without 301 redirects, it’s like moving and then not telling your friends and family your new address.
  • Your own internal website links need to work. If you do a thorough, meticulous job of changing every single hyperlink on your website to your new website URLs, then this won’t be a problem. But if you have hundreds of hyperlinks on your website that direct people to other pages on your website, then it’s likely you won’t have the time to change every link. For the time being, 301 redirects will provide a band-aid until you can change all internal links.
  • 301 redirects help you stay visible to search engines. Search engines spend years and years getting to “know” your website through indexing your site on a regular basis, analyzing links to and from your website, and detecting how much content you produce. If you produce a new website with a new URL with no connection at all to your old website, then search engines will have to get to “know” your new website from scratch. Then it may take a long, long time before you start appearing in search engine results again. For example, if a citizen types in your city’s name into a search engine, your website may not even show up in the first 10-20 search results. 301 redirects smooth things over—essentially letting search engines know that you’ve moved and keeping your city high in the search results.

While just two simple best practices, these are technical aspects of your website that need to be managed and overseen by experienced website professionals. Otherwise, like Springfield, Florida, you may end up losing ownership of your old URL, failing to redirect citizens to your new website, and then letting them think that your city’s website apparently has switched over to providing porn.

Want to ensure that you prevent a similar disaster? Reach out to us today.

Tuesday, May 2, 2017
Brian Ocfemia, Technical Account Manager

Brian OcfemiaWe often talk about data backup as the best remedy for a virus infection. If the worst happens and a virus takes your systems down, then you just restore an uninfected backup.

However, a recent article concerning Bingham County in East Idaho brings up an excellent question: What happens if your backup servers get infected?

The Idaho State Journal reports that “[Bingham] County information technology staff thought the virus was contained but discovered [on February 17, 2017] that one of the backup servers had become infected, knocking the entire system offline.”

Luckily, the county had some other data backups in place to mitigate damage from the ransomware virus attack. But this scenario offers a good lesson. Let’s address several technology pieces that need to be in place to prevent a virus from infecting a backup and permanently destroying your data.

We’ll assume in our discussion that a city already has some type of data backup solution along with antivirus software in place.

1. Monitoring and Alerting

It’s bad enough to get a virus. It’s worse if that virus goes undetected. Many modern viruses often mask themselves, retreat to the background, and do malicious things to your systems such as collect financial information. The longer the virus lurks in the background, the more it can spread and the more damage it can do.

Cities need proactive monitoring and alerting through a combination of automated software that tracks technology health combined with experienced IT professionals watching your systems. Part of that monitoring and alerting involves the right kind of antivirus software. We recommend enterprise-grade antivirus software that offers sophisticated monitoring tools for IT professionals to track and catch viruses.

2. Unlimited Offsite Data Backup Storage

Let’s unpack this phrase a bit.

  • Offsite data backup: In addition to backing up your data onsite, you need an offsite data backup component for worst-case scenario disasters such as tornadoes, flooding, or fires.
  • Storage (and retention): You will need to store various snapshots of your backed up data and make them available in case you need them. For example, you may need to see a snapshot of data as it looked one month ago if certain documents recently went missing. The right storage strategy allows you to maintain all versions of your files and documents while also retaining them for a set period of time.
  • Unlimited: This is key to rarely (if ever) worrying about a virus or ransomware attack. Let’s go back to the example of Bingham County and assume the ransomware virus lingered around for a long time, infecting even backup files. Some recent critical data might unfortunately get lost, but the county could still go back—as far as it wanted—to a snapshot of its data right before the infection hit.

It’s important to note that if you don’t have enough storage for a reasonable backup retention period, you may be stuck in a situation where the only files you can restore are infected ones. We recommend an unlimited offsite data backup storage service that allows you to keep your offsite backups indefinitely. Then, you can go back in time as far as you need to recover files.

3. Employee Education

The Idaho State Journal article goes on to state:

“An information technology director for a neighboring East Idaho county said emails with suspicious attachments can often cause computer systems to become infected. He said his systems manager comes across up to three such emails per week.”

Despite the best cybersecurity protection and data backup, employee education remains an essential part of your strategy. Antivirus and antispam software can help prevent access to many malicious websites and email attachments. But employees still need to learn more about what not to click on and how to spot hacking and phishing attempts.

Some things you need to talk about with employees include:

  • Browsing safely and knowing the signs of a malicious website.
  • Scrutinizing email attachments and understanding that hackers can spoof email addresses (such as an email supposedly coming from their boss).
  • Downloading unnecessary or unauthorized software from untrustworthy sites (such as games, shopping apps, and productivity apps).

As we see from this situation, there’s more to backing up data than just backing up data. You need to stay vigilant through proactive monitoring and alerting. You need to retain data snapshots that go far back in case your backups get infected. And you need to keep training employees who often unknowingly take actions that let in viruses and hackers.

Worried about what would happen to you if a ransomware virus hit? Reach out to us today.

Tuesday, April 25, 2017
Nathan Eisner, COO

Nathan EisnerIn past blog posts, we’ve talked about the importance of data backup for body camera video (and other police department video). We always ask, “What happens if evidence is permanently lost?”

This situation recently arose with the Cockrill Hill Police Department in Dallas, Texas. After a ransomware attack on the police department’s servers, the municipality permanently lost data. According to a Mother Jones article (with my added emphasis in bold):

The police department claimed that they still had paper copies of all the documents on the server and physical copies of much of the video. But in a letter sent to the county prosecutor, the department said "all bodycam video, some photos, some in-car video, and some police department surveillance video were lost." The department tried to recover as much as possible but said that "if requests are made for said material and it has been lost, there is no chance of recovery or producing the material."

The article opens by telling the story of a defense lawyer working for a client who faced prison time. He needed specific evidence to help his client avoid jail time. That evidence? Permanently lost. In other words, the data loss—rooted in a technology problem—could literally send a person to jail or serve a longer prison sentence because important evidence disappeared forever.

What this Mother Jones article doesn’t address is how easily this loss of data could have been prevented.

The Best Weapon Against Ransomware? Data Backup

While you may protect yourself against ransomware in many ways, the worst scenario may still happen. An employee clicks on a malicious email. Hackers break into a server. Lack of up-to-date patches expose your software to a major security flaw. It happens.

Like insurance, your technology needs to prepare for the worst. Only data backup can fully “insure” you against ransomware. Let’s say the worst happens. Ransomware is downloaded, you receive an automated blackmail threat, and you (wisely) decide not to pay the criminals. You permanently lose that data. But luckily, you have a backup you can restore. You may end up losing none of the data, as little as only minutes of data, or, at worst, hours or days of data.

For data backup, you need:

  • Onsite backup that takes frequent snapshots of your data. For smaller disasters (like files lost or a server failure), you can recover quickly.
  • Offsite backup that sends your data to a geographically distant data center (or centers). Then when a disaster wipes out your onsite data, you still have all your data safely stored offsite.
  • Regular testing (such as quarterly) so that you know your data backup works. Too many cities never test their data backup and they often find it doesn’t work when a disaster actually hits.

Because body camera, dashcam, and other police video requires massive amounts of video storage, it’s wise to explore data backup solutions with unlimited offsite storage. You don’t want to lose data arbitrarily because of storage caps or added costs. Unlimited offsite storage also gives you flexibility with data archiving and retention to help you follow the law.

The Second-Best Weapon Against Ransomware? Proactive IT Monitoring and Management

On the preventative side, it’s essential for police departments to hire staff or a vendor that proactively monitors and maintains technology for servers, desktops, and mobile devices. That includes:

  • Shoring up cybersecurity weak points in your network through locking down and properly configuring your computers, servers, switches, routers, and firewalls.
  • Monitoring your technology’s performance and health 24x7x365 and receiving alerts about problems.
  • Using antivirus, antispam, and content filtering software to help employees with safe internet browsing and email.
  • Consistently applying updates and patches to your software.
  • Ensuring any remote access is secure when teleworking.
  • Managing and tracking all technology assets.

While articles like the one we’ve referenced from Mother Jones seem to indicate that failure can be shrugged off without consequences, that may soon no longer be the case. Federal and state laws and regulations increasingly push for higher cybersecurity accountability from government entities. Even at best, these incidents are an embarrassment for cities and, from an ethical perspective, negatively impacting the lives of defendants (especially if they’re innocent of a crime), defense attorneys, and prosecutors who rely on this evidence to uphold the law.

Would your city’s police department survive a ransomware attack? Reach out to us today if you’ve got any doubts.

Wednesday, April 19, 2017
Mike Smith, Network Infrastructure Consultant

Mike SmithLicking County, a county east of Columbus, Ohio, recently experienced a bad ransomware attack on its IT systems. Ransomware is a specialized virus that encrypts files—making them nearly impossible to access unless you pay criminals a ransom. Cybercriminals use ransomware to extort money in return for unlocking your files. Many organizations pay the ransom despite the FBI and other law enforcement agencies recommending against it.

Luckily, Licking County managed to mostly survive the attack based on implementing some important best practices. Let’s look at the good, bad, and ugly of this situation to extract some important lessons.

The Good

Data backups

The difference between getting crippled and devastated by a ransomware attack versus surviving it relatively unscathed all comes down to data backups. Licking County ended up losing only about one day’s worth of data for most systems. Another county referenced in the article ended up paying a ransom of $2,500 to cybercriminals because they did not invest in data backup.

Activating a plan to shut down the network

To stop the spread of the ransomware, Licking County shut down its network. Clearly, the county had a plan in place and enacted it when the ransomware virus hit. By planning ahead, they were best prepared for what to do to keep the virus contained and to minimize impact.

Rebuilding systems based on highest priority data

As part of its disaster recovery plan, the county rebuilt its systems based on the highest priority data first. The article references data such as “servers that house felony-case tracking for the prosecutor's office and the auditor's property-records database.” Any disaster recovery plan needs to have a clear plan as to how data will be restored—and in what order of priority.

The Bad

Rebuilding systems will take a lot of time

Licking County is a big county and so it needs to reformat about 1,000 computers as part of its rebuild. That takes a lot of time. Even smaller organizations will need to spend significant time rebuilding servers and reformatting computers.

Direct and indirect costs

Directly, the costs of billable IT time and possibly enhancing networking equipment and cyber protection software can present a big hit to your budget. Indirectly, lost productivity wastes expensive employee salaries and potentially delays major projects when time is ticking.

Impacts to citizen service

After a disaster, a crippled government entity will not be able to serve citizens at full capacity. The mission of government gets impacted when ransomware hits. County Commissioner Tim Bubb says, “We have lost a large part of our focus on serving the people of Licking County. What price do you put on that?"

Potentially weak firewall and network connections

A Columbus Dispatch article mentions that the county needs to shore up its “firewall and network connections.” An improperly configured firewall can leave ports open that allow hackers to easily gain access to servers and steal information. Setup of switches, routers, and other networking equipment also impacts security.

Potentially weak passwords

The same article mentions that the county needs to encourage employees to change passwords more frequently. In a recent blog post, we said, “The longer a password is in use, the more likely that hackers will be able to crack it. The more you change passwords, the more difficult you make a hacker’s job.”

The Ugly

911 dispatching affected

An article published in the Newark Advocate the day after the incident stated “...the 911 Center has been operating in manual mode since late Tuesday night. The 911 Center phones and radios work, but dispatchers do not have access to their computers. The public can still call 911 for emergency police, fire or medical response.”

While not completely shut down, any impact to 911 or other critical emergency services can literally affect lives in the wake of a ransomware attack.

Employees click on too many suspicious emails

One of the biggest cybersecurity threats is people. No matter how great your data backups, antivirus, firewalls, and security measures, hackers and cybercriminals still often break into a government entity through people clicking on suspicious websites and email attachments.

Note this paragraph in the Columbus Dispatch story:

Fairfield County started working last year to tighten procedures to guard against the type of cyberattack that occurred in Licking County, said Fairfield County IT Administrator Randy Carter. He said he was dismayed when he sent a test phishing email to county employees in September and more than 25 percent clicked on it. Carter plans to provide training to employees on what emails to avoid.

25 percent! One in four people got fooled by these dangerous emails. Each click on one of these emails opens you up to the threat of a virus or ransomware.

Cybercriminals targeting government more and more

Cyberattacks grow more numerous and targeted. Government entities are ripe for these attacks. That includes cities.

Are you prepared?

  • Like Licking County, do you have data backups to recover from a ransomware attack?
  • Do you have the right network equipment and modernized technology to protect yourself?
  • Are your employees trained about the dangers of clicking on malicious emails and websites?

If you need help protecting yourself from a ransomware attack, reach out to us today.

Tuesday, April 11, 2017
Brandon Bell, Network Infrastructure Consultant

Brandon BellA city had operated for a long time with tape backup and decided to upgrade. City administrators heard from their IT staff that they needed something more reliable than a manual solution reliant upon busy people to both conduct the backup and store it offsite.

Spending a lot of money on a modern complex data backup solution, the city was assured by its IT staff that this automated beast would solve all their problems. Indeed, the data backup worked automatically. In a meeting, IT staff showed city department heads the wonder of the data backup system by retrieving a few PDF documents from the backup data storage. To city council and the public, the city administrator proudly said they had ticked data backup off their list. Problem solved!

One day, a fire tragically tore through most of city hall. The building ruined, city staff needed to relocate to a temporary building until a new city hall was built. But thank goodness—despite all the servers destroyed—that the city could retrieve its data.

Or not. When IT staff attempted to restore the city’s data through its backup, most of the major databases, applications, and data would not restore. A few chunks of data—like some people’s individual documents—were okay. But the city’s most important information was not there.

And so...an expensive backup solution became nearly worthless. Why? Upon further investigation, the city administrator was told that nobody ever tested the data backup. “But...it was an expensive solution,” the city administrator said. “And my IT staff said that it was automated. The data backup solution’s reporting even said it worked.”

Well...it didn’t. And that’s all that mattered when the city administrator had to now explain why this expensive investment failed them after a disaster—and failed to do the exact thing it was supposed to do.

Preventing This Disaster

One aspect of data backup and disaster recovery—testing—is nearly as crucial as simply having data backup at all. No matter what kind of data backup you use, you need to test it. Otherwise, you don’t know that it’s working.

Let’s look more closely at the errors in our city scenario above.

Error #1: Assuming the data backup works.

A data backup solution will often look like it’s doing its job. From manual solutions like tape to more sophisticated automated data backup servers, the data backup application will often indicate that the process is a success or failure. But no matter what the application tells you, you don’t know that it works until you test it.

Error #2: Not testing all the backed up data.

Calling up a few files such as PDFs from the data backup storage is not testing. When a disaster hits, you will need to be fully operational with your databases, software applications, website, email, and documents. For example, will your account system work from a backup copy? When you test, test everything. Simulate what would happen if an actual disaster hit.

Error #3: Develop a plan and document it.

Testing needs to be part of your overall disaster recovery and business continuity plan. The act of testing not only guarantees you will access the data but also allows you to practice how data recovery will work. Who does what? How fast will the data be restored? In what order? Where will you access the recovered data?

You want to run into issues during testing and deal with them in a simulation—rather than after a real disaster.

Uncertain about your data backup solution? Are you testing it at least quarterly? Reach out to us today.

Tuesday, April 4, 2017
Jabari Massey, Network Infrastructure Consultant

Jabari MasseyOn the surface, a coastal city did some correct things to back up its data. The city had a few servers in a physically secure basement room that were well-maintained by IT staff. One of the servers backed up important data. In case a server failed, the backup server would run until the city could replace the original server.

A long time had passed since the city last experienced a hurricane. When a hurricane finally seemed eminent, the city was ordered to evacuate until the massive storm passed. The city manager and IT staff didn’t think much about the servers other than placing them upon concrete blocks in case of flooding. As long as the city implemented its emergency action plan and evacuated everyone safely, the city manager assumed its information technology would remain safe.

After the hurricane passed, city staff returned to find that no massive devastation occurred but they did experience heavy flooding. The IT staff had placed the servers upon concrete blocks as a precautionary measure, but they learned an incredibly hard lesson in hindsight.

Located in a basement room, the servers sat below sea level. Although the rest of city hall experienced moderate flood damage in places, the basement had filled up to dangerously high levels. All of the servers—including the backup server—were rendered unusable by the flooding.

With a sinking feeling, the city manager realized all critical data—including financial, public safety, document management, email, and website data—was gone. The only backup server got destroyed along with the others. It might be easy for the city manager to point some blame in the direction of the IT staff, but it was well-known that he had refused requests to explore other data backup options because of “budget concerns.”

Now, the mayor, city council, the media, and public would be asking questions.

Preventing This Disaster

Sure, the city manager and IT staff made a bad decision to place servers in a basement room below sea level. But their errors go deeper than this poor choice of physical location for the servers.

Let’s look at the errors in the story above.

Error #1: Locating servers in a flood-prone area of your building.

Getting the most obvious error out of the way, it’s clear that the servers needed to reside on an upper floor. In addition, the server room needed to be in a room that mitigates flood risks through preventative measures such as water leak sensors or eliminating areas where water can enter.

Error #2: Lack of offsite data backup.

While locating the servers on a higher floor may have prevented this immediate flooding disaster, it’s still not a full disaster recovery plan. Anything can happen to your technology onsite. To guarantee full recovery of your data after a disaster, you need an offsite data backup component to your emergency plan.

We recommend storing your data offsite in geographically dispersed locations (such as in data centers both on the East and West coasts). Then, even if the worst disaster wipes out your buildings, you will be able to recover and access your data.

Error #3: Lack of technology planning.

The lack of offsite data backup also signifies a larger issue—a lack of planning. The city had developed an emergency plan and used it in the case of the hurricane. But when was the plan developed? When was it last updated? Did it include technology-related scenarios? What was the plan to protect data in case of a disaster?

First, the city needed to update its emergency plan and include technology. That would have addressed technology-related gaps in the city’s data backup, disaster recovery, and business continuity plans. Second, the city needed regular technology planning meetings (at least once a quarter) and ongoing monitoring to ensure that data backups were tested and working. This regular monitoring and planning would help the city adapt to changes (such as new technology, more staff, building changes. etc.) and ensure that the risk of data loss is minimal.

Flooding is one of the most common disasters. It can happen anywhere in the country and devastate a city. Because citizens will rely on your city after severe flooding, you must be operational as fast as possible. That means having access to your data—your website, your documents, and your applications that are essential to operations.

By developing a disaster recovery plan that includes an offsite data backup component, you will lessen the risk of permanent data loss and angry “Why?” or “How?” questions after the fact from council, the public, and others.

Concerned about your data backup and disaster recovery? Reach out to us today.

Tuesday, March 28, 2017
Ryan Warrick, Network Infrastructure Consultant

Ryan WarrickIn recent posts, we’ve talked about disasters at cities that result in permanent data loss, incredible damage to city operations, and city department heads wondering if their job is now at risk—all sadly because of preventable risk. The stories we use to illustrate these disasters—and the lessons learned—are based on a combination of many, many scenarios we’ve witnessed at cities throughout the years.

However, we recently saw a story that’s quite specific to one city and a very public, front page news illustration of some important IT-related lessons. Let’s look at what happened to the City of Miami Beach, Florida in December 2016.

Third Parties Steal $3.6 Million—and No One Notices for Six Months

In a nutshell, unknown third parties stole the account and routing numbers from the city’s banking account. According to the Miami Herald, the criminals “[rerouted] automatic payments intended to pay vendors and other government bills.” The criminals did it for six months and stole $3.6 million before staff in the finance department noticed.

We carefully reviewed the Miami Herald article and the city manager’s report. While this crime is a form of cybersecurity, the situation also includes lessons about IT-related processes and controls that are incredibly important to cities. A few bad practices stick out from our analysis of the report that cities need to avoid.

1. Completely ignoring basic, elementary best practices.

The city of Miami Beach was offered free fraud control tools when they set up the account in 2012—the same kind of fraud control tools that many individual banking customers enjoy. Did the city take advantage of these tools? No. Maybe they had a reason at the time such as wanting to implement their own fraud controls. If so, that never happened.

Cities need to stay aware of and implement important best practices that help mitigate information security risks. In this case, both finance and IT staff needed to say “yes” to such an obvious best practice back in 2012.

2. Using easy-to-steal information as authentication for financial transactions.

Think about how many people in a city can take a quick peek at a check. If third parties could steal city money through only this information, then the city had a security vulnerability that was wide open for people to exploit.

We find that cities also have similar weaknesses in areas such as passwords, unencrypted wireless devices, and website hosting that makes it easy for hackers to exploit security vulnerabilities.

3. Apparent lack of financial data oversight.

In a recent post about data processing, we said, “Experienced IT professionals should monitor everything related to your data processing such as transactions and processing, errors and incorrect information, overrides, unauthorized use of the application (especially when it appears that someone is altering data or ignoring/tampering with processes), reconciliations, and application performance (such as after a power outage or server failure).”

Obviously, finance department staff have an even more important role in monitoring this information too. While online banking is great, it’s unwise for even an individual consumer to not regularly review banking transactions. Great risk was introduced by not reviewing for six months and hoping that everything was okay. Cities need to become more proactive at monitoring and reviewing important aspects of their operations where data changes constantly—from accounts payable to information technology.

4. Lack of modernization.

Many cities often hear the word “modernize” and think of it as “unnecessarily wasting money or time on something new and fancy that we don’t need.” Sure, some solutions might fit that definition. But technology modernization is important especially when your old technologies and processes lead to security vulnerabilities, inefficient operations, and significant liability.

In the case of Miami Beach, the city manager’s report includes many “sudden” modernizations in one fell swoop such as ACH fraud controls and using UPIC (Universal Promotional Identification Code) to avoid sharing confidential banking information. The city manager even notes in the report that “the ACH Fraud Control program already prevented an unauthorized ACH transfer.”

I know we beat this drum a lot. But why do cities wait? Why do cities put off modernizing their technology and processes until a massive crisis hits? We see this “putting off” logic holds true at many cities for data backup, disaster recovery, website hosting, records and document management, email, and aging hardware. In all of these cases, lack of modernization increases the risk of a significant city incident or disaster.

Learn from cities like Miami Beach. Are you sure that fraudsters aren’t currently stealing money from you? Is your technology modernized in such a way that you aren’t headed for a major disaster like permanent data loss?

If you are worried about addressing critical technology aspects of your city before a disaster happens, reach out to us today.

Tuesday, March 14, 2017
John Miller, Senior Consultant

John MillerA small city with two servers also stored many paper documents containing critical information. The city backed up its servers with tape-based data backup which a city employee would take home every week or so to store “offsite” at their house. Many of the paper documents were not replicated electronically, and so these paper documents were the only versions in existence.

One night, a fire began that destroyed nearly all the building before firefighters arrived at the blaze. Fire alarms went off but no fire suppression occurred until the fire department showed up.

Assessing the damage the next morning, the city discovered that its paper documents and servers were completely destroyed. With the paper a total loss, the city decided to recover the server data from the tape backups. However, after a two-day attempt at trying to restore the data, the city could only retrieve about 10% of it. Many of the tape backups hadn’t been tested and the city didn’t realize that the backups weren’t running properly for a long time.

As a result, operations ground to a halt and the city found itself in dire trouble. They lost their accounting and billing systems along with many public records and documents. So many critical operational records were lost related to accounting, taxpayers, and businesses. The public outcry had only yet to begin after the admission of data lost—and why the city had not properly backed that data up.

Preventing This Disaster

A fire can happen to any city at any time. Is your city prepared? For such a common disaster, we find that many cities do not have disaster recovery plans that account for a simple yet deadly fire.

Let’s look at the errors in the story above.

Error #1: Using paper as the only copy of important documents.

In today’s electronic information age, relying only on paper for important documents is way too risky. A simple fire can wipe out paper in a matter of minutes. Paper also fails in a flood, tornado, or other natural disaster. Any paper-based documents that are critical to your city need to be scanned electronically and backed up offsite to ensure they are not lost.

Error #2: Poor offsite data backup plan in place.

Relying on a city employee to take tapes offsite every week to their house is not a sure-fire plan. First, these tapes were not tested on a regular basis. When the city actually needed to restore data, most of the tapes failed. Second, too many security and liability risks exist when a city relies on an employee to manually collect backup tapes and store them in a private home. What happens if the employee is negligent or disgruntled? What if they forget one week to take the backups home?

Error #3: Lack of appropriate fire suppression for a server room.

Any room that stores servers needs best-of-breed fire suppression. Fire alarms alone are inadequate. Most data centers feature fire suppression technology that helps eliminate or reduce the severity of a fire. If your city decides to host its own servers, then you need to explore fire suppression options beyond an alarm.

Error #4: Lack of an overall disaster recovery plan.

The city clearly did not think through the consequences of a disaster. Otherwise, it would have identified critical information—such as its paper documents—and planned for a worst-case scenario such as a fire. This plan would include:

  • Identifying which data is most critical and cannot be lost.
  • Estimating the maximum amount of acceptable downtime before restoring city operations.
  • Detailing how the city will get up and running after a disaster.
  • Outlining contingency plans while the data is being restored.
  • Ensuring that any data backups are tested regularly.

While large disasters like tornados can seem more improbable and less likely, cities need to keep in mind that disasters also include more common scenarios like fires. A fire can wipe out critical information quickly. Your disaster recovery plan needs to account for both paper-based and electronic information—ensuring that you can recover your most critical information soon after a fire or other common disaster.

Questions about your city’s ability to protect and recover your most important information after a fire? Reach out to us today.

Wednesday, March 8, 2017
Dave Mims, CEO

Dave MimsWe know. It’s the federal government. Yet, cybersecurity legislative trends show that security risks within government—whether it’s federal, state, or local—are being addressed because they affect national security and the privacy of citizens. There’s an incentive for Congress to help your city shore up its cybersecurity.

The federal bill is called the State Cyber Resiliency Act and it’s in the proposal stage. As a bipartisan bill, it has a higher chance of making it through the House and Senate depending on Congressional priorities. Matt Zone, President of the National League of Cities is quoted as saying:

“Cities manage substantial amounts of sensitive data, including data on vital infrastructure and public safety systems. It should come as no surprise that cities are increasingly targets for cyberattacks from sophisticated hackers. Cities need federal support to provide local governments with the tools and resources needed to protect their citizens and serve them best."

The idea is that FEMA will administer grants for state, local, and tribal governments. Particulars about the grants are not clear at the moment as the text of the bill has not yet been submitted.

We’ve been concerned about city cybersecurity for a long time, and it’s reassuring to us that lawmakers want to help cities address this issue. An article from FCW points out some drivers behind this bill:

  • “[State, local, and tribal governments] typically devote less than two percent of their IT budget to cybersecurity.”
  • “…in 2015, 50 percent of state and local governments had six or more cyber breaches within the last two years.”

We’ll be tracking this bill (S.516) after its introduction last week. Stay tuned!

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 |