CitySmart Blog

Tuesday, May 23, 2017
Nathan Eisner, COO

Nathan EisnerIf ransomware hasn’t gotten your attention yet, then the WannaCry ransomware cryptoworm that ravaged the world for a week in mid-May should make you sit up. The attacks were so devastating to many organizations—from major hospitals to important financial institutions—that ransomware is now mainstream news and the talk of federal and state legislators.

WannaCry 101: Getting You Up to Speed

You may have seen a lot of headlines and articles about WannaCry, but here are the basics to get you caught up.

  • WannaCry is the name of a specific “ransomware cryptoworm.” Ransomware is a type of virus that encrypts your files and documents. The criminal then asks for a ransom within a specific time period (such as 72 hours). If you pay, then they (may) decrypt your files. If you don’t, you permanently lose access to those files. A cryptoworm is a self-replicating virus that encrypts files—meaning that once the virus in inside your IT systems, it can infect other machines without any city employee doing anything.
  • WannaCry originated from a leak of National Security Agency (NSA) data that indicated a security vulnerability in Microsoft Windows operating systems. Hackers stole this information from the NSA and used it to create the ransomware cryptoworm.
  • WannaCry had its biggest impact from May 12-19, 2017 when it affected about 230,000 computers across 150 countries.

Why Your City May Be in Serious Danger from a Future Ransomware Attack

While the media outlined the sophistication and wide reach of this attack, it mostly hit organizations that did not follow three important technology best practices.

This is important for cities to realize: It’s likely that your city has a good chance of experiencing a devastating ransomware attack that leads to permanent data loss if you don’t follow the three best practices below.

1. Failing to regularly patch your software.

Microsoft released a Windows security patch in March 2017 that prevented WannaCry from affecting an organization. According to CNN, “The ransomware is spread by taking advantage of a Windows vulnerability that Microsoft (MSFT, Tech30) released a security patch for in March. But computers and networks that hadn't updated their systems were still at risk.”

Yet, so many organizations—including cities—do not patch their software on a regular basis. Excuses are plentiful. City staff have too much on their plates. Reactive IT vendors do not get paid to do proactive IT maintenance. Nothing appears broken, so why fix it? It’s not a priority. Et cetera.

But when you don’t regularly patch, you miss out on security updates. Software vendors plug holes that hackers can exploit. When you don’t apply patches, it’s like leaving a back door open in your house. Organizations that did not apply the March 2017 Microsoft patch left this back door wide open.

2. Failing to update your operating system.

WannaCry devastated organizations using outdated, unsupported operating systems such as Windows XP, Windows Server 2003, and Windows 7. A newer operating system like Windows 10 wasn’t affected by WannaCry at all.

If your city is running an outdated Windows operating system, consider that:

The older an operating system becomes, the more security issues it will have and there is less of a chance that Microsoft will provide security patching. Many organizations—including cities—stick with older operating systems because of poor practice, older software that’s only compatible with older operating systems, and an unwillingness to budget for the upgrade of operating systems.

Think of your operating system like a car. If Microsoft has stopped supporting it, it’s like driving a car that no professional will officially or possibly be able to repair anymore. You’re essentially just stitching it together with band-aids and waiting for it to break down, at any time.

3. Failing to modernize your technology and get rid of legacy systems.

This issue has become so prevalent across federal, state, and local government that proposed legislation such as the Modernizing Government Technology (MGT) Act specifically addresses IT modernization. In 2017, there is no longer a “nice-to-have” argument about modernizing technology. Instead, modernized technology and cybersecurity are increasingly seen as one and the same thing. The recent WannaCry attacks are now referenced by legislators pushing IT modernization bills—and they see it as both a national security and citizen privacy/protection issue.

For cities, it will become more and more negligent to cling onto old legacy hardware and software that uses obsolete, unsupported, and unsecure technology. While budget is always a concern, the costs of a cyberattack—financially, legally, and politically—can be far worse. States such as Arkansas have even passed laws threatening to revoke a city’s charter if they don’t comply with the law through using appropriate, secure technology.

While the WannaCry attacks might look scary, they really only affected organizations that failed to implement basic IT best practices such as patching, using fully supported Windows operating systems, and keeping their technology modernized.

If your city isn’t following the three best practices above, you are at risk for a ransomware attack. Reach out to us today with any concerns.

Monday, May 15, 2017
Dave Mims, CEO

Dave MimsEven if your city is not located in Arkansas, it’s still worth noting that the state’s Senate Bill 138 was signed into law by Governor Asa Hutchinson on March 29, 2017. For a quick recap on the law, read our March 8 blog post where we summarized and tracked the law while it was going through the state’s House and Senate.

The passing of this bill is important to cities for a few reasons.

1. Arkansas cities can now lose their charter from non-compliance with IT-related accounting practices.

Arkansas already has a Municipal Accounting Law (§ 14-59) that requires compliance with accounting best practices and includes penalties for non-compliance. But now, Senate Bill 138 adds some teeth to the law by clarifying that not following specific IT-related accounting best practices also constitutes “malfeasance.”

Three key points of the law include:

§ 14-59-117 (a) (1) If the Division of Legislative Audit determines that a municipal treasurer is not substantially complying with this chapter, the division shall report the findings to the Legislative Joint Auditing Committee.
§ 14-62-102 (a)(1) If the Legislative Joint Auditing Committee concludes the process under § 14-59-117 on a municipal corporation, and in the immediately subsequent three-year period the Legislative Joint Auditing Committee concludes the process a second time, the Legislative Joint Auditing Committee may notify the Attorney General and the Governor of its actions.
§ 14-62-102 (b) Upon a finding that the conditions under subsection (a) of this section have been met, the circuit court of the Sixth Judicial Circuit shall revoke the charter of a municipal corporation under this section...

Losing one’s city charter is serious. And now ingrained within Arkansas law, a city must make it part of its accounting best practices to take information technology seriously.

2. While the law applies to application controls, it’s wise to follow all IT best practices recommended by the Arkansas Legislative Audit.

Specifically, the new law applies to application controls listed in the Arkansas Legislative Audit best practices. According to the Arkansas Legislative Audit, application controls “relate to the transactions and data for each computer-based automation system; they are, therefore, specific to each application. Application controls are designed to ensure the completeness and accuracy of accounting records and the validity of entries made.”

Application controls include areas such as data input, data processing, data output, and application-level general controls. However, it will help a city if they follow all the IT best practices listed in their document—including areas such as information systems management, contract / vendor management, network security, wireless networking security, physical access security, logical access security, and disaster recovery / business continuity.

That’s because general IT best practices create the foundation for your application systems technology. Without following general IT best practices, you are likely to create too much risk with your applications. Indirectly, you may find yourself in non-compliance with application controls if you don’t plan, invest, and proactively manage your general information technology.

3. Other states should see Arkansas as a sign of what’s to come—and prepare.

The trend for technology-related security, privacy, and best practices legislation is more, not less. Information technology now holds the crucial role of keeping citizen data private and ensuring that government remains operational even during or after a disaster.

Because government entities—including cities—often don’t spend money on implementing IT best practices even when the danger signs are obvious, laws are getting increasingly passed to ensure accountability and compliance. After Arkansas, it’s likely that other states will pass similar or parallel forms of legislation that hold local governments accountable.

In other words, if you’re not a city in Arkansas then that doesn’t mean you should rest on your laurels. Hold yourself accountable to your citizens and city operations proactively—before your state passes stricter laws like in Arkansas.

Concerned about the state of your information security or compliance with the law? Reach out to us today.

Tuesday, May 09, 2017
Victoria Boyko, Software Development Consultant

Victoria BoykoYes, you read our headline correctly. The Associated Press recently reported on the city of Springfield, Florida’s old website taken over by a Japanese pornographer. If citizens checked the old website URL for a period of time, then they would have viewed the homepage of a pornographic website.

The article noted:

Springfield switched its website to a .gov domain about three years ago. The city's information technology department is seeking to buy back the old domain and any domains [sic] names similar to the city's current website,

Before you laugh or judge, consider your city. It’s likely you already have or will do a website redesign in the future. And it’s likely you are switching or have switched from an old URL (such as a .org URL) to a new URL (such as a .gov URL).

In other words, you could make the same mistake.

Whenever you redesign and/or assign new URLs to your website, we recommend having website professionals manage all of the parts and pieces as you go from one URL to another. But two basic website best practices could have prevented Springfield, Florida’s public embarrassment.

1. Keep ownership of all old city website URLs.

Do not give up ownership of these URLs. You will need them.

First, it’s good practice to own these URLs so that other people don’t buy them, use them for websites that have nothing to do with your city, and then unintentionally (or intentionally) confuse citizens. Think of it like a celebrity or a well-known brand buying up URLs that might contains names that people would sensibly search for. It’s a way to make sure that common, incorrect website URL searches all go to your city’s website.

Second, you will need your old URLs to redirect people to your new website. It’s like a store that needs to tell people at the old location that there is a new location.

2. Use 301 redirects.

A “301 redirect” is a technical website term. It means when a person goes to your old URL they are automatically redirected to your new URL. Hubspot uses a great analogy of mail forwarding. When you move, you set up mail forwarding so that your mail goes to your new address. 301 redirects are an online version of that concept.

301 redirects are essential for a few reasons:

  • Citizens will get automatically directed to your new website when they type in your old website. Many of your citizens will not use Google to find your city’s website. They will type in or cut and paste a familiar URL—your old website. Old habits die hard, and many citizens won’t know you changed your URL. You need to make sure that when they type in that old URL they get automatically directed to your new website.
  • Other websites that link to your city’s website need to work. If 100 different websites link to your city’s website, what happens when you change your URL? All of those links won’t work—unless you use 301 redirects. This simple website tactic is the difference between people continuing to visit your website from those 100 external websites versus seeming to go dark. Without 301 redirects, it’s like moving and then not telling your friends and family your new address.
  • Your own internal website links need to work. If you do a thorough, meticulous job of changing every single hyperlink on your website to your new website URLs, then this won’t be a problem. But if you have hundreds of hyperlinks on your website that direct people to other pages on your website, then it’s likely you won’t have the time to change every link. For the time being, 301 redirects will provide a band-aid until you can change all internal links.
  • 301 redirects help you stay visible to search engines. Search engines spend years and years getting to “know” your website through indexing your site on a regular basis, analyzing links to and from your website, and detecting how much content you produce. If you produce a new website with a new URL with no connection at all to your old website, then search engines will have to get to “know” your new website from scratch. Then it may take a long, long time before you start appearing in search engine results again. For example, if a citizen types in your city’s name into a search engine, your website may not even show up in the first 10-20 search results. 301 redirects smooth things over—essentially letting search engines know that you’ve moved and keeping your city high in the search results.

While just two simple best practices, these are technical aspects of your website that need to be managed and overseen by experienced website professionals. Otherwise, like Springfield, Florida, you may end up losing ownership of your old URL, failing to redirect citizens to your new website, and then letting them think that your city’s website apparently has switched over to providing porn.

Want to ensure that you prevent a similar disaster? Reach out to us today.

Tuesday, May 02, 2017
Brian Ocfemia, Technical Account Manager

Brian OcfemiaWe often talk about data backup as the best remedy for a virus infection. If the worst happens and a virus takes your systems down, then you just restore an uninfected backup.

However, a recent article concerning Bingham County in East Idaho brings up an excellent question: What happens if your backup servers get infected?

The Idaho State Journal reports that “[Bingham] County information technology staff thought the virus was contained but discovered [on February 17, 2017] that one of the backup servers had become infected, knocking the entire system offline.”

Luckily, the county had some other data backups in place to mitigate damage from the ransomware virus attack. But this scenario offers a good lesson. Let’s address several technology pieces that need to be in place to prevent a virus from infecting a backup and permanently destroying your data.

We’ll assume in our discussion that a city already has some type of data backup solution along with antivirus software in place.

1. Monitoring and Alerting

It’s bad enough to get a virus. It’s worse if that virus goes undetected. Many modern viruses often mask themselves, retreat to the background, and do malicious things to your systems such as collect financial information. The longer the virus lurks in the background, the more it can spread and the more damage it can do.

Cities need proactive monitoring and alerting through a combination of automated software that tracks technology health combined with experienced IT professionals watching your systems. Part of that monitoring and alerting involves the right kind of antivirus software. We recommend enterprise-grade antivirus software that offers sophisticated monitoring tools for IT professionals to track and catch viruses.

2. Unlimited Offsite Data Backup Storage

Let’s unpack this phrase a bit.

  • Offsite data backup: In addition to backing up your data onsite, you need an offsite data backup component for worst-case scenario disasters such as tornadoes, flooding, or fires.
  • Storage (and retention): You will need to store various snapshots of your backed up data and make them available in case you need them. For example, you may need to see a snapshot of data as it looked one month ago if certain documents recently went missing. The right storage strategy allows you to maintain all versions of your files and documents while also retaining them for a set period of time.
  • Unlimited: This is key to rarely (if ever) worrying about a virus or ransomware attack. Let’s go back to the example of Bingham County and assume the ransomware virus lingered around for a long time, infecting even backup files. Some recent critical data might unfortunately get lost, but the county could still go back—as far as it wanted—to a snapshot of its data right before the infection hit.

It’s important to note that if you don’t have enough storage for a reasonable backup retention period, you may be stuck in a situation where the only files you can restore are infected ones. We recommend an unlimited offsite data backup storage service that allows you to keep your offsite backups indefinitely. Then, you can go back in time as far as you need to recover files.

3. Employee Education

The Idaho State Journal article goes on to state:

“An information technology director for a neighboring East Idaho county said emails with suspicious attachments can often cause computer systems to become infected. He said his systems manager comes across up to three such emails per week.”

Despite the best cybersecurity protection and data backup, employee education remains an essential part of your strategy. Antivirus and antispam software can help prevent access to many malicious websites and email attachments. But employees still need to learn more about what not to click on and how to spot hacking and phishing attempts.

Some things you need to talk about with employees include:

  • Browsing safely and knowing the signs of a malicious website.
  • Scrutinizing email attachments and understanding that hackers can spoof email addresses (such as an email supposedly coming from their boss).
  • Downloading unnecessary or unauthorized software from untrustworthy sites (such as games, shopping apps, and productivity apps).

As we see from this situation, there’s more to backing up data than just backing up data. You need to stay vigilant through proactive monitoring and alerting. You need to retain data snapshots that go far back in case your backups get infected. And you need to keep training employees who often unknowingly take actions that let in viruses and hackers.

Worried about what would happen to you if a ransomware virus hit? Reach out to us today.

Tuesday, April 25, 2017
Nathan Eisner, COO

Nathan EisnerIn past blog posts, we’ve talked about the importance of data backup for body camera video (and other police department video). We always ask, “What happens if evidence is permanently lost?”

This situation recently arose with the Cockrill Hill Police Department in Dallas, Texas. After a ransomware attack on the police department’s servers, the municipality permanently lost data. According to a Mother Jones article (with my added emphasis in bold):

The police department claimed that they still had paper copies of all the documents on the server and physical copies of much of the video. But in a letter sent to the county prosecutor, the department said "all bodycam video, some photos, some in-car video, and some police department surveillance video were lost." The department tried to recover as much as possible but said that "if requests are made for said material and it has been lost, there is no chance of recovery or producing the material."

The article opens by telling the story of a defense lawyer working for a client who faced prison time. He needed specific evidence to help his client avoid jail time. That evidence? Permanently lost. In other words, the data loss—rooted in a technology problem—could literally send a person to jail or serve a longer prison sentence because important evidence disappeared forever.

What this Mother Jones article doesn’t address is how easily this loss of data could have been prevented.

The Best Weapon Against Ransomware? Data Backup

While you may protect yourself against ransomware in many ways, the worst scenario may still happen. An employee clicks on a malicious email. Hackers break into a server. Lack of up-to-date patches expose your software to a major security flaw. It happens.

Like insurance, your technology needs to prepare for the worst. Only data backup can fully “insure” you against ransomware. Let’s say the worst happens. Ransomware is downloaded, you receive an automated blackmail threat, and you (wisely) decide not to pay the criminals. You permanently lose that data. But luckily, you have a backup you can restore. You may end up losing none of the data, as little as only minutes of data, or, at worst, hours or days of data.

For data backup, you need:

  • Onsite backup that takes frequent snapshots of your data. For smaller disasters (like files lost or a server failure), you can recover quickly.
  • Offsite backup that sends your data to a geographically distant data center (or centers). Then when a disaster wipes out your onsite data, you still have all your data safely stored offsite.
  • Regular testing (such as quarterly) so that you know your data backup works. Too many cities never test their data backup and they often find it doesn’t work when a disaster actually hits.

Because body camera, dashcam, and other police video requires massive amounts of video storage, it’s wise to explore data backup solutions with unlimited offsite storage. You don’t want to lose data arbitrarily because of storage caps or added costs. Unlimited offsite storage also gives you flexibility with data archiving and retention to help you follow the law.

The Second-Best Weapon Against Ransomware? Proactive IT Monitoring and Management

On the preventative side, it’s essential for police departments to hire staff or a vendor that proactively monitors and maintains technology for servers, desktops, and mobile devices. That includes:

  • Shoring up cybersecurity weak points in your network through locking down and properly configuring your computers, servers, switches, routers, and firewalls.
  • Monitoring your technology’s performance and health 24x7x365 and receiving alerts about problems.
  • Using antivirus, antispam, and content filtering software to help employees with safe internet browsing and email.
  • Consistently applying updates and patches to your software.
  • Ensuring any remote access is secure when teleworking.
  • Managing and tracking all technology assets.

While articles like the one we’ve referenced from Mother Jones seem to indicate that failure can be shrugged off without consequences, that may soon no longer be the case. Federal and state laws and regulations increasingly push for higher cybersecurity accountability from government entities. Even at best, these incidents are an embarrassment for cities and, from an ethical perspective, negatively impacting the lives of defendants (especially if they’re innocent of a crime), defense attorneys, and prosecutors who rely on this evidence to uphold the law.

Would your city’s police department survive a ransomware attack? Reach out to us today if you’ve got any doubts.

Wednesday, April 19, 2017
Mike Smith, Network Infrastructure Consultant

Mike SmithLicking County, a county east of Columbus, Ohio, recently experienced a bad ransomware attack on its IT systems. Ransomware is a specialized virus that encrypts files—making them nearly impossible to access unless you pay criminals a ransom. Cybercriminals use ransomware to extort money in return for unlocking your files. Many organizations pay the ransom despite the FBI and other law enforcement agencies recommending against it.

Luckily, Licking County managed to mostly survive the attack based on implementing some important best practices. Let’s look at the good, bad, and ugly of this situation to extract some important lessons.

The Good

Data backups

The difference between getting crippled and devastated by a ransomware attack versus surviving it relatively unscathed all comes down to data backups. Licking County ended up losing only about one day’s worth of data for most systems. Another county referenced in the article ended up paying a ransom of $2,500 to cybercriminals because they did not invest in data backup.

Activating a plan to shut down the network

To stop the spread of the ransomware, Licking County shut down its network. Clearly, the county had a plan in place and enacted it when the ransomware virus hit. By planning ahead, they were best prepared for what to do to keep the virus contained and to minimize impact.

Rebuilding systems based on highest priority data

As part of its disaster recovery plan, the county rebuilt its systems based on the highest priority data first. The article references data such as “servers that house felony-case tracking for the prosecutor's office and the auditor's property-records database.” Any disaster recovery plan needs to have a clear plan as to how data will be restored—and in what order of priority.

The Bad

Rebuilding systems will take a lot of time

Licking County is a big county and so it needs to reformat about 1,000 computers as part of its rebuild. That takes a lot of time. Even smaller organizations will need to spend significant time rebuilding servers and reformatting computers.

Direct and indirect costs

Directly, the costs of billable IT time and possibly enhancing networking equipment and cyber protection software can present a big hit to your budget. Indirectly, lost productivity wastes expensive employee salaries and potentially delays major projects when time is ticking.

Impacts to citizen service

After a disaster, a crippled government entity will not be able to serve citizens at full capacity. The mission of government gets impacted when ransomware hits. County Commissioner Tim Bubb says, “We have lost a large part of our focus on serving the people of Licking County. What price do you put on that?"

Potentially weak firewall and network connections

A Columbus Dispatch article mentions that the county needs to shore up its “firewall and network connections.” An improperly configured firewall can leave ports open that allow hackers to easily gain access to servers and steal information. Setup of switches, routers, and other networking equipment also impacts security.

Potentially weak passwords

The same article mentions that the county needs to encourage employees to change passwords more frequently. In a recent blog post, we said, “The longer a password is in use, the more likely that hackers will be able to crack it. The more you change passwords, the more difficult you make a hacker’s job.”

The Ugly

911 dispatching affected

An article published in the Newark Advocate the day after the incident stated “...the 911 Center has been operating in manual mode since late Tuesday night. The 911 Center phones and radios work, but dispatchers do not have access to their computers. The public can still call 911 for emergency police, fire or medical response.”

While not completely shut down, any impact to 911 or other critical emergency services can literally affect lives in the wake of a ransomware attack.

Employees click on too many suspicious emails

One of the biggest cybersecurity threats is people. No matter how great your data backups, antivirus, firewalls, and security measures, hackers and cybercriminals still often break into a government entity through people clicking on suspicious websites and email attachments.

Note this paragraph in the Columbus Dispatch story:

Fairfield County started working last year to tighten procedures to guard against the type of cyberattack that occurred in Licking County, said Fairfield County IT Administrator Randy Carter. He said he was dismayed when he sent a test phishing email to county employees in September and more than 25 percent clicked on it. Carter plans to provide training to employees on what emails to avoid.

25 percent! One in four people got fooled by these dangerous emails. Each click on one of these emails opens you up to the threat of a virus or ransomware.

Cybercriminals targeting government more and more

Cyberattacks grow more numerous and targeted. Government entities are ripe for these attacks. That includes cities.

Are you prepared?

  • Like Licking County, do you have data backups to recover from a ransomware attack?
  • Do you have the right network equipment and modernized technology to protect yourself?
  • Are your employees trained about the dangers of clicking on malicious emails and websites?

If you need help protecting yourself from a ransomware attack, reach out to us today.

Tuesday, April 11, 2017
Brandon Bell, Network Infrastructure Consultant

Brandon BellA city had operated for a long time with tape backup and decided to upgrade. City administrators heard from their IT staff that they needed something more reliable than a manual solution reliant upon busy people to both conduct the backup and store it offsite.

Spending a lot of money on a modern complex data backup solution, the city was assured by its IT staff that this automated beast would solve all their problems. Indeed, the data backup worked automatically. In a meeting, IT staff showed city department heads the wonder of the data backup system by retrieving a few PDF documents from the backup data storage. To city council and the public, the city administrator proudly said they had ticked data backup off their list. Problem solved!

One day, a fire tragically tore through most of city hall. The building ruined, city staff needed to relocate to a temporary building until a new city hall was built. But thank goodness—despite all the servers destroyed—that the city could retrieve its data.

Or not. When IT staff attempted to restore the city’s data through its backup, most of the major databases, applications, and data would not restore. A few chunks of data—like some people’s individual documents—were okay. But the city’s most important information was not there.

And expensive backup solution became nearly worthless. Why? Upon further investigation, the city administrator was told that nobody ever tested the data backup. “ was an expensive solution,” the city administrator said. “And my IT staff said that it was automated. The data backup solution’s reporting even said it worked.” didn’t. And that’s all that mattered when the city administrator had to now explain why this expensive investment failed them after a disaster—and failed to do the exact thing it was supposed to do.

Preventing This Disaster

One aspect of data backup and disaster recovery—testing—is nearly as crucial as simply having data backup at all. No matter what kind of data backup you use, you need to test it. Otherwise, you don’t know that it’s working.

Let’s look more closely at the errors in our city scenario above.

Error #1: Assuming the data backup works.

A data backup solution will often look like it’s doing its job. From manual solutions like tape to more sophisticated automated data backup servers, the data backup application will often indicate that the process is a success or failure. But no matter what the application tells you, you don’t know that it works until you test it.

Error #2: Not testing all the backed up data.

Calling up a few files such as PDFs from the data backup storage is not testing. When a disaster hits, you will need to be fully operational with your databases, software applications, website, email, and documents. For example, will your account system work from a backup copy? When you test, test everything. Simulate what would happen if an actual disaster hit.

Error #3: Develop a plan and document it.

Testing needs to be part of your overall disaster recovery and business continuity plan. The act of testing not only guarantees you will access the data but also allows you to practice how data recovery will work. Who does what? How fast will the data be restored? In what order? Where will you access the recovered data?

You want to run into issues during testing and deal with them in a simulation—rather than after a real disaster.

Uncertain about your data backup solution? Are you testing it at least quarterly? Reach out to us today.

Tuesday, April 04, 2017
Jabari Massey, Network Infrastructure Consultant

Jabari MasseyOn the surface, a coastal city did some correct things to back up its data. The city had a few servers in a physically secure basement room that were well-maintained by IT staff. One of the servers backed up important data. In case a server failed, the backup server would run until the city could replace the original server.

A long time had passed since the city last experienced a hurricane. When a hurricane finally seemed eminent, the city was ordered to evacuate until the massive storm passed. The city manager and IT staff didn’t think much about the servers other than placing them upon concrete blocks in case of flooding. As long as the city implemented its emergency action plan and evacuated everyone safely, the city manager assumed its information technology would remain safe.

After the hurricane passed, city staff returned to find that no massive devastation occurred but they did experience heavy flooding. The IT staff had placed the servers upon concrete blocks as a precautionary measure, but they learned an incredibly hard lesson in hindsight.

Located in a basement room, the servers sat below sea level. Although the rest of city hall experienced moderate flood damage in places, the basement had filled up to dangerously high levels. All of the servers—including the backup server—were rendered unusable by the flooding.

With a sinking feeling, the city manager realized all critical data—including financial, public safety, document management, email, and website data—was gone. The only backup server got destroyed along with the others. It might be easy for the city manager to point some blame in the direction of the IT staff, but it was well-known that he had refused requests to explore other data backup options because of “budget concerns.”

Now, the mayor, city council, the media, and public would be asking questions.

Preventing This Disaster

Sure, the city manager and IT staff made a bad decision to place servers in a basement room below sea level. But their errors go deeper than this poor choice of physical location for the servers.

Let’s look at the errors in the story above.

Error #1: Locating servers in a flood-prone area of your building.

Getting the most obvious error out of the way, it’s clear that the servers needed to reside on an upper floor. In addition, the server room needed to be in a room that mitigates flood risks through preventative measures such as water leak sensors or eliminating areas where water can enter.

Error #2: Lack of offsite data backup.

While locating the servers on a higher floor may have prevented this immediate flooding disaster, it’s still not a full disaster recovery plan. Anything can happen to your technology onsite. To guarantee full recovery of your data after a disaster, you need an offsite data backup component to your emergency plan.

We recommend storing your data offsite in geographically dispersed locations (such as in data centers both on the East and West coasts). Then, even if the worst disaster wipes out your buildings, you will be able to recover and access your data.

Error #3: Lack of technology planning.

The lack of offsite data backup also signifies a larger issue—a lack of planning. The city had developed an emergency plan and used it in the case of the hurricane. But when was the plan developed? When was it last updated? Did it include technology-related scenarios? What was the plan to protect data in case of a disaster?

First, the city needed to update its emergency plan and include technology. That would have addressed technology-related gaps in the city’s data backup, disaster recovery, and business continuity plans. Second, the city needed regular technology planning meetings (at least once a quarter) and ongoing monitoring to ensure that data backups were tested and working. This regular monitoring and planning would help the city adapt to changes (such as new technology, more staff, building changes. etc.) and ensure that the risk of data loss is minimal.

Flooding is one of the most common disasters. It can happen anywhere in the country and devastate a city. Because citizens will rely on your city after severe flooding, you must be operational as fast as possible. That means having access to your data—your website, your documents, and your applications that are essential to operations.

By developing a disaster recovery plan that includes an offsite data backup component, you will lessen the risk of permanent data loss and angry “Why?” or “How?” questions after the fact from council, the public, and others.

Concerned about your data backup and disaster recovery? Reach out to us today.

Tuesday, March 28, 2017
Ryan Warrick, Network Infrastructure Consultant

Ryan WarrickIn recent posts, we’ve talked about disasters at cities that result in permanent data loss, incredible damage to city operations, and city department heads wondering if their job is now at risk—all sadly because of preventable risk. The stories we use to illustrate these disasters—and the lessons learned—are based on a combination of many, many scenarios we’ve witnessed at cities throughout the years.

However, we recently saw a story that’s quite specific to one city and a very public, front page news illustration of some important IT-related lessons. Let’s look at what happened to the City of Miami Beach, Florida in December 2016.

Third Parties Steal $3.6 Million—and No One Notices for Six Months

In a nutshell, unknown third parties stole the account and routing numbers from the city’s banking account. According to the Miami Herald, the criminals “[rerouted] automatic payments intended to pay vendors and other government bills.” The criminals did it for six months and stole $3.6 million before staff in the finance department noticed.

We carefully reviewed the Miami Herald article and the city manager’s report. While this crime is a form of cybersecurity, the situation also includes lessons about IT-related processes and controls that are incredibly important to cities. A few bad practices stick out from our analysis of the report that cities need to avoid.

1. Completely ignoring basic, elementary best practices.

The city of Miami Beach was offered free fraud control tools when they set up the account in 2012—the same kind of fraud control tools that many individual banking customers enjoy. Did the city take advantage of these tools? No. Maybe they had a reason at the time such as wanting to implement their own fraud controls. If so, that never happened.

Cities need to stay aware of and implement important best practices that help mitigate information security risks. In this case, both finance and IT staff needed to say “yes” to such an obvious best practice back in 2012.

2. Using easy-to-steal information as authentication for financial transactions.

Think about how many people in a city can take a quick peek at a check. If third parties could steal city money through only this information, then the city had a security vulnerability that was wide open for people to exploit.

We find that cities also have similar weaknesses in areas such as passwords, unencrypted wireless devices, and website hosting that makes it easy for hackers to exploit security vulnerabilities.

3. Apparent lack of financial data oversight.

In a recent post about data processing, we said, “Experienced IT professionals should monitor everything related to your data processing such as transactions and processing, errors and incorrect information, overrides, unauthorized use of the application (especially when it appears that someone is altering data or ignoring/tampering with processes), reconciliations, and application performance (such as after a power outage or server failure).”

Obviously, finance department staff have an even more important role in monitoring this information too. While online banking is great, it’s unwise for even an individual consumer to not regularly review banking transactions. Great risk was introduced by not reviewing for six months and hoping that everything was okay. Cities need to become more proactive at monitoring and reviewing important aspects of their operations where data changes constantly—from accounts payable to information technology.

4. Lack of modernization.

Many cities often hear the word “modernize” and think of it as “unnecessarily wasting money or time on something new and fancy that we don’t need.” Sure, some solutions might fit that definition. But technology modernization is important especially when your old technologies and processes lead to security vulnerabilities, inefficient operations, and significant liability.

In the case of Miami Beach, the city manager’s report includes many “sudden” modernizations in one fell swoop such as ACH fraud controls and using UPIC (Universal Promotional Identification Code) to avoid sharing confidential banking information. The city manager even notes in the report that “the ACH Fraud Control program already prevented an unauthorized ACH transfer.”

I know we beat this drum a lot. But why do cities wait? Why do cities put off modernizing their technology and processes until a massive crisis hits? We see this “putting off” logic holds true at many cities for data backup, disaster recovery, website hosting, records and document management, email, and aging hardware. In all of these cases, lack of modernization increases the risk of a significant city incident or disaster.

Learn from cities like Miami Beach. Are you sure that fraudsters aren’t currently stealing money from you? Is your technology modernized in such a way that you aren’t headed for a major disaster like permanent data loss?

If you are worried about addressing critical technology aspects of your city before a disaster happens, reach out to us today.

Tuesday, March 21, 2017
Victoria Boyko, Software Development Consultant

Victoria BoykoDespite the perceived importance of ADA-compliant websites, many city websites do not comply with best practices that help disabled people access content. While ADA, W3C, and other organizations provide detailed guidelines and best practices, very few enforceable laws exist to keep cities accountable. Plus, even if a website designer follows all ADA best practices, a city employee may upload content to the city's website that doesn’t meet these requirements.

While some signs exist that the Department of Justice may create enforceable ADA-related website regulations in 2017, it’s not definite at this time. But that doesn’t mean your city should ignore ADA-compliant website best practices.

By making your website ADA-compliant, you:

  • Help extend your website services to disabled people.
  • Improve the overall functionality of your website.
  • Anticipate following future laws and regulations that may be expensive to correct later.

If you haven’t thought about ADA compliance for your website, then where should you start? While existing guidelines cover a lot of technical ground, here are some best practices that should be easy to tackle with the help of your website designer and whoever creates and uploads content to your website.

1. Describe images with text.

Many people just upload an image to a website as quickly and simply as possible. However, there should be an option on the back end of your website to provide alternative text (or “alt text”) for an image. For example, if you place a picture of city hall on your website then the alt text may say “Picture of city hall on a sunny day.” If someone is blind or cannot see very well, they may use a screen reader tool that describes all images on a page. When you fill out the alt text, you make images “readable” and accessible to people with vision problems.

2. Provide alternate ways to access video and audio content.

Videos and audio files (like podcasts) have become more and more embraced by cities. But what if someone can’t see a video? Or what if someone can’t hear the audio? Provide alternate ways for people to access the content. For example:

  • Offer closed-captioning for videos with audio content. Some video services will do this automatically for you (although it’s a good idea to spot check the quality of the closed-captioning) or you can do it manually.
  • Offer transcripts for videos and audio files.
  • In some cases, a summary description may be sufficient for visually-heavy videos with little spoken word or a lack of heavy substance.

3. Provide a clean, simple navigation and website structure.

If your website is a structural mess, then it will be even worse for people with disabilities who try to navigate it with screen readers or keyboards alone. Your website’s information architecture (meaning the way your webpages are structured and organized) needs to be as simple and clean as possible. For example, you wouldn’t want to clutter your homepage with a dozen things about your city’s history while barely mentioning or providing links to your most important city services.

4. Work with your designers to ensure that people can adjust colors and font sizes with ease.

Many disabled people with vision problems often need to adjust the contrast and sizing on their computers to see what’s on their screen. While the design specifications for ensuring ADA compliance are complex, most modern websites allow disabled people to adjust contrast and sizing. If you’re not sure about your city’s website (especially if you haven’t modernized it in a long time), then ask someone with website design experience to help you assess this aspect of accessibility.

5. All content should be accessible by keyboard alone.

Some disabled people cannot use a mouse and click on website content such as buttons or links. They need to rely only on a keyboard to get to it. If you have content on your website inaccessible by keyboard, then make it accessible as soon as possible. You should also consider adding a “skip navigation” link so that keyboard users can skip the often long navigation tabs (the ones seen on every page). That will save those people from wasting a lot of time.

6. Avoid flashing images.

Luckily, most modern websites avoid flashing images because they look tacky. However, if you are tempted to use them then consider that they may cause seizures in some people.

7. Follow writing best practices.

Write simply, clearly, and concisely. This is a good best practice anyway but it also helps disabled people who need information stated as clearly as possible. Rambling text, typos, and bad grammar prevent you from communicating to your audience. Consider hiring a professional writer to write your content if you’re unable to ensure a high writing standard.

8. If you hyperlink text, then make sure it’s descriptive.

“Click here” is not descriptive. “January 5, 2017 City Council Agenda” is descriptive. When disabled people use screen readers, they often look for links to take them to another webpage. Make the text you hyperlink contain a specific description instead of something vague.

9. Offer an alternate version of PDF documents.

Unfortunately, screen readers cannot read PDF documents. If the thought of converting tons of PDF documents to HTML or rich-text format horrifies you, then talk to your IT staff or vendor. You may be able to find a tool that can convert your PDFs to HTML. Then, it’s a matter of going through the PDFs you offer on your website and creating HTML versions of each document.

10. Avoid cutting and pasting pre-formatted content to your website.

When city employees upload content to websites, we often find that they make the mistake of posting pre-formatted content. For example, people may cut and paste content from a Microsoft Word document to the city’s website. The problem? Microsoft Word content contains a lot of HTML code that makes sense when you’re working in Microsoft Word—and not so much sense when you transfer it somewhere else. That’s why what looked great in your word processing software can look awful on your website.

Usually, cutting and pasting into Notepad first (a free application that comes with nearly all computers) and then cutting and pasting the Notepad version into your website’s content management system will remove junk formatting and convert your words into clean, plain text.

Following these best practices will give you a good head start for making your website ADA-compliant. For more detailed best practices, refer to the following resources.

Website Accessibility Under Title II of the ADA

Web Content Accessibility Guidelines (WCAG) 2.0

Need help assessing the ADA compliance of your website? Reach out to us today.

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 |
Contact a Sophicity Consultant Now To Find Out How We Can Help Reduce Your IT Costs Go