Thanks to the recent 2016 Data Breach Digest from the Verizon RISK team, a story about poor cybersecurity at a water company has recently been in the media. The company, given the fictional name of the Kemuri Water Company (KWC), is responsible for the water supply of a number of counties.
Verizon was retained by KWC to assess their networks for indications of a security breach. Of concern was an unexplained pattern of valve and duct movements that had occurred over the previous 60 days. The Verizon team identified that breaches had occurred and that KWC’s SCADA system had been manipulated to alter the amount of chemicals that went into their water system as well as the flow rate.
Several significant security weaknesses were identified, including poor network segmentation and the lack of use of Defense in Depth best practices. Let’s take a closer look at the KWC situation and compare it with another water/wastewater company that proactively segmented and secured its systems.
A hactivist cyberattack impacted one water system by altering flow rates and chemical quantities. The cyber incident was not immediately identified as such, which is unfortunately common with industrial networking systems.
SCADA Platform Hosts Critical OT and IT Applications
The Verizon team looked at all of KWC’s IT and OT systems. These systems support end users and corporate functions, as well as the distribution, control and metering of the regional water supply.
The OT end of the water district relies heavily on computer systems running operating systems from ten-plus years ago (possibly Windows XP). Critical OT and IT functions run on a single AS400 system referred to as the “SCADA platform.” Prior to the breach being identified, this system:
- Functioned as a router with direct connections into several networks
- Ran the water district’s valve and flow control application that was responsible for manipulating hundreds of Programmable Logic Controllers (PLCs)
- Housed customer billing and personally identifiable information as well as KWC’s financials
- Was administered by only a single employee
- Had open access to the Internet
- Exposed its internal IP address and administrative credentials in clear text within an initialization (.ini) file
Data Breaches Steal Customer Information and Modify ICS
The Verizon investigation showed that at least four data breaches had occurred in a 60 day period and 2.5 million unique records of data containing personal and payment information were stolen. Fortunately the data was not used for fraud or financial gain; rather the attackers seemed to be hactivists whose motive was to create business disruption.
The same credentials used to steal customer information were used to interface with the water district’s valve and flow control application. Application settings were modified in a manner that suggested little knowledge of how the system worked.
In at least two instances, the attackers managed to manipulate the system to alter the amount of chemicals that went into the water supply. This impacted water treatment and production, increasing the time it took to replenish water supplies. Fortunately, based on alert functionality, KWC was able to quickly identify and reverse the chemical and flow changes, largely minimizing the impact on customers.
If the attackers had more time and more knowledge of the ICS/SCADA system, KWC and the local community could have suffered serious consequences.
Verizon Recommends Defense in Depth
Verizon worked with KWC’s IT administrators to:
- Shut down access to and from the account management web front end that was the entry point of the attacks
- Block Internet and outbound connectivity from the AS400
- Rebuild affected systems from baseline images
The RISK team then recommended:
- Replacing older systems with newer ones and applying patches as necessary
- Having multiple AS400 administrators working under oversight
- Moving away from convenient configuration and security settings to ones that are more secure
- Improving network design so SCADA management systems are not directly connected to the Internet and so that critical automation assets are isolated and protected
- Implementing a layered Defense in Depth strategy which could have prevented the attacks or limited their success
ISA IEC 62443 Standards for Effective ICS Security
Let’s now look at a contrasting story. It starts when a Dept. of Water Resources upgraded their SCADA network to industrial Ethernet in 2012. Like the KWC situation, at the time there was little protection or separation of the SCADA network from the IT network.
However, the team involved, led by the plant electronics technician, recognized there was a security issue and took the initiative to find ways to improve cyber defenses. This was started by attending the ISA Water/Wastewater and Automatic Controls Symposium and taking a one-day security course on the fundamentals of the ISA IEC 62443 cybersecurity standards.
Subsequently this city decided to implement the zones and conduits network segmentation model that is part of ISA IEC 62443. They then used industrial firewalls (specifically Tofino Security Appliances) as the conduits protecting SCADA systems. Below is a simplified network diagram of their solution.
This SCADA wastewater network is segmented according to ISA/IEC 62443 standards and is secured using Belden’s Tofino Security Appliances. Downloadable Image
Although the Tofino Security Appliances were easily installed in the network, support was required when it came to configuring and managing the quantity of traffic on the network.
Through Belden’s strategic partnership with exida – a firm specializing in industrial automation safety and cyber security services – exida Senior Cyber Security Engineer Eric Persson arrived on-site for two days of training and testing. The training included everything from baseline networking knowledge to hands-on installation and troubleshooting.
“The goal of our training was twofold – first, to have fully functional Tofinos in place to protect the critical areas and assets of the plant from malicious traffic and activity, and second, to have our customer fully competent and comfortable in the networking and configuration knowledge necessary to maintain these devices moving forward,” said Persson.
During the configuration, several key steps were taken to ensure no disruption to the active network. This included notifying the control room that maintenance was underway on the network and using Tofino’s test and passive modes to see the traffic flowing through before putting the security appliance in operational mode.
The training, commissioning and testing process also included the creation of custom rules to manage the facility’s network traffic. Firewall and traffic rules were set up to meet their specific needs.
Finally, the exida team ensured that the security implementation met the recommendations of the ISA IEC 62443 cyber security standards.
A Practical Approach to Industrial Cyber Security
The intention here is not to embarrass “KWC” – their situation is by no means uncommon. Instead, we want to highlight the risks associated with weak cyber defenses and show how it is possible to move forward to protect critical infrastructure assets with minimal disruption to system operations.
Nowadays, the risk to water utilities of a cyber incident impacting operations is significant. While plant staff may not be experts in cyber security, it is possible to improve cyber defenses through training, adopting best practices, partnering with vendors with the right expertise and using security technologies designed for industrial facilities.Is limited cyber security knowledge holding back your team from improving protection measures? What challenges do you face in implementing Defense in Depth? I look forward to hearing you.
- Verizonenterprise.com Download Page: Data Breach Report
- Isssource.com Water Company Operations Breached
- Blog: PLC Security for Water / Wastewater Systems
- IIConsortium.com: Belden Wastewater Systems Case Study
- ISA.org: 2016 ISA Water/Wastewater and Automatic Controls Symposium
- Product Bulletin: Tofino Xenon Security Appliance
- Blog: Improving Control System Security with ANSI/ISA-99 Standards