The Need to Change the Paradigm of Control Systems Cyber Security – Article 1/3: Background
Visit Article 2/3 Visit Article 3/3
Control Systems Cyber Security – There have been many articles, webinars, and even books written on cyber security of control systems. I put them into three bins. The first bin includes those presentations and papers that apply to “keeping lights on and water flowing”. The second bin includes those presentations and papers that apply to Operational Technology (OT) networks but do not include “keeping lights on and water flowing”. This is generally where the IT/OT convergence discussions lie. The third bin are those presentations and papers that are factually not correct or not applicable to control systems. Unfortunately, there are very few articles and discussions that fit into the first bin. Most fit into the second bin.
Consequently, I have prepared a three-part series with the first part describing the history of control system cyber security and differences between control systems and networking. The second will provide actual control system cyber incidents to demonstrate control system cyber security is a real issue. It will also address the lack of control system cyber forensics and cyber security training for the engineers. The third will address the hole in control system cyber security – the lack of cyber security, authentication, or cyber logging in Purdue Reference Model Level 0,1 devices (e.g., process sensors, actuators, drives, etc.). Without the ability to validate the integrity and authenticity of the sensor input, OT cyber security is based on untrusted input.
Control system cyber security has been an identified issue since Presidential Decision Directive 63 in 1998. The purpose of control system cyber security is to protect the control systems and the processes they monitor and control from electronic threats. That is “keep lights on and water flowing”. Networks are a support function in the overall objective of safety, reliability and productivity of the process. Yet, the instrumentation and control (I&C) systems used in the physical infrastructures are still not cyber secure. In fact, much of instrumentation and low-level instrumentation networks may not be able to be cyber secured. Control system cyber security is more than just attacks – they can also be unintentional incidents. Control system cyber security does not need to be just against “critical infrastructure”.
Examples of control system cyber attacks include damage to not only traditional critical infrastructures, but also to buildings/data centers, medical devices, transportation, and space.
Cyber Incident – The unofficial IT definition of a cyber incident is the system is connected to the Internet, is using Windows, and the attacker is maliciously compromising the data. Effectively, protecting data is referred to as “Information Assurance.” This also implies that cyber vulnerabilities are important for network security and need to be expeditiously addressed regardless of process system impact. Yet, the most important factors for facility operations are reliability and safety – not data. The NIST definition of a cyber incident in FIPS PUB 200 is: “An occurrence that actually or potentially jeopardizes the confidentiality, integrity, or availability (CIA) of an information system or the information the system processes, stores, or transmits or that constitutes a violation or imminent threat of violation of security policies, security procedures, or acceptable use policies.”
This definition is more relevant to the control system community with one critical modification: the definition needs to add the letter S (Safety). It is also important to note that the term “malicious” is not mentioned in the NIST definition. Effectively, this is Mission Assurance, which means cyber vulnerabilities are important if they can impact the mission.
The additional reasons for not using the term malicious is the lack of adequate control system cyber forensics as well as lack of sufficient control system cyber security technologies. In many cases, the only difference between an incident being malicious versus unintentional is the motivation of the individual involved.
Operational Technology (OT) – Hardware and software that detects or causes a change, through the direct monitoring and/or control of industrial equipment, assets, processes and events. OT is not the pumps, valves, or other hardware nor does it include the engineers and technicians responsible for the equipment.
IT/OT Convergence – The integration of IT technology with OT systems. Generally, this occurs at the network level.
Cyber security became an Information Technology (IT) issue after the Morris worm of November 2, 1988 – usually considered the first computer worm and certainly the first to gain significant mainstream media attention – was distributed via the Internet. This worm resulted in the first conviction in the US under the 1986 Computer Fraud and Abuse Act. IT cyber attacks have continued unabated, leading to wide spread attention and legislation. IT cyber security threats also led to the development of cyber security policies within ISO/IEC27000. This is part of a growing family of ISO/IEC Information Security Management Systems (ISMS) standards within the field of information and IT security. Standards include general methods, management system requirements, techniques and guidelines to address both information security and privacy. These standards are IT-focused and do not address the unique issues associated with control systems including reliability and safety. This led to the establishment of ISA99 which is developing the suite of IEC62443 series of Automation and Control System Cyber Security Standards specific to automation and control systems (Figure 1). IT security also has a focus on the Open System Interconnection (OSI) model SI 7-layer model which is a conceptual model that characterizes and standardizes the communication functions of a computing system whereas control systems use the OSI model but also use the Purdue Reference Model.
Figure 1 ISA/IEC62443 Control System Cyber Security Standards
In the mid-1990s, while managing the Electric Power Research Institute (EPRI) Fossil Plant I&C Program, I was conducting research and development to increase the use of digital controls and develop predictive controls and predictive maintenance methodologies. The intent was to eliminate “islands of automation”, which would result in reliability, safety, and productivity improvements. However, there was an unforeseen side – cyber vulnerabilities. Because of the lack of cyber security as a design consideration many I&C systems utilized “design features” that provided what was considered needed functionality but later would be considered cyber vulnerabilities. These design features included “back doors” for the control system supplier to have remote access, hard-coded default password that could not be changed, hardcoding the frequencies to avoid variable frequency drives, use of Bluetooth with no security to access distribution reclosers, and capabilities to remotely disable battery operation in Uninterruptible Power Supplies (UPSs). Moreover, even when cyber security was included, the cyber security features were generally turned off by default. This is because security features such as anti-virus software could shut down older plant Distributed Control Systems (DCS) as they did not have sufficient computing resources to use the Anti-Virus software. However, the concept of cyber security often was consciously ignored as cyber threats were generally viewed as e-mail problems. What could cyber threats to e-mail possibly have to do with power plant or substation operation?
Following 9/11, cyber became national security (I do not a have a specific date or document). However, at this time, the cyber security function for control systems was moved to the IT organization. Engineering was no longer involved. Consequently, all cyber security monitoring and mitigation were at the IP network layer – network anomaly detection. As a result, control system cyber security went from being Mission Assurance to Information Assurance. As the engineering systems were not included under IT’s purview, the Level 0,1 devices were not included in cyber security considerations. These legacy engineering systems have no cyber security, authentication, or cyber logging nor can they be upgraded. The lower level sensor networks such as Highway Addressable Remote Transducer (HART), Profibus, Fieldbus, etc. also have no cyber security. This lack of focus on control system devices is still occurring. This total focus on the OT networks has led to what I call the second coming of the Maginot Line (Figure 2) That is, a disregard for any potential threat that is not IP network-focused. At least one nation-state has used that approach to bypass all cyber security protections that led to the issuance of Presidential Executive Order 13920.
Figure 2 The Maginot Wall and OT Cyber Security
A control system maintains or optimizes a process by monitoring and controlling physical inputs such as temperature, level, flow, voltage, current, etc. to meet desired physical outputs such as flow, power, color, size, etc. A safety system ensures the process does not exceed prescribed safety limits. Networks are a support function in the overall objective of safety, reliability and productivity. That is to optimize the process.
Control systems consist of process sensors connected to controllers, actuators, and HMIs (effectively, the control system network) as shown in Figure 3. The sensors and actuators operate almost exclusively in near-real-time (microseconds to milliseconds), whereas the Human Machine Interface- HMI (operator displays) provides operator information on the order of seconds to minutes. The sensors and actuators can operate, and in most cases were designed to function, without the Internet Protocol (IP) network.
Figure 3 Control Systems
Control and safety systems were designed to meet seismic, environmental, fire, operational, safety, regulatory, and other specified compliance issues. However, they were not designed to meet cyber security considerations. Consequently, cyber security needs to be assessed.
From a cyber security perspective:
– Sensor input: The input to controllers are generated from process sensors such as temperature and level sensors which continuously measure in milliseconds. The sensors have no cyber security, authentication or cyber logging capabilities nor do the sensor networks.
– Sensor protocols: The sensing input is generally serial communication that uses some form of communication protocol which is generally cyber insecure such as RS232, Modbus, Highway Addressable Remote Transducer (HART), etc. to communicate with the controller which can be a programmable logic controller (PLC). The PLC is a multipurpose machine that takes discrete (serial analog) or digital inputs and outputs Ethernet packets.
– Controller: The Programmable Logic Controller (PLC) accepts the insecure and unauthenticated process sensor serial input into the control logic as a “discrete input”. The PLC control logic compares the sensor input to the control requirements to determine if the controller needs to take real time action.
– If necessary, the PLC communicates with a “final element” such as a motor controlling a heater or a valve in milliseconds if action needs to be taken to meet the control requirements. The communication can be via insecure serial point-to-point or by secure Ethernet packets to the cyber insecure final element.
– The PLC also communicates with the HMI (HMI refresh rates can be on the order of several seconds or longer) using secure Ethernet packets based on insecure sensor inputs. The communications can assist the operator in making control decisions, provide system alarms, and provide detailed process data.
– From a personnel perspective, Operations and Maintenance is responsible for the process sensors, controllers, motors, heaters, and valves (there is generally no cyber security at this level). The cyber security organization is responsible for monitoring the Ethernet OT network, network infrastructure, and the HMIs.
There have been numerous catastrophic (upwards of several billion dollars) control system cyber incidents, many of which have resulted in injuries and deaths. In many cases, these accidents have not come from network problems, but from compromises or problems with control system devices. The real safety and reliability impacts come from manipulating physics, not data. This will be covered in more detail in Part 2.
Gaps in Understanding
There is a convergence of highly integrated industrial automation sharing more constructs with IT. As opposed to IT security, control system cyber security is still a developing area. Control system cyber security is an interdisciplinary field encompassing computer science, networking, public policy, and engineering control system theory and applications. Unfortunately, today’s computer science curriculum often does not address the unique aspects of control systems. Correspondingly, the electrical engineering power systems focus, chemical engineering, mechanical engineering, nuclear engineering, and industrial engineering curricula do not address computer security. Consequently, there is a need to form joint interdisciplinary programs for control system cyber security, both in the university setting as well as in industry. The cultural gap between the cyber security and engineering organizations is alive and well. The impact of this gap will be discussed in several actual cases with potential catastrophic consequences.
As previously mentioned, following 9/11, cyber security was moved to the IT organizations and became a network issue with engineering field devices no longer part of the cyber security process. The purpose became protecting networks, not physical processes. The move to IT effectively removed the engineering organizations from the cyber security process. Over the past few years, a new term has been added to the lexicon – “Operational Technology” (OT). OT ostensibly is everything that is not IT but that is not really the case. OT is the organization and people responsible for control system networks, not control/safety system field devices. OT is generally networking not engineering. Most control/safety and automation engineers, substation engineers, instrumentation technicians, relay technicians, etc. do not consider themselves to be OT. The control system software and logic are engineering software.
For example, Proportional-Integral-Derivative (PID) control system logic would not be considered OT, but the network infrastructure and monitoring of the computers containing the PID logic would be OT. Similarly, process sensors would not be OT but the sensor data after conversion to Ethernet packets would be OT. As such, the focus of OT cyber security has been on the network and workstation layers ASSUMING the sensors are secure (uncompromised), accurate, and authenticated, which may not be the case. Consequently, the culture gap between “packets (IT/OT) vs. process (Engineering)”. Monitoring of OT networks is necessary, but not sufficient, to protect control systems and processes. Another culture issue is the siloing of multiple functional areas affected by process sensors including cyber security, safety, alarm management, and device management organizations.
However, this doesn’t have to continue. A working group consisting of members from ISA84 (Process Safety), ISA99 (ICS Cyber Security), and ISA108 (Device Management) are currently working on the 3rd edition of ISA TR84.00.09, Cybersecurity Related to the Safety Lifecycle. This group has recognized that control system cybersecurity encompasses the entire system from the overlap with IT, boundary devices, networking devices, servers, Basic Process Control System (BPCS) controllers, Safety Integrated System (SIS) controllers, HMIs, AND the field devices. As such, this represents a broad set of disciplines, including, but not limited to engineering such as process control, process safety, instrumentation, electrical engineering, as well as operations, maintenance and IT aspects of control/electrical systems/networks.
Network vulnerabilities are often assumed to correspond to physical system impacts. They do not. It is generally not possible to correlate the severity of a network vulnerability to the potential for hardware impact. It is also not possible to correlate a network vulnerability to specific equipment such as pumps, motors, or protective relays. Consequently, the questions is what should engineers do when they are apprised of cyber vulnerabilities?
Cyber security is often equated to process safety. They are related but not the same. The process can be cyber secure, but not safe as there are other features besides cyber security that can make a process unsafe. Conversely, process can be safe, but not cyber secure if devices on an IP network are used for process safety.
The gap between networking (whether IT or OT) and Engineering can be seen from Table 1.
Table 1 Differences Between Networking and Engineering
As can be seen from Table 1, the differences between Networking and Engineering are, in many cases, fundamentally different. Issues such as Zero Trust vs 100% trust fundamentally affect architecture, training, and policies. The difference between networking systems that are non-deterministic and control systems that are deterministic directly affects technology and testing. This difference has resulted in control systems having been shutdown or even damaged by using inappropriate network technology or testing tools.
Attackers are becoming better system engineers than the defenders as they generally don’t have organization charts, and the resultant silos, to meet. Often, sophisticated attackers work “backwards” by determining what damage they want to cause and then look for tools to enable that to occur. For control systems, older network vulnerabilities are often sufficient to cause the desired impacts. Whereas defenders often focus on the latest network vulnerabilities without considering the physical impacts that may or may not be created. Consequently, there is a need to understand and adapt to the myriad approaches attackers are using. Unfortunately, cyber forensics does not exist for Level 0,1 devices nor is there training for the control system engineers. There has been a reticence by government to share actual control system cyber incidents. Control system and equipment vendors are often made aware of control system cyber incidents with their equipment but cannot share the information because of non-disclosure agreements. Consequently, there has been minimal identification or disclosure of actual control system cyber incidents.
Part 2 will address sample case histories and Part 3 will address the paradigm shift using process sensor monitoring.
2 Presidential Decision Directive 63, Critical Infrastructure Protection https://fas.org/irp/offdocs/pdd/pdd-63.htm
3 Gartner – https://www.gartner.com/en/information-technology/glossary/operational-technology-ot
9 Presidential Executive Order 13920, Securing the US Bulk Power Systems, May 1, 2020, https://www.whitehouse.gov/presidential-actions/executive-order-securing-united-states-bulk-power-system/
10 Weiss, Joseph, “The Need for Interdisciplinary Programs for Cyber Security of Industrial Control Systems”, WorldComp 2010, Las Vegas, NV.
The Need to Change the Paradigm of Control Systems Cyber Security