A Configurable Hardware Framework for a Trusted Computing

A Configurable Hardware Framework for a Trusted Computing Base: Application to the Power Grid Ravi K. Iyer Information Trust Institute Coordinated Science Laboratory University of Illinois at Urbana-Champaign 1

Objectives • Develop enabling technology to provide customizable level of trust to a significant critical infrastructure as exemplified by the Power Grid computing and communication systems. • The focus is on design methods and runtime techniques to achieve application-specific level of reliability and security, while delivering optimal and timely performance. 2

TCIP- NSF Cyber Trust Center-Scale Project. Trustworthy Cyber Infrastructure for Power www. iti. uiuc. edu Address technical challenges motivated by domain specific problems in Ubiquitous exposed infrastructure Real-time data monitoring and control Wide area information coordination and information sharing By developing science in Secure and Reliable Computing Base Trustworthy infrastructure for data collection and control Wide-Area Trustworthy Information Exchange Quantitative Validation

Present Day Power Grid Cyber Infrastructure - Peer coordinators may exchange information for broad model - Degree of sharing may change over time 10’s of control areas feed data to coordinator Coordinator - State estimator creates model from RTU/IED data - 1000’s of RTU/IEDs - Monitor and control generation and transmission equipment Control Area Photos courtesy of John D. Mc. Donald, KEMA Inc.

U. S. Power Grid Cyber Infrastructure Complexity ISO-NE NYSO RTO WEST PJM MISO CAISO TVA GRID SOUTH ERCOT GRID FLORIDA Copyright 2003 Edison Electric Institute. Source: POWERmap, © Platts, a Division of the Mc. Graw Hill Companies.

Increased Power Grid Trustworthiness via Secure and Reliable Computing Base Control Center (EMS) Level 3 (Enterprise) Control Center (EMS) ISO LAN New Types of Platforms Customizable Reconfiguration Vendor Application Specific Architectures Gateway Level 2 (Substation 1) Substation 4 IEDs meter Level 1 (Sensors/Actuators) Substation 2 Substation 3 6

Approach • Explore processor/OS/Application level solutions to achieve low-cost, high-performance, scalable security and reliability checking in the same framework • Provide small footprint solutions that not require large amount of extra hardware or software • Provide solutions that can coexist with the current generation of critical infrastructure • Ensure timely detection and recovery to prevent loss of service or damage to critical infrastructure 7

Approach Vision: • Systematically transform the computing base • for application-level security and reliability Main idea: • Derive application-centric checks • embed them in the HW • access them with OS/middleware support • validate them in power-grid cyber infrastructure Considering: • Both COTS and new architectures • technical challenges raised by deployment/management

Current Generation of Low-end Devices (1) • • New generation devices for substation automation: Network Terminal Units (NTUs) Op. En Connect NTU-7500 – Combines traditional RTU (remote terminal unit) functionality with superior processing power and advanced tools for data collection and control. Capabilities Advanced – innovative data management Control – easy-to-use configuration program, Systems • employs Windows interface and drag-and-drop database editing – preservation of legacy components, subsystems, and data concentration applications • assemble multiple databases from the available data points. • ability to monitor real-time transfer of raw data to and from your NTU – support for reconfiguration or expansion for changes and additions – grow from a basic site to one with many IEDs, • IED configuration and on- site setup – upgrade of RTUs to NTU capabilities with minimal difficulty.

Current Generation of Low-end Devices (2) • NTU-Substation Controller – high-performance – large database capacity – data concentrator and protocol converter applications – ability to process a large amount of data from IEDs, – interface a large number of discrete data acquisition and control devices in the substation. – use of virtual RTUs • sort incoming IED and RTU data into discrete databases • can be based on data type, e. g. , critical and non-critical • can configure Virtual RTUs to provide appropriate data subsets • • Design Features – distributed processing architecture; – multiple 32 -bit microprocessors, – linked using a peer-to-peer type network – multiple IED isolated serial communication interfaces Operating Systems – IED: RTOS, e. g. , Thread X – NTU/RTU: Linux, XP

n Achieving Secure and Reliable Computing Base Reconfigurable operating system-level kernel module to support OS/application aware security and reliability services n Current features n Two level hierarchy: - low-level pins interfacing with OS and hardware - high-level modules providing application-specific security and reliability techniques n Available modules n Application/OS hang/crash detection n Transparent application checkpoint Reconfigurable processor-level hardware framework to support security and reliability n Current features n On-core approach n Framework and modules implemented on an FPGA n Available modules n Malicious attack detection - Pointer taintedness detection - Information-flow signatures n Transparent hang/crash detection for OS and applications Reliability and Security Microkernel (RSM) n Reliability and Security Engine (RSE)

Automated Design Flow Application Source Code profiling Dynamic Execution Profile Fanouts analysis Application code annotated with critical variables Reliability Augmented Compiler Application code instrumented with interface to invoke H/W checks Regular compiler General-purpose Processor Path-tracking state machines Checking Expressions VHDL Translation & synthesis RSE Interface Reliability and Security Engine (RSE)

Hardware Implementation: RSE Module Static-Detector Module PATH_CHECK Instruction Committed Register File DLX Superscalar with RSE EXPR_CHECK Instruction Committed Path Tracking Runtime Path Main Memory Static-Checking Write Buffer Error Detected

Security Partitioned Applications Intelligent Electronic Device - SEL 3351 Data Aggregator Main Processor Database Application Secure Coprocessor Protected with RSE Secure Data Secured Code Kernels Streaming Video Distribution Application Access Control Functions 14

Example of Security Partitioned Applications Intelligent Electronic Device - SEL 3351 Data Aggregator Main Processor Main application runs on COTS processor. Database Application Secure Coprocessor Protected with RSE Secure Data Secured Code Kernels Streaming Video Distribution Application Access Control Functions 15

Security Partitioned Applications Intelligent Electronic Device - SEL 3351 Data Aggregator Main Processor Database Application Coprocessor allows utilization of custom checking hardware. Secure Coprocessor Protected with RSE Secure Data Secured Code Kernels Streaming Video Distribution Application Access Control Functions 16

Security Partitioned Applications Intelligent Electronic Device - SEL 3351 Data Aggregator Main Processor Database Application Secure Coprocessor Protected with RSE Secure Data Secured Code Kernels Streaming Video Distribution Application Access Control Functions Secure kernels provide high levels of trust 17

Security Partitioned Applications Intelligent Electronic Device - SEL 3351 Data Aggregator Main Processor Database Application Clearly Defined Interfaces Support Development Secure Coprocessor Protected with RSE Secure Data Secured Code Kernels Streaming Video Distribution Application Access Control Functions 18

Security Partitioned Applications Intelligent Electronic Device - SEL 3351 Data Aggregator Main Processor Database Application Secure Coprocessor Protected with RSE Secure Data Secured Code Kernels Streaming Video Distribution Application Access Control Functions Secured Data Remains on Coprocessor 19

Coprocessor Integration within the Testbed • • • Augment SEL 3351 with FPGAbased coprocessor. Nallatech FPGA Card available in PC-104+ Demonstrate Coprocessor in various applications: – Undervoltage Relay – Video Streaming Distribution SEL 3351 Data Aggregator PC-104+ Computer FPGA Coprocessor SEL 20

Applications Undervoltage Load Shedding Relay: • Monitor critical power data and take corrective action before system is affected. • Protect against accidental and malicious data corruption using RSE. • Integrate FPGA coprocessor into SEL-3351 Data Aggregator using PC -104+ interface. • FPGA Coprocessor with RSE will execute security critical kernels within protected application with a high degree of Trust. Streaming Video Distribution: • Provide protection for distributed distribution of Substation security camera video. • Prevent malicious tampering with video feed due to bandwidth strangling by limiting each individual users bandwidth usage. • RSE FPGA Coprocessor will protect small security critical kernels within the streaming application. 21

Schweitzer SEL-3351 Data Aggregator TCIP Testbed: Synchrophaser Relay Protected with RSE Nallatech DIME-II with Xilinx FPGA Xilinx JTAG Debug Cable Coprocessor Integration within the Testbed Schweitzer SEL-421 Relay

Backups 23

Approach Vision: • Systematically transform the computing base • for application-level security and reliability Main idea: • Derive application-centric checks • embed them in the HW • access them with OS/middleware support • validate them in power-grid cyber infrastructure Considering: • Both COTS and new architectures • technical challenges raised by deployment/management

Research and Engineering Issues Control Centers and above Substations Application-Oriented Threats General Platform Threats Exploring HW support and use cases for “Tunable Trust” • Between components of substation • Between RTUs and the rest of the system Goal: • Restrictions in bandwidth and processing power prevent security from being applied in current system Setting up testbed to experiment with emerging hardware approaches: • Physical security attacks Goal: • Increase assurance that IEDs/RTUs can maintain secrets despite physical attack • Explore ways to update crypto and sw for devices that must last decades, unmanned HW mechanisms to enable trustworthy sharing of computation on private data • “Tiny Trusted Third Parties” • Draws on secure coprocessing, oblivious RAM, blinded circuits • Prototyped on IBM 4758 • Implementing smaller, cheaper, faster version on FPGAs • Application-aware security and reliability services Setting up testbed to experiment with emerging hardware/software approaches: • RSE (reliability and security engine) • AEGIS • RSM (reliability and security microkernel) • Multicore Goal: • Increase availability and resilience for general-purpose platforms exposed to network attacks

Automated Design Flow Application Source Code profiling Dynamic Execution Profile Fanouts analysis Application code annotated with critical variables Reliability Augmented Compiler Application code instrumented with interface to invoke H/W checks Regular compiler General-purpose Processor Path-tracking state machines Checking Expressions VHDL Translation & synthesis RSE Interface Reliability and Security Engine (RSE)

Hardware Implementation: RSE Module Static-Detector Module PATH_CHECK Instruction Committed Register File DLX Superscalar with RSE EXPR_CHECK Instruction Committed Path Tracking Runtime Path Main Memory Static-Checking Write Buffer Error Detected