Host-based Intrusion Detection Work in Progress The SOS

Host-based Intrusion Detection (Work in Progress) The SOS Group School of Computing and Information Sciences Florida International University

Talk outline } } Background on intrusion detection Current Approaches Challenges in Intrusion Detection Holistic Intrusion Detection § Philosophy § Design & Implementation } Open Questions

Intrusions } Typically occur as sequence of operations § Exploit vulnerabilities in software § Operations affect various subsystems (e. g. , memory, network, or filesystem) § A seemingly normal operation can be part of a intrusive operation sequence } Intrusions result in system compromise § Unauthorized access to system resources § Exploitation domains: User Root

Intrusion Detection } Detection generation § Current and/or Past intrusions } Detection point § “At entry” : Vulnerability exploited (Legal entry? ) § “Post entry”: Ongoing misuse, unusual user or process behavior } Detection scope § User / Root domains } Detection automation § Human initiated, Human aided, Fully automated, etc. } Detection accuracy § Metrics: false+/- rates

Current Approaches } Subsystem-centric § E. g. , Snort [Roe 99], Bro [Pax 99], Tripwire [KS 94], Stackguard [CMH+98], etc. § Handicapped by local view of system state Detection inaccuracy § Incorrect inferences about system status § Increased false+ and false§ E. g. False+: /bin/ls is modified => Rootkit § E. g. False- : rhupdate contacts unlisted server } Detection scope and automation § Too broad scope encompassing user and root domains § Human initiated [KC 05] } Detection technique § Vulnerability-specific entry point triggers § Post-intrusion anomaly detection }

Challenges in Intrusion Detection Global state monitoring } Subsystem activity correlation } Generic activity signatures } § Infinite good-set ? § User-specific behavior modeling Framework for combining malicious and anomalous activity detection } Detection accuracy } Detection efficiency }

Approach Philosophies Holistic System View I. A. B. Division of responsibility with subsystem sensors Low-overhead correlation and signature detection Ignore User, Protect ROOT! II. A. B. Monitor everything Detect root domain compromise only Two-Level Detection III. A. B. Perimeter detector to monitor entry-points Internal detector to monitor system misuse

I. Holistic System View Global System State Network Filesystem Memory Process

Subsystem Events } Network activity § E. g. , Connection established, Connection bandwidth exceeds threshold, Connection closed, etc. } Filesystem activity § E. g. , File is opened, Permissions/Owner changed, Symbolic link created, Write/Read performed, etc. } Process activity § E. g. , Process created, Executable loaded, Process defunct, Process terminated, EUID changed, etc. } Memory activity § E. g. , Memory allocated/deallocated, Protected memory overwritten, etc.

Detection Errors } False § /etc/passwd modifed (by non-root process) § rhupdate contacts unlisted server for updates § TOCTTOU exploit } False + § rhupdate modifies ls § Unusual network activity (backup process)

Holistic View } Subsystem Sensors: Monitor low-level events occurring within each subsystem § § } Filter and aggregate into high-level semantic events Escalate high-level events to the central analyzer Sensor policies can be tuned by central analyzer Allow integration of third-party subsystem profilers with standard interfaces Central Analyzer: Sink for high-level events § § § Integrates and correlates them Tracks global system state Uses both good and malicious signatures for intrusion detection

Holistic Detection Architecture Process Sensor Memory Sensor Central Analyzer Sensor Network Sensor Filesystem

Terminology } Event: Any significant occurrence § Low-level § High-level Activity: Set of ordered sequence of events } Signature: Generic activity (with wildcards) } Signatures database: Collection of signatures } § Malicious § Good

High-level Events } High-level event: [start: end] [subsystem] [pid] [event-type] <sub-type> <attr-list> } Examples: § § § § § 21000: 21019 P 666 Start backup SYSTEM 38450: 38900 P 25 Start rhupdate SYSTEM 39453: 39454 P 46 Defunct 46 httpd SERVER 40039: 41001 F 25 Access /bin/ls W 45233: 52001 F 562 Delete /var/log/messages 54883: 54889 N 25 Conn. Est TCP 128. 111. 58. 143 65900: 66005 N 666 Ld. Spike TCP 152. 41. 20. 1 70001: 70010 M 46 Write RA_STACK 0 x. F 04986 FF 0 x. F 04986 AF 89000: 89001 M 46 Write CODE_SEG 0 x. FF 98754 A 551

Activity Complex co-operating events from multiple subsystems } Can be simplified into (one or more) event sequences } § E. g. False+ 38450: 38900 P 25 Start rhupdate SYSTEM Access /bin/ls W 45091: 45092 P 25 Stop 40039: 41001 F 25

Signatures Generic activity pattern } Malicious signatures database } § Infer malicious activity § Increase detection accuracy (Reduce False-) } Good signatures database § Infer non-malicious activity § Increase detection accuracy (Reduce False+) } Example Specific False+ 38450: 38900 P 25 Start rhupdate SYSTEM 40039: 41001 F 25 Access /bin/ls W 45091: 45092 P 25 Stop Generic False+ P X Start Update. Class: A(Y) SYSTEM F X Access Protected. Class: Y W P X Stop

II. Ignore User, Protect ROOT! } Intrusions can effect two domains § User § Root } Approach: Ignore user, protect root! § § § § Too many application-specific vulnerabilities New applications introduce new entry-points Users make mistakes: cannot be controlled! No uniform user behavior model (also application dependent) User compromise does not affect other users or system OS must implement fairness and it does (mostly) Significantly reduced signature set and detection overhead

III. Two-Level Detection Internal operation of the Central Analyzer Detector } } § Input: Global event log Perimeter Detector: Monitor entry points into system (i. e. vulnerabilities) for compromise I. § § § Bad signature DB entries Vulnerability-generic class signatures What about? Unknown vulnerabilities Entry through legitimate entry point (careless admin? ) Internal Detector: Monitor for misuse/anomalies II. § Good signature DB entries Potentially infinite Only register: Apparently bad, but Actually OK.

Two-Level Detection Framework Perimeter Detector: Vulnerability-generic bad signatures Feedback (backtracking) Internal Detector: Misuse-specific good signatures Legend: Known vulnerability Unknown vulnerability

Perimeter Detector } Entry points are system vulnerabilities § Class 1: Improper protection Incorrect initial assignment (incorrect permissions, config, etc. ) Improper isolation of implementation detail (bypass OS) Improper change (TOCTTOU) Improper naming (symbolic link ambiguity) Improper allocation/de-allocation (old data reused) § Class 2: Improper validation E. g. , memory overflows, format string, type clashes § Class 3: Improper synchronization Improper indivisibility (atomicity, cache consistency) Improper sequencing (writing during reading, etc. ) § Class 4: Improper choice of operand or operation E. g. , Not resetting GID allowing read of kernel memory

Perimeter Detector } Example § Vulnerability: TOCTTOU § Bad signature (p: pid, f: filename): F p Access f F !p Delete f F !p Sym. Link * f F p {Rd, Mdfy} f

Internal Detector } Monitor system resource usage § Need only monitor root activity Good signature: Apparently BAD, but OK! } Policy framework for capturing Good Root behavior } § § Protected resource, Authorized set (users, executables) Must be administrator tunable } Password protected console interface Trigger Events § Access to protected resource § Check good DB for eliminating false + § E. g. Good Signature P X Start Update. Class: A(Y) SYSTEM F X Access Protected. Class: Y W (Trigger Event) P X Stop

Open Questions (? ) } Signature generation § Source: public signature db, sysadmin (? ) § Automation (from specific activity) § Generality Monitoring overhead } Analysis overhead } Intrusion response } § Automatic vulnerability removal (? )

The SOS Group @ FIU } Faculty § § Dr. Raju Rangaswami Dr. Geoff Smith Dr. Masoud Sadjadi Dr. Tao Li } Students § § § Krista Merrill Joseph Marrero Igor Hernandez Mandy Barsilai Ruben Duque

Thank You!

Back-up Slides