Processing Intelligence Feeds with Open Source Software Chris

Processing Intelligence Feeds with Open Source Software Chris Horsley, SC Leung, Tomas Lima, L. Aaron Kaplan, Raphael Vinot

Overview • Current topics in automatic incident handling for CERTs • IFAS • HKCERT , IFAS and use-cases • IHAP project • Contact. DB project • Current R&D

IFAS • Information Feed Analysis System

Knowing what’s going on

How do national CSIRTs know what’s happening? National CSIRTs need visibility on network in their economy However, many national CSIRTs don’t operate networks themselves, and normally don’t have global (or any) direct visibility How does the CSIRT know what’s going on in their country?

The kindness of strangers Luckily, there a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs. And here comes the “but”. . .

So much data, so many formats feeds, all with their own data formats and mediums: There are many Formats: CSV, JSON, XML, STIX, IODEF Mediums: HTML, RSS, email, HTTP APIs While there are efforts to standardise data formats, this will take a long time, and will likely never cover 100% of feeds We can’t change the format of remote feeds - we can only change what we do with the data.

The need for standards Different feeds use many terms to mean the same thing: ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip. . . If we receive events from many feeds, we need to normalise so we can compare them together.

The need for storage As a national CSIRT, we’re concerned with the health of national networks: which means measurement. We can only measure longterm if we store events, enabling us to analyse them. We also want to search through events, like: C&C servers in domestic networks in last week Bots infected with Trojan. abc on Big. ISP Defaced web sites targeting gov. zz

Need for automation There’s way too much network event data out there to manually process Options: a) use lots of analyst time doing tedious log processing b) write lots of small, independent scripts c) ignore inbound logs completely d) use an automated processing system

So what do we need? We need something which automatically: Gathers many different types of feeds Normalises the data in those feeds Stores that data somewhere Allows search and performs statistical analysis

IFAS = Information Feed Analysis System Project sponsored by HKCERT and developed by HKCERT and CSIRT Foundry An integration of open source tools, released as open source for CSIRTs

Architecture

Architecture Abusehelper: gather, process, and enrich feeds, generate events Logstash: process and normalise feeds Elasticsearch: store events in schema-free index server Kibana: search through events IFAS Reporter: get overall statistics, build realtime dashboards

Kibana event searches

Freeform statistical reporting

Nesting, filtering, deduplication

IFAS – Dashboard l Visualize information *Drill down right at the chart

What you need to start

Software Open source under Apache 2. 0 License Only possible with the hard work released under open source licenses from Abusehelper and Elasticsearch teams Contributions, bug reports, feature requests most welcome!

Hardware Production: 8 -16 GB memory machine Dev: 4 GB possible Multi-core machine (4+ ideal) Runs in a VM no problem

Out of the box feeds Out of Box Feed Plugins (4 publicly available) Abuse. ch Clean. MX Millersmiles Phishtank Other developed Plugins Malc 0 de Malicious Domain List Arbor SRF Shadowserver Zone-H Future … more, and your own

Where to get it Currently under closed pilot to trusted CSIRTs Eventually public release Please contact@ifas. io for details

DEMOS

IFAS and Use Cases SC Leung, HKCERT

Give a sense of Today’s Events

IFAS - Log Search l Powerful search on all the information collected Keywords here Feed Details Add columns of interests

IFAS - Reporter l Statistical analysis-Trends & Distributions l Free form statistical reports 5. 2. 1. 6. 3. 4.

Nesting, filtering, deduplication Number of phishings in “. AU” in each ASN by brand

IFAS - Alert l Set tracking criteria – get notify ASAP l domain: *. gov. hk l Alert lists : educational institutions (hkedu), NGOs (hkorg) !

Dashboard Real-time situational awareness for CERT management

Public Situational Awareness on Compromised Servers / PCs

Hong Kong Security Watch Report

Analysis of Trend with Events • Correlate Cryptolocker 2013 -Oct with Zeus

Engage ISPs for large scale incident handling ISP • • Data do help HKCERT engaging ISPs (their sales team) Data do help a server hosting SP understand their customers’ security problems

Converting security events into incident reports • Defacement • Phishing • Export to CSV for batch processing, with some other scripts • Malware hosting – a bit difficult • Large volume of incidents – need prioritisation

Future of IFAS - a collaboration platform • All you can use • All you can contribute • • Add new plug-in modules • Add new chart and visualization • Integrate with other systems, e. g. RTIR • • Add input filters for new feeds … Standard language: STIX, taxonomy of ENISA

DSMS (Decision Support & Monitoring System) • An ongoing project that turn security events into Actionable Data • Set Priority, Choose Monitors, Consolidate Results Decision Support Sub-system IFAS Interfaces to Monitors Input URL Profile Tasks Monitoring Services Request to monitor Interface Module Status ? (online /offline) Status Check (HTTP, DNS) via proxy Interface Modules Output Incident Mgmt Story Public analysis sys (Virus. Total, Threat. Expert) Private analysis Interface Modules Web reputation (D-Shield) Consolidated Results sys

Incident handling automation project IHAP

IHAP • Very similar to IFAS, developed in parallel by CERT. pt, CERT. at • Also uses Logstash, Elastic Search and Abusehelper • Less work on the Webinterface, more work on Ontology, „Data harmonisation document“

IHAP - History • Discussions about CERT. AT developments/documents • Discussions about cooperation between CERTs • ENISA support

IHAP - Goals • Open Source • Maintainable • • • Flexible and Modular - must be possible to integrate existing software and modules (Pastemon, Abuse. Helper, etc. . ) Reusable Easily Extendable - should require little knowledge and basic programming skills Easily Deployable Easily Updatable – easy to share new developments with other CERTs and update the system with that new code • Easily Configurable - config files that can be easily modified to fit CERT‘s needs • Documented - must be well documented

Links & Code http: //www. enisa. europa. eu/activities/cert/support/incident-handlingautomation

Common field names for AH • https: //bitbucket. org/clarifiednetworks/abusehelper/wiki/Data%20 Har monization%20 Ontology • A standard set of well defined field names within Abusehelper (AH) • Allows CERTs to: • Write bots which are interoperable within AH • Measure in identical ways • Easier to parse different feeds („generic santizer bot“) : you just have to define the mappings

CONTACTDB

Background/ problem • abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c: ) • Getting the right lookup is non-trivial, complex • Many (national) CERTs create their own abuse contact lookup DBs. • National CERT DB, TI directory, FIRST data can not be looked up automatically via scripts.

Idea • A caching contact database with more specific internal data • Some of this data (tel nos, etc) will never be in the public whois • Unify with TI, FIRST etc data • Make it query-able by scripts

What databases exist? What can we query? ABUSE CONTACT LOOKUP - FLOW

Number based resource: IP addr, netblock, ASN Name based resource: domain name, hostname TI, FIRST, CERT. org DBs Get country() Gethostbyname() Extract cc. TLD Whois DB (RIPE, ARIN, . . ) Maxmind RIPE DB Cymru, . . . Whois DB (registrant, registrar) National CERT DB Country code CERT. org IRT object, abuse-c, . . . National CERT for country Email Address IANA cc. TLD list Country code

What exists now? • Public code repo ; -) • Whois server (thx Mauro) • RESTful API (Mauro, Rafiot) • Some scripts to import TI data (Aaron, David) • Still some bugs ; -)

Code & document with RIPE • • Document (WIP): https: //github. com/certtools/contactdb/blob/master/doc/contactdatabases-for-abuse-handling. mkd Codebase: https: //github. com/certtools/contactdb (thx Rafiot, David, Mauro!)

SUMMARY

Summary • The CERT community has limited ressources for development • We re-implement the same thing all the time • Let‘s share code or at least exchange ideas on how to automate incident handling! • Let‘s share on how to measure success • Thanks HKCERT, ENISA, CERT. at, CERT. pt, CIRCL, etc. . • Mailinglist: https: //tiss. trustedintroducer. org/mailman/listinfo/ihap

Thanks!