Beruflich Dokumente
Kultur Dokumente
Agenda
Simple Big Data A Child Becomes a Teenager Can Data Science Solve Business Problems Visibility and Insight Visibility and Insight, Context and Analytics A Use Case for Smart Data
2
When the size of the data itself becomes part of the problem*
* Mike Loukides OReilly Radar
GPS, RFID, Hypervisor, data Web Servers, Email, Messaging, Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, Telematics, Storage, Servers, Security Devices, Desktops
Unstructured Platforms
RDBMS Sharding
Data Science is more than storing data in HDFS, NoSQL, or a cloud offering. its getting value, insight and analysis out
9
Traughs
10
Data scientists are the next-generation analytics professional; they are responsible for turning the data into insight. They extract meaning from Big Data to help the business.
11
Data Analyst
Conduct analytics on structured data with traditional tools
Data Architect
Guide designs, set standards, and manage developers
Data Scientist
Extract meaning from big data to help the business
Developer
Build scalable applications based on data in Big Data platforms
Data Scientarchilystoper
Advanced Analytics
Algorithmic Engine Knowledge Management Collaboration Platform Data Driven Thinking
15
18
19
Show Me
20
21
22
Better awareness of ecosystems (how large? Geographic understanding? visibility into C2 servers, signatures and attribution) take down services (identify and degrade hostile botnets prior to an attack, more law enforcement, law enforcement/agreements to stop attacks.) proactive ISP assistance (ASNs, router/flow data) Full view for geographic perspective, what controls, IPs, protocols more visibility into global actors - capabilities, weapons of choice, etc. Sharing threat intelligence built into multiple vendor products across others. Real-time and proactive DDoS forecasting, behavioral modeling with historical context Deep Technical Expertise
23
DNS Analytics
25
Analytics + Context
After looking at our 24 hours of DNS traffic, we applied some heuristics to get an idea what is BotNet traffic and what is legitimate Now we have some analytics, what next?
Lets compare with our watchlist of malicious domains Lets look for new requests we havent seen before Lets look for requests with the same variance between the last request Lets look combine insight from proxy logs to validate potential bad traffic by domain, IP, or Top Level Domain Lets use some geospatial
26
Analytics + Context
All Heuristic Bot Net DNS run through Threat Intelligence watchlists for matches
27
Threat Intelligence
Common Challenges
28
response = conn.getresponse()
print response.status, response.reason data = response.read() print(data) conn.close()
29
Reducing the TTL for these DNS entries will help prevent targeted attack from Syria now that we know what causes that
30
Hybridization
Internal apps, customer-facing apps, mobile apps
Analysis tools (SAS, SPSS, R, Tableau)
Relational DBs
MapReduce HDFS
ETL
Time Series
Enterprise DW
Real-time analytics
Web
Files
Social
Logs
ERP
CRM
31
LDAP, AD
Watch Lists
CMDB
Correlate across multiple data sources and data sets using indexes and keys
32
ocessing and analyses of a Our approach incorporates the use of a NoSQL database as the here are a myriad of open key element in the mitigation of these three tension points (see n and display of network Figure 1). ://www.wireshark.org), and r ones, with more listed at ovide a comprehensive set the stream, which 1. network Start with Security Ops model ards of 20,000 rules. Work 2. Big Data + Internal context data gs is also common, Splunk 3. Add Threat Intelligence Analysis Farm [13] uses adata Collaboration/Communication on4. and query performance. 4. Automation vel localized rule-based 5. Iteration her-level analysis functions zational email traffic or r.
d streaming tools is to save ize one of the analytically as WEKA [6], Orange [3], rical data. The significant NoSQL Databases for Streaming Network Analysis 1: Conceptual diagram of the approach used for our cyber s that * Excerpt the - Using analysis becomes Figure ers may find themselves defense system illustrating the central role served by our NoSQL database. against static data sources 33
Brian Wylie*, Daniel Dunlavy*, Warren Davis IV*, Jeff Baumes**
*Sandia National Laboratories, **Kitware Inc
ABSTRACT
different programming languages and scripts are welcome, components are interchangeable, and most importantly its
MACHINE DATA
Manufactured
Patient Behavior
Prescribed
to patient
Prescription Patterns
Shipped
to Physician
Returned
to iRhythm
Aggregate Data from Vehicles Remotely Vehicles Acceleration, Braking, Battery Charge and Location
Insights
into customers driving habits
Frequency
of charging and charging locations
Thank You!
Fred@Splunk.com
Further Reading
OpenDNS Dynamic DNS Fast Flux
DNS
Botnetshttp://www.elsevierdirect.com/companions/9781597491358/casestudies/D NS.pdf
http://www.syssec-project.eu/media/page-media/3/dietrich-ec2nd11.pdf
40
Hybridization
Internal apps, customer-facing apps, mobile apps
Analysis tools (SAS, SPSS, R, Tableau)
Relational DBs
MapReduce HDFS
ETL
Time Series
Enterprise DW
Real-time analytics
Web
Files
Social
Logs
ERP
CRM
41
Mobile Methodology
Client Application
Static Analysis
Web Application
Static Analysis
Network
Dynamic Analysis Dynamic Analysis
{"tags": {"UTIL": ["Lcom/jumptap/adtag/actions/BrowserAdAction;", "Lcom/inmobi/androidsdk/ai/container/IMWebView$2;", "Lcom/flurry/android/ai;", "Lcom/jumptap/adtag/media/JTMediaPlayer;", "Lcom/jumptap/adtag/media/JtVideoAdView$3;", "Lcom/jumptap/adtag/media/JtVideoAdView$4;", "Lcom/inmobi/androidsdk/IMAdInterstitial$1$1;", "Lcom/jumptap/adtag/media/JtVideoAdView;", "Lcom/rovio/ka3d/GLSurfaceView$DefaultContextFactory;", "Lcom/millennialmedia/android/MillennialMediaView;", "Lcom/jumptap/adtag/utils/JtAdFetcher;", "Lcom/jumptap/adtag/utils/JtAdManager;", "Lcom/burstly/lib/component/networkcomponent/burstly/ormma/OrmmaDisplayContr oller;", "Lcom/inmobi/androidsdk/ai/container/IMWebView$TimeOut;", "Lcom/burstly/lib/component/networkcomponent/burstly/ormma/OrmmaSensorContro ller;", "Lcom/jumptap/adtag/utils/JtSettingsParameters;", "Lcom/flurry/android/v;", "Lcom/inmobi/androidsdk/ai/controller/JSAssetController;", "Lcom/jumptap/adtag/JtAdView;", "Lcom/millennialmedia/android/BasicMMAdListener;", "Lcom/google/ads/AdActivity;", "Lcom/millennialmedia/android/HandShake$AdTypeHandShake; 43
44
45
46
Redis Lookup
###CHANGE PATH According to your REDIS install ###### sys.path.append(/Library/Python/2.6//redis-2.4.5-py.egg) import redis def main() #Connect to redis Change for your distribution pool = redis.ConnectionPool(host=localhost,port=6379,db=0) redp = redis.Redis(connection_pool=pool)
47
except: return
48