Big

 Brother  for  Enterprises:    

Log  Analysis  Use  Cases  

Samisa  Abeysinghe  
VP  Developer  Evangelism  
19  Feb  2014
 About  the  Presenter  
๏  Samisa  Abeysinghe  
VP  Developer  Evangelism  
samisa@wso2.com    

๏  Samisa  Abeysinghe,  Vice  

President  of  Developer  
Evangelism  joined  the  company  in  
September  2005.    Prior  to  the  
current  role,  Samisa  used  to  be  
VP  of  Engineering  and  managed  
the  development  of  WSO2  
Carbon  based  product  plaOorm.    

2  
 About  WSO2  
๏  Driven  by  InnovaXon  
๏  Global  enterprise,  founded  in  2005  
by  acknowledged  leaders  in  XML,   ๏  Launched  first  open  source  API  
web  services    technologies,   Management  soluXon  in  2012  
standards    and  open  source  
๏  Launched  App  Factory  in  2Q  2013  
๏  Provides  only  open  source  
plaOorm-­‐as-­‐a-­‐service  for  private,  
public  and  hybrid  cloud   ๏  Launched  Enterprise  Store  and  
deployments   first  open  source  Mobile  soluXon  
in  4Q  2013  
๏  All  WSO2  products  are  100%  open  
source  and  released  under  the  
Apache  License  Version  2.0.  

๏  Is  an  AcXve  Member  of  OASIS,  

Cloud  Security  Alliance,  OSGi  
Alliance,  AMQP  Working  Group,  
OpenID  FoundaXon  and  W3C.  

3  
 What  WSO2  Deliver  
4  
 NSA like Monitoring for
Your Enterprise
๏ Analyze volumes of data

๏ Address correlation complexities in analytics

๏ Off line vs Real time operations implications

๏ Some operations got to be in real time, else the value is lost

๏ Summarized data over time (and other) dimensions for analytics

A “Big Brother” that keeps an eye on

the whole enterprise

5  
 Why should I bother?
๏ Deal with high volume (terra bites) of information

๏ In order to make decisions

๏ Real time & Offline

๏ Take action

6  
 WSO2 :ONLY COMPLETE
& INTEGRATED Platform
๏ Complete and integrated for

๏ Data capture

๏ Analysis: both real-time and batch

๏ Visualization

๏ Action taking business process execution

Making data driven intelligence for

your enterprise easy

7  
WSO2 Big Data Analytics
Platform for Your Enterprise

WSO2  Big  Data  Analy0cs  

Pla4orm  for  Your  Enterprise  

8  
 Key  Elements  
Data  
CollecXon  

Taking   Data  
AcXon   Analysis  

Data  
VisualizaXon  

9  
 Use  Case  Scenario  1:  
Monitor  your  Java  ApplicaXon  System  Logs    
with  BAM  &  CEP

10  
 & WSO2 CEP

11  
 Overview  of  SoluXon    
o  Send Log Events to

o  Business Activity Monitor (BAM) &

o  Complex Event Processor (CEP)

o  Real time Log Event Processing

o  With CEP

o  Batch Processing of Log Data

o  With BAM analytics

o  Visualization of Log Data

o  With Gadgets on Dashboards

12  
 Log  Event Publishing  (BAM)    

13  
 Event  Streams  &  AlerXng  (CEP)  

WSO2 CEP

14  
 LogEvent  Stream  
Meta Data



q 

clientType {String}


q 

Meta Data


q 

tenantID {String}


q 

q  ServerName {String}

appName {String}



q 

logTime {Long}


q 

priority {Long}


q 

message {String}


q 

q  logger {String}

ip {String}



q 

instance {String}


q 

q  stacktrace {String}

15  
 CEP  Query  


from LogEvents [priority == "ERROR"]

select message, stacktrace, serverName

insert into ExceptionStream

Email Body  
Error Occurred in {{serverName}} – {{message}}

{{stacktrace}}

16  
 AnalyXcs  &  Batch  Processing    
(BAM)  

17  
 Hive Query  
CREATE EXTERNAL TABLE IF NOT EXISTS LogEventInfo (key STRING,

tenantID INT,serverName STRING,

appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY

'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" =

"localhost",

"cassandra.port" = "9160","cassandra.ks.name" = "EVENT_KS",

"cassandra.ks.username"

= "admin","cassandra.ks.password" = "admin",

"cassandra.cf.name" = "log_0_AS_2014_01_23",

"cassandra.columns.mapping" =

":key,payload_tenantID,payload_serverName,payload_appName,

payload_priority,payload_logTime,payload_logger,payload_message"

);



CREATE EXTERNAL TABLE IF NOT EXISTS Logs(tenantID INT,serverName STRING,

appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING)

STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES (

'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver',

'mapred.jdbc.url' =

'jdbc:mysql://localhost:3306/MYBAMDB',

'mapred.jdbc.username' =

'root','mapred.jdbc.password' = 'root',

'hive.jdbc.update.on.duplicate' = 'true',

'hive.jdbc.table.create.query' =

'CREATE TABLE LogEvent(tenantID INT,serverName

VARCHAR(200),

appName VARCHAR(200), priority VARCHAR(200),logTime DOUBLE,logger VARCHAR(800),message VARCHAR(3800))');



insert overwrite table Logs select tenantID, serverName, appName, priority, logTime,

logger, message from LogEventInfo;

select tenantID, serverName, appName, priority, logTime,

logger, message from LogEventInfo;

18  
 VisualizaXon -­‐  Gadgets  

19  
 Demo  …..  
http://wso2.com/library/demonstrations/2014/02/screencast-monitoring-system-logs-with-wso2-business-activity-monitor/

20  
 Use  Case  Scenario  2:  
HTTP  Log  Monitoring  With  WSO2  BAM  

21  
 HTTPD  Logs  and  Use  Cases    
•  Monitor every web request information
–  HTTP method
–  URI
–  Status code

•  Monitor request properties

–  Message size
–  Host / IP address
–  Geo location
–  Date and time

22  
 HTTPD Logs:  Scenarios  
•  Publish HTTP logs to BAM from a data agent

•  Split log entry for each fields and store in another big
data column family

•  Resolve IP addresses to geo locations

•  Aggregate request for geo locations and store into a

relational database (RDB)

•  Visualize from gadgets

23  
 SoluXon  Architecture    

24  
 Key  Performance  Indicator  (KPI)  
 Use  Cases  
•  Analysing request count variation for each host

•  Analysing request count variation for each hour

•  Analysing request message size variation

25  
 Final  Dashboard  

26  
 Demo  …..  
http://wso2.com/library/demonstrations/2014/02/screencast-http-log-monitoring-with-wso2-business-activity-monitor/

27  
 Business  Model  

28  
 Call  to  acXon  page  
๏  Be  your  own  NSA:  hhp://wso2.com/landing/nsa/    

๏  Big  Data  Webinar  Series:  

hhp://wso2.com/landing/wso2-­‐bigbrother-­‐webinar-­‐series/    

๏  WSO2  Business  AcXvity  Monitor:  

hhp://wso2.com/products/business-­‐acXvity-­‐monitor/    

๏  WSO2  BAM  Docs:  

hhp://docs.wso2.org/display/BAM240/WSO2+Business+AcXvity+Monitor
+DocumentaXon    

๏  WSO2  Complex  Event  Processor:  

hhp://wso2.com/products/complex-­‐event-­‐processor/    

๏  WSO2  CEP  Docs:  

hhp://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor
+DocumentaXon    
29  
Contact  us  !