Mapping and Analyzing Data Matrices in Real Time

INTRODUCTION
Among other things, Security Information and Event Management (SIEM) is a system capable of data aggregation, correlation, analysis and reporting of information security threats in an organization which deals with mammoth and labyrinthine data. One major function of the SIEM tool is to organize and log data collected from multiple sites and then to provide real time, actionable analysis of attacks. The Matrix Mapper (MM) entails the real time incorporation, enhancement and extension of SIEM to complex data streams across varying hardware and software platforms to provide relationships between events and entities coupled with predictive analytics.

The SIEM
SIEM is a combination of the following systems: SIM: Security Information Management - provides storage, powerful analysis and reporting of log data. SEM: Security Event Management- provides real-time monitoring, link formation and correlation among events and generation of alerts. SIEM covers two major functionalities: 1. Log Consolidation: An organisation receives numerous inputs in different forms from various sources creating a diffuse data alluvium which becomes difficult to handle. The logs are the records of the activities performed by the software running on a system. These log records cover normal activities, errors, configuration changes, alerts, authorized and unauthorized user access, behavior patterns etc. 2. Threat intelligence and effective analysis: The log and other activities in the systems are associated through a link and form a consolidated network which makes the analysis of occurrence of threats and their preventive measures that can be adopted. If such threats are repeated in the same fashion, the patterns are correlated to the previous incidents and preventive measures are either performed or predicated. Market Overview : According to a 1Gartner Report, 2012: Magic Quadrant for Security Information and Event Management
1

Gartner Report, 2012: Magic Quadrant for Security Information and Event Management http://www.gartner.com/technology/reprints.do?id=1-1ATPEL3&ct=120608&st=sg

The SIEM market is mature and very competitive and thus SIEM technology demands are increasing. According to Gartner, it is the fastest rising sub-section of the security sector with a growing rate of 21% a year. During 2011 the SIEM market grew from $987 million to $1.1 billion, achieving a growth rate of 15%. The multiple vendors meet the basic log management, compliance and event monitoring requirements of a typical customer. The greatest area of unmet need is effective targeted attack and breach detection. Organizations are failing at early breach detection, with more than 85% of breaches undetected by the breached organization. The situation can be improved with better threat intelligence, the addition of behavior profiling and better analytics. According to 2Frost & Sullivan Report The Asia Pacific (APAC) security information and event management (SIEM) market witnessed a healthy growth momentum in 2010 and is expected to grow at a strong compound annual growth rate (CAGR) of 27.0 percent during 2010-2014. Enterprises have recognized the importance of SIEM in ramping up their security posture. The Asia Pacific Security Information and Event Management (SIEM) Market CY 2010, finds that the market earned revenues of $93.4 million in 2010 and estimates this to reach $242.7 million in 2014. According to an 3IDC Report An IDC study examines the worldwide IT security products market for the period from 2010 to 2015, with vendor revenue trends and market growth forecasts. Worldwide market sizing is provided for 2010, and a growth forecast for this market is shown for 20112015. A vendor competitive analysis, with vendor revenue and the market shares of the leading vendors, is provided for 2010. "Following a difficult 2009, security revenue rebounded well in 2010 with an overall growth rate of 9.1%. This was a full percentage point higher than what had been expected. Organizations and enterprises upgraded their security in 2010 due to a much more difficult threat environment and technology innovation. In 2011, we expect security spending to remain brisk with growth expected to be nearly 10%," said Charles Kolodgy, research vice president for Security Products. "Security is required for organizations that want to expand into cloud computing, increase their use of mobile and virtual technologies, and deal with increasing regulatory requirements.

Asia Pacific SIEM Market to Reach $243 Million in 2014

http://www.frost.com/prod/servlet/press-release.pag?docid=226373777
3

Worldwide IT Security Products 2011-2015 Forecast and 2010 Vendor Shares http://www.idc.com/getdoc.jsp?containerId=232221

The MATRIX MAPPER (MM)

Matrix Mapper is under development as a strong, effective and advanced analytics tool having its application in sectors as diverse as mining, oil & gas, power (generation & distribution), airlines, land and water transport, shipping, chain stores, agriculture/ food distribution, warehousing, courier concerns, accounting, banking and insurance Government agencies/ departments, defence forces, police, forest conservation, counter terrorism units, municipalities, railways and a variety of Public Sector Undertakings would find Matrix Mapper to be a kind of force multiplier. Matrix Mapper as an SIEM solution can be seen as one out of hundreds of application areas with area specific integrations and wider implementations in log management, analysis of data, threat intelligence, network forensics etc.

Theoretical Basis of the MM

1. The collation of enormous data which could be text, audio, video, images, graphics etc. received by the police agency through diverse means and sources. 2. This data is then warehoused i.e. all the data is stored together, in once place, with a homogenous structure which allows interactive analysis. In effect, the data is reduced to

the common denominator of a uniform database by way of what are called normalization tables. This data is then ready to be analyzed. 3. The data is now to be mined i.e. knowledge is to be extracted from the warehoused data through the application of appropriate algorithms which will be able to categorize the data on the strength of defining parameters like say individuals, places, transport used, channel of monetary remittance etc., such categorization (or gradation or clustering or classification or association) depending on what the programme user is looking for. This exercise is also known in the cyber world as big data analytics. 4. Next come algorithms that subject the categorized data to network analysis viz. the linkages, connectivities or relationships between elements in the various categories. For instance, who X is in touch with, how is he in touch with and how often is he in touch with and then how are these relationships related to each other and to what degree and in what manner. 5. Finally comes the turn of predictive analysis. It includes the application of "what if" scenarios to the relationships that have been networked to plot the logical direction of "what next" viz for example what transport is 'X' likely to take to travel to which place, with whom and when. Predictive analysis was earlier done using legacy data. Now it is also used to correlate legacy data with real time event reporting applying SIEM (Security Information and Event Management) technologies so as to be able to secure logical pointers for the direction (or conclusion) towards which ongoing events are headed.

Types of data collected: i) ii) iii) iv) v) vi) vii) viii) ix) Business transactions including money movements. Scientific data including RFID, technical surveillance Medical & personal data images, voice clips, personal profile & history Surveillance video and pictures Satellite sensing Legacy data Digital media scanned material, films, voice and video collections Graphics data maps, drawings, schemes , sketches Virtual data in cloud, digital repositories, mail, SMS

Imagery databases The interpretation of images involves categorising or identifying data, in an image, by correlating it with domain knowledge. So the key to good image interpretation is domain knowledge.

One way by which such correlation can be achieved is to identify the correspondence and coincidence between captured image data and stored models. In the case of sequential images, e.g. videos, each sequence has to be taken and examined in terms of correspondence/ coincidence/ relationship with the stored models. Another method can be to mimic the understanding/ interpretative process of the human eye and the categories in which images, captured by the eye, are categorised and interpreted by the brain. All in all, image interpretation and analysis necessarily predicates the availability of stored models or prior reference points regardless of the precise procedure adopted for identification, categorization and/ or recognition
Domain Knowledge

Feature Extraction & Algorithm

Data Labelling Tuples Wrinkles Dimples Jawline Lip contour etc.

Interpretative Conclusions & Inferences

Audio Databases This part of data mining works on simple principles. Firstly, there have to be sample audios which already stand identified and labeled in relation to persons, things or entities. Content analyzing software, part of the MM, uses a captive algorithm to analyze the tempo, beat, amplitude and frequency of the audio rather than the encoding language per se. Each of these factors, taken together, go to produce a complete fingerprint for the audio sample which, in turn, makes it possible to match with other audio files in the database that yield an identical fingerprint.

Spatial Databases Spatial data mining (SDM) looks for patterns rather than random features. The common spatial features are location prediction, feature interaction and hot spots. SDM is, therefore, the search for unexpected, interesting patterns in large databases. Techniques used in SDM include classification, associations, clustering and outlier detection. It involves discovering the nuggets of useful, unexpected spatial patterns in large databases; very like looking for a needle in a haystack. Examples of vast amounts of spatial data are inputs through satellite imagery, sensors on highways, GPS tracks etc. The basics of the probability calculus in SDM can be stated as:
4

Spatial Databases a tour by Shashi Shekhar and Sanjay Chawla : www.spatial.cs.umn.edu/Book/slides/

Given a set of events , the probability P is a function which satisfies the following two axioms: P() = 1 and If A and B are mutually exclusive events then P(AB) = P(A)P(B) Conditional Probability: Given that an event B has occurred the conditional probability that event A will occur is P(A|B). A basic rule is: P(AB) = P(A|B)P(B) = P(B|A)P(A)
5

Associations, Spatial Associations, Co-location

To determine patterns from the following dataset:

Spatial Databases a tour by Shashi Shekhar and Sanjay Chawla : www.spatial.cs.umn.edu/Book/slides/

Features of the MATRIX MAPPER Matrix Mapper is a single tool with multiple technologies as its components. The features of Matrix Mapper are: (i) Handling of big data by the MM The tsunami of unstructured data such as contact details, financial transactions, remittances, video, audio, graphics, mails, logs, web files, records etc. is fed as input for MM. The MM supports data aggregation and collection from disparate sources such as mobile device, fixed line services, Geospatial information, relational database records, which is quickly sorted into structured form for further processing. It is capable of handling big colossal data, from gigabyte to petabyte and exabyte scales which has to be efficaciously, reliably and quickly analysed to yield useful results. The major functions covered are as follow: a) Collection of logs and events from different sources like security devices (firewall, antivirus, and other UTMs), applications and software, access management products etc. b) Collection of network flow data from switches and routers. (ii) Easy to use workspace IDE Thousands of 2D and 3D icons to accommodate large number of different entities at one time. Drag and drop facility for quicker response and chart generation. Information filtering and cluster analysis. Option of searching maps and accessing charts simultaneously. Zooming capability. Timeline, charts, report etc. generation. Automatic animation. Different linkages and network options Find links, Find path, Entity search and Visual search options are available. Dynamic group association through SNA. Highlight key players. Flexible controls

(iii) Knowledge Discovery: Matrix mapper then collates the big data for aggregation and correlation on which knowledge discovery of entities and events is performed. It is a highly domain specific tool where different knowledge discovery methodologies are applied for intelligent data retrieval. Various queries are operated to obtain simplified results from complex data. The queries help in identifying trends and patterns by building links and relations among the entities. The highly efficient semantic capabilities of Matrix Mapper also ensure the coverage of uncommon and duplicated data. The analysis of trends and patterns provides the key players and entities from network for deeper forensic investigation.

(iv) Social Network Analysis (SNA): Huge and nested complex networks of relationships among the different groups of entities and events. Powerful visualization by way of various techniques such as heat matrix where some bold lines represents strong or direct relationship or a lighter link showing interconnections among contacts of contacts of contacts etc. Matrix mapper would incorporate powerful visual tool to comprehend, decipher and derive intelligent conclusions from networks.

(v) Predictive Analysis: Matrix Mapper is capable of predictive analysis whereby historical facts enable the mapping out, in advance, the probability of future events. It would address issues like what could happen next? so important to successful strategic management. Predictive Analysis could be approached through a number of methodologies. Some of these are:

The greatest number of incidents or heat in relationships or frequency of incidents are plotted on relevant area maps to show up kernels of intensity where the likelihood of
Analytics in Policing: Predictive Policing & Location Intelligence National Police Academy, Mahesh Narayan, May 2, 2013
6

recurrence would be high. Thus, geo spatial data is matched with incident and imagery data. For instance, corridors and landmarks in a certain area can be associated with greater risk. In such places, robbery risk can be a function of prior crimes plus disorderly and suspicious activity calls.

Connaught Place Delhi Crime Prediction based on event data, local geography and high risk locations.

The thumb rule of Hot Spot and Modus Operandi analysis is: Future Crime ~= [past crime] (vii) Customization:

The Matrix Mapper is highly customizable for the use for different businesses in sectors as diverse as mining, oil & gas, power (generation & distribution), airlines, land and water transport, shipping, chain stores, agriculture/ food distribution, warehousing, courier concerns, accounting, banking and insurance. Government agencies/ departments which would find Matrix Mapper to be a kind of force multiplier include the defence forces, police, forest conservation, counter terrorism units, municipalities, railways and a variety of Public Sector Undertakings.

(viii) Database: A flexible centralized database is used for effective analysis from pool of data. From higher growth perspective Matrix Mapper can have capabilities that aid in targeted attack detection, including support for data access, user activity, application activity monitoring, profiling and anomaly detection, threat intelligence and effective analytics.

Architecture: The architecture is comprised of following major entities: (a) Data Accumulator; (b) Log Manager; (c) Analytics Engine; (d) Predictive Results

1. Data Accumulator: The data accumulator is an agent that collects data from firewalls, software applications, IDS/IPS, antivirus, UTMs, router information, IP addresses, normal activities, errors, configuration changes, alerts, authorized and unauthorized user access etc. It splits the data streams to form logs.

2. Log Manager: The log manager is dedicated to the log related activities in database. There are two log managers each for real time activities and long term activity analysis. The real time log manager acts on the operation directly on the single log database and

expects real time results while the other log manager operated on centralized database pool. 3. Analytics Engine: The analytics engine is responsible for Detailed data access: The engine is capable of scanning and parsing the data on which semantics are applied. It performs events and flow data searches in near real-time streaming mode or on a historical basis to enhance investigation. Event identification: The engine identifies the potential events mined from the database for correlation and network formation. Incident and event correlation: It tracks significant incidents and threats, providing links to all supporting data and context. Recognize trends and behavior patterns: It automatically discovers most log source devices and monitors network traffic to find and classify hosts and serverstracking the applications, protocols, services and ports they use. It also includes a view to access near real-time analysis, incident management and reporting. Intuitive Report and chart generation: It produces visually dexterous reports and charts for the real time as well as long term analysis results. 4. Predictive results: The results are based on the historical evidences available in the log and the alerts, notification and updates are generated accordingly.

MATRIX MAPPER- STAR: COMING TOGETHER OF THE SIEM AND THE SBMS Matrix Mapper- Star is a proposed Secure Messaging cum Analytics Tool to be developed by Aarken Technologies (Aarktech). It is an integration of Aarktechs most promising and unique, secure messaging system - SBMS and the powerful analytical tool, Matrix Mapper. SBMS ensures secure real time communication of complex data (text, graphics, audio and video) even in low bandwidth with low latency and Matrix Mapper is a powerful analytics tool. Integration of the two produces a powerful tool that improves the scalability of an SIEM tool and can fit into organizations with globally connected IT infrastructure. The secure data transmission is the need of organizations dealing with critical data and decisions are rely heavily upon analysis of that data. For

example, an oil company with global presence requires regular analysis of various onshore and off shore activities but the communication has to be achieved in captive and secure environment which signifies the need and importance of integration of Advanced Matrix Mapper with SBMS thus producing intuitive Matrix Mapper-Star. This innovative concept would usher in a revolution in the SEIM and Big Data Analysis industry.

Architecture: The diagram below would throw more light on the design principles of this concept.

Suppose a company has several Regional/Zonal offices each having its own SIEM. This SIEM would spot and flag attacks that would otherwise go unnoticed. The flagged data in the log database at each region/zone is then sent to centralized database located in the country office via SBMS that ensures low latency at low bandwidth transmission. The data streaming in is coalesced into the normalization table of the legacy database and subjected to the analytics engine of Matrix Mapper which performs the analytics in real time and extracts useful patterns The real time pattern identification and predictive results with directions for actions to be taken thereon are routed back to regional/zonal offices securely with assured delivery that is achieved by SBMS. SUMMARY CONCLUSIONS

As an incident management tool, SIEM can be highly effective at increasing organisations security layer to identify and handle a large number of events while simultaneously analyzing them to improve the accuracy of threat identification thus increasing effectiveness in detecting and responding proactively to security threats. The innovative concept of Secure messaging cum Analytics tool reflected in proposed Matrix Mapper- Star is an intuitive need of Security and Vulnerability Management domain to ensure internal as well as network security for industries that are engaged in critical communication and exchange.

BIBLIOGRAPHY: 1. Gartner Report, 2012: Magic Quadrant for Security Information and Event Management http://www.gartner.com/technology/reprints.do?id=1-1ATPEL3&ct=120608&st=sg
2 1

2.

Asia Pacific SIEM Market to Reach $243 Million in 2014 http://www.frost.com/prod/servlet/press-release.pag?docid=226373777

3.

Worldwide IT Security Products 2011-2015 Forecast and 2010 Vendor Shares http://www.idc.com/getdoc.jsp?containerId=232221

4.

4,5

Spatial Databases a tour by Shashi Shekhar and Sanjay Chawla www.spatial.cs.umn.edu/Book/slides Analytics in Policing: Predictive Policing & Location Intelligence National Police Academy, Mahesh Narayan, May 2, 2013

5.

6. http://en.wikipedia.org/wiki/Siem 7. http://en.wikipedia.org/wiki/Security_information_and_event_management 8. http://searchsecurity.techtarget.com/definition/security-information-and-eventmanagement-SIEM 9. http://www.certconf.org/presentations/2005/files/WC4.pdf 10. http://www.slideshare.net/stijnvdc/siem-evolution-a-day-in-the-life-of-a-securityarchitect 11. http://www.slideshare.net/vikasraina/SIEM 12. http://www.sans.org/reading_room/whitepapers/logging/practical-application-simsem-siem-automating-threat-identification_1781 13. http://www.techrepublic.com/whitepapers/gartner-2012-siem-magicquadrant/32874165

14. http://www.aarktech.net/solutions-news/matrix-mapper.html