V8.

0 HyBoost

Quick Trouble Shooting Guide

Including General Information
and ICAP

Distribution Date: Nov 2016

Author: SOMANSA Technologies, Inc

This document is a guide provided along with the Mail-i V8.0 HyBoost
product, a network data loss protection, of SOMANSA Co.,Ltd., and has
been written as a reference of the product after sale to be used in the
actual operation environment from initial configuration to
maintenance.

*Expected Readers: Engineers, Administrators

Somansa Co.,Ltd. holds patents and copyrights over this document.

1. General for the troubleshooting

1.1. Overall Mail-i Configuration Diagram and Modules

Mail-i 8.0

Agent (NDLP-Agent)
refers to Traffic-Agent processing network traffics and Content-Agent processing protocols.

CM(Configuration Manager; henceforth CM)

provides web based user interface for the operation and control of the product such as database configuration, Agent
execution and termination.

DLP+Center
provides web based user interface for tasks such as incident (log) view, policy management and reports.

Query Server
views the incidents and deliver the policies and HR data to the Agent.

Indexing Server (Indexer)

saves the incident an Agent has created in ElasticSearch.

Cache Server (Redis)

works as a temporary storage for viewed incidents.

SMSAnalyzer
detects the data patterns such as resident registration numbers from the saved incidents (Content/Attachment files).

SMSSummary
performs scheduled summary task with incidents with patterns.

ElasticSearch (henceforth ES)

saves the incidents in the form of an index.

GlusterFS (henceforth GFS)

saves the attachment file of the incidents.

PostgreSQL
saves system configuration, HR data, policies, data mining (reports) and audit logs.

2
1. General

1.2. Folder Structure of /somansa

cm
contains necessary files for configuring and operating CM.
common
contains common files and modules (Query server, Indexing server etc) used in the products.
data
saves the incidents and attachment files.
dlpcenter
contains files required in configuration and operation of DLP+Center.
elasticsearch
contains files required in configuration and operation of ElasticSearch
intergrityi
contains configuration and modules of integrity check.
maili
contains product configuration of Mail-I product.
ndlp
contains configuration and modules of an Agent.
temp_index
temporarily saves the incident from the Indexing Server.

3
1. General

1.2.1. CM Folder Structure and Execution Files

bin
Contains the executable modules.
conf
Contains configuration files.
logs
Contains the logs of CM modules.
tomcat
Is a tomcat folder to execute the files.

1. To RUN / STOP CM
[root@tproxy]# service cm [start | stop]
OR
[root@tproxy]# /somansa/cm/tomcat/bin/catalina.sh [start | stop]

2. To replace cm.war
1) [root@tproxy]#service cm stop
2) [root@tproxy]#cp Filenametocopy/somansa/cm/tomcat/webapps/cm.war
3) Delete the CM folder in tomcat
[root@tproxy]#rm rf /somansa/cm/tomcat/webapps/cm
4) [root@tproxy]#service cm start

* Stopping may cause an error(s) during the operation, please do not stop while it
running normal.

4
1. General

1.2.2. CM Folder Structure and Execution Files

bin
Contains the common module JAR files
conf
Contains common setting files
lib
Contains shared library.
license
Reads license files.
log
Contains logs of SMS modules.
tomcat_indexer
Tomcat folder of the indexer.
tomcat_queryserver
Contains the Query Server.

1. To RUN / STOP the indexer and Query Server

[root@tproxy]#/somansa/common/tomcat_indexer/bin/catalina.sh [start | stop]
[root@tproxy]#/somansa/common/tomcat_queryserver/bin/catalina.sh [start | stop]

2. To replace war
1) Stop the service of a war file to replace (catalina.sh stop)
2) [root@tproxy]#cp Filenametocopy /somansa/common/servicetoreplace/webapps/
3) Delete the original file in tomcat
- If it is an Indexer
[root@tproxy]# rm rf /somansa/common/tomcat_indexer/webapps/SMSIndexerWeb_Spring
- If it is a Query Server
[root@tproxy]# rm rf /somansa/common/tomcat_queryserver/webapps/DLPQueryServer
4) Start the service of the replaced war file (catalina.sh start)

3. To RUN / STOP the SMSAnalyzer

[root@tproxy]#/somansa/common/script/SMSAnalyzer.sh [start | stop]

4. To RUN / STOP the SMSSummary

[root@tproxy]#/somansa/common/script/SMSSummary.sh [start | stop]

* Stopping may cause an error(s) during the operation, please do not stop while it
running normal.

5
1. General

1.2.3. Data Folder Structure

es_data
Saves the ElasticSearch indexes
es_log
Saves the ElasticSearch logs
es_tmp
Temporary storage of ElasticSearch
gfs_brick1
Actual storage of GlusterFS
gfs_data
Mount path of GlusterFS
productdata
Database file folder of postgresql
worm
Log forgery prevention folder

6
1. General

1.2.4. Folder Structure of DLP+ Center and Execution Files

conf
Contains configuration files.
fileupload
Is a temporary folder used to upload files.
logs
Saves logs of DLP+ Center modules.
tomcat
Tomcat folder to execute war files.

1. To RUN / STOP DLP+Center

- Service
[root@tproxy]#service dlpcenter [start | stop]
- Command to run
[root@tproxy]# /somansa/dlpcenter/tomcat/bin/catalina.sh [start | stop]

2. To replace DLPCenter.war
1) [root@tproxy]#service dlpcenter stop
2) [root@tproxy]#cp Filenametocopy /somansa/CM/tomcat/webapps/DLPCenter.war
3) Delete the DLPCenter folder in tomcat
[root@tproxy]#rm rf /somansa/CM/tomcat/webapps/DLPCenter
4) [root@tproxy]#service dlpcenter start

* Stopping may cause an error(s) during the operation, please do not stop while it running
normal.

7
1. General

1.2.5. Folder Structure of ElasticSearch and Execution Files

bin
Contains the executable files and service
configuration.
conf
Contains general settings.
plugins
Contains plugins.

1. To RUN / STOP Elasticsearch

[root@tproxy]#service elasticsearch [start | stop]

* Stopping may cause an error(s) during the operation, please do not stop while it running
normal.

8
1. General

1.2.6. Folder Structure of Mail-i and Execution Files

conf
Contains Mail-i configuration set from CM
script
Contains Mail-i Index Creation script

9
 1. General

1.2.7. Folder Structure of NDLP and Execution Files

built
Is a folder where the Agents are linked.
env/default/bin
Contains executable modules
env/default/config
Contains the Agent configuration.
env/default/dump
Saves the dump files.
env/default/log
Saves the Agent execution logs.
env/default/modules
Contains the protocol modules.
env/default/scripts
Contains the execution scripts.
env/default/servicemodules
Contains the service modules.

1. To RUN / STOP the agent

- Entire Agent
[root@tproxy]#/somansa/ndlp/env/default/script/ndlp-agent [start | stop]
- Content Analyzing Agent
[root@tproxy]#/somansa/ndlp/env/default/script/content-agent [start | stop]
- Traffic Processing Agent
[root@tproxy]#/somansa/ndlp/env/default/script/traffic-agent [start | stop]

2. To Manually update the Agent

1) Decompress the tar file of the agent to /somansa/ndlp/built folder
[root@tproxy]#tar xvf ndlp.8.0.1.XXX
2) Restart the agent
[root@tproxy]#/somansa/ndlp/env/default/scripts/ndlp-agent restart

* Stopping may cause an error(s) during the operation, please do not stop while it running
normal.

10
1. General

1.3.1. /somansa/cm/conf Files

conf.info
Contains the hash values of the configuration files.
database.info
Contains postgresql connection information.
password
Contains CM login password.
scm.info
Contains configured products used in the system.
timeout
Contains IP information which has been approved
to connect with CM.

1.3.2. /somansa/common/conf Files

DLPQueryServer.conf
Contains IP information of the Query Server
used by DLP+Center.
Hyboostinfo.xml
Saves the excess of bandwidth notification
configurations.
indexer.conf
Contains Indexing Server configuration.

11
1. General

1.3.3. /somansa/dlpcenter/conf Files

dlpcenter.properties
Contains the IP, language, product information
used in DLP+Center.

1.3.4. /somansa/elasticsearch/config Files

elasticsearch.yml
Contains basic configuration of ElasticSearch.

[Reference]
/somansa/elasticsearch/bin/service/ela
sticsearch.conf
Contains the service execution configuration.

12
1. General

1.3.4. /somansa/ndlp/env/default/config Files

default.property.script.xml
Contains basic configuration files.

property.script.xml
Contains Configuration files such as IP
connection, storage location.

traffic_agent.property.script.xml
Contains traffic process configuration such as
CN, IP Bypass.

13
1. General

1.4 Ports of Mail-i 8.0 & WebKeeper 9.0

Module Port

Agent 9600, 45123

Query Server 9500

Indexing Server 9700

ElasticSearch 9200, 9300

GlusterFS 111, 24007~8, 49152~49156

Redis 9800

HTTP 80

HTTPD 443

SSH 22

Time Sync 37, 123

WebKeeper Server 5959

WebKeeper Agent 5858

WebKeeper Realtime Monitoring 5860

WebKeeper Malicious Code 6000, 8787

14
1. General

1.5. FionaPerformanceMonitor Web

FionaPerformanceMonitor Web is an integration of FionaPerformanceMonitor and

Packet Record Commander.

1. FionaPerformanceMonitor Web Address

https://ip:9610

2. FionaPerformanceMonitor Screen Capture

Location : /somansa/ndlp/env/default/log
Method
1) Enter an interval time in the [sec] textbox at the left bottom corner.
2) Click [Save], [Stop] : Clicking [Save] will change the button to [Stop].

15
1. General

1.5.1. PerformanceMonitor (Left)

Packet Process (A/B/C)

(Current Value/Max Value on Graph/Max Value
after activation)

Memory Usage (A/B/C)

(Current Value/Max Value on Graph/Max Value
after activation)

1.5.2. PerformanceMonitor (Right)

Core Status
Packet Pump, Packet Sstop (A/B/C)
Currently flowing into Agent every 3 second
(Packet count/Packet size(KB)/Traffic
size(Mbps))

Result Status
DIP : Saving counts on the log(ES)
FIP : Number of attachment being saved(GFS)

16
1. General

1.5.3. Packet Recorder

Packet Recorder is a web service to capture the packets entering the Agent. It can be beneficial
when trying to capture the packet due to errors such as logging errors by protocol.

1. Packet Capturing Order

- Path : /somansa/ndlp/env/default/log The file will be saved as xxx.pcap in pcap file format.
- Method
1) Select the Protocol
2) Select the file size and counts
3) Specify the IP range : Enter in the format of ip-ip.
4) Click [Start], [Stop] : Try not to click the Start button multiple times to get the accurate
capture.

To ensure the smooth download from the web, please maintain the download path as a
default /somansa/ndlp/env/default/log.

17
2. Structure and Configuration of Modules by Modes

2.1.1 Single Structure (Mirror)

Execute all files in case of single structure.

18
2. Module Structure and Configuration by Modes

2.1.3 Multiple Structure (Agent+Storage 1)

It is a structure with an Agent and one storage.

Only CM, Indexing Server and NDLP-Agent will be executed in the Agent.

Creation of a gfs_volume to mount will occur in the storage.

Mount the gfs_volume of the storage in all equipment.

Execution of SMSAnalyzer, SMSSummary, QueryServer, and DLP+Center will take

place only in the storage equipment.

19
2. Module Structure and Configuration by Modes

2.1.4 Multiple Structure (Agent+ N Storages)

It is a configuration of an Agent and Storage n.

Creation of the gfs_volume to mount will occur in the main storage equipment.

After initializing the glusterfs in all STOAGE equipments, connect the

/somansa/data/gfs_brick1 folder of each equipment to the gfs_volume of the
main storage equipment.

For the case of ElasticSearch, the storage equipment should have their MASTER
and DATA mode enabled.

In the case of N main storage equipment, more memory should be allocated

compared to other storages.

20
3. Trouble Shooting

3.1. Known Issues Could not present the Reports

Measures

1) Check Process Performance

ps -ef | grep SMSSummary
If the process is performing, move to 5)

2) Check Crontab Registration

Perform crontab -e and check for the statement below.
*/10 * * * * /somansa/common/script/SMSSummaryD_check.sh >>
/somansa/common/log/SMSSummaryD_Restart.log 2>&1

3) Execute Process
[root@tproxy]#/somansa/common/script/SMSSummary.sh start

4) Check Executed Logs

[root@tproxy]#vi /somansa/common/log/SMSSummary_trace.log .
Inquire QA when error logs are found

5) Check DB(PostgreSQL) Task Status

SQL Query
SELECT task_stoptime, task_type, task_procstatus FROM mi_info.mi_taskrange WHERE
task_type='reporter';
When the value of task_procstatus is N, the task will not be performed. The value must
be changed to Y.

21
3. Trouble Shooting

3.2. Known Issue Could not inspect the Data

Measures

1) Check Process Performance

ps -ef | grep SMSSummary
If the process is performing, move to 4)

2) Check Crontab Registration

Perform crontab -e and check for the statement below.
*/3 * * * * /somansa/common/script/SMSAnalyzerD_check.sh >>
/somansa/common/log/SMSAnalyzerD_Restart.log 2>&1

3) Execute Process
[root@tproxy]#vi /somansa/common/log/SMSAnalyzer_trace.log
Inquire QA when error logs are found.

4) Check DB(PostgreSQL) Task Status

SQL Query
SELECT task_endtime, task_type, task_procstatus FROM mi_info.mi_taskrange WHERE
task_type=analyzer';
The final time of the logs pattern analyzation will be saved into the task_endtime column.
Inquire QA when the time does not change after going through the steps above.

22
3. Trouble Shooting

3.3. Known Issues Could not insert the Logs

Possible Causes
1) When the ElasticSearch does not work or has problems
2) Indexing Server Issues
A. When the indexing server does not work or has problems
B. When the ElasticSearch IP set in the indexing server is wrong
3) Agent Issues
A. When the Agent does not work or has problems
B. When the indexing server IP set in the Agent is wrong
4) When HR data matching was not done

Measures

1) Check Process Performance

After checking ps -ef | grep elasticsearch, when there is no running process, move to 2).
After checking ps -ef | grep tomcat_indexer, when there is no running process, move to 3).
After Checking ps -ef | grep Fiona, when there is no running process, move to 4).

2) ElasticSearch Issues
Execute ElasticSearch Service
[root@tproxy]#service elasticsearch start
Check the log below when the process does not execute
[root@tproxy]#tail f /somansa/data/es_log/SMS_LogServer.log

3) Indexing Server Issue 1

A. When the Indexing Server does not work or has problems
Execute the Indexing Server Process
[root@tproxy]#/somansa/common/tomcat_indexer/bin/startup.sh
Check the log below when the process does not execute
[root@tproxy]#tail f /somansa/common/tomcat_indexer/log/catalina.out

23
3. Trouble Shooting

3.3. Known Issues Could not insert the Logs

4) Indexing Server Issue 2

B. When the ElasticSearch IP set in the Indexing Server is wrong
Check the configuration values
[root@tproxy]#vi /somansa/common/conf
Check if the IP set in index server in configuration file is same with the ElasticSearch IP.

5) Agent Issue 1
A. When the Agent does not work or has problems
Execute the Agent
[root@tproxy]#/somansa/ndlp/env/default/scripts/ndlp-agent start
Check the log below when the process does not execute
[root@tproxy]#tail f /somansa/ndlp/env/default/log/ndlp_agent_ Date.rmk

6) Agent Issue 2
B. When the Indexing Server IP set in the Agent is wrong
Check the configuration values
[root@tproxy]#vi /somansa/ndlp/env/default/config/property.script.xml
Check if the IP set in mms.es_db.ip in configuration file is same with the ElassSearch IP.

24
3. Trouble Shooting

3.4. Known Issues Could not match the HR Data

How to check whether HR Data matching was not successful

http://ip:9200/_plugin/head/ > Check the date log from browser

1) Check the HR Data policy provided to the Agent

Check the /somansa/ndlp/env/default/log/rule_dump_personal.xml file
Inquire QA if HR Data is already reflected.

2) Check the HR Data policy provide by the Query Server

Check https://ip:9500/DLPQueryServer/NDLP/getPersonalInfo.do.
If the HR Data policy is not reflected, it is likely that there is an error in HR Data.

25
3. Trouble Shooting

3.4. Known Issues Could not match the HR Data

3) Policy Application
Click on POLICIES > Net App Prevent > [Apply Policy]
Apply the policy as an Agent in DLP+Center

4) Check whether the HR data has been applied in the Agent.

Check /somansa/ndlp/env/default/log/rule_dump_personal.xml.
Inquire QA if the HR data policy has been applied.

26
3. Trouble Shooting

3.5. Known Issue Could not export the Logs

Possible Causes
1) When the Redis does not work or has problems.

Measures

1) Check whether the process is running

Check whether the process is running
[root@tproxy]#ps ef | grep redis | grep 9800
Execute the process
[root@tproxy]#/somansa/common/script/redis_check.sh

27
3. Trouble Shooting

3.6. Known Issues Could not view the Incidents

Possible Causes
1) When the Query Server does not work or has problems
2) When the DLPQueryServer IP is wrong.

Measures

1) When the Query Server does not work or has problems

Check the logs of the Query Server
[root@tproxy]#tail -f /somansa/common/tomcat_queryserver/log/catalina.out
A. When there are Stop/Detroys in the logs
[root@tproxy]#/somansa/common/tomcat_queryserver/bin/catalina.sh stop
[root@tproxy]#/somansa/common/tomcat_queryserver/bin/catalina.sh start
B. When there are listenfails in the logs
Check the status of the Query Server Process
[root@tproxy]#ps ef | grep tomcat_query
Add the following statement when the ES_JAVA configuration has not been registered in
the /etc/profile when the return value of java is not 1.8 version.
ES_JAVA=/usr/java/jdk1.8.0_60
export ES_JAVA

2) When the Query Server IP of DLP+Center is wrong

Check the configuration files
[root@tproxy]#vi /somansa/common/conf/DLPQueryServer.conf
Check if the configuration values are set as ip=http://Query Server IP . If not, change
accordingly before restarting the DLP+ Centre.

28
3. Trouble Shooting

3.7 Known Issues Could not connect with T-Proxy In/Out Ports

Symptoms
T-Proxy Servers In/Out port link and the agent status remain normal but the
Internet is not connected.

Possible Causes

F.O.D. or T-Proxy Servers In/Out Port connections are reversed.

How to Check

1) Make dumps for In/Out packets

[root@tproxy]#tcpdump I eth2 w eth2.pcap host T-Proxy IP and host Intranet GW IP
[root@tproxy]#tcpdump I eth3 w eth3.pcap host T-Proxy IP and host Internet GW IP

2) Try Arping Intranet GW (While making Dumps)

[root@tproxy]#arping I br23 Intranet GW

3) Check the packets

If the arping leaves from GW(or eth3), the In/Out port connection is reversed.

Measures

Connect the In/Out connection lines in reverse.

29
3. Trouble Shooting

3.8. Known Issues Could not run ElasticSearch

Possible Causes
1) Incorrect configuration value of ElasticSearch
2) Lack of memory allocated to ElasticSearch

Measures

1) Incorrect configuration value of ElasticSearch

Check the Configuration Values
[root@tproxy]#vi /somansa/elasticsearch/config/elasticsearch.yml
Check if any of the configuration contains wrong values

2) Lack of memory allocated to ElasticSearch

[root@tproxy]#vi /somansa/elasticsearch/bin/service/elasticsearch.conf
Provide extra memories to Heapsize among configuration values (Unit : KB)

30
3. Trouble Shooting

3.9. Known Issue Could not create GlusterFS

Possible Causes
1) Firewall Issues
2) When a Brick to connect is already connected to another volume
When an error occurs when creating a volume
failed: Brick: 192.168.208.241:/somansa/data/gfs_brick1 not available. Brick may be
containing or be contained by an existing brick

Measures

1) Firewall Issues
Check the firewall
[root@tproxy]#vi /etc/sysconfig/iptables
Check whether the ports from 49152~49156 are allowed from firewall configuration

2) When a Brick to connect is already connected to another volume

Try again after initializing the GlusterFS setting from the Brick. (Refer to
/hyboost/common/es.gfs.init.sh, Page 27).

31
4. Trouble Shooting Useful Information

4.1. tail Commands

Purpose
1) To check the real-time log.
2) To check the logs of Actions.

Usage

1) How to Use
tail [Option] <File Name>

2) Main Functions
-n numbers : Print all numbers from the end
-f : Print added information whenever the file size changes

3) Example
SMSAnalyzer Check current Log
[root@tproxy]#tail f /somansa/common/log/SMSAnalyzer.out
SMSAnalyzer Check the last 20 logs from below
[root@tproxy]#tail n 20 /somansa/common/log/SMSAnalyzer.out

32
4. Trouble Shooting Useful Information

4.2. ebtables Command

Purpose
1) To use to configure the IP to be logged in T-Proxy

Usage

1) How to use the Commands

ebtables t broute [Option] BROUTING Configuration

2) Main Option
-L : Print ebtables List
-I : Add to ebtables
-D : Delete from ebtables

3) Examples
Exclude 192.168.10.67 from T-Proxy
[root@tproxy]#ebtables t broute I BROUTING --p IPv4 --ip-src 192.168.10.67 j
ACCEPT
[root@tproxy]#ebtables t broute I BROUTING --p IPv4 --ip-dst 192.168.10.67 j
ACCEPT
Deactivate excluding 192.168.10.67 from T-Proxy
[root@tproxy]#ebtables t broute --D BROUTING p IPv4 ip-src 192.168.10.67 j
ACCEPT
[root@tproxy]#ebtables t broute --D BROUTING --p IPv4 --ip-dst 192.168.10.67 j
ACCEPT

33
4. Trouble Shooting Useful Information

4.3. Bypass specific CN/IP from HTTPS

How to Use

1) Add /somansa/ndlp/env/default/config/traffic_agent.property.script.xml option

<list name="tproxy.ssl.bypass_cn_list">
<string value="*.python.org"/>
<string value="*.update.microsoft.com"/>
<string value="itunes.apple.com"/>
</list>
<list name="tproxy.ssl.bypass_network_list">
<string value="103.246.57.52/25"/>
<string value="210.103.240.0/24"/>
</list>

2) Restart Agent
[root@tproxy]#/somansa/ndlp/env/default/scripts/ndlp-agent restart

34
5. Trouble Shooting ICAP/HTTPS Proxy

5.1. ICAP Integration

1) Precondition
- ICAP-Agent Running
/somansa/ndlp/env/default/scripts/icap-agent start

2) ICAP Port : 1344

3) ICAP Config File

- /somansa/ndlp/env/default/config/icap.property.script.xml

4) Mirroring Mode (default off)

- Option for only Mirroring
[root@tproxy]#vim /somansa/ndlp/env/default/script/config/icap.property.script.xml
change the value bellow to on
<string name="icap.mirror_mode" value="on"/>

35
5. Trouble Shooting ICAP/HTTPS Proxy

5.2. Somansa HTTPS Proxy Integration

1) Precondition
- Traffic-Agent Running
/somansa/ndlp/env/default/scripts/traffic-agent start

2) HTTPS Config File

- /somansa/ndlp/env/default/config/traffic_agent.property.script.xml

3) Proxy Setting for User

1) IE/Chrome
(1) Open Internet Explorer by clicking the Start button. In the search box, type Internet
Explorer, and then, in the list of results, click Internet Explorer.
(2) Click the Tools button, and then click Internet Options.
(3) Click the Connections tab, and then click LAN settings.
(4) Select the Use a proxy server for your LAN check box.
(5) Click the Advanced button, and then type https proxy server addresses to use.
(6) In the Port box, type 13128.
(7) When you are finished making changes, click OK until you return to Internet Explorer.

2) Firefox
(1) Open Firefox by clicking the Start button. In the search box, type Firefox, and then, in
the list of results, click Firefox.
(2) Click the Options button, and then click Advanced.
(3) Click the Network tab, and then click Settings.
(4) Select the Manual proxy configuration.
(5) In the SSL Proxy box, type the address of the proxy server.
(6) In the Port box, type 13128.
(7) When you are finished making changes, click OK until you return to Firefox.

36