Document 1088414.

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=fzv7ps55o_4&id=1088414.1

How to Troubleshoot Communication From the Oracle Management Service (OMS) to a Grid Agent in 10g Enterprise Manager Grid Control? (Doc ID
1088414.1)
Modified: Sep 5, 2013

Type: TROUBLESHOOTING

In this Document
Purpose
Troubleshooting Steps
Quick Checklist
1. OMS / Agent Component level issues
2. Hostname/IP Address Resolution Issues
3. Firewall Setup / Proxy Server Issues
References

APPLIES TO:
Enterprise Manager Base Platform - Version 10.1.0.2 to 10.2.0.5 [Release 10.1 to 10.2]
Information in this document applies to any platform.
Checked for relevance on 05-Sep-2013

PURPOSE
The 10g Enterprise Manager Grid Control includes the following Components:
Oracle Management Service (OMS)
Grid Agent, which is responsible for monitoring the certified targets.
Grid Control Repository Database, hosted in a certified Oracle Database.
Grid Console, which is the User-interface rendered in a web-browser.
This document describes the troubleshooting steps to be followed when there is a communication problem between the Oracle Management Service (OMS) and the Grid Agent.
Additional references:
Note 951076.1: How to Troubleshoot Communication From a Grid Agent to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?
Note 1089443.1: How to Troubleshoot Communication From the Grid Console (UI) Machine to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?
Note 1089693.1: How to Troubleshoot Communication From the Oracle Management Service (OMS) to the Grid Control Repository Database in 10g Enterprise Manager Grid Control?

TROUBLESHOOTING STEPS
Incorrect communication setup between the OMS and the remote Agents can result in the following problems:
In the Grid Console, Setup -> Agents -> clicking on the 'Agent name' displays a red warning/error message such as:

Communication between the Oracle Management Service host to the Agent host is unavailable. Any functions or displayed information requiring this communication link will be unavailable. For example: deleting/configuring/adding targets,
uploading metric data, or displaying Agent home page information such as Agent to Management Service Response Time (ms).
All the targets monitored by the Agent, including the Agent itself are not shown in the correct status in the Grid Console.
Many pages related to this Agent's monitored targets, in the Grid Console return errors such as:

1 of 6

22/05/2014 10:28

Document 1088414.1

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=fzv7ps55o_4&id=1088414.1

Error getting data for target <Host target name>. Exception: No route to host
OR
oracle.sysman.emSDK.emd.comm.CommException: UnknownHostException sending request:: agentmachine.domain:agentmachine.domain
Trying to configure a database monitored by the remote Agent from Grid Console results in the error:
oracle.sysman.emSDK.emd.comm.CommException: Cannot establish proxy connection: 302 Moved Temporarily
OR
oracle.sysman.emSDK.emd.comm.CommException: javax.net.ssl.SSLException: SSL handshake failed: SSLIOClosedOverrideGoodbyeKiss
OR
Communication from the Oracle Management Service host to the Agent host failed. Refer to help for details. IOException in sending Request :: Connection refused: connect
Trying to access any of the database target pages or testing the preferred credentials could return:

Io exception: The Network Adapter could not establish the connection

Any EM-level job submitted against the targets monitored by the agent machine hangs in 'Running' status. The <OMS_HOME>/sysman/log/emoms.trc shows errors such as:

2007-06-27 15:08:08,841 [JobWorker 4046:Thread-30] INFO em.jobs executeCommand.526 - remoteOp:

oracle.sysman.emSDK.emd.comm.CommException: Agent HTTP version HTTP/1.0 not HTTP/1.1, can't send 0
byte streaming request.
at oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse_
(EMDClient.java:2986)
at oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse_
(EMDClient.java:2870)
....

This affects operations such as cloning, creating a standby database etc.

The troubleshooting steps are grouped as per the following areas which can cause the communication failure:
1. OMS / Agent Component level issues.
2. Hostname/IP Address Resolution Issues
3. Firewall Setup / Proxy Server Issues.
Log/Trace files to check
<OMS_HOME>/sysman/log/emoms.trc
<OMS_HOME>/sysman/log/emoms.log
Note 229627.1: How to Locate 10g Grid Control OMS Log / Trace Files and Control their Size and Other Details

Quick Checklist
Verify that the Agent on the target machine is up and running using:

cd <AGENT_HOME>/bin
emctl status agent

2 of 6

22/05/2014 10:28

Document 1088414.1

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=fzv7ps55o_4&id=1088414.1

The command should return output such as:

emctl status agent

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
--------------------------------------------------------------Agent Version : 10.2.0.5.0
OMS Version : 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home : /home/oracle/OracleHomes/agent10g
Agent binaries : /home/oracle/OracleHomes/agent10g
Agent Process ID : 24465
Parent Process ID : 24449
Agent URL : https://agentmachine.domain:1830/emd/main/
Repository URL : https://omsmachine.domain:1159/em/upload
Started at : 2010-04-22 15:35:39
Started by user : oracle
....
---------------------------------------------------------------

which indicates that the Agent has started up fine. Also review the <AGENT_HOME>/sysman/log/emagent.nohup to ensure that the Agent is not re-starting frequently, which can affect the OMS to Agent communication.
Refer to Note 548928.1: Enterprise Manager Grid Control Agent 10g, Process Control (Start, Stop & Status) Troubleshooting Guide
Verify that the Agent's URL, as seen in the Grid Console -> Setup -> Agent name page is the same as the value configured for the EMD_URL in the <AGENT_HOME>/sysman/config/emd.properties file.
Refer Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent?

1. OMS / Agent Component level issues

If the Agent machine is configured with DHCP and/or the IP address of the machine has recently changed, the OMS will not be able to communicate with the Agent.
Refer Note 605009.1: Problem: OMS Cannot Communicate with Agent if IP Address of the Grid Agent Machine is Changed
If there is a rogue emagent process on the target machine, then the OMS log/trace files could show communication errors. Refer Note 733879.1: Communication: OMS Log/Trace Files Show 'ERROR eml.OMSHandshake processFailure' for Agent
Already Removed from Grid Console
If the Agent is not capable of accepting incoming connection requests from the OMS, then the communication will fail. Refer Note 550452.1: Communication: OMS to Agent Communication Fails with 'IOException in sending Request :: Broken pipe'
Verify if there are multiple Agents installed / discovered from this machine. Refer to Note 435728.1: Communication: OMS to Agent Communication Fails with "Connection refused" if Multiple Agent Targets are Discovered

2. Hostname/IP Address Resolution Issues

If the OMS and Agent Components are located in separate machines, then the hostname/IP address resolution should work correctly from the OMS to the Agent machine.
Refer Note 763844.1: How to Verify the Hostname/IP Address Resolution Between the 10g Enterprise Manager Grid Control Components?
If the OMS is unable to resolve the hostname / IP address of the Agent machine, the <OMS_HOME>/sysman/log/emoms.trc will show errors such as below, when trying to access the Agent Homepage in the Grid Console:

2010-04-26 12:01:51,405 [EMUI_12_01_26_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd] ERROR emdConfig.EmdConfigTargetsData getEmdUploadData.1732 - IOException in sending

Request :: No route to host

To verify the Hostname / IP Address resolution from OMS to Agent machine, follow below steps:
Collect the following details on the Agent machine:
Hostname and the corresponding IP Address on which the Agent is configured:

3 of 6

22/05/2014 10:28

Document 1088414.1

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=fzv7ps55o_4&id=1088414.1

cd <AGENT_HOME>/bin
emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
--------------------------------------------------------------Agent Version : 10.2.0.5.0
OMS Version : 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home : /home/oracle/OracleHomes/agent10g
Agent binaries : /home/oracle/OracleHomes/agent10g
Agent Process ID : 24465
Parent Process ID : 24449
Agent URL : https://agentmachine.domain:1830/emd/main/
Repository URL : https://omsmachine.domain:1159/em/upload
Started at : 2010-04-22 15:35:39
Started by user : oracle

The hostname is the one seen in the 'Agent URL' field.

Obtain the IP address for this hostname using:

ping <agentmachine.domain>

Output of these commands:

ping <IP address of the Agent machine>

ping <hostname.domain of the Agent machine>
ping <hostname of the Agent machine>
nslookup <IP address of the Agent machine>
nslookup <hostname.domain of the Agent machine>
nslookup <hostname of the Agent machine>

Collect the following details from the OMS machine:

ping <IP address of the Agent machine>

ping <hostname.domain of the Agent machine>
ping <hostname of the Agent machine>
nslookup <IP address of the Agent machine>
nslookup <hostname.domain of the Agent machine>
nslookup <hostname of the Agent machine>

Compare the output of the above commands on OMS and Agent machines - the outputs should match. If there is a difference or an error, please enlist the help of your System / Network Administrator to correct the configuration in the hosts file or the
DNS.
Note:
1. If all the above commands work fine but the OMS still fails to communicate with the Agent, then stop and restart the OMS once to reset the TCP caching

<OMS_HOME>/opmn/bin
opmnctl stopall
<OMS_HOME>/opmn/bin>
opmnctl startall

2. If the Agent machine has multiple NIC cards / IP addresses, the Agent can be bound to a particular hostname / IP address combination using steps in:

4 of 6

22/05/2014 10:28

Document 1088414.1

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=fzv7ps55o_4&id=1088414.1

Note 390444.1: How to: Tell the agent to listen to only one specific NIC Network Interface Card?
If the hostname / resolution works fine from the OMS to Agent but the communication still fails, then check for the presence of Firewall or Proxy Server in the setup using the steps below.

3. Firewall Setup / Proxy Server Issues

For details about configuring the Firewall and using the Proxy Server for the EM components, refer
Note 1088393.1: How to Verify the Communication Between the 10g Enterprise Manager Grid Control Components via Firewall/Proxy?
If the Agent port is blocked, then the <OMS_HOME>/sysman/log/emoms.trc will show:

2008-12-01 11:21:25,535[EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd] ERROR emdConfig.EmdConfigTargetsData

getEmdTargetsList.1767 - CommException:
Unable to get list of targets from emd-getEmdTargetsList()
2008-12-01 11:21:25,541 [EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd] ERROR emdConfig.EmdConfigTargetsData
getEmdTargetsList.1769 - Connection timed out oracle.sysman.emSDK.emd.comm.CommException: Connection timed out

The following error is displayed when trying to look at the Targets -> Agent Host -> Performance page:

An error has occurred!

Unable to obtain data for target solaris.oracle.com. The target may be down. Switching to the last 24 hrs view
Incorrect Proxy server configuration at the OMS side, can cause problems described in
Note 395717.1: Communication: OMS to Agent Communication Fails With 'Cannot Establish Proxy Connection' Due to Proxy-Related Settings
To verify the communication between OMS to Agent machine, when Firewall / Proxy server is in use:
Identify the Agent port and URL using the steps in
Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent?
Test the connectivity to the Agent URL from the OMS machine, using one of the following methods:
Open a web-browser on the OMS machine and try to access these URL's:

http://agentmachine.domain:agentport/emd/main
OR
https://agentmachine.domain:agentport/emd/main

The URL must return an output similar to:

EMAgent10.1.0.2.0
Congratulations, EMAgent is working!
Use telnet

telnet agentmachine.domain <agent port>

Sample output:

telnet agentmachine.domain 3872

Trying 20.20.20.20...
Connected to agentmachine.domain.

5 of 6

22/05/2014 10:28

Document 1088414.1

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=fzv7ps55o_4&id=1088414.1

Escape character is '^]'.

If the access to the port is blocked due to a firewall, then the above command will fail with:

telnet agentmachine.domain 3872

Trying 20.20.20.20...
telnet: connect to address 20.20.20.20: Connection refused

Use wget

wget <agent http url>

OR
wget --no-check-certificate <agent https url>

If any of the above commands fail, please contact your Network Administrator to determine if there is a Firewall / Proxy Server in use and check the configuration.

REFERENCES
NOTE:1088393.1 - How to Verify the Communication Between the 10g &11g Enterprise Manager Grid Control Components via Firewall/Proxy?
NOTE:1089443.1 - How to Troubleshoot Communication From the Grid Console (UI) Machine to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?
NOTE:1089693.1 - How to Troubleshoot Communication From the Oracle Management Service (OMS) to the Grid Control Repository Database in 10g Enterprise Manager Grid Control?
NOTE:229627.1 - Enterprise Manager Grid Control Management Service (OMS) 10g - Locate and Manage the Log and Trace Files
NOTE:235290.1 - Understanding the Enterprise Manager Management Agent 10g 'emd.properties' File
NOTE:358953.1 - What ports are used in communication between the Grid Control OMS and a Management Agent?
NOTE:390444.1 - Grid Agent Configuration: How to Configure the Agent to Listen to Only One Specific Network Interface Card (NIC)?
NOTE:395717.1 - Communication: OMS to Agent Communication Fails With 'Cannot Establish Proxy Connection' Due to Proxy-Related Settings
NOTE:435728.1 - Communication: OMS to Agent Communication Fails with "Connection refused" if Multiple Agent Targets are Discovered
NOTE:471842.1 - Understanding Proxy Settings in Enterprise Manager Grid Control
NOTE:548928.1 - Troubleshooting Grid Control Agent 10g, Process Control (Start, Stop & Status)
NOTE:550452.1 - Communication: OMS to Agent Communication Fails with 'IOException in sending Request :: Broken pipe'
NOTE:605009.1 - Communication: OMS to Agent Communication Fails if IP Address of the Grid Agent Machine is Changed
NOTE:733879.1 - Communication: The emoms.trc File Shows 'ERROR eml.OMSHandshake processFailure' for an Agent Removed from Grid Console
NOTE:763844.1 - How to Verify the Hostname/IP Address Resolution Between the Enterprise Manager Grid Control Components?
NOTE:951076.1 - How to Troubleshoot Communication From a Grid Agent to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?

6 of 6

22/05/2014 10:28