11gR2 RAC: Why SCAN & Node Listeners in 11gRAC

Hello,

A long notes (confusing) on listeners in 11gR2 RAC.

You all aware of we have two listeners running in database servers in Grid Infrastructure (Aka RAC)
namely scan listeners and node listeners.

How does they work?

The SCAN works by being able to resolve to multiple IP addresses reflecting multiple listeners in the
cluster handling public client connections. When a client submits a request, the SCAN listener listening on
a SCAN IP address and the SCAN port is contracted on a client’s behalf. Because all services on the
cluster are registered with the SCAN listener, the SCAN listener replies with the address of the local
listener on the least-loaded node where the service is currently being offered. Finally, the client
establishes connection to the service through the listener on the node where service is offered. All of
these actions take place transparently to the client without any explicit configuration required in the
client.

A picture can save 100 words of explanation

We have 3 scan listeners , registering the IPC(TCP) end points as the RAC instances
(remote_listener) registered with PMON of rac instances and the PMON also registers
to node listeners (local_listener) which actually points to your database.

 Indeed they both run from the same home (Grid Home), but for different functionality. From
11gR2 onwards these listeners are part of clusterware and managed by oraagent. This oraagent
take care of listeners. This oraagent spawned by crsd takes care of our listeners in terms of
configuration and monitoring.
 Especially when the listener is started by oraagent you can see the lines in the listener.ora file
#line added by agent.
  Another important aspect of the listener.ora files are these files are managed by oraagent,
whenever there is a clusterware command is used to manage this listeners. for example srvctl
etc.
 Further, the listeners will be monitored by oraagent process every 60 seconds, which you can
find in the  $GRID_HOME/log/nodename}/agent/crsd/oraagent_oracle/oraagent_oracle.log
 You must set the local_listener and remote_listener parameter to scan listener, description or
address list of the same. Since these is the only way now to register your database/instance to
scan listener. Even if you dont specify the oraagent will automatically add local_listener values ,
but you must exclusively set remote_listener parameter to cluster scan name which will provide
you the TAF. 

For example,

If you delete the listener.ora and restart the listener with srvctl start listener, a listener.ora will reappear.
The original configuration will reappear in listener.ora and the manually modified listener.ora will be
renamed (with a timestamp suffix) 

 The agent also creates and maintains the file: endpoints_listener.ora this file is there for
backward compatibility,
 Oraagent comes into action also at instance startup, when the instance is started with srvctl (as
opposed to ‘manually’ started instance from sqlplus) and sets LOCAL_LISTENER  parameter,
dynamically (this is done with an alter system command and only if the parameter has not been
set on spfile).

You may also has observed in listener.ora file there were no tcp/ip settings, Where are the TCP/IP
settings of my listeners in listener.ora? and Only IPC endpoints are listed in listener.ora,  As you see
below the listener.ora contains only scan listener which contain IPC protocol 

Notes:- Before proceed further

What is dynamic listening end point?

dynamic listening endpoint=the address or connection point to the listener. The most known endpoint in
the oracle DB world being TCP, port 1521.

Oraagent connects to the listener via IPC and activates the TCP (TCPS, etc) endpoints as
specified in the clusterware configuration

Example Listener.ora settings:-

[oracle@rac1]$ cat /u01/app/11.2.0.2/grid/network/admin/listener.ora 

LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent 
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent 
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent 
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent 
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent 
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent 
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent 
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent 
WALLET_LOCATION = 
(SOURCE = 
(METHOD = FILE) 
(METHOD_DATA = 
(DIRECTORY = /u01/app/11.2.0.2/grid/network/admin/cost) 
) 
) 
So here no SID, no Port, no host address configured, this is due to the settings whole managed in the
clusterware.

SCAN Listeners has an endpoints to the scan listeners in endpoint_listener.ora via IPC or TCP looks
like.

endpoints_listener.ora

LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=strac201a-vip)(PORT=1522))
(ADDRESS=(PROTOCOL=TCP)(HOST=172.24.21.89)(PORT=1522)(IP=FIRST)))) # line added by Agent 
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=strac201a-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=172.24.21.89)(PORT=1521)(IP=FIRST)))) # line added by Agent

tnsnames.ora

MYDB1522 = 
(DESCRIPTION = 
(ADDRESS = (PROTOCOL = TCP)(HOST = strac201-scan.frxntnyc.frx2.com)(PORT = 1521)) 
(CONNECT_DATA = 
(SERVER = DEDICATED) 
(SERVICE_NAME = MYDB1522) 
) 
)

As you can see the listener.ora files points to IPC scan listeners end points of vip/hostnames in
endpoint_listener.ora and once these listeners started , the database pmon registers the sid/services to
this listeners (see picture above) as per tnsentries or the local_listener parameter, the pmon register to
both node and scan listeners.

For example,

When you just start a listener ,

$GRID_HOME/bin/lsnrctl stop listener; $GRID_HOME/bin/lsnrctl start listener;

This command starts only IPC endpoint

However oraagent is posted at listener startup and makes active the rest of the endpoints (notably
listening on the TCP port), this can be seen for example by running the following a few seconds after
listener restart: $GRID_HOME/bin/lsnrctl status listener (which will list all the active endpoints), the active
endpoints are infact the database instances. This is possible due to the listener.ora parameter you usually
see ENABLE_GLOBAL_DYNAMIC_ENDPOINT_{LISTENER_NAME}=ON 

Another check,

when you disable the ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=OFF in listener.ora

and manually stop the listener (manually means not using srvctl)

$GRID_HOME/bin/lsnrctl stop listener 

$GRID_HOME/bin/lsnrctl start listener

When you check listener.ora and check that the parameter edited above has not changed, that is  in this
case the TCP endpoint will not be started, that is the listener will be listening only on IPC.
Check with: $GRID_HOME/bin/lsnrctl status listener

If we try do the same exercise by stopping and starting the listener with srvctl, as it would be
the typical way to do it, we will see that the parameter
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER in listener.ora will be set again to ON. 

Now, morale of the post,

Always, use srvctl to add listeners,stop,start

And Always, Use srvctl to start instances/db as it set the local_listeners parameter automatically to
endpoints.

So that the listener.ora file manages promptly/neatly by oraagent process.

-Thanks

Suresh