UNDERSTANDING SCADA FAILOVER

Operators view process data received from a SCADA server using a View node. Should the SCADA server become unavailable, process data can become unavailable to the operator at the View node. SCADA failover increases the availability of data to the View node and minimizes the amount of time that data is unavailable. It does this by allowing the View node to connect to more than one SCADA server. By recognizing multiple paths to the data, FIX32/iFIX can switch from one path to another automatically, improving overall system availability for critical operations. Switching from one connection to another is known as failover. The backup SCADA server feature allows you to connect a View node to both primary and backup SCADA servers, or SCADA server pair, that are connected to the same PLC. This feature provides two paths to the same process data instead of just one. The View node establishes and maintains the connections to both the primary and backup SCADA servers, either of which can be the active serverthe node with which the View node is currently communicatingor non-active SCADA server. When the connection to the primary SCADA server is lost, FIX32/iFIX automatically fails over to the backup SCADA server. Sometimes SCADA failover and SCADA redundancy are used interchangeably. Unfortunately, this may cause customers to think that the feature synchronizes tag values between PDB files. This, of course, is not the case. The only synchronization that takes place is with alarm acknowledgement. Failover actually accomplishes two things; alarm acknowledgments are synchronized and View nodes can fail over to another SCADA in the event of connection loss or needed downtime.

FIX32/iFIX Failover Explained

When FIX32/iFIX starts, the View node attempts to establish communication with its primary and backup SCADA servers. The primary node is the SCADA node you want the View node to communicate with at startup. The backup node is the SCADA node you want the View node to communicate with if the primary node becomes unavailable. If both nodes are available, the View node establishes a connection with both of them. If only one SCADA server is available, the View node establishes a connection with it. If neither SCADA server is available, the View node polls both nodes until it establishes a connection with at least one SCADA server. If failover occurs to the backup SCADA, the View node remains connected to the backup SCADA even when the primary SCADA becomes available. Automatic failover to the primary SCADA only occurs if there is a connection loss to the backup SCADA and the primary SCADA is available. Data is written to the active SCADA server only. For example, if the active node is PACKER1 and the nonactive node is PACKER2, before the failover the View node writes data to PACKER1. After the failover, the View node writes data to PACKER2. When an alarm occurs on a SCADA server, the alarm is sent to View node. The View node accepts alarms from the active SCADA only, regardless of whether it is the primary or backup SCADA. You cannot view alarms generated by the non-active SCADA. When you configure networking and SCADA options in the SCU for both SCADA servers, the Alarm Startup Queue Service is automatically enabled. This service: Ensures that alarms are not lost during session loss and reconnection. Synchronizes alarms, so you only have to acknowledge an alarm once.

Comparison between FIX32 and iFIX Failover

iFIX offers improved display portability by using logical node names to represent the primary and backup SCADA pair. With iFIX failover, the backup SCADA server feature uses a logical node name to represent the physical names of the primary and backup SCADA nodes. The primary node is the SCADA server you want the View node to communicate with at startup. The backup node is the SCADA server you want the View node to communicate with if the primary node becomes unavailable. The View node communicates to the logical node name, and iFIX substitutes the physical node name at run time based on which SCADA server is available. The combination of the logical node name and physical primary and backup SCADA server names is referred to as the primary and backup grouping. You configure the primary and backup grouping in the View node's SCU as well as on the primary and backup SCADA nodes. FIX32 does not use a logical node name for failover. Any alarms received from the backup node are modified to appear as if they came from the primary node. Compare this with iFIX where no modification takes place and the logical node name is always used. The example below demonstrates how to configure FIX32 failover.

FIX32 Failover Configuration Example

The following figure shows a View node (VIEW1) connected to a SCADA redundant pair (SCADA1 and SCADA2).

In this example, to configure VIEW1 for failover, perform the following: 1. Open VIEW1s SCU file and choose Configure->Network. 2. In the Network Configuration dialog box that opens up, add SCADA1 as a remote node. 3. Click to highlight the SCADA1 entry in the Configured Remote Node list box, and choose the Configure button. This displays the following dialog box shown below. Notice how SCADA2 is entered as the backup node. 4. Save SCU changes.

The only configuration needed on SCADA1 and SCADA2 (if you want alarm acknowledgements to be synchronized) is to fill in the Partner SCADA field in the SCU. To fill in this field on SCADA1 and SCADA2, perform the following: 1. Open the SCU on SCADA1 and choose Configure->SCADA. 2. Enter SCADA2 in the Partner SCADA field and click Ok button. 3. Save SCU changes. 4. Perform steps 1 through 3 on SCADA2, but instead enter SCADA1 in the Partner SCADA field. The purpose of the Partner SCADA field is that it tells the SCADA to establish a connection with its partner to exchange alarm information and to synchronize alarm acknowledgements. If you require that SCADA1 and SCADA2 function as View nodes as well, only the following would need to be configured on SCADA2: 1. Open SCADA2s SCU file and choose Configure->Network. 2. In the Network Configuration dialog box that opens up, add SCADA1 as a remote node. 3. Click to highlight the SCADA1 entry in the Configured Remote Node list box, and choose the Configure button. Enter SCADA2 as the backup node and click Ok. Click Ok again in the Network Configuration dialog box. 4. Save SCU changes.

iFIX Failover Configuration Example

The following figure shows a View node connected to a SCADA redundant pair (PACKER1 and PACKER2).

In this example, to configure the View /PACKER1/PACKER2 nodes for failover, perform the following on each: 1. Open the SCU file and choose Configure->Network. 2. In the Network Configuration dialog box that opens up, add LGCL_ND1 as a remote node. 3. Click to highlight the LGCL_ND1 entry in the Configured Remote Node list box, and choose the Configure button. This displays the following dialog box shown below. 4. Save SCU changes.

If you need to synchronize alarm acknowledgments on PACKER1 and PACKER2, fill in the Partner SCADA field in the SCU. To fill in this field on PACKER1 and PACKER2, perform the following: 1. Open the SCU on PACKER1 and choose Configure->SCADA. 2. Enter PACKER2 in the Partner SCADA field and click Ok button. 3. Save SCU changes. 4. Perform steps 1 through 3 on PACKER2, but instead enter PACKER1 in the Partner SCADA field. The purpose of the Partner SCADA field is identical to the same field in FIX32 in that it tells the SCADA to establish a connection with its partner and synchronize alarm acknowledgement information.

The ALM_SYNC Service

The function of ALM_SYNC is to synchronize the acknowledgement of alarms between two SCADA nodes set up in a SCADA redundant fashion. By default, the ALM_SYNC application will launch if the SCU file has a partner SCADA node defined AND the following entry in the FIX.INI file: [SCADA ALARMS] RUN=%ALM_SYNC.EXE When ALM_SYNC launches, it creates the Alarm Sync queue that is examined on a continuous basis for alarms received from the partner node. Also, knowledge of who the partner SCADA node will be is obtained from the SCU file. Once startup of ALM_SYNC completes, it watches for alarm messages to appear in the Alarm Sync queue. Upon receipt of an alarm message, the ALM_SYNC program will check if it is an alarm acknowledgement type of message, which it always should be if it is placed in this queue. If it is an acknowledgement message, then we use the Variable Specification Parameter (VSP) data structure from the acknowledgement message to determine which tag in the local database needs to be acknowledged. It is important to note that the Alarm Sync queue does not appear in Almstat.exe utility, but the utility may be useful for tracking which alarms are actually arriving at the computer and being delivered to the other alarm services. Another name used interchangeably with VSP is "index". You will see the word "index" used at times in various knowledge base articles that pertain to alarm synchronization and the process database. The fact that we use the VSP from the partner node to find the tag in the local database is the reason that both databases must be identical. In order for this to work, they must not only have the same tags but the tags must have been added in the same order which will give them the same VSP. This is why we recommend that a database import be done on both nodes from the same GDB file (or PDB file if iFIX), or the database file itself should be copied to both nodes. If the VSP's do not match, then the alarm cannot be acknowledged locally and synchronization will not work. Even worse, you could acknowledge the wrong alarm! Once the local database tag has been identified, its A_NALM field is examined to determine if it is truly in an "unacknowledged alarm" state. If it is not, then nothing is done. If the alarm is in an unacknowledged state, then we write to the tag's A_NALM field to indicate that it is now acknowledged. Each alarm message received will be handled in this way.

Working with the NSD tag

FIX32/iFIX provides an NSD tag that you can use when designing displays for monitoring and controlling SCADA server failover. This tag is not a database block. It is a special tag residing on each networked node that displays diagnostic, failover, and network information. You can monitor/control failover of the network using the NSD tags diagnostic and control fields. This feature allows you to monitor sessions on your network, determine which nodes are active, and display the name of the local node. In the event that a SCADA server becomes unavailable, you can also display an error code and text describing the current state of the connection with each SCADA server.

NSD tag fields differ slightly between the FIX32 and iFIX products. Make sure to consult the electronic books for each product on NSD tag usage. These books also provide examples of how to programmatically control failover. In addition to providing NSD fields, GE Fanuc supplies diagnostic displays you can use with FIX32/iFIX. These displays are pre-built pictures with links that reference the diagnostic NSD fields. By using these displays, you can quickly display diagnostic information for any View node. FIX32 supplies the following diagnostic displays. These displays are provided in the Picture path of your View node. You need to modify these pictures to reference your local node name (replace all instances of the word LOCAL with your View node name): NSD.ODF - a network status display showing the local node name, each incoming connection, each outgoing connection, and each connection's status. This display always shows the primary node name even if the backup node is the active node. NSDREDUN.ODF - a network status display showing the information in NSD.ODF plus the names of the active node, the primary node, and the backup node. iFIX supplies a diagnostic display that contains links referencing diagnostic NSD fields. This display, called NetworkStatusDisplay.GRF, allows you to view the local node name, each incoming connection, each outgoing connection, and each connection's status. This display always shows the logical node name even if the backup node is the active node. The NetworkStatusDisplay.GRF file is provided in the Picture path of your iClient. You need to modify this file to reference your local node name by replacing all instances of the word THISNODE with the name of your View node. Make sure to replace all instances of THISNODE in the VBA code as well in the picture. If you need to view diagnostic data on the SCADA nodes, iFIX supplies two pictures; LocalAsBackup.grf (to be used on the backup SCADA), and LocalAsPrimary.grf (to be used on the primary SCADA). You cannot use NetworkStatusDisplay.GRF on SCADA nodes.

Adjusting Alarm Queues for Failover

The implementation of logical node names is more complete in iFIX than in FIX32. Alarms are sent with logical names from the SCADA. View nodes simply receive and display the alarms. Contrast this with FIX32 Auto failover where the name substitution is done at the view node. iFIX Alarming Tips (as it pertains to failover): Use Alarm ODBC and look at the physical node name field if you want to know which SCADA really sent an alarm. There is no other way to know which node actually sent the alarm. Since Logical Node Names requires the same PDB on each SCADA, you will also need the same AAD file on each SCADA or put it into a shared directory. Since a FIX32/iFIX View node and active SCADA node can receive alarms from the non-active SCADA, proper consideration needs to be given to alarm queue sizes. For example, on the View node you must size your summary queue correctly. It has to be twice as big as you would think. The reason is that it can

contain a full set of alarms from the non-active node. These alarms are in the summary queue but are marked as hidden until that node becomes active. The ALMSTAT.EXE Utility in the FIX32/Dynamics root directory can be used to configure the alarm queues. Run ALMSTAT.EXE from the run prompt and press Q. The Alarm Queue sizes should be increased if an OVER condition exists in the ALMSTAT.EXE Utility. You should not optimize your Alarm Queues by just bumping them up to the maximum of 9999. The larger you make your queues the more memory is allocated to them. The proper way is to set the queues to the maximum of 9999 on both SCADA nodes and restart. Then, force all your alarms in each database and view the ALMSTAT utility. The peak column will display the maximum number of alarms coming into each queue. The key is to get the queue slightly higher than the peak displayed in the utility. A simpler method in adjusting the alarm queue sizes would be to simply count the total number of tags in the PDB that can alarm, and multiply this value by 2. Then adjust your queues accordingly with the result.

Security Rights and Partnered SCADAs

If redundant SCADAs have a logged in user with different levels of security rights, then it is possible that the write could happen on one SCADA and not the other partnered SCADA. To remedy this, you can allow the security rights of the logged in user to be overridden so that the ALM_SYNC alarm synchronization can be done correctly. Note that this override of the logged in user security rights is limited to the writing of the A_NALM block field only. In order to enable this feature, create a security user account called "ALMSSU" with full-on security rights. The existence of this account enables the functionality the logged in user can remain as before. The ALMSSU account is a new account you create in iFIX security. Full-on rights means you want to give it access to all security areas, groups, and application feature rights. It is basically the same type of account as the default ADMIN account, but with a different name. Give it any password that you want; it will be fetched dynamically. IMPORTANT: iFIX 4.0 has this account built-in. iFIX 3.0 requires iFIX30_270021 and iFIX 3.5 requires iFIX35_270021.

Name Resolution Considerations

Your name resolution configuration does not need to be altered once SCADA failover has been implemented. This means that from an iFIX perspective you do not have to add the logical name to your name resolution scheme. TCPTASK talks to actual iFIX node names and not logical names.