FIX32/iFIX Networking

Architecture
 Contents

Discussion of FIX32 Client/Server

Network Architectural Components…………………………………4

Network Component Breakdown………………………………………4

TCPTASK………………………………………………………………………….5
NBTASK……………………………………………………………………………6
NDK/NOH………………………………………………………………………….6
NNTABLE………………………………………………………………………….6
CONMGR………………………………………………………………………….7
DBASRV……………………………………………………………………………8
RDA…………………………………………………………………………………8
UDBA……………………………………………………………………………….9
LDBA………………………………………………………………………………..9
EDA…………………………………………………………………………………9
NAM…………………………………………………………………………………9
NAC…………………………………………………………………………………9
FTK/STK/LTK……………………………………………………………………..11

Tweaking the Communication Process……………………………11

RDAThrottle………………………………………………………………..12

Session Timers……………………………………………………………13

Dynamic Connections……………………………………………………14

IPC Mechanisms Used By

FIX32…………………………………………………………………………….15

Windows Sockets Interface……………………………………………15

NetBIOS Interface…………………………………………………………16

Differences Between FIX32 and iFIX Network

Communication……………………………………………………………..16

Multiple LAN Support……………………………………………………16

LAN Redundancy…………………………………………………………17

2
 Performance Improvements……………………………………………17

Tying Everything Together: A Close

Examination of FIX32 Client/Server
Communication…………………………………………………………..…17

3
Discussion of FIX32 Client/Server
Network Architectural Components

Probably the best approach to learning FIX32/iFIX networking is to have an understanding

of the “big picture”. Being able to visually depict a working block diagram of FIX server and
client networking components in your mind will help make your troubleshooting efforts much
more efficient.
The contents and examples contained in this white paper are geared towards the FIX32
networking model. No troubleshooting is covered; just a detailed discussion of the various
networking components, how they interact with one another, and how FIX32 client/server
communication is accomplished. It is assumed that we have a working client/server connection
regardless of transport protocol.
What you will learn can also be applied to iFIX networking as well. There are some minor
differences between FIX32 and iFIX networking. These changes are discussed in the section
“Differences Between FIX32 and iFIX Network Communication” later on in this paper.

Network Component Breakdown

A block diagram of the FIX32 client/server model is shown in Figure 1. Components colored in
gray represent those found on a FIX32 node with SCADA support enabled. All other remaining
blocks are related to both the FIX client/server model. From this point forward, the word “client”
represents a FIX node without SCADA support enabled in the SCU file. The word “server” means
just the opposite.

One essential component not pictured in Figure 1 is the license key. Keys come in two forms, a
physical parallel port key or a soft key (keyless option). From a customer support standpoint, the
keyless option should never be advertised to customers. This type of key is only provided to
customers by our sales division under special circumstances.

It is important to be aware that just because a key has been affixed to a FIX32/iFIX node does not
necessarily suggest that networking has been enabled for that machine. Among the various
options available that can be enabled on keys, networking is one of them. Some FIX packages
sold to OEMs by our sales division will not have the networking option enabled; probably because
the machine(s) will be standalone or will be using our STK.

4
 Figure 1: FIX32 Client/Server Model

TCPTASK

TCPTASK is a service that has the NDK/NOH interface on one side and Windows Sockets
Interface (WINSOCK) on the other. Its duties include constructing WINSOCK library calls,
perform bookkeeping on session timers (Receive) and outstanding messages, and
handling/routing NDK/NOH communication for NAC, NAM, CONMGR, and DBASRV.

When messages come in on the server side, TCPTASK looks at the contents of the packets to
find out which are database requests or alarm requests. These requests are then placed into the
appropriate queues in the global memory buffer space.

Before client connection requests can be received TCPTASK needs to set up a TCP listening port
by calling WINSOCK. This results in the creation of a socket with the server’s IP address and
TCP port number 2010. When the listening port detects a client connection request, a new socket
is generated that contains the client side port number and IP address. All sockets created for
client requests flow through the same port number 2010. Each socket created for a client will
actually have two logical connections (assuming Alarm Network Service is enabled on the server
SCU). The first logical connection is for data and the other is for alarming.

TCPTASK has one table associated with it: the Socket Table (SKT). This table exists in the global
memory buffer space. The SKT stores the list of socket records in use, which represent
connections to remote nodes. Information on timeouts, IP address, and transactions per second

5
is recorded. Each socket record contains various counters that TCPTASK updates with
communication data to manage the connection.

NBTASK

NBTASK is a service that has the NDK/NOH interfaces on one side and NetBIOS interface on the
other. Its responsibilities include constructing calls to the NetBIOS interface, perform bookkeeping
on session timers (Send and Receive) and outstanding messages, and handling/routing
NDK/NOH communication for NAC, NAM, CONMGR, and DBASRV.

With existing client connections on the server side, NBTASK looks at the contents of the packets
to find out which are database requests or alarm requests. These requests are then placed into
the appropriate queues in the global memory buffer space. If the Alarm Network Service on the
server SCU has been enabled, there will be two sessions per client connection. One will be for
data and the other for alarming. The lack of bi-directional communication on a NetBIOS session is
the reason why two sessions are required.

NBTASK has one table associated with it: the NetBIOS Table (NBT). This table exists in the
global memory buffer space. The NBT stores the list of NBT records, which represent
connections with remote nodes. Information on LANA 0, rate of transactions, Network Control
Block (NCB) usage, and maximum number of records in use at one time is also retained. Each
record contains data on whether the connection should be cleaned up or not and helpful
reply/request logic.

NDK/NOH

Network Driver Toolkit (NDK) and Network Object Handler (NOH) contain dynamic link library
files to allow exe-to-exe communication between some other executable and
TCPTASK/NBTASK. NDK/NOH calls set up global memory buffer space that allow this
interaction between two disparate executables and also indicate to RDA how the packets will be
sized for transmission. Global memory can be accessed by any executable. From a Windows NT
security perspective, two executables cannot directly communicate with one other, so global
memory acts as a communication conduit between the two.

NDK/NOH roots come from a past decision to allow the possibility of other transport protocols
(IPX for example) to be used besides TCP/IP and NetBEUI. This was supposed to come in the
form of a toolkit to be provided to developers, but the idea for a toolkit was rejected.

NNTABLE

NNTABLE is an executable launched automatically during the FIX32 startup process. It is

responsible for creating two tables that CONMGR examines to form remote connections. The two
tables reside in the global memory buffer space and are named the Logical Name Table (LNT)
and Network Name Table (NNT).

6
CONMGR

CONMGR is responsible for establishing/reestablishing communications and periodically checks

for connection status. It is responsible for managing the data connection and it accomplishes this
through a heartbeat mechanism on the client node using the SCU configurable session timers
Keep Alive and Inactivity.

Two tables are used by CONMGR: the Logical Name Table (LNT) and Network Name Table
(NNT). The records in these tables represent nodes on the FIX network. The NNT records
represent actual nodes on the network while the LNT records represent “virtual nodes”, which are
groups of one or more actual nodes that can be treated as a single node for the purpose of data
and alarm communications. SCADA redundancy relies on the entries in LNT to work correctly.

The CONMGR on a client is responsible for detecting communication losses and reestablishing
them, without having to restart client nodes. Even if data will not be requested, it will connect or
reconnect. CONMGR was designed to be very efficient in managing connections because from a
TCP/IP standpoint, WINSOCK is not very reliable in doing this.

When the CONMGR on a client detects a connection loss to a server, a immediate connect
attempt (one shot) is sent. If there is no response to this connect, then the client will try to connect
every 20 seconds. Also, any period of inactivity that exceeds 20 seconds results in CONMGR
checking the connection. By default, the Keep Alive timer is set to 20 seconds, and this is the
timer CONMGR uses to check for inactivity.

Each record stored in NNT contains connection statistics for a remote node. It is this information
that CONMGR uses to manage this connection. Figure 3 displays a flowchart of CONMGR
managerial duties.

7
 Figure 3: Logic that CONMGR uses to manage connections

DBASRV

Database Server (DBASRV) is a service that runs on server nodes only. When the SCADA
support has been enabled in the SCU, this service is made available. DBASRV is responsible for
handling incoming process database value read and writes from clients. As requests are
received, they are placed into the global memory buffers defined by NDK/NOH. This memory acts
as a temporary storage location to hold these requests until DBASRV can process them.

RDA

Remote Data Access (RDA) is responsible for breaking up and piecing together FIX information
for transmission. RDA is a dynamic link library that is called by EDA applications. Knowledge of
how to break up FIX information for transmission (packet size) is provided by NDK/NOH. All EDA

8
traffic flows through RDA. This traffic flow is controlled by RDAThrottle; a FIX registry entry that
tells RDA how many requests can be made at a single time to a server.

UDBA

Universal Database Access (UDBA) is a dynamic link library that is responsible for deciding
where the information that EDA is requesting resides. The information may come from the local
process database or from a data system integrated into our product via System Extension Toolkit
(STK). If the data is requested locally, UDBA calls the Local Database Access (LDBA) dynamic
link library.

LDBA

Local Database Access (LDBA) is a dynamic link library that is responsible for deciding if the
information requested will come from a standard FIX block or from one created with the Loadable
Block Toolkit (LTK).

EDA

Easy Database Access (EDA) is a dynamic link library that receives requests from VIEW or from
any EDA compliant application created with the FIX Integration Toolkit (FTK). If the request is to
the local node, EDA calls UDBA, otherwise RDA is called.

NAM

Network Alarm Manager (NAM) is a service that is also available once SCADA support has been
enabled. NAM is responsible for establishing alarm sessions, handling local and remote NAC
requests, and sending new alarms.
There are four alarm queues used by NAM and are contained in the same global memory buffer
space that DBASRV uses; NAM Receive Queue, NAM Control Queue, Startup Queue, and
Summary Startup Queue. The NAM Receive Queue stores alarm information from the server
itself and from other servers. These are not requests, but actual alarm or message data. This
data is then available to any alarm client that has an established alarm session request fulfilled
and listed in the NAM Control Queue.

When a session is first established between NAM and an alarm client, NAM may also be
responsible for sending past alarm data to clients if the server has been configured to do so. This
is dependent on whether or not the Alarm Startup Queue service has been enabled in the server
SCU. If so, alarm data contained in the Startup Queue and Summary Startup Queue will be sent
to the alarm client.

NAC

Network Alarm Client (NAC) is a service that runs on both server and client machines. Its
responsibilities include initiating alarm connections, sending operator messages, receiving

9
alarm/message data either locally or over the network, and managing alarm connection(s) by
sending “heartbeats” periodically to remote NAM(s). Depending on what is needed in managing
the alarm connection, NAC will send a heartbeat or connect roughly every 2 minutes. The interval
for the heartbeat timeout is hard coded and not configurable. A response to the heartbeat must
be received within the Receive session timer setting; otherwise a NAC request to reconnect is
performed. In maintaining alarm connections, NAM is responsible for establishing/reestablishing
connections, but it will not do so until told to by a remote NAC.

NAC has two alarm queues; NAC Send Queue and NAC Control Queue. Both these queues
reside in the global memory buffer space. The NAC Control Queue will contain NAM responses to
requests that NAC had made to form alarm connections. The NAC Send Queue only exists on
server machines and receives alarms that SAC generates. This alarm data is then made
available to alarm clients by placing it into the NAM Receive Queue. Figure 2 displays a flowchart
of NAC alarm communication management.

Figure 2: Flowchart of NAC alarm communication management

10
FTK/STK/LTK

FIX32 has 3 major toolkits: FIX Integration Toolkit (FTK), System Extension Toolkit (STK), and
Loadable Block Toolkit (LTK). These products allow seasoned developers to tailor their FIX32
implementation as required.

The FIX Integration Toolkit is a set of application program interfaces and libraries that allow a
customer with development experience in VB or C++ to write programs that directly read, write,
and modify their process database. This toolkit also contains a help file that briefly describes each
API and how it is used, and a set of functions that allow the customer to access their historical
data as well. We recommend the FIX Integration Toolkit to customers who have project
specifications that require high-speed database access and modification.

The System Extension Toolkit is a set of application program interface descriptions and libraries
that guide customers with extensive development experience on how to adapt a native data
engine to our graphical front end. The STK enables customers to integrate their data systems
onto our product. It replaces our process database, including SAC. For customers who
understand our product and the components that make up the CORE, this toolkit sits at the LDBA
level. The toolkit contains libraries and a WORD doc that describes the APIs and how to develop
them.

The Loadable Block Toolkit is a set of libraries and instructions on how to write a custom block for
our database system. This toolkit is designed for experienced developers who understand pointer
usage, structures, and function tables. It allows customers to create blocks for specific data,
native systems, and custom designed data manipulation.

Tweaking the Communication Process

Although the default FIX32 communication settings are usually sufficient for proper FIX network
communication, there may be situations where these settings need to be adjusted to better suite
your implementation. These situations may involve a large FIX32 network that contain a myriad of
clients, a slow communication medium between two separate FIX32 subnets or networks, an
overloaded server, connection time charges, or occasional interest in data or alarms.

This section discusses ways to tweak the FIX32 networking. These adjustments do not in any
way remove or bypass any of the FIX network architectural objects discussed in section
“Discussion of FIX32 Client/Server Network Architectural Components”. All they do is affect how
some of the components function.

11
RDAThrottle

Around the FIX 6.0 era, it was noted that if a good number of clients where connected to a server,
any of the clients opening a picture (especially one that was unresolved) would dominate more
server time that the other clients. This was due to the flurry of activity View generated to initially
populate a picture with values or to map object data sources in pictures to associated tags in a
process database. When this situation occurred, other client nodes noticed a performance drop in
how their picture values were updated. It was decided that some sort of “throttle” was needed to
control how RDA behaved. This throttle should be configurable, so an entry for this throttle was
placed into the registry. The outcome resulted in the RDAThrottle registry setting.

RDAThrottle applies only to FIX nodes that make outgoing data requests (clients). FIX 6.0 had a
default value of 5 for this setting. Later FIX32 versions now use 30. All FIX HMI products besides
iFIX use a packet size of 1,400 bytes, so FIX 6.0 will send (5) 1,400 byte packets, and later
versions would send (30) 1,400 byte packets. If we use the term “message” instead of packet,
one message is 1,400 bytes.

An example of when RDAThrottle should be used would be a picture opening for the first time on
client A. Say that this picture is unresolved and will require 70 messages to build the picture. All
other clients are just requesting picture values. If you could see the contents of the DBASRV
queue in global memory on the server, you would see a chunk of 30 messages from client A, a
few small requests from other clients, another chunk of 30 messages from client A, a few small
requests again from other clients, and finally a chunk of ten messages from client A. Figure 4
gives a pictorial representation of the example just described. Notice how an RDAThrottle setting
of 5 on client A does not allow it to dominate DBASRV server resources.

RDAThrottle is really designed for pictures that are not resolved. Once a picture is open, requests
for picture values are small when compared to requests for building a picture. The only time this
setting should be altered on all clients is if there are a tremendous number of client nodes
communicating with a server. It is important to apply the RDAThrottle change to ALL clients so
that they play nice together. In this situation, the default value (30) would be changed to a lower
value and apply this change to all clients, not just to machines experiencing performance drops.

12
Figure 4: Pictorial example of RDAThrottle usage

Session Timers
FIX32 uses network session timers to manage connection lifetimes. These timers are configured
in the SCU and apply to all outgoing connections.
The four time out values are described below:
Send - defines the amount of time that a client waits for a request to the server to be
acknowledged. This acknowledgement is different than the message response. If this timer
expires, the session ends.
Receive - defines the amount of time that a client waits for a reply from the server.
Keep Alive - defines the amount of time that, if no activity has occurred over an established
connection, a client waits before sending a heartbeat message.
Inactivity - defines the amount of time that, if no data activity has occurred over an established
dynamic connection, a client waits before removing the dynamic connection from the list of
outgoing connections. If this timer expires, the session ends. The Inactivity timer only applies to
outgoing connections and specifically Dynamic Connections. Connections listed in the Remote
Node List in the SCU will not be disconnected after a period of inactivity.

13
TCPTASK uses only the Send and Receive session timers. When running FIX32 over TCP/IP,
the effective session timeout value is either the Send timer or the Receive timer; whichever is
greater. If this timer expires, the session ends. The Send/Receive session timers do not alter any
TCP/IP protocol registry settings. They are only used in the TCPTASK communication
management process. NBTASK provides the NetBIOS interface with the values of the Send and
Receive timers when it needs to form a session. CONMGR only uses the Keep Alive and
Inactivity session timers for its connection management.

Dynamic Connections

Dynamic Connections is a FIX32 feature that creates server connections “on the fly” if a picture
should be opened that contains a server name not listed on the client SCU as a Configured
Remote Node. Once a dynamic connection is made to the server, it remains connected even if
the picture or View is closed.

You may wish to use Dynamic Connections to manage connection lifetime for either or both of the
following reasons:

You are charged for connection time and you want to minimize your connection time:
Wide area connections such as ISDN can charge for connect time. You should enable inactivity
timer on the client that is connecting to the server via ISDN.

You have only occasional interest in data or alarms:

Even if you have only occasional interest in server data, it makes sense to use dynamic
connection rather than add that server name to your Remote Node List. Using dynamic
connections will ensure that you stop receiving server alarms when you stop requesting data.
You should enable inactivity timer on the client to close the connection when data requests stop

The definition of inactivity is that there are 0 data messages for the configured amount of time.
Data messages are any messages initiated by any application but Connection Manager or
NAC/NAM. The following messages will keep a connection alive:
 Real time data requests and writes
 Database Builder messages, including database reload messages.
 Alarm acknowledgments

Alarms are not data and they will not keep a session alive. Alarms for this purpose includes:
 Block alarms
 Alarm management messages (NAC management messages)
 Operator messages.
 Recipe messages.
 SAC restart messages

There is also connection management and "heartbeat" messages sent periodically and these will
not keep the connection alive either.

14
The Inactivity timer does not affect incoming connections or alarm connections. A server should
not terminate the connection just because the view node is not requesting data. Server
connections should however be cleaned up after the session is dropped by the client. Alarm
connections should be cleaned up when their accompanying data session is cleaned up.
A dynamic connection that never gets successfully established is inactive. An example of this is
a mistyped SCADA node name or the name of a SCADA that no longer exists. When the
inactivity timer expires, its allocated session/socket should be released, and its LNT_REC and
NNT_REC should be removed.

IPC Mechanisms Used By FIX32

Interprocess Communications (IPC) is mechanisms allow communications and data sharing
between applications. Most current operating systems provide the following IPC mechanisms:
 named pipes
 mailslots
 NetBIOS
 Windows Sockets,
 Remote Procedure Calls (RPC)
 Network Dynamic Data Exchange (NetDDE)

NBTASK uses the NetBIOS interface while TCPTASK uses the Windows Sockets Interface.

Windows Sockets Interface

The Windows Sockets specification defines a network-programming interface for Microsoft

Windows. Using WINSOCK permits your application to communicate across any network that
conforms to the Windows Sockets API.

Sockets are a bi-directional way data can be exchanged between networked computers. When a
WINSOCK compliant application (FIX32 for example) creates a socket, it specifies the IP address
of the intended host, the transport protocol used (TCP/UDP), and the port that the particular
application will use.

When a server first starts up, TCPTASK create a socket with listening port 2010. Each client
connection results in the construction of another socket that uses the same port number. The port
numbers on the client side are assigned dynamically by the operating system when a request for
a socket is made by TCPTASK on that node.

A port functions as a multiplexed message queue (can receive more than one message at a time)
and this is what allows two logical connections to be formed on the same socket: the first
connection by TCPTASK for data, and the other for alarming by NAC.

NetBIOS Interface

15
Network Basic Input/Output System (NetBIOS) defines a software interface and a naming
convention, not a protocol. The NetBEUI protocol, introduced by IBM in 1985, provided a protocol
for programs designed around the NetBIOS interface. However, NetBEUI is a small protocol with
no networking layer and because of this, it is not a routable protocol suitable for medium-to-large
intranets.

A NetBIOS “session” is a logical connection between any two processes on the network. For
FIX32 to establish a session, NBTASK on the server side issues a NCBLISTEN command to
prepare to open a session. The server is now ready for client connection requests. When a client
wants to connect to the server, it issues a NCBCALL command. After a session is established,
the computers can exchange data using NetBIOS commands. Since NetBIOS sessions are uni-
directional, separate sessions are needed for data and alarming.

Differences Between FIX32 and iFIX Network Communication

FIX32 and iFIX network architecture are very similar in design. If you fully understand how the
underlying FIX32 network components function and communicate with one another, you
understand iFIX networking. There is only one minor exception though, which we will discuss
shortly.

Advances in networking technology, Windows operating systems, and computer resources

helped shape decisions on iFIX development. These changes allowed iFIX to make use of
multiple network interface cards and introduced two major network performance enhancements
for iFIX data/alarming transmission.

Multiple LAN Support

FIX32 NBTASK only attempted to communicate over LANA 0 when the SCU was configured to
use the NetBEUI protocol. The problem with this was that other protocols and additional network
interface cards could be installed on a system, and if this was the case, it introduced additional
network paths (LANA numbers) through the NetBIOS interface. Since FIX32 only used LANA 0,
there was a possibility that LANA 0 was not using NetBEUI. If a decision was made to use
NetBEUI, and this protocol was not tied to LANA 0, reconfiguration was needed. This involved
either changes to LANA number configuration, command line switches when loading NBTASK, or
changes in the NETWORK.INI file. TCPTASK did not have this issue because WINSOCK was
used, not NetBIOS over TCP/IP (NetBT).

A similar issue existed with TCPTASK. If a machine contained more than one network interface
card (NIC), this meant that there was more than one IP address. TCPTASK, in this case, would
use the first IP address listed in the operating system, and this sometimes resulted in FIX32 trying
to communicate with a completely different network instead of the FIX network on the other NIC
card.

Fortunately, iFIX supports multiple LAN support. NBTASK in iFIX examines all LANA numbers to
see which one is using NetBEUI. No user configuration is required for LANA numbers. With
TCPTASK, all IP addresses and NIC cards are noted. Now, when iFIX starts, it examines all
available paths for the protocol configured in the SCU and uses whatever one can connect with
the server first.

16
Multiple LAN support allows path exclusion. More than likely, only one path is associated with an
iFIX network, so the ability to disable other network paths is available. What is special about this
feature is that it inhibits the possibility of a non-iFIX network connection and especially, unwanted
bandwidth usage on this network.

LAN Redundancy

LAN redundancy is another new network feature in iFIX that takes advantage of the ability of iFIX
to detect and work with multiple network paths. With LAN redundancy, you can install two NIC
cards onto a machine where each will connect to the same iFIX network/subnet. If a client
detected a path communication failure, it would simply use the other one.

The difference between multiple LAN support and LAN redundancy is that LAN redundancy
monitors the next available non-active route. A heartbeat is sent over the non-active route by
CONMGR based on the value of the Keep Alive session timer.

A new table in global memory is required to keep track of redundant LAN connections. This new
table is called the Logical Connection Table (LCT). The information in this table indicates LAN
failovers and which connection is the active one.

So in the end you have two available paths; one active and the other inactive. What makes LAN
redundancy stand apart from multiple LAN support is the managed backup connection.

Performance Improvements

Two major performance improvements introduced into the iFIX product. The first is the elimination
of DBASRV. It was noted that DBASRV created a bottleneck because TCPTASK in FIX32 could
only communicate to DBASRV through global memory. DBASRV functionality has now been
incorporated into TCPTASK in iFIX.

The second performance enhancement was packet size. In the FIX32 era, packet size was
limited to 1,400 bytes. Even though packet size was no longer a issue as FIX32 advanced, it was
never taken advantage of because backwards compatibility would be needed if earlier FIX32
node versions existed on the network. The overhead of little packet sizes was causing
performance degradation. A decision was made to increase the packet size to 16K in iFIX. This is
possible due to new logic incorporated in RDA. RDA is smart enough to know how to format or
piece together 1,400 byte and 16K packets.

Tying Everything Together: A Close Examination

of FIX32 Client/Server Communication
Now that we have a good understanding of FIX32 network architecture, let us see how everything
ties together. Our examination assumes a single client/server model, network alarming, and that
the TCP/IP protocol is configured in the SCU on both machines (since TCP/IP is the most
commonly used protocol with FIX32). Figure 5 displays a flowchart that can be applied to either a
client or server node. It describes the successive communication tasks performed on startup of a
FIX32 node.

17
Figure 5: Flowchart of client/server startup tasks to form FIX network connection

18