Interconnection networks in parallel computers

4/23/12

Introduction
What are interconnection networks and why are they important?
n

Processors in a parallel computer need to communicate in order to solve a problem. Therefore, there is a need for some kind of communication highway or interconnection network, i.e. the processors to be connected in some pattern. Performance in multiprocessor systems are highly dependent on communication processes between processors and memory, I/O devices, and other 4/23/12

IN Design Considerations
Network topologies: From a simple linear array to high-

performance INs, various topologies are available. Selecting a good topology is one of the most important issues for parallel computing/systems. used to set switches and plays a crucial role in the performance of multistage interconnection networks. packets are handled on their way from a source to a destination processor.

Switching technologies: Routing strategies are procedures

Routing algorithms: Switching techniques are ways that data

4/23/12

Performance Requirements
Processes executing in different processors synchronized and

communicated through the IN. These operations are usually performed by explicit message passing or by accessing shared variables. generated at its source node and the time the message is delivered at the destination node. network per time unit.

Message latency: the time elapsed between the time a message is

Throughput: the maximum amount of information delivered by the

The diameter is the minimal number of links between the

two farthest nodes in the network.

4/23/12

Contd..
The bisection width of a network is the number of links that must

be cut to divide the network of n PEs into two (almost) equal parts, n/2 and n/2 . time (i.e., bits per second). through the network.

The bandwidth is the number of bits that can be transmitted in unit The network latency is the time required to transfer a message The communication latency is the total time required to send a

message, including software overhead and interface delay.

4/23/12

Interconnects are classified as static or dynamic.

Static networks consist of point-to-point communication

links among processing nodes and are also referred to as direct networks.
Dynamic networks are built using switches and

communication links. Dynamic networks are also referred to as indirect networks.

Processor-to-memory connections usually employ dynamic

interconnections. These can be blocking or nonblocking.

v

Nonblocking interconnections allow connections to occur simultaneously.

4/23/12 Processor-to-processor message-passing interconnections

Static and Dynamic Interconnection Networks

4/23/12

Static interconnects topologies

88

4/23/12

Completely Connected Network

Each of n nodes has a link to every other node. Requires n(n-1)/2 links Impractical, unless very few processors

Line/Ring Network

A line consists of a row of n nodes, with connection to adjacent nodes. Called a ring when a link is added to connect the two end nodes of a line. The line/ring networks have many applications. 4/23/12

Interconnection Network Examples

Completely Connected Network

Each of n nodes has a link to every other node. Requires n(n-1)/2 links Impractical, unless very few processors

Line/Ring Network

A line consists of a row of n nodes, with connection to adjacent nodes. Called a ring when a link is added to connect the two end nodes of a line. The line/ring networks have many applications. 4/23/12

The Mesh Interconnection Network

Each node in a 2D mesh is connected to all four of its nearest neighbors. The diameter of a n n mesh is 2(n - 1) Has a minimal distance, deadlock-free parallel routing algorithm: First route message up or down and then right or left to its destination. If the horizonal and vertical ends of a mesh to the opposite sides, the network is called a torus. Meshes have been used more on actual computers than any other network. A 3D mesh is a generalization of a 2D mesh and has been used 4/23/12 in several computers.

Binary Tree Network

A binary tree network is normally assumed to be a complete binary tree. It has a root node, and each interior node has two links connecting it to nodes in the level below it. The height of the tree is lg n and its diameter is 2 lg n . In an m-ary tree, each interior node is connected to m nodes on the level below it. The tree is particularly useful for divide-and-conquer algorithms.
4/23/12

Hypercube Network

A 0-dimensional hypercube consists of one node. Recursively, a d-dimensional hypercube consists of two (d-1) dimensional hypercube, with the corresponding nodes of the two (d-1) hypercubes linked. Each node in a d-dimensional hypercube has d links. Each node in a hypercube has a d-bit binary address. Two nodes are connected if and only if their binary address differs by one bit. A hypercube has n = 2d PEs

4/23/12

Advantages/disadvantages of the hypercube

its low diameter of lg(n) or d its large bisection width of n/2 its regular structure.

An important practical disadvantage of the hypercube is that the number of links per node increases as the number of processors increase.

Large hypercubes are difficult to implement. Usually overcome by increasing nodes by replacing each node with a ring of nodes.

4/23/12

Single Shared Bus Network

v The single shared bus interconnection structure is widely used in

parallel architectures.

A number of PEs and memory units are connected to a single bus in this case, through which communication is established among the PEs and memory units connected to it.

4/23/12

The operation of the single bus interconnection structure

when a processor issues a read request to a memory location it

holds the bus until it receives the expected data from the memory module. from the appropriate location.

It will require some time for the memory module to access the data The transfer scheme also will need some time and another request

from any processor will not be initiated until the transfer is completed, which means the bus will remain idle for a certain amount of time

4/23/12

Split transfer protocol scheme

Split transfer protocol scheme whereby the bus can handle a

number of requests from different processors to different memory modules. starts the transfer of the next request so that two requests are executed in parallel. the cost of complexity.

In this case after transferring the address of the first request the bus

The split transfer protocol increases the performance of the bus at One of the limitations of single bus interconnection is that a large

number of processors and memory modules cannot be connected to a bus.

4/23/12

Multiple Shared Buses Network

processor and memory are connected to one or more

of the available buses, each of

which possesses all the attributes of an independent

system bus. Besides reducing

the communication load per bus, a degree of fault

tolerance is provided, since the

system can be designed to continue operation,

possibly with reduced performance,

if an individual bus fails (Hays, 1988).
4/23/12

Multiple Shared Buses Network

4/23/12

Tree Interconnection Network

Tree structure is another important and useful

interconnection topology.
There could be a number of levels in a tree structure. In

this case any intermediate node acts as medium to establish communication between its parents and children.
Through this mechanism communication could also be

established between any two nodes in the structure. A tree structure can be highly effective if a small portion of traffic goes through the root node otherwise due to bottleneck problems performance and4/23/12 Control

Complete Graph Interconnection Network

In a complete graph interconnection structure several processors

are connected in the complete graph format

Each node can directly communicate with any other node without

going through or touching any intermediate node. However, it requires many buses. For a complete graph with n nodes the number of buses required is n(n 1) / 2 and the maximum internode distance is 1.

4/23/12

Complete Graph Interconnection Network

4/23/12

Star Interconnection Network

v The star interconnection is one of the simplest

interconnection networks. In this configuration n 1 buses are required to connect n nodes and the maximum internode distance is 2.
v A node in this structure can communicate with any

other node through the node in the centre.

4/23/12

Star Interconnection Network

4/23/12

Summary of measures: p nodes

25 25

Network
CompletelyConnected Star Ring Complete binary tree Hypercube

Diameter
1 2 p/2 2log((p+1)/2) log(p)

Bisectionwi Arc dth connectivity

p2/4 p/2 * 2 1 p/2 p-1 1 2 1 log(p)

#links
p(p-1)/2 p-1 p p-1 p.log(p)/2

CS575 lecture 3

Network Topologies: Crossbars

A crossbar network uses an pm grid of switches to connect p inputs to m outputs in a non-blocking manner.

4/23/12

Crossbar Switch Connections

Has n inputs and m outputs; n and m are n Data can same. either usually the flow in directions. are usually n The inputs connected to processors and outputs connected to memory, I/O, or other n These switches have processors. complexities of O(n2); doubling the number of inputs and outputs also n To solve this problem, doubles the size of the multistage switch. interconnection networks were developed.
n

inp uts

0 1 . . . n1

. m1

1 .. . outp uts

4/23/12

Network Topologies: Multistage Networks

Crossbars have excellent performance scalability but poor cost scalability. vBuses have excellent cost scalability, but poor performance scalability. vMultistage interconnects strike a compromise between these extremes.
v

4/23/12

Multistage Interconnection Networks (MIN)

MINs connect input devices to output devices through a number of switch stages, where each switch is a crossbar network. vThe number of stages and the connection patterns between stages determine the routing capability of the networks. vMINs are good for constructing parallel computers with hundreds of processors and have been used in some commercial parallel computers.
v

4/23/12

Multistage Interconnection Networks (MIN)

Multistage interconnection (or shuffle) networks are
30 30

the most advanced class of switching networks.

They can be used in loosely-coupled distributed systems, or in tightly-coupled processor-tomemory configurations.

4/23/12

The End

4/23/12