Serial Attached SCSI

Architecture

by Rob Elliott
HP Industry Standard Servers
Server Storage Advanced Technology
elliott@hp.com http://www.hp.com
30 September 2003
 Notice

• These slides are freely distributed by HP through the

SCSI Trade Association (http://www.scsita.org)
• STA members are welcome to borrow any number of the
slides (in whole or in part) for other presentations,
provided credit is given to the SCSI Trade Association
and HP
• This compilation is © 2003 Hewlett-Packard Corporation

9/30/2003 SAS architecture page 2

 SAS clause 4 – Architecture
•Standard layers •SAS address
•SAS object model •Reset sequences
•Physical links and phys •State machines
•Ports •Transmit data paths
•SAS devices •Expander device model
•Expander devices
•Domains
•Edge expander device set
•Pathways
•Connections
9/30/2003 SAS architecture page 3
 SAS standard layering

Management
SCSI application layer ATA application layer Clause 10
application layer

SSP transport layer STP transport layer SMP transport layer Clause 9

SAS port layer Clause 8

SSP link layer STP link layer SMP link layer

Clause 7
SAS link layer

SAS phy layer Clause 6

SAS physical layer Clause 5

9/30/2003 SAS architecture page 4

 SATA 1.0a standard layering

• For SATA 1.0a from the private Serial ATA working

group

9/30/2003 SAS architecture page 5

 ATA/ATAPI-7 standard layering

• Forthe public standard ATA/ATAPI-7

• Subject to change by T13 standards committee

9/30/2003 SAS architecture page 6

 SCSI and ATA terminology differences

Common SCSI ATA

SCSI device <none>
HBA SCSI initiator device ATA host
Disk drive SCSI target device ATA device
SCSI port ATA port
SCSI initiator port ATA host port
SCSI target port ATA device port

Common SAS
port
port phy

9/30/2003 SAS architecture page 7

 SAS object model

• This figure describes

SCSI domain

classes of objects
1 SAS domain

1 1

• Diamonds mean 1
Service delivery

“contains”
subsystem

• Arrows mean “subclass

1
0..127

of”
1..* Expander device
SAS address
1
SCSI device
1 1 1

• Examples
SCSI device name
1..* 1..*
1 SAS device

– SAS domain contains 1 1 2..128

or more SAS devices 1..*

SCSI port
Expander port
Attached SAS address

– SAS device contains

SCSI port identifier
1..* 1 1

one or more SAS ports

SAS port SMP port
SAS address Attached SAS address
Attached SAS address

– SAS port contains 1 to

1 1
Note: See SAM-3
1..128 1..128 1..128
for more details on

128 phys SCSI domain,

SCSI device, and
SCSI port.
Phy
1..*
Phy identifier (device scope)

9/30/2003 SAS architecture page 8

 Physical links and phys

•A phy contains one transceiver

• A physical link attaches two phys together
Phy Phy
Transceiver Physical link Transceiver
Differential signal pair
transmitter receiver
-Signal

+Signal

Differential signal pair

receiver transmitter
-Signal

+Signal

9/30/2003 SAS architecture page 9

 Physical link rate

• Each direction runs 1.5 Gbps or 3.0 Gbps (150 MB/sec or 300
MB/sec)
– Both directions use the same physical link rate
• Dual simplex (full duplex) operation – 600 MB/sec total bandwidth
• Example: peak bandwidth needs of an HBA with 8 phys
– 2400 MB/sec half duplex, 4800 MB/sec full duplex

9/30/2003 SAS architecture page 10

 Ports

Ports contain phys

Narrow link
• Narrow port
SAS address A
Narrow port

• An “expander port” is phy Physical link phy

not a “SAS port”

SAS address X

Wide link
• Each SAS port has a
SAS address B
phy Physical link phy
SAS address SAS address Y

• Ports are abstract Wide port Wide port

– A set of phys with SAS address B

matching SAS phy Physical link phy

addresses attached to
SAS address Y

another set of phys SAS address B Wide link

with matching SAS phy Physical link phy

addresses SAS address Z

– Determined at Wide port Wide port

initialization time SAS address B

phy Physical link phy
SAS address Z

Each horizontal line represents a differential signal pair

9/30/2003 SAS architecture page 11

 SAS devices
Device Device

• SAS devices contain Narrow port Narrow port

ports phy phy

• An “expander device” is
not a “SAS device”
Device

• Each phy in a SAS

phy phy

device has a phy

Wide port Wide port

identifier unique within

phy phy

that device Device

phy phy

Wide port Wide port

phy phy

Each horizontal line represents a differential signal pair

9/30/2003 SAS architecture page 12

 End devices

• End device is a SAS device that is not an expander

device
• Sample end devices
– HBA – 8 phys
• One SAS address for all 8 phys
– Potentially all one (very) wide port
• One SAS address for 4 phys, another SAS address for 4 phys
– Guarantees at least two ports
– Good match for 4-wide connectors
• Eight SAS addresses
– Disk drive - 2 phys
• Separate SAS address for each phy
– Guarantees two ports
– Never a wide port
9/30/2003 SAS architecture page 13
 Expander devices
Expander
• Expander device device

contains expander Expander port

SAS device
(internal/optional)
ports SMP
target
(internal/
optional)

• May contain SAS port with virtual SAS port

phys
devices too (e.g. for
enclosure
management) Expander function

• Each expander
device has a SAS
address STP/SATA
Expander port

• Each expander phy

(narrow port)
bridge
(internal/
has a phy identifier Expander
port
optional)
Expander
phy
unique within that
expander device Either SSP, STP and SMP SATA

SAS initiator ports, SAS target ports, or SATA device

expander ports port

9/30/2003 SAS architecture page 14

 Expander device types – edge vs. fanout
Expander device
SAS address

An expander device is an edge expander device or a

fanout expander device

Edge expander device Fanout expander device

• Edge expander device

– Always part of an “edge expander device set”
– May perform subtractive routing
• Fanout expander device
– Never does subtractive routing
– Usually supports larger tables for table routing
• Topologies described later
9/30/2003 SAS architecture page 15
 Domains

• A simple SSP ATA

SAS initiator SSP
SSP
initiator
initiator
STP
initiator
SMP
initiator
host
domain port
port(s)
port(s) port(s) port(s)
port

contains SSP STP SMP

SAS
devices Service delivery
and SSP subsystem SATA
expander with expander
devices devices

• An ATA
domain SSP
SSP STP SMP
ATA
contains a target
SSP
SSP STP SMP
device
SATA host port
initiator
target target target
port
and a SAS port(s)
port(s) port(s) port(s)ATA domain
SATA domain SAS domain (informative)
device Note: When expander devices are present, SAS target ports may be
located in SAS devices contained in expander devices.
9/30/2003 SAS architecture page 16
 SAS domain bridged to ATA domains

SSP
SSP STP SMP
initiator
initiator initiator initiator
port(s)
port(s) port(s) port(s)
STP SMP
SSP

Service delivery
subsystem STP STP/SATA SATA ATA device
with expander bridge(s) port
devices
ATA domain
(informative)
SSP STP SMP

SSP
SSP STP SMP
initiator
target target target
port(s)
port(s) port(s) port(s)
SAS domain
9/30/2003 SAS architecture page 17
 SAS devices in multiple SAS domains

9/30/2003 SAS architecture page 18

 Edge expander device set

• Set of edge expander devices

• 128 SAS addresses per set
• Typically bounded by a subtractive port (to a fanout
expander device, or to another edge expander device
set)
• Edge expander devices uses table routing and direct
routing “downstream” and subtractive routing
“upstream”
• Wide links between expanders are allowed
• No loops

9/30/2003 SAS architecture page 19

 Edge expander device set diagram

Expander port or SAS

port

Edge expander device set

Edge
Expander port
expander Expander port
(internal)(optional)
device (subtractive routing attribute)
(direct routing attribute)

SAS device
(internal)
Expander port Expander port (optional)
(table routing attribute) (direct routing attribute) SAS port

Edge
Edge
expander Expander port
expander
device (subtractive routing attribute)
device
(internal
Expander port expander ports/
(direct routing attribute) SAS ports may
also be present)

SAS ports

9/30/2003 SAS architecture page 20

 Expander topologies

• Maximum of one fanout expander device in a SAS

domain
• If no fanout expander, maximum of two edge expander
device sets (attached via subtractive decode ports)
• End devices may be attached at any level
– Directly to fanout expander device
– Any level edge expander device
• Wide links possible between any two devices
• No loops
• No multiple paths

9/30/2003 SAS architecture page 21

 Edge expander device set and fanout expander
device
One fanout
expander Fanout expander device
device

Edge Edge
expander Edge expander
device expander device
set device set Maximum
of 128
Edge Edge SAS
Edge expander device
expander expander addresses
device device per
edge
expander
device
set

End device End device End device End device

128 edge expander device sets

9/30/2003 SAS architecture page 22

 Two edge expander device sets
The root edge expander device in Upstream phys use the
each edge expander device set subtractive routing method;
uses the subtractive routing downstream phys use table
method on the expander phys routing method or direct routing
attached to its peer. methods.

Edge Edge
expander Edge expander Edge
device expander device expander
set device set device

Edge Edge Edge Edge

expander expander expander expander
device device device device

End device End device End device End device End device

Two edge expander device sets

9/30/2003 SAS architecture page 23
 Pathways

• Potential pathway = set of physical links between an initiator phy

and a target phy
• Pathway = set of physical links used by a connection

9/30/2003 SAS architecture page 24

 Connections

• Connection = temporary association between an initiator

phy and a target phy
– Source phy transmits an OPEN address frame
• Contains a destination SAS address
– Expanders route it to a matching destination phy
– Destination phy replies with an OPEN_ACCEPT primitive
– Connection is established
– Both sides exchange CLOSE primitives to close
• Connections are addressed to ports but established phy-
to-phy
• N-wide ports may establish N connections at a time (to
up to N other ports)
• Wide ports may establish multiple connections to other
wide ports simultaneously
9/30/2003 SAS architecture page 25
 Connection examples
SAS initiator Expander device SAS target
device device
Expander port Expander port
A
A
Phy Expander phy Expander phy Phy SAS
SAS target
initiator C port
port Phy Expander phy Expander phy Phy
B

B SAS
Expander phy Phy target
C port
Phy Expander phy
Expander port

Expander port SAS

Phy Expander phy target
Expander phy Phy
D D port

SAS Expander port

initiator
Target device
port Expander port
E
Phy Expander phy F
Expander phy Phy
SAS
target
F port
Phy Expander phy Expander phy Phy
E

Key: Single physical link Wide link X X Connection

Notes: The expander device has a unique SAS address. Each SAS initiator port and SAS target port
has a unique SAS address. Connections E and F represent a wide SAS initiator port with two
simultaneous connections to a wide SAS target port.

9/30/2003 SAS architecture page 26

 Connection rules

• Connections are addressed to SAS ports but are

established from phy to phy
• Wide ports may establish multiple connections at a time
(to up to one per phy) to different destinations
• Wide ports may establish multiple connections to other
wide ports simultaneously (wide initiator port to wide
target port)
– SAS disk drives will offer two narrow ports
– Only HBAs and RAID controllers will offer wide ports

9/30/2003 SAS architecture page 27

 Connection rate

• Connection runs at 1.5 Gbps or 3.0 Gbps

• Connection rate <= physical link rate
• If the connection rate is slower than the physical link
rate, rate matching is used
– e.g. 1.5 Gbps connection rate over a 3.0 Gbps physical
link
– Inserts ALIGN primitive every other dword to slow down the
effective throughput
– Lets 3.0 Gbps initiators talk to 1.5 Gbps targets through
expanders

9/30/2003 SAS architecture page 28

 SAS address

• Each SAS port and expander device has a worldwide unique 64-
bit SAS address
• Same namespace as the Fibre Channel Port_Name

Byte\Bit 7 6 5 4 3 2 1 0

0 NAA (5h)
1
IEEE Company ID (24 bits)
2

5
Vendor-Specific Identifier (40 bits)
6

9/30/2003 SAS architecture page 29

 Reset sequences

9/30/2003 SAS architecture page 30

 State machines

• SAS uses state machines for detailed definition of rules

• Intended to reduce interoperability issues from reading
vague text descriptions (e.g. that haunted parallel SCSI)
• SAS device state machines differ from expander device
state machines
• SAS state machines do not provide:
– Detailed integration of SATA state machines into SAS
– Detailed implementation of expander functionality
– Hardware-implementable descriptions (zero-time states,
“state machines” with only one state, state transitions
passing arguments, and other atrocities)

9/30/2003 SAS architecture page 31

 State machines for SAS devices
SAS device

• SAS device object SCSI application

layer (SA)
ATA application
layer (AA)
Management application
layer (MA)

contains application
layer state machines
SAS port
SSP transport layer STP transport layer SMP transport layer

(and SAS port objects) (ST) (TT) (MT)

• SAS port object contains Port layer (PL)

transport layer and port
layer state machines SAS phy

(and SAS phy objects) SSP link layer

(SSP)
STP link layer
(STP)
SMP link layer
(SMP)

• SAS phy object contains SAS Port Control (PC)

SAS Port Control (PC)

link layer and phy layer

state machines
Link layer (SL)

Phy layer (SP)

One or more SAS phys (with matching

number of TTs and MTs) per SAS port

One or more SAS ports per SAS device

9/30/2003 SAS architecture page 32

 State machines for expander devices

• Expander phy object contains link layer and phy layer state machines
• Expander port object contains expander phys
• Expander function is architecturally at a lower level than the link layer

Expander device

External expander port External expander port

External External
expander phy expander phy
Expander Expander
link layer link layer
(XL) (XL)

Phy layer Phy layer

(SP) Expander function (SP)

One or more expander One or more expander

phys per expander port phys per expander port

One or more expander ports One or more expander ports

per expander device per expander device

9/30/2003 SAS architecture page 33

 Transmit data path in SAS phy

• SP transmits dwords
during phy reset
sequence
• SL_IR transmits dwords
after phy reset sequence
• SL transmits dwords after
identification sequence,
outside connections
• SSP, STP, or SMP link
layer transmits dwords
during connections Port layer

9/30/2003 SAS architecture page 34

 Transmit data path in expander phy
Expander phy

• SP transmits dwords
SL_IR Clock skew SP (phy layer)
(identification and management state machine
hard reset) state

during phy reset sequence

ALIGN
machine SP transmitter
insertion
(OOB signals,
SL_IR transmitter D10.2, ALIGNs)

• SL_IR transmits dwords

(IDENTIFY ALIGNs
address frame,

after phy reset sequence

HARD_RESET,
XL (link layer for idle dword, rate
expander phys) matching ALIGNs

• XL transmits dwords after

state machine and NOTIFYs)

XL transmitter

identification sequence (AIP, BREAK,

BROADCAST,
CLOSE,
OPEN_ACCEPT,
OPEN_REJECT,
OPEN address
frame, dword,
idle dword) Phy reset
sequence
Link reset complete
sequence
complete

At each level, there is

feedback to indicate when a
dword has been transmitted
Key: and the next dword may be
Controls presented.
Data path

9/30/2003 SAS architecture page 35

 Expander device model

Expander function
Expander device

• Internal expander port Expander function

divided into 3 blocks

Virtual phy

Expander link layer (XL)

Narrow ports or

– Expander connection Protocol-specific

link and transport layers
wide ports

router (ECR)
– Expander connection
manager (ECM)
External expander port - SATA host port
External SATA phy
Expander Expander
STP/SATA transport

Broadcast primitive
Narrow ports connection connection

–
layer
router manager
Expander link layer (XL (ECR) (ECM)

processor (BPP)
and SL_IR)

SATA phy layer

External expander port Narrow ports or

External phy wide ports
Expander link layer
(XL and SL_IR)

Phy layer
(SP and DWS)
Broadcast primitive
processor (BPP)

9/30/2003 SAS architecture page 36

 Expander device interface details

XL state machine in
Source Expander function Destination

• expander
phy
expander
phy

Requests
each expander phy
Request Path

interfaces to the

Responses
Arbitrating Phy Status

Confirmations
expander function
ECM
Arb Won
Arb Lost

• Requests forwarded to
Arb Reject

peer phys as
indications Transmit Open
ECR
Transmit Open

Indications
Requests
• Responses forwarded
Transmit Close Transmit Close
Transmit Break Transmit Break

to peer phys as
Transmit Dword Transmit Dword
Arb Status Arb Status

confirmations
Confirmations

Open Accept Open Accept

Responses
Open Reject Open Reject

• ECR just forwards;

Backoff Retry Backoff Retry
Backoff Reverse Path Backoff Reverse Path

ECM and BPP

Indications
Requests

generate own replies

Broadcast Event Transmit Broadcast
Notify BPP Primitive

9/30/2003 SAS architecture page 37

 Expander routing attributes and methods

• Each expander phy has an expander routing attribute

• The attribute determines the routing methods the ECM uses with
each phy
Routing attribute When attached to Routing method used
Direct End device Direct
Table Expander device Table + Direct
End device Direct
Subtractive Expander device Subtractive (+ Direct)
End device Direct

• Direct = route requests to the attached SAS port through this phy
• Table = route requests that match in routing table through this phy
• Subtractive = route unresolved requests through this phy

9/30/2003 SAS architecture page 38

 Expander route table contents

Phy identifier

A phy identifier for each expander phy

Expander route table of the expander device.

0 1 2 ... N N ≤ number of phys - 1

0
1
2 Expander route entry

... Includes:
! Routed SAS address
M ! Enable/disable bit

Expander route index

An expander route index for each

expander route entry.

M = expander route indexes - 1

9/30/2003 SAS architecture page 39

 Discover process

Probe the SAS

Management application client in an
• end device or expander device

domain one
level at a time 1
Expander
device

• When
expander
devices with 2 Expander
device
3 End
device
4 Expander
device
5 Expander
device

configurable
routing tables 6 Expander 7 Expander 8 End 9 Expander 10 End

are found, fill device device device device device

in the tables 11 End 12 Expander 13 End 14 End

device device device device

15 End 16 End
device device

Assume that the lowest phy identifier in each expander device is on the top
right, and the remaining phys are numbered counter-clockwise
9/30/2003 SAS architecture page 40
 Expander route table example

No matter
Edge
•
Edge Entries for N
expander expander Entries for U
device ...

which initiator
device
set R Entries for V
Expander
Level 0 (portion) route table Entries for W

fills in the table

(from for phy A ...
device R) Entries for X
Entries for Y

for a phy, it Phy A

...
Entries for Z

ends up with (Assume that the

lowest phy identifier is Edge expander device N

the same on the top and phy

identifiers increase in
(n phys)
Expander route
table for phy B

entries in the
Level 1 counter-clockwise
(from order) Entries for U
Phy B
device R) ...

same locations
Entries for W
...
Entries for X

Edge expander Edge expander

Level 2
device U device V
(from
(u phys) (v phys)
device R)

Edge expander Edge expander Edge expander Edge expander

Level 3 device Z
device W device X device Y
(from (z phys)
(w phys) (x phys) (y phys)
device R)

Level 4
(from
device R)

9/30/2003 SAS architecture page 41

 Wrap up

9/30/2003 SAS architecture page 42

 Serial Attached SCSI tutorials

• General overview (~2 hours)

• Detailed multi-part tutorial (~3 days to present):
– Architecture
– Physical layer
– Phy layer
– Link layer
• Part 1) Primitives, address frames, connections
• Part 2) Arbitration fairness, deadlocks and livelocks, rate matching,
SSP, STP, and SMP frame transmission
– Upper layers
• Part 1) SCSI application and SSP transport layers
• Part 2) ATA application and STP/SATA transport layers
• Part 3) Management application and SMP transport layers, plus port
layer
– SAS SSP comparison with Fibre Channel FCP
9/30/2003 SAS architecture page 43
 Key SCSI standards

• Working drafts of SCSI standards are available on

http://www.t10.org
• Published through http://www.incits.org
– Serial Attached SCSI
– SCSI Architecture Model – 3 (SAM-3)
– SCSI Primary Commands – 3 (SPC-3)
– SCSI Block Commands – 2 (SBC-2)
– SCSI Stream Commands – 2 (SSC-2)
– SCSI Enclosure Services – 2 (SES-2)
• SAS connector specifications are available on
http://www.sffcommittee.org
– SFF 8482 (internal backplane/drive)
– SFF 8470 (external 4-wide)
– SFF 8223, 8224, 8225 (2.5”, 3.5”, 5.25” form factors)
– SFF 8484 (internal 4-wide)
9/30/2003 SAS architecture page 44
 Key ATA standards

• Working drafts of ATA standards are available on

http://www.t13.org
– Serial ATA 1.0a (output of private WG)
– ATA/ATAPI-7 Volume 1 (architecture and commands)
– ATA/ATAPI-7 Volume 3 (Serial ATA standard)
• Serial ATA II specifications are available on
http://www.t10.org and http://www.serialata.org
– Serial ATA II: Extensions to Serial ATA 1.0
– Serial ATA II: Port Multiplier
– Serial ATA II: Port Selector
– Serial ATA II: Cables and Connectors Volume 1

9/30/2003 SAS architecture page 45

 For more information

• International Committee for • SCSI Trade Association

Information Technology – http://www.scsita.org
Standards • Serial ATA Working Group
– http://www.incits.org – http://www.serialata.org
• T10 (SCSI standards) • SNIA (Storage Networking Industry
– http://www.t10.org Association)
– Latest SAS working draft – http://www.snia.org
– T10 reflector for developers • Industry news
• T13 (ATA standards) – http://www.infostor.com
– http://www.t13.org – http://www.byteandswitch.com
– T13 reflector for developers – http://www.wwpi.com
• T11 (Fibre Channel standards) – http://searchstorage.com
– http://www.t11.org • Training
• SFF (connectors) – http://www.knowledgetek.com
– http://www.sffcommittee.org
9/30/2003 SAS architecture page 46