Hardware

Read-Write Lock
Elision
Shady Issa Alexander
Pascal Felber Paolo Romano Matveev
 Multicores are everywhere

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 2

 Parallel programming

Main memory

Core 1 Core 2 Core 3 Core 4

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 3

 Parallel programming

Main memory

Core 1 Core 2 Core 3 Core 4

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 4

 Parallel programming

complexity
- deadlocks
Main memory - livelocks
- priority inversions
- convoy effects

Core 1 Core 2 Core 3 Core 4

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 5

 Parallel programming

Main memory

Transactional memory
atomic{
Core 1 Core 2 Core 3 Core 4 if(bal>amount)
withdraw(amount)
}
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 6
 Hardware Transactional Memory

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 7

 Hardware lock elision

r(A) w(B)
Thread 1
lock unlock

w(X) w(Y)
Thread 2
lock lock unlock

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 8

 Hardware lock elision

r(A) w(B)
Thread 1
lock unlock
Begin Commit
H/W Tx H/W Tx

w(X) w(Y)
Thread 2
lock unlock
Begin Commit
H/W Tx H/W Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 9

 Hardware lock elision

r(A) r(X)
Thread 1
lock
Begin
acquire
H/W Tx abort lock
w(X) normally
Thread 2
lock
Begin
H/W Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 10

 Hardware lock elision

r(A) r(X)
Thread 1
lock
Begin
acquire
H/W Tx abort lock
w(X) normally
Thread 2
lock capacity
Begin prohibited instructions
H/W Tx page faults
TLB miss

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 11

 Read-write Locks

read mode write mode

✔ concurrent ✘ sequential writers

readers
✘ blocks readers
read
dominated
workloads

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 12

 Hardware read-write lock elision

readers run without

instrumentation: writers run in H/W Txs:
• no H/W Txs • no lock acquisition
• no S/W tracking of • H/W tracking of
read locations read/write locations
• no lock acquisition

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 13

 Hardware read-write lock elision

r(X) r(?)
Reader
r-lock r-unlock

w(X) w(Y)
Writer
w-lock w-unlock
Begin Commit
HW Tx HW Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 14

 Hardware read-write lock elision

r(X) r(?)
Reader
r-lock r-unlock

? = Y
w(X) w(Y)
Writer
w-lock w-unlock
Begin Commit
H/W Tx H/W Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 15

 Hardware read-write lock elision

r(X) r(?)
Reader
r-lock r-unlock

? = Y
w(X) w(Y)
Writer
w-lock w-unlock w-unlock
Begin Commit
H/W Tx H/W Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 16

 Hardware read-write lock elision

r(X) r(Y)
Reader
r-lock r-unlock

w(X) w(Y)
Writer abort
w-lock w-unlock
Begin Commit
H/W Tx H/W Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 17

 Hardware read-write lock elision

r(X) r(?)
Reader
r-lock r-unlock
wait for concurrent readers
active here
w(X) w(Y)
Writer
w-lock w-unlock w-unlock
Begin Commit
H/W Tx H/W Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 18

 Hardware read-write lock elision
R1
r-unlock
R2
r-lock r-unlock

reader state reader state

R1 inactive R1 inactive
R2 active R2 inactive
. . . .

Writer abort
21/4/16
w-unlock Hardware Read-Write Lock Elision - Eurosys 2016
w-unlock 19
 Hardware read-write lock elision

r(X) r(?)
Reader
r-lock r-unlock
wait for concurrent readers
active here
w(X) w(Y)
Writer
w-lock w-unlock
Begin Suspend Resume Commit
HW Tx HW Tx HW Tx HW Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 20

 Hardware read-write lock elision
Writes use
H/W Txs

✔ concurrency ✘ Txs may never commit

among writers • fallback lock is a must
• Readers must synchronize
with lock holder

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 21

 Hardware read-write lock elision
check old
wait for lock
lock X
r(X) r(Y)
Reader
r-lock r-unlock r-lock

wait for concurrent readers

Writer w(X) w(Y)
in lock
w-lock w-unlock
fallback
Acquire Release
lock lock

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 22

 Rollback Only Transactions

r(X) r(X) r(X)

Thread 1
lock lock unlock
Begin abort Begin Commit
H/W Tx ROT ROT
w(X) w(X)
Thread 2
lock lock unlock
Begin Begin Commit
H/W Tx ROT ROT

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 23

 Rollback Only Transactions
old new
X X
r(X) r(X) r(X)
Thread 1
lock lock unlock
Begin abort Begin Commit
H/W Tx ROT ROT
w(X) w(X)
Thread 2
lock lock unlock
Begin Begin Commit
H/W Tx ROT ROT

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 24

 Using Rollback Only Transactions (ROTs)

✔ atomic ✘ no tracking of reads

✔ allow larger Txs ✘ not serializable

✔ no need for Suspend/Resume ✘ single writer

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 25

 Experiments
complex
benchmarks
synthetic and
benchmarks applications
10 cores
degree of STMBench7
contention 80 H/W
threads TPC-C
length of
Transactions Kyoto
Cabinet
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 26
 Optimistic Pessimestic
10% write locks

RW-LEOPT
RW-LEPES
HTM HLE
ROT
1
BRLock
ROT RWL
Time (s)

GL SGL

GL
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 27
 Optimistic Pessimestic
10% write locks

RW-LEOPT
RW-LEPES
HTM HLE
ROT
1
BRLock
ROT RWL
Time (s)

GL SGL

GL
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 28
 Synthetic benchmarks

10% write locks

RW-LEOPT
RW-LEPES
Size of Txs

1 HLE
BRLock
RWL
Time (s)

SGL

21/4/16
Degree of contention
Hardware Read-Write Lock Elision - Eurosys 2016 29
 Stress test – performance
10% writers
RW-LE
OPT 1
RW-LEPES
HLE
BRLock
1 RWL
SGL 10% write locks
Large Txs

Time (s)

Time (s)
0.1

RW-LEOPT
8X 7X RW-LEPES
0.1 0.01
16 32
1
Number of threads
64 80 16 32
Number of threads
64 80 HLE
10 BRLock
1 RWL
Time (s)
Small Txs

SGL
Time (s)

Time (s)
1

10X
0.1

0.1
16 32 64 80
Number of threads 16 32 64 80
Number of threads
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 30
Low contention High contention
 Stress test – abort rate
10% writers
Large Txs

Aborts (%)

Aborts (%)
ROT capacity
ROT conflicts

RW-LE RW-LE HLE RW-LE RW-LE HLE Lock aborts

OPT PES OPT PES

HTM capacity
HTM non-tx
Small Txs

Aborts (%)

Aborts (%) HTM tx

RW-LE RW-LE HLE RW-LE RW-LE HLE

OPT PES OPT PES
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 31
Low contention High contention
 Stress test
90% writers
RW-LEOPT 1
RW-LEPES
HLE
BRLock
1 RWL
SGL 10% write locks
Large Txs

Time (s)

Time (s)
0.1

RW-LEOPT
RW-LEPES
0.01
0.1
16 32
1 64 80 16 32 64 80 HLE
Number of threads Number of threads

100
10
BRLock
RWL
Time (s)
Small Txs

10
SGL
Time (s)

Time (s)
1

-25% 0.1
-10%
0.1
16 32 64 80 16 32 64 80
Number of threads Number of threads
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 32
Low contention High contention
 Synthetic benchmarks
HTM ROT SGL Uninstrumented HTM ROT SGL Uninstrumented
1% write locks 10% write locks 90% write locks 1% write locks 10% write locks 90% write locks
100 100
High capacity

Commits (%)

Commits (%)
80 80
60 60
40 40
20 20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES OPT PES OPT PES OPT PES
Number of threads (2,4,8,16,32,64,80) Number of threads (2,4,8,16,32,64,80)

HTM ROT SGL Uninstrumented HTM ROT SGL Uninstrumented

1% write locks 10% write locks 90% write locks 1% write locks 10% write locks 90% write locks
Low capacity

100 100
Commits (%)

Commits (%)
80 80
60 60
40 40
20 20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES OPT PES OPT PES OPT PES
Number of threads (2,4,8,16,32,64,80) Number of threads (2,4,8,16,32,64,80)

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 33

Low contention High contention
 Synthetic benchmarks

1% write locks 10% write locks 90% write locks 1% write locks 10% write locks 90% write locks
100 100
High capacity

ROT capacity ROT capacity

ROT conflicts ROT conflicts
80 Lock aborts 80 Lock aborts
Aborts (%)

Aborts (%)
HTM capacity HTM capacity
60 HTM non-tx 60 HTM non-tx
HTM tx HTM tx
40 40
20 20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES OPT PES OPT PES OPT PES
Number of threads (2,4,8,16,32,64,80) Number of threads (2,4,8,16,32,64,80)

1% write locks 10% write locks 90% write locks 1% write locks 10% write locks 90% write locks
100 ROT capacity
100 ROT capacity
Low capacity

ROT conflicts ROT conflicts

80 Lock aborts 80 Lock aborts
Aborts (%)

Aborts (%)
HTM capacity HTM capacity
60 HTM non-tx 60 HTM non-tx
HTM tx HTM tx
40 40
20 20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES OPT PES OPT PES OPT PES
Number of threads (2,4,8,16,32,64,80) Number of threads (2,4,8,16,32,64,80)

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 34

Low contention High contention
 TPC-C 6X

1% write locks 10% write locks 50% write locks

Speedup (vs. SGL 1 thr.)

100
OPT
RW-LE HLE
PES BRLock
RW-LE
10 RWL
SGL
1

0.1
16 32 64 80 16 32 64 80 16 32 64 80
Number of threads
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 35
 TPC-C
HTM ROT SGL Uninstrumented
1% write locks 10% write locks 90% write locks
100
Commits (%)

80
60
40
20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES
Number of threads (1,4,8,16,32,64,80)
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 36
 TPC-C
1% write locks 10% write locks 50% write locks
100 ROT capacity
ROT conflicts
80 Lock aborts
Aborts (%)

HTM capacity
60 HTM non-tx
HTM tx
40
20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES
Number of threads (1,4,8,16,32,64,80)
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 37
 STMbench7 4X

10% write locks 50% write locks 90% write locks

Throughput (103 Tx/s)

10
OPT HLE
RW-LE PES
8 RW-LE BRLock
RWL
6 SGL
4
2
0
16 32 64 80 16 32 64 80 16 32 64 80
Number of threads
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 38
 STMbench7
HTM ROT SGL Uninstrumented
10% write locks 50% write locks 90% write locks
100
Commits (%)

80
60
40
20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES
Number of threads (2,4,8,16,32,64,80)
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 39
 STMbench7
10% write locks 50% write locks 90% write locks
100 ROT capacity
ROT conflicts
80 Lock aborts
Aborts (%)

HTM capacity
60 HTM non-tx
HTM tx
40
20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES
Number of threads (2,4,8,16,32,64,80)
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 40
 Kyoto Cabinet 2X

<1% write locks 5% write locks 10% write locks

Throughput (106 Tx/s)

8 RW-LEOPT HLE
RW-LEPES BRLock
6 Orig
SGL
4
2
0
16 32 64 16 32 64 16 32 64
Number of threads
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 41
 Kyoto Cabinet
HTM ROT SGL Uninstrumented
<1% write locks 5% write locks 10% write locks
100
Commits (%)

80
60
40
20
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES
Number of threads (1,4,8,16,32,64)
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 42
 Kyoto Cabinet
<1% write locks 5% write locks 10% write locks
50 ROT capacity
ROT conflicts
40 Lock aborts
Aborts (%)

HTM capacity
30 HTM non-tx
HTM tx
20
10
0 RW-LE RW-LE HLE RW-LE RW-LE HLE RW-LE RW-LE HLE
OPT PES OPT PES OPT PES
Number of threads (1,4,8,16,32,64)
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 43
 Conclusions

readers without
instrumentation

writers using
H/W Tx

21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 44

 Conclusions

readers without
instrumentation

writers using
H/W Tx
10X
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 45
 Conclusions

readers without
instrumentation
suspend/resume ROTs
writers using
H/W Tx
10x
21/4/16 Hardware Read-Write Lock Elision - Eurosys 2016 46