Recent Developments in Microarchitectural

Attacks: Meltdown, Spectre, and Rowhammer

Daniel Gruss
September 19, 2018
Graz University of Technology

1 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

printf("%d", i);
printf("%d", i);

2 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

h e miss
Cac
printf("%d", i);
printf("%d", i);

2 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

h e miss Req
uest
Cac
printf("%d", i);
printf("%d", i);

2 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

h e miss Req
uest
Cac
printf("%d", i);
onse
printf("%d", i); Resp

2 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

h e miss Req
uest
Cac
printf("%d", i);
i onse
printf("%d", i); Resp

2 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

h e miss Req
uest
Cac
printf("%d", i);
i onse
printf("%d", i); Resp

e hit
Cach

2 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

ess,
DRAM acc
slow

h e miss Req
uest
Cac
printf("%d", i);
i onse
printf("%d", i); Resp

e hit
Cach

2 Daniel Gruss — Graz University of Technology

 CPU Cache www.tugraz.at

ess,
DRAM acc
slow

h e miss Req
uest
Cac
printf("%d", i);
i onse
printf("%d", i); Resp

e hit
Cach
o D R A M access,
N
er
much fast
2 Daniel Gruss — Graz University of Technology
 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

flush access
access

3 Daniel Gruss — Graz University of Technology

 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

ca
flush ch e d access
e d ch
ca
Shared Memory

access

3 Daniel Gruss — Graz University of Technology

 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

flush Shared Memory access

access

3 Daniel Gruss — Graz University of Technology

 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

flush access
access

3 Daniel Gruss — Graz University of Technology

 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

flush access
access

3 Daniel Gruss — Graz University of Technology

 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

flush Shared Memory access

access

3 Daniel Gruss — Graz University of Technology

 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

flush Shared Memory access

access

3 Daniel Gruss — Graz University of Technology

 Flush+Reload www.tugraz.at

Shared Memory
ATTACKER VICTIM

flush Shared Memory access

access

fast if victim accessed data,

slow otherwise
3 Daniel Gruss — Graz University of Technology
 Memory Access Latency www.tugraz.at

Cache Hits

107
Number of accesses

104

101

50 100 150 200 250 300 350 400

Access time [CPU cycles]

4 Daniel Gruss — Graz University of Technology

 Memory Access Latency www.tugraz.at

Cache Hits Cache Misses

107
Number of accesses

104

101

50 100 150 200 250 300 350 400

Access time [CPU cycles]

4 Daniel Gruss — Graz University of Technology

 Out-of-order Execution www.tugraz.at

int width = 10 , height = 5;

float diagonal = sqrt ( width * width

+ height * height ) ;
int area = width * height ;

printf ( " Area % d x % d = % d \ n " , width , height , area ) ;

5 Daniel Gruss — Graz University of Technology

 Out-of-order Execution www.tugraz.at

Parallelize
ency

int width = 10 , height = 5;

Depend

float diagonal = sqrt ( width * width

+ height * height ) ;
int area = width * height ;

printf ( " Area % d x % d = % d \ n " , width , height , area ) ;

5 Daniel Gruss — Graz University of Technology

 Out-of-Order Execution www.tugraz.at

ITLB
L1 Instruction Cache

Branch Instruction Fetch & PreDecode

Frontend

Predictor
Instruction Queue

µOP Cache 4-Way Decode

µOPs µOP µOP µOP µOP

MUX

Allocation Queue
µOP µOP µOP µOP
Instructions are
CDB Reorder buffer
µOP µOP µOP µOP µOP µOP µOP µOP
• fetched and decoded in the front-end
Execution Engine

Scheduler
µOP µOP µOP µOP µOP µOP µOP µOP
ALU, FMA, . . .
ALU, AES, . . .

ALU, Vect, . . .

Store data
Load data

Load data
ALU, Branch

AGU

Execution Units

Load Buffer Store Buffer

Subsystem
Memory

DTLB STLB
L1 Data Cache
L2 Cache

6 Daniel Gruss — Graz University of Technology

 Out-of-Order Execution www.tugraz.at

ITLB
L1 Instruction Cache

Branch Instruction Fetch & PreDecode

Frontend

Predictor
Instruction Queue

µOP Cache 4-Way Decode

µOPs µOP µOP µOP µOP

MUX

Allocation Queue
µOP µOP µOP µOP
Instructions are
CDB Reorder buffer
µOP µOP µOP µOP µOP µOP µOP µOP
• fetched and decoded in the front-end
Execution Engine

Scheduler
µOP µOP µOP µOP µOP µOP µOP µOP

• dispatched to the backend

ALU, FMA, . . .
ALU, AES, . . .

ALU, Vect, . . .

Store data
Load data

Load data
ALU, Branch

AGU

Execution Units

Load Buffer Store Buffer

Subsystem
Memory

DTLB STLB
L1 Data Cache
L2 Cache

6 Daniel Gruss — Graz University of Technology

 Out-of-Order Execution www.tugraz.at

ITLB
L1 Instruction Cache

Branch Instruction Fetch & PreDecode

Frontend

Predictor
Instruction Queue

µOP Cache 4-Way Decode

µOPs µOP µOP µOP µOP

MUX

Allocation Queue
µOP µOP µOP µOP
Instructions are
CDB Reorder buffer
µOP µOP µOP µOP µOP µOP µOP µOP
• fetched and decoded in the front-end
Execution Engine

Scheduler
µOP µOP µOP µOP µOP µOP µOP µOP

• dispatched to the backend

ALU, FMA, . . .
ALU, AES, . . .

ALU, Vect, . . .

Store data
Load data

Load data
ALU, Branch

AGU

• processed by individual execution units

Execution Units

Load Buffer Store Buffer

Subsystem
Memory

DTLB STLB
L1 Data Cache
L2 Cache

6 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

char data = *( char *) 0 xffffffff81a000e0 ;

printf ( " % c \ n " , data ) ;

7 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

char data = *( char *) 0 xffffffff81a000e0 ;

printf ( " % c \ n " , data ) ;

segfault at ffffffff81a000e0 ip
0000000000400535
sp 00007 ffce4a80610 error 5 in reader

7 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

Adapted code

*( volatile char *) 0;
array [84 * 4096] = 0; // unreachable

8 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

Flush+Reload over all pages of the array

Access time
500

[cycles]
400
300

0 50 100 150 200 250

Page

9 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

Flush+Reload over all pages of the array

Access time
500

[cycles]
400
300

0 50 100 150 200 250

Page

This also works on AMD and ARM!

9 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

• Out-of-order instructions leave microarchitectural traces

10 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

• Out-of-order instructions leave microarchitectural traces

• We can see them for example through the cache

10 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

• Out-of-order instructions leave microarchitectural traces

• We can see them for example through the cache
• Give such instructions a name: transient instructions

10 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

• Out-of-order instructions leave microarchitectural traces

• We can see them for example through the cache
• Give such instructions a name: transient instructions
• We can indirectly observe the execution of transient instructions

10 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

• Combine the two things

char data = *( char *) 0 xffffffff81a000e0 ;

array [ data * 4096] = 0;

11 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

Flush+Reload again...

Access time
500

[cycles]
400
300

0 50 100 150 200 250

Page

... Meltdown actually works.

12 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

• Flush+Reload over all pages of the array

Access time
500

[cycles]
400
300

0 50 100 150 200 250

Page

• Index of cache hit reveals data

13 Daniel Gruss — Graz University of Technology

 Building Meltdown www.tugraz.at

• Flush+Reload over all pages of the array

Access time
500

[cycles]
400
300

0 50 100 150 200 250

Page

• Index of cache hit reveals data

• Permission check is in some cases not fast enough

13 Daniel Gruss — Graz University of Technology

Kernel Address Isolation to have Side channels Efficiently Removed
 KAISER /ˈkʌɪzə/
1. [german] Emperor,
ruler of an empire
2. largest penguin,
emperor penguin

Kernel Address Isolation to have Side channels Efficiently Removed

 KAISER Illustration www.tugraz.at

Without KAISER:
Shared address space
User memory Kernel memory
0 −1

context switch

16 Daniel Gruss — Graz University of Technology

 KAISER Illustration www.tugraz.at

Without KAISER:
Shared address space
User memory Kernel memory
0 −1

context switch

With KAISER:
User address space
User memory Not mapped
0 −1

addr. space
switch
context switch Interrupt
dispatcher

SMAP + SMEP Kernel memory

0 −1
Kernel address space

16 Daniel Gruss — Graz University of Technology

 Foreshadow / Foreshadow-NG1 [Van+18; Wei+18] www.tugraz.at

1
Jo Van Bulck et al. Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient
Out-of-Order Execution. In: USENIX Security Symposium. 2018.
17 Daniel Gruss — Graz University of Technology
L1TF/Foreshadow Demo
 Mitigating L1TF/Foreshadow www.tugraz.at

Either:

19 Daniel Gruss — Graz University of Technology

 Mitigating L1TF/Foreshadow www.tugraz.at

Either:
• hyperthreading: only schedule mutually trusting threads on
same physical core

19 Daniel Gruss — Graz University of Technology

 Mitigating L1TF/Foreshadow www.tugraz.at

Either:
• hyperthreading: only schedule mutually trusting threads on
same physical core
• context switch: flush L1 when switching to guest

19 Daniel Gruss — Graz University of Technology

 Mitigating L1TF/Foreshadow www.tugraz.at

Either:
• hyperthreading: only schedule mutually trusting threads on
same physical core
• context switch: flush L1 when switching to guest
Or:

19 Daniel Gruss — Graz University of Technology

 Mitigating L1TF/Foreshadow www.tugraz.at

Either:
• hyperthreading: only schedule mutually trusting threads on
same physical core
• context switch: flush L1 when switching to guest
Or:
• disable EPTs

19 Daniel Gruss — Graz University of Technology

 Spectre (variant 1) www.tugraz.at

index = 0;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 0;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 0;

char* data = "textKEY";

if (index < 4)
Speculate
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 0;

char* data = "textKEY";

if (index < 4)
Execute
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 1;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 1;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 1;

char* data = "textKEY";

if (index < 4)
Speculate
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 1;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 2;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 2;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 2;

char* data = "textKEY";

if (index < 4)
Speculate
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 2;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 3;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 3;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 3;

char* data = "textKEY";

if (index < 4)
Speculate
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 3;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 4;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 4;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 4;

char* data = "textKEY";

if (index < 4)
Speculate
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 4;

char* data = "textKEY";

if (index < 4)
Execute
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 5;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 5;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 5;

char* data = "textKEY";

if (index < 4)
Speculate
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 5;

char* data = "textKEY";

if (index < 4)
Execute
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 6;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 6;

char* data = "textKEY";

if (index < 4)
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 6;

char* data = "textKEY";

if (index < 4)
Speculate
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 1) www.tugraz.at

index = 6;

char* data = "textKEY";

if (index < 4)
Execute
els
en e
th
Prediction

LUT[data[index] * 4096] 0
21 Daniel Gruss — Graz University of Technology
 Spectre (variant 4) www.tugraz.at

index = 0;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 0;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 0;

index = index & 0x3; // sanitization

char* data = "textKEY";

Speculate
er ig
id no
s re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 0;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 1;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 1;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 1;

index = index & 0x3; // sanitization

char* data = "textKEY";

Speculate
er ig
id no
s re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 1;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 2;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 2;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 2;

index = index & 0x3; // sanitization

char* data = "textKEY";

Speculate
er ig
id no
s re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 2;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 3;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 3;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 3;

index = index & 0x3; // sanitization

char* data = "textKEY";

Speculate
er ig
id no
s re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 3;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 4;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 4;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 4;

index = index & 0x3; // sanitization

char* data = "textKEY";

Speculate
er ig
id no
s re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 4;

index = index & 0x3; // sanitization

char* data = "textKEY";

Execute 0 ig
= no
x re
nde
i Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 5;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 5;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 5;

index = index & 0x3; // sanitization

char* data = "textKEY";

Speculate
er ig
id no
s re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 5;

index = index & 0x3; // sanitization

char* data = "textKEY";

Execute 1 ig
= no
x re
nde
i Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 6;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 6;

index = index & 0x3; // sanitization

char* data = "textKEY";

er ig
no
sid re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 6;

index = index & 0x3; // sanitization

char* data = "textKEY";

Speculate
er ig
id no
s re
con
Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 4) www.tugraz.at

index = 6;

index = index & 0x3; // sanitization

char* data = "textKEY";

Execute 2 ig
= no
x re
nde
i Prediction

LUT[data[index] * 4096] LUT[data[index] * 4096]

22 Daniel Gruss — Graz University of Technology

 Spectre (variant 1.1) www.tugraz.at

“Speculative Buffer Overflows”

• Speculatively write to memory locations

→ Many more gadgets than previously anticipated
• Very interesting for sandboxes
• Causes some protection mechanisms to fail

23 Daniel Gruss — Graz University of Technology

 Spectre (variant 1.2) www.tugraz.at

“Speculative Buffer Overflows”

• Speculatively write to memory locations which are not writable

• Actually a variant of Meltdown
• A permission bit is ignored during out-of-order execution
• But no scenario where it makes sense without speculative execution?

24 Daniel Gruss — Graz University of Technology

 Spectre (variant 2) www.tugraz.at

Animal* a = bird;

a->move()
sw
) im
fl y( swim()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = bird;

a->move()
Speculate
sw
() swim() im
fly ()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = bird;

a->move()
sw
) im
fl y( swim()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = bird;

a->move()
Execute sw
) im
fl y( swim()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = bird;

a->move()
sw
) im
fl y( fly()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = bird;

a->move()
Speculate
sw
) im
fl y( fly()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = bird;

a->move()
sw
) im
fl y( fly()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = fish;

a->move()
sw
) im
fl y( fly()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = fish;

a->move()
Speculate
sw
) im
fl y( fly()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = fish;

a->move()
sw
) im
fl y( fly()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = fish;

a->move()
sw Execute
() fly() i m
fly ()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 2) www.tugraz.at

Animal* a = fish;

a->move()
sw
) im
fl y( swim()
()

Prediction
LUT[data[index] * 4096] 0
25 Daniel Gruss — Graz University of Technology
 Spectre (variant 5) www.tugraz.at

• “SpectreRSB”
• Similar to Spectre variant 2:
• Redirect an indirect branch (a return in this case)
• Fill buffer with “wrong” values

26 Daniel Gruss — Graz University of Technology

 Mitigating Spectre www.tugraz.at

• Trivial approach: disable speculative execution

27 Daniel Gruss — Graz University of Technology

 Mitigating Spectre www.tugraz.at

• Trivial approach: disable speculative execution

• No wrong speculation if there is no speculation

27 Daniel Gruss — Graz University of Technology

 Mitigating Spectre www.tugraz.at

• Trivial approach: disable speculative execution

• No wrong speculation if there is no speculation
• Problem: massive performance hit!

27 Daniel Gruss — Graz University of Technology

 Mitigating Spectre www.tugraz.at

• Trivial approach: disable speculative execution

• No wrong speculation if there is no speculation
• Problem: massive performance hit!
• Also: How to disable it?

27 Daniel Gruss — Graz University of Technology

 Mitigating Spectre www.tugraz.at

• Trivial approach: disable speculative execution

• No wrong speculation if there is no speculation
• Problem: massive performance hit!
• Also: How to disable it?
• Speculative execution is deeply integrated into CPU

27 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

28 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Workaround: insert instructions stopping

speculation

28 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Workaround: insert instructions stopping

speculation
→ insert after every bounds check

28 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Workaround: insert instructions stopping

speculation
→ insert after every bounds check
• x86: LFENCE, ARM: CSDB

28 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Workaround: insert instructions stopping

speculation
→ insert after every bounds check
• x86: LFENCE, ARM: CSDB
• Available on all Intel CPUs, retrofitted to existing
ARMv7 and ARMv8

28 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

29 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Speculation barrier requires compiler supported

29 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Speculation barrier requires compiler supported

• Already implemented in GCC, LLVM, and MSVC

29 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Speculation barrier requires compiler supported

• Already implemented in GCC, LLVM, and MSVC
• Can be automated (MSVC) → not really reliable

29 Daniel Gruss — Graz University of Technology

 Spectre Variant 1 Mitigations www.tugraz.at

• Speculation barrier requires compiler supported

• Already implemented in GCC, LLVM, and MSVC
• Can be automated (MSVC) → not really reliable
• Explicit use by programmer: builtin load no speculate

29 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Indirect Branch Restricted Speculation (IBRS):

30 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Indirect Branch Restricted Speculation (IBRS):
• Do not speculate based on anything before entering IBRS mode

30 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Indirect Branch Restricted Speculation (IBRS):
• Do not speculate based on anything before entering IBRS mode
→ lesser privileged code cannot influence predictions

30 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Indirect Branch Restricted Speculation (IBRS):
• Do not speculate based on anything before entering IBRS mode
→ lesser privileged code cannot influence predictions
• Indirect Branch Predictor Barrier (IBPB):

30 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Indirect Branch Restricted Speculation (IBRS):
• Do not speculate based on anything before entering IBRS mode
→ lesser privileged code cannot influence predictions
• Indirect Branch Predictor Barrier (IBPB):
• Flush branch-target buffer

30 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Indirect Branch Restricted Speculation (IBRS):
• Do not speculate based on anything before entering IBRS mode
→ lesser privileged code cannot influence predictions
• Indirect Branch Predictor Barrier (IBPB):
• Flush branch-target buffer
• Single Thread Indirect Branch Predictors (STIBP):

30 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Indirect Branch Restricted Speculation (IBRS):
• Do not speculate based on anything before entering IBRS mode
→ lesser privileged code cannot influence predictions
• Indirect Branch Predictor Barrier (IBPB):
• Flush branch-target buffer
• Single Thread Indirect Branch Predictors (STIBP):
• Isolates branch prediction state between two hyperthreads

30 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Software) www.tugraz.at

Retpoline (compiler extension)

31 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Software) www.tugraz.at

Retpoline (compiler extension)

push < call_target >
call 1 f
2: lfence ; speculation barrier
jmp 2 b ; endless loop
1: lea 8(% rsp ) , % rsp ; restore stack pointer
ret ; the actual call to <
call_target >

→ always predict to enter an endless loop

31 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Software) www.tugraz.at

Retpoline (compiler extension)

push < call_target >
call 1 f
2: lfence ; speculation barrier
jmp 2 b ; endless loop
1: lea 8(% rsp ) , % rsp ; restore stack pointer
ret ; the actual call to <
call_target >

→ always predict to enter an endless loop

• instead of the correct (or wrong) target function

31 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Software) www.tugraz.at

Retpoline (compiler extension)

push < call_target >
call 1 f
2: lfence ; speculation barrier
jmp 2 b ; endless loop
1: lea 8(% rsp ) , % rsp ; restore stack pointer
ret ; the actual call to <
call_target >

→ always predict to enter an endless loop

• instead of the correct (or wrong) target function →
performance?

31 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Software) www.tugraz.at

Retpoline (compiler extension)

push < call_target >
call 1 f
2: lfence ; speculation barrier
jmp 2 b ; endless loop
1: lea 8(% rsp ) , % rsp ; restore stack pointer
ret ; the actual call to <
call_target >

→ always predict to enter an endless loop

• instead of the correct (or wrong) target function →
performance?
• ret may fall-back to the BTB for prediction

31 Daniel Gruss — Graz University of Technology

 Spectre Variant 2 Mitigations (Software) www.tugraz.at

Retpoline (compiler extension)

push < call_target >
call 1 f
2: lfence ; speculation barrier
jmp 2 b ; endless loop
1: lea 8(% rsp ) , % rsp ; restore stack pointer
ret ; the actual call to <
call_target >

→ always predict to enter an endless loop

• instead of the correct (or wrong) target function →
performance?
• ret may fall-back to the BTB for prediction
→ microcode patches to prevent that

31 Daniel Gruss — Graz University of Technology

 Spectre Variant 4 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

32 Daniel Gruss — Graz University of Technology

 Spectre Variant 4 Mitigations (Microcode/MSRs) www.tugraz.at

Intel released microcode updates

• Disable store-to-load-forward speculation
• Performance impact of 2–8%

32 Daniel Gruss — Graz University of Technology

 Spectre Variant 5 Mitigations (Software) www.tugraz.at

• Already implicitly patched on some architectures

• RSB stuffing (part of retpoline)

33 Daniel Gruss — Graz University of Technology

 What does not work www.tugraz.at

• Prevent access to high-resolution timer

34 Daniel Gruss — Graz University of Technology

 What does not work www.tugraz.at

• Prevent access to high-resolution timer

→ Own timer using timing thread

34 Daniel Gruss — Graz University of Technology

 What does not work www.tugraz.at

• Prevent access to high-resolution timer

→ Own timer using timing thread
• Flush instruction only privileged

34 Daniel Gruss — Graz University of Technology

 What does not work www.tugraz.at

• Prevent access to high-resolution timer

→ Own timer using timing thread
• Flush instruction only privileged
→ Cache eviction through memory accesses

34 Daniel Gruss — Graz University of Technology

 What does not work www.tugraz.at

• Prevent access to high-resolution timer

→ Own timer using timing thread
• Flush instruction only privileged
→ Cache eviction through memory accesses
• Just move secrets into secure world

34 Daniel Gruss — Graz University of Technology

 What does not work www.tugraz.at

• Prevent access to high-resolution timer

→ Own timer using timing thread
• Flush instruction only privileged
→ Cache eviction through memory accesses
• Just move secrets into secure world
→ Spectre works on secure enclaves

34 Daniel Gruss — Graz University of Technology

 Meltdown vs. Spectre www.tugraz.at

Meltdown attacks Spectre attacks

35 Daniel Gruss — Graz University of Technology

 Meltdown vs. Spectre www.tugraz.at

Meltdown attacks Spectre attacks

• Meltdown, LazyFP (v3.1), • v1, v1.1, v2, v4, SpectreRSB (v5)
Foreshadow, Foreshadow-NG, ...

35 Daniel Gruss — Graz University of Technology

 Meltdown vs. Spectre www.tugraz.at

Meltdown attacks Spectre attacks

• Meltdown, LazyFP (v3.1), • v1, v1.1, v2, v4, SpectreRSB (v5)
Foreshadow, Foreshadow-NG, ... • Speculative Execution ⊂
• Out-of-Order Execution Out-of-Order Execution

35 Daniel Gruss — Graz University of Technology

 Meltdown vs. Spectre www.tugraz.at

Meltdown attacks Spectre attacks

• Meltdown, LazyFP (v3.1), • v1, v1.1, v2, v4, SpectreRSB (v5)
Foreshadow, Foreshadow-NG, ... • Speculative Execution ⊂
• Out-of-Order Execution Out-of-Order Execution
• no prediction required • fundamentally rely on prediction

35 Daniel Gruss — Graz University of Technology

 Meltdown vs. Spectre
Meltdown attacks
• Meltdown, LazyFP (v3.1),
Foreshadow, Foreshadow-NG, ...
• Out-of-Order Execution
• no prediction required
→ melt down isolation by ignoring access
permissions (e.g., page table bits)
35
www.tugraz.at
Spectre attacks
• v1, v1.1, v2, v4, SpectreRSB (v5)
• Speculative Execution ⊂
Out-of-Order Execution
• fundamentally rely on prediction
• difficult to mitigate because it does
not violate access permissions
Daniel Gruss — Graz University of Technology
 Meltdown vs. Spectre
Meltdown attacks
• Meltdown, LazyFP (v3.1),
Foreshadow, Foreshadow-NG, ...
• Out-of-Order Execution
• no prediction required
→ melt down isolation by ignoring access
permissions (e.g., page table bits)
• practical mitigation in software (e.g.,
KAISER)
35
www.tugraz.at
Spectre attacks
• v1, v1.1, v2, v4, SpectreRSB (v5)
• Speculative Execution ⊂
Out-of-Order Execution
• fundamentally rely on prediction
• difficult to mitigate because it does
not violate access permissions
Daniel Gruss — Graz University of Technology
What if we want to modify data?
 DRAM organization www.tugraz.at

37 Daniel Gruss — Graz University of Technology

 DRAM organization www.tugraz.at

channel 0

channel 1

37 Daniel Gruss — Graz University of Technology

 DRAM organization www.tugraz.at

back of DIMM: rank 1

channel 0
front of DIMM:
rank 0

channel 1

37 Daniel Gruss — Graz University of Technology

 DRAM organization www.tugraz.at

back of DIMM: rank 1

channel 0
front of DIMM:
rank 0

chip
channel 1

37 Daniel Gruss — Graz University of Technology

 DRAM organization www.tugraz.at

bank 0
chip
row 0
row 1
row 2
...
row 32767

row buffer

38 Daniel Gruss — Graz University of Technology

 DRAM organization www.tugraz.at

bank 0
chip
row 0
row 1
row 2
...
row 32767 64k cells
1 capacitor,
row buffer
1 transitor each

38 Daniel Gruss — Graz University of Technology

 Rowhammer www.tugraz.at

DRAM bank
• Cells leak → repetitive refresh
11111111111111
necessary
11111111111111
11111111111111
• Maximum interval between
11111111111111 refreshes to guarantee data
... integrity
11111111111111
• Cells leak faster upon
proximate accesses →
row buffer
Rowhammer

39 Daniel Gruss — Graz University of Technology

 Rowhammer www.tugraz.at

DRAM bank
• Cells leak → repetitive refresh
11111111111111
necessary
activate
11111111111111
11111111111111
• Maximum interval between
11111111111111 refreshes to guarantee data
... copy integrity
11111111111111
• Cells leak faster upon
proximate accesses →
row buffer
Rowhammer

39 Daniel Gruss — Graz University of Technology

 Rowhammer www.tugraz.at

DRAM bank
• Cells leak → repetitive refresh
11111111111111
necessary
11111111111111
11111111111111
• Maximum interval between
activate
11111111111111 refreshes to guarantee data
... integrity
11111111111111 copy
• Cells leak faster upon
proximate accesses →
row buffer
Rowhammer

39 Daniel Gruss — Graz University of Technology

 Rowhammer www.tugraz.at

DRAM bank
• Cells leak → repetitive refresh
11111111111111
necessary
activate
11111111111111
11111111111111
• Maximum interval between
11111111111111 refreshes to guarantee data
... copy integrity
11111111111111
• Cells leak faster upon
proximate accesses →
row buffer
Rowhammer

39 Daniel Gruss — Graz University of Technology

 Rowhammer www.tugraz.at

DRAM bank
• Cells leak → repetitive refresh
11111111111111
necessary
11111111111111
11111111111111
• Maximum interval between
activate
11111111111111 refreshes to guarantee data
... integrity
11111111111111 copy
• Cells leak faster upon
proximate accesses →
row buffer
Rowhammer

39 Daniel Gruss — Graz University of Technology

 Rowhammer www.tugraz.at

DRAM bank
• Cells leak → repetitive refresh
11111111111111 bit flips in row 2! necessary
11111111111111
10111110101111
• Maximum interval between
11111111111111 refreshes to guarantee data
... integrity
11111111111111
• Cells leak faster upon
proximate accesses →
row buffer
Rowhammer

39 Daniel Gruss — Graz University of Technology

 Hammering techniques www.tugraz.at

• There are two different hammering techniques

40 Daniel Gruss — Graz University of Technology

 Hammering techniques www.tugraz.at

• There are two different hammering techniques

• #1: Hammer one row next to victim row and other random rows

40 Daniel Gruss — Graz University of Technology

 Hammering techniques www.tugraz.at

• There are two different hammering techniques

• #1: Hammer one row next to victim row and other random rows
• #2: Hammer two rows neighboring victim row

40 Daniel Gruss — Graz University of Technology

 Hammering techniques www.tugraz.at

• There are three different hammering techniques

• #1: Hammer one row next to victim row and other random rows
• #2: Hammer two rows neighboring victim row
• #3: Hammer only one row next to victim row

40 Daniel Gruss — Graz University of Technology

 #1 - Single-sided hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

41 Daniel Gruss — Graz University of Technology

 #1 - Single-sided hammering www.tugraz.at

DRAM bank
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

41 Daniel Gruss — Graz University of Technology

 #1 - Single-sided hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

41 Daniel Gruss — Graz University of Technology

 #1 - Single-sided hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
activate
11111111111111

41 Daniel Gruss — Graz University of Technology

 #1 - Single-sided hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

41 Daniel Gruss — Graz University of Technology

 #1 - Single-sided hammering www.tugraz.at

DRAM bank

11111111111111 bit flips

11111111111111
10111110101111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

41 Daniel Gruss — Graz University of Technology

 #2 - Double-sided hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

42 Daniel Gruss — Graz University of Technology

 #2 - Double-sided hammering www.tugraz.at

DRAM bank

11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

42 Daniel Gruss — Graz University of Technology

 #2 - Double-sided hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

42 Daniel Gruss — Graz University of Technology

 #2 - Double-sided hammering www.tugraz.at

DRAM bank

11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

42 Daniel Gruss — Graz University of Technology

 #2 - Double-sided hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

42 Daniel Gruss — Graz University of Technology

 #2 - Double-sided hammering www.tugraz.at

DRAM bank

11111111111111 bit flips

activate
11111111111111
10111110101111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

42 Daniel Gruss — Graz University of Technology

 #3 - One-location hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

43 Daniel Gruss — Graz University of Technology

 #3 - One-location hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

43 Daniel Gruss — Graz University of Technology

 #3 - One-location hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

43 Daniel Gruss — Graz University of Technology

 #3 - One-location hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

43 Daniel Gruss — Graz University of Technology

 #3 - One-location hammering www.tugraz.at

DRAM bank

11111111111111
11111111111111
11111111111111
activate
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

43 Daniel Gruss — Graz University of Technology

 #3 - One-location hammering www.tugraz.at

DRAM bank

11111111111111 bit flips

11111111111111
10111110101111
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

43 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

44 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

44 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

44 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

• They are not random → highly reproducible flip pattern!

44 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

• They are not random → highly reproducible flip pattern!

1. Choose a data structure that you can place at arbitrary memory
locations

44 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

• They are not random → highly reproducible flip pattern!

1. Choose a data structure that you can place at arbitrary memory
locations
2. Scan for “good” flips

44 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

• They are not random → highly reproducible flip pattern!

1. Choose a data structure that you can place at arbitrary memory
locations
2. Scan for “good” flips
3. Place data structure there

44 Daniel Gruss — Graz University of Technology

 How to exploit random bit flips? www.tugraz.at

• They are not random → highly reproducible flip pattern!

1. Choose a data structure that you can place at arbitrary memory
locations
2. Scan for “good” flips
3. Place data structure there
4. Trigger bit flip again

44 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

45 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

45 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

45 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

• Many applications perform actions as root

45 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

• Many applications perform actions as root

45 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

• Many applications perform actions as root

• They can be used by unprivileged users as well

45 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

• Many applications perform actions as root

• They can be used by unprivileged users as well

45 Daniel Gruss — Graz University of Technology

 What if we cannot target kernel pages? www.tugraz.at

• Many applications perform actions as root

• They can be used by unprivileged users as well
• sudo

45 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE HLT
0 1 1 1 0 1 0 0 1 1 1 1 0 1 0 0

46 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE XORB
0 1 1 1 0 1 0 0 0 0 1 1 0 1 0 0

46 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE PUSHQ
0 1 1 1 0 1 0 0 0 1 0 1 0 1 0 0

46 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE <prefix>
0 1 1 1 0 1 0 0 0 1 1 0 0 1 0 0

46 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE JL
0 1 1 1 0 1 0 0 0 1 1 1 1 1 0 0

46 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE JO
0 1 1 1 0 1 0 0 0 1 1 1 0 0 0 0

46 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE JBE
0 1 1 1 0 1 0 0 0 1 1 1 0 1 1 0

46 Daniel Gruss — Graz University of Technology

 Opcode Flipping - Conditional Jump www.tugraz.at

JE JNE
0 1 1 1 0 1 0 0 0 1 1 1 0 1 0 1

46 Daniel Gruss — Graz University of Technology

 Page Cache www.tugraz.at

• If a binary is loaded the first time, it is loaded to the memory

47 Daniel Gruss — Graz University of Technology

 Page Cache www.tugraz.at

• If a binary is loaded the first time, it is loaded to the memory

• It stays in memory (in the page cache) even after execution

47 Daniel Gruss — Graz University of Technology

 Page Cache www.tugraz.at

• If a binary is loaded the first time, it is loaded to the memory

• It stays in memory (in the page cache) even after execution
• Only evicted if page cache is full

47 Daniel Gruss — Graz University of Technology

 Page Cache www.tugraz.at

• If a binary is loaded the first time, it is loaded to the memory

• It stays in memory (in the page cache) even after execution
• Only evicted if page cache is full
• Page cache is huge - usually all unused memory

47 Daniel Gruss — Graz University of Technology

 Memory Waylaying www.tugraz.at

(1) Start

B
X

48 Daniel Gruss — Graz University of Technology

 Memory Waylaying www.tugraz.at

(2) Evict Page Cache

48 Daniel Gruss — Graz University of Technology

 Memory Waylaying www.tugraz.at

(3) Access Binary

48 Daniel Gruss — Graz University of Technology

 Memory Waylaying www.tugraz.at

(4) Evict + Access

B

48 Daniel Gruss — Graz University of Technology

 Memory Waylaying www.tugraz.at

(5) Evict + Access

48 Daniel Gruss — Graz University of Technology

 Memory Waylaying www.tugraz.at

(6) Stop if target reached

X
B

48 Daniel Gruss — Graz University of Technology

 How well does it work? www.tugraz.at

• New pages cover most of the physical memory

49 Daniel Gruss — Graz University of Technology

 How well does it work? www.tugraz.at

• Great advantage over memory massaging: only negligible memory footprint

50 Daniel Gruss — Graz University of Technology

Rowhammer + SGX = Cheap Denial of Service
 Intel SGX www.tugraz.at

• Instruction-set extension
• Integrity and confidentiality of code and data in untrusted
environments
• Run with user privileges and restricted, e.g., no system calls
• Run programs in enclaves using protected areas of memory

51 Daniel Gruss — Graz University of Technology

 SGX Encrypted Memory www.tugraz.at

EPC (128 MB)

Physical Memory
0 GB 16 GB

52 Daniel Gruss — Graz University of Technology

 SGX Encrypted Memory www.tugraz.at

EPC (128 MB)

Physical Memory
0 GB 16 GB

52 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• What happens if a bit flips in the EPC?

53 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• What happens if a bit flips in the EPC?

• Integrity check will fail!

53 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• What happens if a bit flips in the EPC?

• Integrity check will fail!
→ Locks up the memory controller

53 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• What happens if a bit flips in the EPC?

• Integrity check will fail!
→ Locks up the memory controller
→ Not a single further memory access!

53 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• What happens if a bit flips in the EPC?

• Integrity check will fail!
→ Locks up the memory controller
→ Not a single further memory access!
→ System halts immediately

53 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• What happens if a bit flips in the EPC?

• Integrity check will fail!
→ Locks up the memory controller
→ Not a single further memory access!
→ System halts immediately

53 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• If a malicious enclave induces a bit flip, . . .

54 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• If a malicious enclave induces a bit flip, . . .

• . . . the entire machine halts

54 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• If a malicious enclave induces a bit flip, . . .

• . . . the entire machine halts
• . . . including co-located tenants

54 Daniel Gruss — Graz University of Technology

 Bit Flips in the EPC www.tugraz.at

• If a malicious enclave induces a bit flip, . . .

• . . . the entire machine halts
• . . . including co-located tenants
• Denial-of-Service Attacks in the Cloud [Gru+18; Jan+17]

54 Daniel Gruss — Graz University of Technology

SGX + One-location Hammering + Opcode Flipping =
Undetectable Exploit
 (Ab)using SGX Protection www.tugraz.at

• SGX protects software from malicious environments

55 Daniel Gruss — Graz University of Technology

 (Ab)using SGX Protection www.tugraz.at

• SGX protects software from malicious environments

• Thwarts static and dynamic (= performance counters) analysis

55 Daniel Gruss — Graz University of Technology

 (Ab)using SGX Protection www.tugraz.at

• SGX protects software from malicious environments

• Thwarts static and dynamic (= performance counters) analysis
• Hammering from SGX defeats countermeasures relying on this

55 Daniel Gruss — Graz University of Technology

 Bypassing the Defenses www.tugraz.at

n er
s
nter

Patt

t
imit
Cou

prin
cess
sis

Prox

foot
ance
naly

y Ac
ic A

ory
orm

sica
or
Mem

Mem
Stat

Perf

Phy
Defense Class
Bypass

Intel SGX
One-location hammering
Opcode flipping
Memory waylaying
Defense class defeated

57 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

= 671 875 accesses per second

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

= 671 875 accesses per second
• Network packets access memory location up to 6 times
(depending on kernel)

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

= 671 875 accesses per second
• Network packets access memory location up to 6 times
(depending on kernel)
→ 111 979 packets per second

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

= 671 875 accesses per second
• Network packets access memory location up to 6 times
(depending on kernel)
→ 111 979 packets per second
• Network packets are a least 64 B

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

= 671 875 accesses per second
• Network packets access memory location up to 6 times
(depending on kernel)
→ 111 979 packets per second
• Network packets are a least 64 B
= 7 166 656 B/s = 7 MB/s = 57 Mb/s

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

= 671 875 accesses per second
• Network packets access memory location up to 6 times
(depending on kernel)
→ 111 979 packets per second
• Network packets are a least 64 B
= 7 166 656 B/s = 7 MB/s = 57 Mb/s
→ That sounds doable on “modern” networks

58 Daniel Gruss — Graz University of Technology

 Just comparing some performance numbers... www.tugraz.at

• ≥ 43 000 hammering attempts (within 64 ms) for a bit flip [GMM16]

= 671 875 accesses per second
• Network packets access memory location up to 6 times
(depending on kernel)
→ 111 979 packets per second
• Network packets are a least 64 B
= 7 166 656 B/s = 7 MB/s = 57 Mb/s
→ That sounds doable on “modern” networks

58 Daniel Gruss — Graz University of Technology

 Nethammer www.tugraz.at

59 Daniel Gruss — Graz University of Technology

 Nethammer www.tugraz.at

59 Daniel Gruss — Graz University of Technology

 Nethammer www.tugraz.at

Inducing bit flips:

59 Daniel Gruss — Graz University of Technology

 Nethammer www.tugraz.at

Inducing bit flips:

• Network stacks on ARM often use uncached memory
(perfect for hammering)

59 Daniel Gruss — Graz University of Technology

 Nethammer www.tugraz.at

Inducing bit flips:

• Network stacks on ARM often use uncached memory
(perfect for hammering)
• Intel recommends Intel CAT for QoS (perfect for hammering)

59 Daniel Gruss — Graz University of Technology

 Nethammer www.tugraz.at

Inducing bit flips:

• Network stacks on ARM often use uncached memory
(perfect for hammering)
• Intel recommends Intel CAT for QoS (perfect for hammering)
• Network reachable code might use clflush or non-temporal
stores
(both great for hammering)

59 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”
• attacks on ASLR

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”
• attacks on ASLR → “ASLR is broken anyway”

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”
• attacks on ASLR → “ASLR is broken anyway”
• attacks on SGX and TrustZone

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”
• attacks on ASLR → “ASLR is broken anyway”
• attacks on SGX and TrustZone → “not part of the threat model”

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”
• attacks on ASLR → “ASLR is broken anyway”
• attacks on SGX and TrustZone → “not part of the threat model”
• Rowhammer

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”
• attacks on ASLR → “ASLR is broken anyway”
• attacks on SGX and TrustZone → “not part of the threat model”
• Rowhammer → “only affects cheap sub-standard modules”

60 Daniel Gruss — Graz University of Technology

 How did we get here? www.tugraz.at

We have ignored microarchitectural attacks for many years:

• attacks on crypto → “software should be fixed”
• attacks on ASLR → “ASLR is broken anyway”
• attacks on SGX and TrustZone → “not part of the threat model”
• Rowhammer → “only affects cheap sub-standard modules”
→ for years we solely optimized for performance

60 Daniel Gruss — Graz University of Technology

 ... and we’re still optimizing for performance www.tugraz.at

• lower refresh rate = lower energy but more bit flips

61 Daniel Gruss — Graz University of Technology

 ... and we’re still optimizing for performance www.tugraz.at

• lower refresh rate = lower energy but more bit flips

• ECC memory → fewer bit flips

61 Daniel Gruss — Graz University of Technology

 ... and we’re still optimizing for performance www.tugraz.at

• lower refresh rate = lower energy but more bit flips

• ECC memory → fewer bit flips
→ it’s an optimization problem

61 Daniel Gruss — Graz University of Technology

 ... and we’re still optimizing for performance www.tugraz.at

• lower refresh rate = lower energy but more bit flips

• ECC memory → fewer bit flips
→ it’s an optimization problem
• what if “too aggressive” changes over time?

61 Daniel Gruss — Graz University of Technology

 ... and we’re still optimizing for performance www.tugraz.at

• lower refresh rate = lower energy but more bit flips

• ECC memory → fewer bit flips
→ it’s an optimization problem
• what if “too aggressive” changes over time?
→ difficult to optimize with an intelligent adversary

61 Daniel Gruss — Graz University of Technology

 Conclusions www.tugraz.at

A unique chance to
• rethink processor design

62 Daniel Gruss — Graz University of Technology

 Conclusions www.tugraz.at

A unique chance to
• rethink processor design
• many problems to solve around microarchitectural attacks

62 Daniel Gruss — Graz University of Technology

 Conclusions www.tugraz.at

A unique chance to
• rethink processor design
• many problems to solve around microarchitectural attacks
• dedicate more time into identifying problems and not solely in
mitigating known problems

62 Daniel Gruss — Graz University of Technology

 Recent Developments in Microarchitectural
Attacks: Meltdown, Spectre, and Rowhammer

Daniel Gruss
September 19, 2018
Graz University of Technology

63 Daniel Gruss — Graz University of Technology