www.iaik.tugraz.

at

Hello from the Other Side:

SSH over Robust Cache
Covert Channels in the Cloud
Clémentine Maurice, Manuel Weber, Michael Schwarz, Lukas Giner,
Daniel Gruss, Carlo Alberto Boano, Stefan Mangard, Kay Römer
Graz University of Technology
February ���� — NDSS ����

� / ��
 www.iaik.tugraz.at

Outline

cache covert channels

how do we get a covert channel working in the cloud?

how do we get a covert channel working in a noisy environment?

what are the applications of such covert channel?

� / ��
 www.iaik.tugraz.at

CPU cache

main memory is slow compared to the CPU

� / ��
 www.iaik.tugraz.at

CPU cache

main memory is slow compared to the CPU

caches buffer frequently used data

� / ��
 www.iaik.tugraz.at

CPU cache

main memory is slow compared to the CPU

caches buffer frequently used data

every data access goes through the cache

� / ��
 www.iaik.tugraz.at

CPU cache

main memory is slow compared to the CPU

caches buffer frequently used data

every data access goes through the cache

caches are transparent to the OS and the software

� / ��
 www.iaik.tugraz.at

Caches on Intel CPUs

L� and L� are private

core � core � core � core �

L� L� L� L�
ring
L� L� L� L� bus

LLC LLC LLC LLC

slice � slice � slice � slice �

� / ��
 www.iaik.tugraz.at

Caches on Intel CPUs

L� and L� are private

core � core � core � core �
last-level cache
L� L� L� L�
ring
L� L� L� L� bus

LLC LLC LLC LLC

slice � slice � slice � slice �

� / ��
 www.iaik.tugraz.at

Caches on Intel CPUs

L� and L� are private

core � core � core � core �
last-level cache
L� L� L� L� divided in slices
ring
L� L� L� L� bus

LLC LLC LLC LLC

slice � slice � slice � slice �

� / ��
 www.iaik.tugraz.at

Caches on Intel CPUs

L� and L� are private

core � core � core � core �
last-level cache
L� L� L� L� divided in slices
ring
L� L� L� L� bus shared across cores

LLC LLC LLC LLC

slice � slice � slice � slice �

� / ��
 www.iaik.tugraz.at

Caches on Intel CPUs

L� and L� are private

core � core � core � core �
last-level cache
L� L� L� L� divided in slices
ring
L� L� L� L� bus shared across cores
inclusive
LLC LLC LLC LLC
slice � slice � slice � slice �

� / ��
 www.iaik.tugraz.at

Caches on Intel CPUs

L� and L� are private

core � core � core � core �
last-level cache
L� L� L� L� divided in slices
ring
L� L� L� L� bus shared across cores
inclusive
LLC
slice �
LLC
slice �
LLC
slice �
LLC
slice �
hash function maps a
physical address to a slice

� / ��
 www.iaik.tugraz.at

Set-associative caches
� �� �� �� �� ��

Address Index Offset

Cache

� / ��
 www.iaik.tugraz.at

Set-associative caches
� �� �� �� �� ��

Address Index Offset

Cache set

Cache

Data loaded in a speci�c set depending on its address

� / ��
 www.iaik.tugraz.at

Set-associative caches
� �� �� �� �� ��

Address Index Offset

way � way �

Cache set

Cache

Data loaded in a speci�c set depending on its address

Several ways per set

� / ��
 www.iaik.tugraz.at

Set-associative caches
� �� �� �� �� ��

Address Index Offset

way � way �

Cache set

Cache line

Cache

Data loaded in a speci�c set depending on its address

Several ways per set

Cache line loaded in a speci�c way depending on the replacement policy

� / ��
 www.iaik.tugraz.at

Timing differences

cache hits cache misses

Number of accesses

���

���

���
�� ��� ��� ��� ��� ��� ��� ���
Access time [CPU cycles]

� / ��
 www.iaik.tugraz.at

Cache-based covert channels

cache attacks ! exploit timing differences of memory accesses

� / ��
 www.iaik.tugraz.at

Cache-based covert channels

cache attacks ! exploit timing differences of memory accesses

covert channel: two processes communicating with each other

not allowed to do so, e.g., across VMs

� / ��
 www.iaik.tugraz.at

Cache-based covert channels

cache attacks ! exploit timing differences of memory accesses

covert channel: two processes communicating with each other

not allowed to do so, e.g., across VMs

literature: stops working with noise on the machine

� / ��
 www.iaik.tugraz.at

Cache-based covert channels

cache attacks ! exploit timing differences of memory accesses

covert channel: two processes communicating with each other

not allowed to do so, e.g., across VMs

literature: stops working with noise on the machine

solution? “Just use error-correcting codes”

� / ��
 www.iaik.tugraz.at

Prime�Probe

attacker knows which cache set the victim accessed, not the content

� / ��
 www.iaik.tugraz.at

Prime�Probe

attacker knows which cache set the victim accessed, not the content

works across CPU cores as the last-level cache is shared

� / ��
 www.iaik.tugraz.at

Prime�Probe

attacker knows which cache set the victim accessed, not the content

works across CPU cores as the last-level cache is shared

does not need shared memory, e.g., memory de-deduplication

� / ��
 www.iaik.tugraz.at

Prime�Probe

attacker knows which cache set the victim accessed, not the content

works across CPU cores as the last-level cache is shared

does not need shared memory, e.g., memory de-deduplication

! works across VM in the cloud, e.g., on Amazon EC�

� / ��
 www.iaik.tugraz.at

Prime�Probe

Victim address space Cache Attacker address space

� / ��
 www.iaik.tugraz.at

Prime�Probe

Victim address space Cache Attacker address space

Step �: Attacker primes, i.e., �lls, the cache (no shared memory)

� / ��
 www.iaik.tugraz.at

Prime�Probe

loads data

Victim address space Cache Attacker address space

Step �: Attacker primes, i.e., �lls, the cache (no shared memory)

Step �: Victim evicts cache lines while running

� / ��
 www.iaik.tugraz.at

Prime�Probe

loads data

Victim address space Cache Attacker address space

Step �: Attacker primes, i.e., �lls, the cache (no shared memory)

Step �: Victim evicts cache lines while running

� / ��
 www.iaik.tugraz.at

Prime�Probe

cess
fast ac

Victim address space Cache Attacker address space

Step �: Attacker primes, i.e., �lls, the cache (no shared memory)

Step �: Victim evicts cache lines while running

Step �: Attacker probes data to determine if set has been accessed

� / ��
 www.iaik.tugraz.at

Prime�Probe

cess
slow ac

Victim address space Cache Attacker address space

Step �: Attacker primes, i.e., �lls, the cache (no shared memory)

Step �: Victim evicts cache lines while running

Step �: Attacker probes data to determine if set has been accessed

� / ��
 www.iaik.tugraz.at

Why can’t we just use error correcting codes?

Sender � � � � � �

Receiver � � � � � �

(a) Transmission without errors

�� / ��
 www.iaik.tugraz.at

Why can’t we just use error correcting codes?

Sender � � � � � � Sender � � � � � �

Receiver � � � � � � Receiver � � � � � �

(a) Transmission without errors (b) Noise: substitution error

�� / ��
 www.iaik.tugraz.at

Why can’t we just use error correcting codes?

Sender � � � � � � Sender � � � � � �

Receiver � � � � � � Receiver � � � � � �

(a) Transmission without errors (b) Noise: substitution error

Sender � � � � � �

Receiver � � � � � � � � �

(c) Sender descheduled: insertions

�� / ��
 www.iaik.tugraz.at

Why can’t we just use error correcting codes?

Sender � � � � � � Sender � � � � � �

Receiver � � � � � � Receiver � � � � � �

(a) Transmission without errors (b) Noise: substitution error

Sender � � � � � � Sender � � � � � �

Receiver � � � � � � � � � Receiver � � �

(c) Sender descheduled: insertions (d) Receiver descheduled: deletions

�� / ��
 www.iaik.tugraz.at

Our robust covert channel

physical layer:

transmits words as a sequence of ‘�’s and ‘�’s

deals with synchronization errors

data-link layer:

divides data to transmit into packets

corrects the remaining errors

�� / ��
 www.iaik.tugraz.at

Physical layer: Sending ‘�’s and ‘�’s

sender and receiver agree on one set

�� / ��
 www.iaik.tugraz.at

Physical layer: Sending ‘�’s and ‘�’s

sender and receiver agree on one set

receiver probes the set continuously

�� / ��
 www.iaik.tugraz.at

Physical layer: Sending ‘�’s and ‘�’s

sender and receiver agree on one set

receiver probes the set continuously

sender transmits ’�’ doing nothing

! lines of the receiver still in cache ! fast access

�� / ��
 www.iaik.tugraz.at

Physical layer: Sending ‘�’s and ‘�’s

sender and receiver agree on one set

receiver probes the set continuously

sender transmits ’�’ doing nothing

! lines of the receiver still in cache ! fast access

sender transmits ’�’ accessing addresses in the set

! evicts lines of the receiver ! slow access

�� / ��
 www.iaik.tugraz.at

Eviction set generation

need a set of addresses in the same cache set and same slice

�� / ��
 www.iaik.tugraz.at

Eviction set generation

need a set of addresses in the same cache set and same slice
problem: slice number depends on all bits of the physical address

�� / ��
 www.iaik.tugraz.at

Eviction set generation

need a set of addresses in the same cache set and same slice
problem: slice number depends on all bits of the physical address

cache set cache line

cache tag index offset

physical address xxxx

�MB page offset

we can build a set of addresses in the same cache set and same slice

�� / ��
 www.iaik.tugraz.at

Eviction set generation

need a set of addresses in the same cache set and same slice
problem: slice number depends on all bits of the physical address

cache set cache line

cache tag index offset

physical address xxxx

�MB page offset

we can build a set of addresses in the same cache set and same slice
without knowing which slice
�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets
��
��

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pri
me Cache Sets
��
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets

e
prim
��
�� S S S S S S S S
R R R R R R R R

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pro
be Cache Sets
��
�� S S S S S S S S
R R R R R R R R

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets

e
prob
��
�� S S S S S S S S
R R R R R R R R

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pri
me Cache Sets
��
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets
��

e
prim
�� S S S S S S S S

R R R R R R R R

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pro
be Cache Sets
��
�� S S S S S S S S

R R R R R R R R

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets
��

e
prob
�� S S S S S S S S

R R R R R R R R

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pri
me Cache Sets
��
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets
prime
�� R R R R R R R R
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pro
be Cache Sets
�� R R R R R R R R
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets
probe
�� R R R R R R R R
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pri
me Cache Sets
��
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets
��
prime
�� R R R R R R R R

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� pro
be Cache Sets
��
�� S S S S S S S S

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
��
�� Cache Sets
��
probe
�� R R R R R R R R ��

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
�� 3
�� Cache Sets
��
�� ��

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
�� 3
�� Cache Sets
��
�� ��

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
�� 3
�� repeat!
��
�� ��

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
�� 3
�� 3 repeat!
�� ��
�� ��

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
�� 3
�� 3 repeat! ��
�� 3 ��
�� ��

�� / ��
 www.iaik.tugraz.at

Jamming agreement

sender receiver
eviction sets eviction sets
�� 3 ��
�� 3 repeat! ��
�� 3 ��
�� 3 ��

�� / ��
 www.iaik.tugraz.at

Sending the �rst image

�� / ��
 www.iaik.tugraz.at

Handling synchronization errors

Physical layer word Data

�� bits

�� / ��
 www.iaik.tugraz.at

Handling synchronization errors

deletion errors: request-to-send scheme that also serves as ack

�-bit sequence number
request: encoded sequence number (� bits)

Physical layer word Data SQN

�� bits � bits

�� / ��
 www.iaik.tugraz.at

Handling synchronization errors

deletion errors: request-to-send scheme that also serves as ack

�-bit sequence number
request: encoded sequence number (� bits)
’�’-insertion errors: error detection code ! Berger codes
appending the number of ’�’s in the word to itself
! property: a word cannot consist solely of ’�’s

Physical layer word Data SQN EDC

�� bits � bits � bits

�� / ��
 www.iaik.tugraz.at

Synchronization (before)

�� / ��
 www.iaik.tugraz.at

Synchronization (after)

�� / ��
 www.iaik.tugraz.at

Synchronization (after)

�� / ��
 www.iaik.tugraz.at

Synchronization (after)

�� / ��
 www.iaik.tugraz.at

Data-link layer: Error correction

Reed-Solomon codes to correct the remaining errors

�� / ��
 www.iaik.tugraz.at

Data-link layer: Error correction

Reed-Solomon codes to correct the remaining errors

RS word size � physical layer word size � �� bits
packet size � ��� � � ���� RS words
��� error-correcting code: ��� parity and ���� data RS words
���� RS-words ��� RS-words

Data-link layer packet Data Parity

Physical layer word Data SQN EDC

�� bits � bits � bits

�� / ��
 www.iaik.tugraz.at

Error correction (after)

�� / ��
 www.iaik.tugraz.at

Evaluation

Environment Bit rate Error rate Noise

Native ��.�� KBps �.��� –

�� / ��
 www.iaik.tugraz.at

Evaluation

Environment Bit rate Error rate Noise

Native ��.�� KBps �.��� –
Native ��.�� KBps �.��� stress -m 1

�� / ��
 www.iaik.tugraz.at

Evaluation

Environment Bit rate Error rate Noise

Native ��.�� KBps �.��� –
Native ��.�� KBps �.��� stress -m 1
Amazon EC� ��.�� KBps �.��� –

�� / ��
 www.iaik.tugraz.at

Evaluation

Environment Bit rate Error rate Noise

Native ��.�� KBps �.��� –
Native ��.�� KBps �.��� stress -m 1
Amazon EC� ��.�� KBps �.��� –
Amazon EC� ��.�� KBps �.��� web server serving �les on sender VM
Amazon EC� ��.�� KBps �.��� stress -m 2 on sender VM
Amazon EC� ��.�� KBps �.��� stress -m 1 on receiver VM
Amazon EC� ��.�� KBps �.��� web server on all � VMs, stress -m
4 on �rd VM, stress -m 1 on sender
and receiver VMs
Amazon EC� ��.�� KBps �.��� stress -m 8 on third VM

�� / ��
 www.iaik.tugraz.at

Building an SSH connection

VM � VM �
TCP Client TCP Server
(e.g. ssh) (e.g. sshd)
Socket Socket
TCP$File TCP$File
File System File System
Covert Channel Covert Channel
Hypervisor
Prime�Probe Prime�Probe
Last Level Cache (LLC)

�� / ��
 www.iaik.tugraz.at

SSH evaluation
Between two instances on Amazon EC�

Noise Connection
No noise 3
stress -m 8 on third VM 3
Web server on third VM 3
Web server on SSH server VM 3
Web server on all VMs 3
stress -m 1 on server side unstable

�� / ��
 www.iaik.tugraz.at

SSH evaluation
Between two instances on Amazon EC�

Noise Connection
No noise 3
stress -m 8 on third VM 3
Web server on third VM 3
Web server on SSH server VM 3
Web server on all VMs 3
stress -m 1 on server side unstable

Telnet also works with occasional corrupted bytes with stress -m 1

�� / ��
 www.iaik.tugraz.at

Conclusion

cache covert channels are practical

�� / ��
 www.iaik.tugraz.at

Conclusion

cache covert channels are practical

even in the cloud, even in presence of extraordinary noise

�� / ��
 www.iaik.tugraz.at

Conclusion

cache covert channels are practical

even in the cloud, even in presence of extraordinary noise

our robust covert channel supports an SSH connection

�� / ��
 www.iaik.tugraz.at

Conclusion

cache covert channels are practical

even in the cloud, even in presence of extraordinary noise

our robust covert channel supports an SSH connection

we extended Amazon’s product portfolio :)

�� / ��
 www.iaik.tugraz.at

Conclusion

cache covert channels are practical

even in the cloud, even in presence of extraordinary noise

our robust covert channel supports an SSH connection

we extended Amazon’s product portfolio :)

�� / ��
 www.iaik.tugraz.at

Conclusion

cache covert channels are practical

even in the cloud, even in presence of extraordinary noise

our robust covert channel supports an SSH connection

we extended Amazon’s product portfolio :)

�� / ��
 www.iaik.tugraz.at

Hello from the Other Side:

SSH over Robust Cache
Covert Channels in the Cloud
Clémentine Maurice, Manuel Weber, Michael Schwarz, Lukas Giner,
Daniel Gruss, Carlo Alberto Boano, Stefan Mangard, Kay Römer
Graz University of Technology
February ���� — NDSS ����

�� / ��