When Irresistible Forces Attack

Security In The Cloud

Dan Kaminsky
Director of Penetration Testing
IOActive, Inc.
Introduction
• Hi! I’m Dan Kaminsky
– Director of Penetration Testing for IOActive
– Well known security researcher
• Found major flaw in DNS last year
• Coordinated massive multivendor patch event across IT
last year
• Concerned with all issues related to scale
– DNS was such a problem precisely because it’s the only
solution for cross-organizational trust, and people were
using it whether or not it was secure for that purpose
• So, this Cloud thing is getting pretty popular
– Why?
Deploying Software Sucks
• There’s really no better way to say it.
– Procuring the hardware is a pain
– Integrating with existing systems is a pain
– Monitoring operations is a pain
– Scaling to load is a pain
– Securing the data inside is a real pain
• Christofer Hoff has an amazing blog on the subject
• http://rationalsecurity.typepad.com
Can’t Someone Else Do It?
Basic Cloud Taxonomy
• Infrastructure As A Service
– “Deploying and scaling hardware is hard. We’ll
do that for you.”
• Platform As A Service
– “Building software that will smoothly scale across
a cloud is hard. We’ll make that easier for you.”
• Software As A Service
– “Deploying software is hard, especially if you
need to interact with people outside your
company. Why don’t all of you come to our
website?”
Depends What “It” Is
Another Perspective: The Starcraft Model
• All IT projects require a certain amount of the
following resources
– CPU
– Bandwidth
– Storage
• Possibly fast (RAM)
• Possibly slow (Disk)
• Possibly somewhat fast (SSD)
• Possibly somewhat slow (Tape)
– “Glue”
• Everything that makes the above invisible / manageable
• IaaS -> PaaS -> SaaS == More Glue, Less Direct
Management Of Resources
How Security Plays In
• Security, in general: When bad guys extract value
from your resources
– Information Disclosure: Read data they’re not supposed to
– Elevation of Privilege: Modify data they’re not supposed to
– Denial of Service: Prevent legitimate access to resources
• Can also be economic, i.e. make it too expensive to host
services
• Security, in the cloud: When bad guys extract value
from your resources…hosted in the cloud (or a cloud)
– Is this any different?
– Yes and no.
What Hasn’t Changed
• It’s the same data at risk
• It’s the same applications (generally)
protecting access to the data
– Written in the same languages, with the same
classes of bugs
• Administrative scope remains larger than one
individual piece of hardware
– In theory, individual servers are independent
fiefdoms
• Breaking into one doesn’t imply access to another
– In reality, we abandoned that years ago, because
we couldn’t patch machines fast enough that way
Security Has Demanded Cloud-Scale Central
Manageability For Years
• This is what Enterprise Patch Management means
– All systems are asset-tracked
– When a critical vulnerability is identified, the patch is
pushed out as quickly as possible
• Days or weeks, not months or years
– Shared administrative accounts on all systems, specifically
to facilitate this
• Security has been using proto-cloud functionality for
ages
– Cloud Computing simply represents other major pain
areas – deployment, performance, cost control, reliability,
manageability – getting into the inter-server control plane
• It’s almost like there are other important things in life than
security 
What has changed (from a security
perspective)
• Other People’s Data
– Other people managing your data
– Other people’s data on your hardware
• There are private cloud concepts that alter the above in
significant ways
– We’ll discuss them later in this talk
– We are in a temporary period of heavy development where fully
public clouds are sucking up most of the oxygen
• Other People’s Data is pretty much the big shift in the cloud
– We’ve long had the ability to use other organization’s resources,
but for a long time, our use of these resources was limited
because there was no operational advantage for most of our
internal operations
• We’ve been using external resources to host external public content,
through what increasingly looks like a Cloud without us knowing/caring
Recognition of the
three clouds in use in
many large scale
websites.

http://tinyurl.com/9ce
3e4
Other People Managing Your Data
• Even when there are third parties managing systems,
they tend to be either very tightly scoped appliances
(SPAM filters) or trusted contractors managing internal
resources
• A surprising number of companies even disable
Windows Update
• IaaS and some PaaS systems suggest moving your
internal applications to an amorphous, external cloud
– Where is your data?
– What jurisdiction controls legal access to the data?
– Who can get physical access for illegal access to the
data?
– What happens when an intrusion is detected?
Uncomfortable Fact #1: Cloud Applications Do Not Have A
Strong Track Record For Delivering Intrusion Data

• Gmail is the only webmail provider to offer “Last

Account Activity” feature, allowing some degree of
intrusion detection
– This is a very recent feature
– Email account passwords are (surprisingly) among
the most hunted pieces of PII
• All cloud providers have (necessarily!) a
management layer where individual customers
must not be allowed to see what happens
– Cloud providers are not necessarily incentivized to
disclose vulnerabilities detected at this layer
– This can of course be negotiated contractually
Security Management Gets A Little Tricky
• EC2 giveth: “Attacks such as ARP cache
poisoning do not work within EC2.”
• EC2 taketh away: “Port scans by Amazon
EC2 customers are a violation of the
Amazon EC2 Acceptable Use Policy”
– You’re not allowed to sweep your own servers
– fundamental difference between your
infrastructure and their infrastructure
Security Appliances Become Very Very Tricky
• Where do you put the firewall, when your code is
highly mobile?
• How do you create a perimeter – even a virtual
perimeter – around your applications?
– Most authentication in real world applications comes from
the ability to send packets to, or not to send packets to, a
given host
• This is grossly insufficient, but it’s what we have
– Web Application Firewalls proving more scalable (and
more popular) then difficult / expensive code audits on web
apps
– Surprisingly difficult to migrate appliance model to clouds
• Code has to follow images around
• Performance is a problem
• See Hoff’s Four Horsemen of the Virtualization Apocalypse talk
The SaaS Quirk: Other People’s Data Is The
Whole Point
• There are actually two ways to scale systems across
organizational boundaries
– Acquire credentials / encryption keys through a mutually
trusted provider, and share data when asked by the trusted
identity (SSL CA’s, DNSSEC)
– Allow a mutually trusted provider access to all your data,
and trust it to share out the appropriate subset (MySpace,
BaseCamp Salesforce.Com, All SaaS Cloud Apps)
• It’s actually much easier to build systems using the
second model
– DNSSEC will eventually make the first model scale better
– SSL CA’s work “OK” but not perfectly
• Put another way: There are many systems that are
legendarily difficult to build in a distributed fashion, but
(relatively) trivial to build centralized
Uncomfortable Fact #2: The more centralized
the data, the more valuable the break
• Breaking into one user’s email is valuable
• Breaking into one company’s email is very
valuable
• Breaking into every company’s email is off
the charts
• Breaking into every company’s email, when
there’s actually no incentive for whoever
discovers the attack notifying the owner of
the email account…
– Yikes.
Uncomfortable Fact #3: Most intrusions probably go
undetected, even today

• Attacks have gotten much more stealthy

• No longer finding the attacks when the
network happens to go down
• Error rates with IDS (Intrusion Detection
Systems) have always been too high
Other People’s Data On Your Hardware Is
Also A Problem
• There are many sorts of Cloud concepts, but one
unifying theme is – you don’t build software to a
host, rather you build it to the cloud and the cloud
puts your code/data where it needs to be
– Clouds tend to desire elasticity – the ability to
increase or decrease resources according to load
• On public clouds, your data is shuffled with the
data of others by the management layer
– Two classes of attack
• 1) Break the management layer -- force it to bring you other
people’s data
• 2) Escape the management layer – be somewhere you’re not
supposed to be
There’s No Good Time For A VM Escape
• Before: An attacker’s code might run on
hardware before yours does, and might
leave something behind
• During: An attacker’s code might run on
hardware while yours is, and might cross
over and grab stuff
• After: An attacker’s code might run on
hardware after yours does, and recover
whatever you left behind
Uncomfortable Fact #4: There are many
layers to virtualize at, and they’re all buggy.
• Google App Engine: Virtualized Python,
vulnerable to bugs in Python interpreter
• Microsoft IIS: Virtualized on user
accounts w/ >Standard User capabilities,
vulnerable to “Token Kidnapping” attacks
• Beneath-The-Kernel Virtualization is its
own subject entirely
Virtualization Is Hard
• Multitenancy: The state of having
separate execution domains, with an
enforced and stable security boundary
between them
• Virtualization solutions need to create
multitenancy without their children
knowing
– Excluding Xen – we’ll return to it
• This is hard, because…
Uncomfortable Fact #5: There’s no such
thing as a hardware bug.
• Hardware is much more expensive to fix than software
• So hardware is just not fixed; software has to work around it
• That means software that thinks it’s talking to hardware has to do
very weird things
• Suppose you’re trying to make software, thinking its talking to one
piece of hardware, talk to hardware, thinking it’s talking to totally
trusted software (a kernel or a driver)
• To make this safe, you must either:
– A) Emulate, meaning you have to parse whatever weird stuff comes
from either the software above or the hardware below, and do so without
falling over
– B) Pass through, meaning you have to hope that the hardware doesn’t
have any buggy/undefined behavior that will violate your security model
• Ahahahahhahahahahahha
• Oh, and you’re under massive pressure from customers to do this because it’s
faster
Virtualization Not Actually Required
• Xen’s Paravirtualization creates idealized
hardware interfaces, and asks guest OS’s to
speak in a clear and concise way to those
interfaces
– Xen, in theory, is thus much easier to secure, while
remaining performant
– Still vulnerable to processor bugs, of course
• Can also speak directly to hardware, and have the
management layer populate the bare metal
– LoudCloud was doing this years ago
– Amusing history: LoudCloud offered me my first full
time job in 2001 – scared me so much I went back to
school 
Uncomfortable Fact #6: Virtualization Not
Actually Required For Cloud Compromise
• Not using VM’s only stops the “During” subclass of attack
• Before and After still exist, for any solution that wants to
support elasticity
• In theory, you can return a node to a “known state” after it
runs arbitrary code
• In reality, there’s persistent storage in each of the various
hardware components (Motherboard, Network Card, Hard
Drive, etc) and they tend to require cooperation from whatever
code is already on the device to reflash them
– CPU is the Central processing unit, not the only processing unit
– Not even the most powerful unit – that’s often the GPU!
– Any processor in the system can take over kernel memory and
alter a system in arbitrary ways
• Even the hard drive – especially, it has DMA (Direct Memory Access) for
obvious reasons
Regarding Virtual Appliances
• The whole reason to have a firewall is –
whatever happens on the outside, the path
in is tightly controlled
• What happens when the firewall has an
attack surface not via TCP/IP, but through
the VM or the Cloud Management layer?
So should you avoid Cloud Computing?
• OF COURSE NOT! This is a way better way to
deploy software at scale
– Model #1: Ship software, install it on customer boxes.
• Heavy variability, heavy support costs
– Model #2: Ship appliances, install them on customer
networks
• Much less variability, still need to integrate with customer
systems, actual physical hardware has to move so it’s
manual labor to scale
– Model #3: Ship virtual appliances and services, let
customers access them from the cloud
• Much, much less variability, can scale up arbitrarily
• Again, genuine messiness with virtual security appliances,
which have to follow nodes around and can never offer quite
the same guarantees
Towards A More Secure Cloud
• Have to recognize the risks in order to
mitigate, and in the long term eliminate
them
– Hardware for multitenancy is spreading
throughout the PC architecture
• Intel and AMD have their processors on board
today
• Rest of the PC coming soon (I hope)
What Of Private Clouds?
• There are three classes of private cloud
– Fully private, where they’re your own servers in your
own facilities but they’re managed (as per
IaaS/PaaS/SaaS) using cloud models
• No sharing whatsoever, no elasticity outside your own assets
– Outsourced Private, where they’re still your own
servers, but they’re in foreign data centers
• No sharing, but you have to worry about management plane
attacks
– Nonsecure, where it doesn’t matter that other people
can access or change the data, it’s encrypted,
authenticated, or at least public already
• S3 is being looked at a lot for this
Costs
• The actual cost of delivering an enhanced
“private cloud” solution is not wildly higher
than building the cloud out in the first place
– Expect to see every cloud vendor with a private
offering
• “Outsourced Cloud” makes a lot of sense
– This is basically paying more for a slightly less
flexible solution, that still beats the status quo
• If you’re large enough, you’re already using a massive
enough dedicated set of resources that you might as
well call it private
• Still need to worry about intrusion detection
incentives across organizational boundaries!
Conclusions
• Cloud computing is a fascinating realm, that
really does make it easier to deploy software
and increase productivity
• There are some engineering (and non-
engineering!) realities that make security
somewhat difficult to deliver in a cloud
– It can be expected that, over the next few years,
these problems will be addressed
• Private clouds ameliorate many of the
security threats, while maintaining the value
proposition of the cloud