Edge and Hybrid Storage

Everett Dolgner
Senior Storage BD Manager, Global Platform BD - Today
Storage Specialist, Solutions Architect EMEA – Starting 2 July 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complete Set of Data Building Blocks

Data movement Data security

AWS Storage and management
Gateway Family
Amazon Kinesis
Video Streams Amazon Macie
Amazon Kinesis AWS QuickSight
Data Streams
Online

Amazon Amazon
Amazon EFS File Sync AWS Lambda
EBS EFS
Amazon S3 AWS CloudFormation
Transfer Acceleration
AWS CloudTrail
AWS Direct Connect

Amazon Amazon
AWS CloudWatch
Storage Partners
S3 Glacier
AWS IAM
Offline

AWS Snow family AWS KMS

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage Gateway Hybrid Storage Solutions
Use standard storage protocols to access AWS storage

Amazon S3 & S3 –
Files
infrequent access

On-premises

AWS Cloud
Volumes Amazon Glacier

Amazon EBS
Tapes AWS Storage snapshots
Gateway

AWS Amazon AWS Identity and Access AWS Key Management

CloudTrail CloudWatch Management (IAM) Service (KMS)

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A Few of Our Customers

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway

Files
AWS Storage Gateway Amazon S3 Standard &
(NFS v3 / v4) Infrequent Access
HTTPS
Volumes Amazon EBS
(iSCSI) snapshots

Tapes Amazon Glacier

(iSCSI VTL) Amazon EC2

Service uses an appliance to provide Data stored in Amazon S3 with

• Protocol conversion and device emulation • Native S3 management (File)
• Caching (read-through / write-back) • Snapshots to EBS (Volume)
• Optimized data transfer • Archive to Amazon Glacier (Tape)

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Tape Gateway

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape Gateway
Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management
Customer Premises Region
Tape Gateway Appliance (VM)

iSCSI HTTPS
Local
Cache Virtual Tapes
Backup Tape Archived Tapes in
Media Changer stored in Amazon Glacier
Server Gateway Amazon S3
Tape Drive

• Easy cloud adoption: Virtual tape storage in S3 & Glacier accessed via gateway
• Secure & efficient: Data encrypted and compressed in-transit and at-rest
• Scalable: Unlimited archive capacity; up to 1 PB total tape storage per gateway
• Dependable: 3-5-hour tape retrieval from Amazon Glacier with 99.999999999% data durability
• Low-cost: Simple licensing, predictable storage costs, and reduced management time

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Drop-in replacement for tape backups
• Works with leading common backup applications
• Emulates a tape library
• Supports existing backup workflows

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Backup, archive, and disaster recovery
Cost-effective storage in AWS with local or cloud restore

“Tapes are a headache, prone with hardware

failures, offsite storage costs, and constant
maintenance needs. Storage Gateway
provided the most cost-effective and simple
alternative. We even got disaster recovery by
using a bi-coastal data center.”

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Your tape archive today
Customer Premises

Backup
Server

Backup
Disk
Appliance
Tape
Library

Offsite Vault

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Move your tape archive
Customer Premises
Region
Backup
Tape Gateway VM
Server

iSCSI HTTPS

Virtual Tapes Archived Tapes

Media Changer Storage stored in stored in
Tape Drive Gateway Amazon S3 Amazon Glacier
Service
Backup
Disk
Appliance
Tape
• Deploy Storage Gateway VTL as an archive target
Library • Old archives can expire, or be recalled and written to SGW
• Decommission Tape Library when all tapes are expired
Offsite Vault

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway
• Cloud-backed block storage that can be presented on-premises
• Tier snapshots or whole volumes to the cloud to reduce SAN/NAS mgt.
• Flexible recoveries in-cloud or on-prem. with snapshots and clones
• Common use cases: Tiered storage for Windows file servers, backup and
restore, disaster recovery, data migrations

Customer Premises Region

iSCSI HTTPS

Application Storage Volume Amazon EBS

Volume
Server Gateway Stored in snapshots
Gateway
appliance Service Amazon S3
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway – Stored Mode
Using Stored Volumes a full copy of the volume is stored on-premises and
in the cloud

Customer Premises Region

iSCSI HTTPS

Application Storage Volume Amazon EBS

Volume Stored in
Server Gateway snapshots
Gateway Amazon S3
appliance Service

100% of volume stored

in AWS & on-prem
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Volume Gateway – Cached Mode
Using Cached Volumes a full copy of the volume is stored in AWS and a
portion (hot data) is stored on-premises

Customer Premises Region

iSCSI HTTPS

Application Storage Volume Amazon EBS

Server Gateway Stored in snapshots
Volume Gateway VM Service Amazon S3

virtual volume cache frequent accessed 100% of volume stored

blocks (reduce onsite storage) in AWS & on-prem
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High availability Windows Server storage
Local HA with remote DR
Data Center us-west-1
us-west-1

iSCSI HTTPS
us-east-1
SMB

iSCSI HTTPS
Windows
Clients Storage
Microsoft DFS Storage Gateway Service
Gateway VM

us-east-1

iSCSI HTTPS

Microsoft DFS Storage Gateway Service

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Flexible storage
Easily expand storage capacity on-demand

“Storage Gateway allows us to store backups offsite,

create online archives of big files, and easily expand
our file servers. With cached volumes, we have the
most used files local, and all of the files are stored safely
in AWS. We use Storage Gateway whenever we need
a quick way to have big storage available.”

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway + EBS Snapshots

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway – EBS Snapshots
Snapshots are point-in-time backups of a volume
Created on-demand or on a configurable schedule
Can be used to create an EBS or Storage Gateway volume

Customer Premises Region

iSCSI HTTPS

Application Storage Volume

Volume
Server Gateway Stored in Amazon EBS
Gateway
Service Amazon S3 snapshots
appliance

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recovery with Volume Gateway

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recovery with EBS Snapshots
Customer Premises Region

Storage Gateway Service

volume

A B C D volume
iSCSI A B C D

A B C
App A B C volume
Storage Gateway
Server
appliance

A B EC2
A B C D CD
EBS
snap1 snap2 snap3 AZ

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 File Gateway

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway
Store files as objects in Amazon S3; Access via S3 API or file protocols
Customer Premises

NFS
v3 / v4.1 HTTPS

Application Storage Gateway Objects in your Glacier

Server appliance S3 bucket

• Storage in your S3 buckets with 1:1 mapping from files-to-objects

• File metadata stored in object metadata
• Full control over how your data is accessed and stored
• Use cases: Hybrid cloud data processing (Big Data/Data Lakes), data
distribution, e.g. for content or media, backup & recovery, etc.

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway
One-to-one mapping from files-to-objects

NFS client Objects in

your S3 bucket
File Gateway
appliance

nfs://{IP}/{share name}/{folder1}/{file1} s3://{bucket name}/{folder1}/{file1}

nfs://{IP}/{share name}/{folder1}/{file2} s3://{bucket name}/{folder1}/{file2}
nfs://{IP}/{share name}/{folder1}/{file3} s3://{bucket name}/{folder1}/{file3}
nfs://{IP}/{share name}/{folder1}/{file4} s3://{bucket name}/{folder1}/{file4}

nfs://{IP}/{share name}/{folder2}/{file1} s3://{bucket name}/{folder2}/{file1}

nfs://{IP}/{share name}/{folder2}/{file2} s3://{bucket name}/{folder2}/{file2}

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway
Metadata and workload automation

File Gateway VM

Amazon
Metadata Rekognition
and object Amazon
inventory Athena
File
share AWS
Lambda
NFS client Data Objects in your
cache Amazon
S3 bucket EMR Amazon
QuickSight

NotifyWhenUploaded Amazon
API CloudWatch
Events
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway
Metadata and workload automation

File Gateway VM
ls /foo/*
Metadata
and object
inventory
File
share
NFS client Data Objects in your
cache S3 bucket

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway
Metadata and workload automation

File Gateway VM

Metadata
and object
inventory
File
share
NFS client Data Objects in your
cache S3 bucket

RefreshCache Amazon
API CloudWatch
Events
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content distribution with File Gateway
New York regional office
Los Angeles headquarters

File Gateway NFS client

VM (read-only)

NFS client File Gateway Amazon S3

(writer) VM 2 bucket
1
3 Miami regional office

Request
upload Amazon
File Gateway NFS client
notification CloudWatch (read-only)
VM

1 Client writes files and calls NotifyWhenUploaded API… 2 When files are uploaded, the gateway sends a
CloudWatch Event…

3 Event triggers RefreshCache API on gateways in the 4 When RefreshCache completes, the gateway
remote offices sends a CoudWatch Event notification
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enabling cloud workloads
Move data to AWS storage for big data, cloud bursting, or migration

“Storage Gateway has the promise to transform the way we move

data into the cloud. The NFS interface lets us easily integrate data
files from analytical instruments, and the transparent S3 storage
lets us easily connect our cloud-based applications and leverage the
powerful storage capabilities of S3.
With Storage Gateway, we can now unleash the full power of AWS
on our instrument data.”

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway

• Seamlessly connect on-premises applications with AWS Cloud storage; No

changes to existing applications

• Local cache for low-latency access to data stored durably in AWS

• Supports use cases throughout your cloud journey:

Backup & DR, tiered storage, hybrid processing workflows, migration

• Login to your AWS account, download the VM (or launch an EC2

instance), deploy new on-premises storage in minutes

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.