11cloud Spectator Report - Final PDF

Cloud IaaS Performance & Price-Performance
Comparing Linux Compute Performance of 1&1, Amazon AWS, Aruba Cloud,

CloudSigma, and Microsoft Azure
Prepared for 1&1 on Behalf of SolidFire

Commercial Report
Published on 6/2015
Cloud Spectator Performance Report
Contents
1. Introduction
2
Why Performance Matters
2. Executive Sum m ary
4
Findings
3. M ethodology
6
Process
Tests Used
Provider Data Center/Region Locations
VM Configurations and Pricing
Understanding Performance Results
10
Understanding The CloudSpecs Score (Price-Performance)
10
Key Considerations
11
4. Detailed Perform ance Findings
12
Processor & Memory Bandwidth
12
Disk IOPS: Sequential and Random Operations
14
Internal Network
17
5. Detailed Price-Perform ance Findings
19
Processor & Memory Bandwidth
19
Disk IOPS: Sequential and Random Operations
21
Internal Network
24
6. Conclusion
26
7. About
26
1
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Introduction
In an effort to simulate an end-user experience regarding performance of virtual machines across various cloud providers, Cloud Spectator
ran its iterative benchmark suite for 72 hours on each of the following providers: 1&1, AWS, Aruba Cloud, Microsoft Azure, and CloudSigma.
SolidFire sponsored this study on behalf of its client, 1&1.
In most cases, much of the work was straightforward regarding provisioning and the setup process. Occasionally, vendor-side issues
occurred during the provisioning process, and the Cloud Spectator team contacted the corresponding vendors support team in order to resolve
issues such as VM provisioning errors. Three primary VMs of each size were tested on all providers for 24 hours each (72 hours total). This was
done sequentially; once one VM had run the test suite for 24 hours, that VM was terminated and a new VM was created.
This study not only examined the performance of each vendor, but also tracked performance variability for each of the three 24-hour
periods. The methodology allowed Cloud Spectator to capture performance variability both over time on the same VM as well as across different
VMs on multiple physical hosts. Some providers, such as 1&1, show strong processor and memory bandwidth performance stability for all of its VMs
throughout the course of the study. Other providers, such as AWS, exhibited controlled periods of burst followed by throttled performance on network
storage depending on the size of the storage volume. Others, such as CloudSigma, exhibited unstable performance across all resources throughout
the study, possibly due to server-side issues at the time of the study, which contributed to provisioning problems as well.
Taking performance and stability a step further, price-performance analyses are conducted to help readers understand the value ratio
between the cost of the VM and the performance output. While the performance output is limited to the data points collected in the study, by
comparing the price-performance ratio, readers can gain better insight into the overall user experience seen on these providers.
Why Performance Matters
Performance and pricing are both key considerations in the public cloud industry, together having a substantial impact on a companys annual
operating costs. Cloud users may need fewer resources on better performing services, which can lower costs. Since many users only consider price
and not price-performance, these users may be paying more because they require additional resources to achieve a desired level of performance.
While some providers try to differentiate their offerings by cutting prices, others try to differentiate by focusing on improved performance and user
experience.
Differences in performance outputs of VMs across IaaS providers
can greatly impact quality of service as well as annual operating
costs. The graph on the right illustrates an example of the
average processor performance from a sample of six Cloud
Service Providers (CSPs) as studied by Cloud Spectator. CSP 1
has a processor performance three times as high as CSP 6
(names removed), which gives CSP 1 a notable advantage in
many processor-intensive workloads. CSPs 2-5 exhibit a closer
resemblance in processor performance, but do not offer nearly as
much processing power as CSP 1 does.
2
The table below lists the 3 hardware components studied in this project, and each purpose as a function in the server.
CPU & MEMORY PERFORMANCE
STORAGE PERFORMANCE
NETWORK PERFORMANCE
highly
Because most applications and all data reside
In a cloud environment, network performance
dependent on the CPU. The CPU is responsible for
on the disk, having fast disk performance is a
is a critical piece. Scalability, in many cases,
the processing and orchestration of all applications.
key
is dependent on the availability of additional
The relationship between CPU performance and RAM
performance in many cases.
The
performance
of
all
applications
is
is also observed by examining RAM bandwidth. While
consideration
for
best
application
VMs that must maintain a strong network

backbone.
memory performance is not considered one of the key

bottlenecks in performance for many applications, a
subset of applicationsparticularly HPC and inmemory databasesis highly dependent on large
sustained memory bandwidth.
3
Executive Summary
On behalf of 1&1, SolidFire commissioned Cloud Spectator to gauge the performance of VMs on five different cloud providers European data
centers: 1&1, Amazon AWS, Aruba Cloud, CloudSigma, and Microsoft Azure. Both performance and price-performance were examined to evaluate
the value of each providers VMs. The purpose of the study was to understand, from an end-user perspective, the disparity of performance and value
(defined as price-performance) among cloud providers with similarly sized VMs. Overall, 1&1 exceled in performance and price-performance for all
component resources of the VMs tested. Its high performance rankings, combined with hourly pricing, introduces powerful, scalable cloud
infrastructure at low cost to its users.
Findings
vCPU & M em ory Perform ance Findings
For this study, Cloud Spectator evaluated vCPU and memory
vCPU & Memory Performance Key Findings:
bandwidth performance by benchmarking the VMs using Geekench
1&1s VMs achieved the highest performance across processor
3, a suite of benchmark tests that simulate tasks such as

cryptographic encoding and image processing. Testing occurred over
and memory performance in the study.

1&1s VMs achieved the highest CloudSpecs ScoreTM in the test
the course of a 72-hour testing period. Pricing was examined in
group, indicating the strongest price-performance value for
conjunction with the performance tests.
processor and memory bandwidth.

CloudSigmas processor performance varied the most. Its
coefficients of variation (CV), which is a percentage expressing the
relationship between the average and standard deviation (useful
for determining variability in performance), ranges up to 43.7%.
The virtual processors on 1&1, AWS, and Azure exhibited the most
stability throughout the study, resulting in less than 3% coefficients
of variation.
4
Storage Perform ance Findings

Storage was evaluated using the FIO disk benchmark, which tests
Storage Performance Key Findings:
the sequential read/write and random read/write operations of
1&1 VMs offered the highest disk performance among all included
storage. In this study an 8KB block size was used. Testing occurred
in this study.
over a 72-hour test period. Persistent storage (offered as block
Although AWS offers SSD technology on its block storage offering,
storage or redundant storage) was used in all storage tests. Pricing
EBS, the performance of that offering is more dependent on the
was examined in conjunction with the performance tests.
size of the block storage volume provisioned.

Despite being one of the lower-tier performers in disk IOPS, Azure
displayed the most stable disk performance throughout the study.
Out of all providers examined in the study, only AWS appeared to
provide a period of burst performance for its block storage. On the
2vCPU VMs, which have 100GB of block storage, AWS volumes
displayed a burst behavior. After the period of burst, sequential
read/write operations and random read/write operations dropped to
become 10% and 20%, respectively, of the initial IOPS achieved
during burst. This burst behavior was not seen on VMs with 400GB
and 800GB of block storage, due to the larger number of IOPS.
1&1 displayed the best price-performance value for disk IOPS. Its
high-performance SAN disk offering is designed with SolidFire
technology.
Internal Network Perform ance Findings

Internal network performance was measured as the throughput
Internal Network Performance Key Findings:
between VMs within the same data center of the cloud provider
CloudSigma VMs achieved the highest internal network
(measured using iperf and ping respectively) over the course of a 72-
throughput, although the high throughput is unstable and
hour test period. Pricing was examined in conjunction with the
fluctuates between less than 500Mbits to over 10Gbits.
performance tests.
CloudSigma displayed the best price-performance value for

internal network. Despite its large fluctuation in internal network
throughput, its median score range significantly outperformed all
other providers.
1&1 achieved the second-highest internal network CloudSpecs
ScoreTM.
Other than CloudSigma, only certain Azure VMs exceeded 1 GB/s
throughput.
5
Methodology
Cloud Spectator strives to create a transparent and detailed methodology to allow readers to understand the testing process and recreate any
studies. If any information is unclear or if you have any questions, please email the team at contact@cloudspectator.com or call +1 (617) 300-0711.
Process
1.
Three iterations of 24-hour test cycles were run for each VM on each provider for a total of 72 hours of testing per VM size. After each 24hour block, VMs were terminated before beginning another cycle of tests on newly provisioned machines.
2.
Each VM was provisioned with a Linux Ubuntu 14.04 OS by default, available from all providers. For AWS, the HVM image was used.
3.
Before each 24-hour test period, and after provisioning the VMs, system updates and upgrades were conducted via apt-get.
4.
The following dependencies were installed for testing:

a.
Git. Git was used to clone the test repository on the VM.
b.
MySQL. For automation purposes, mysql-server was installed to automate data uploads.
c.
Pip. Used to download the appropriate libraries for Python in order to run the testing. SQL Alchemy was downloaded to interact
with MySQL and upload data.
d.
5.
Libmysqlclient-dev. MySQL databases development files, which are necessary for the SQL Alchemy and MySQL interaction.
Each test cycled through in the following sequence: Geekbench 3 (process & memory), fio sequential operations, fio random operations,
Iperf internal network throughput (for more information on testing, see Tests Used).
a.
For fio testing (to measure disk IOPS), sequential operations ran first. Files from the sequential tests were deleted, and fio
recreated files before running random operations. Once random operations completed, the files were also deleted. Thus, before
each disk IOPS test, the files associated with the tests were deleted and recreated.
6.
Internal network testing was conducted in one of the following manners:

a.
On AWS and Azure, where VMs demonstrated varying internal network throughput depending on size and/or instance type, a
clone of that VM was created in the same region/availability zone. The cloned server listened for a TCP connection via Iperf.
E.g., two c4.large instances were created in Amazon AWSs EU West 1 region to test throughput. The cloned VM was
terminated alongside the tested VMs at the conclusion of each 24-hour test cycle.
b.
On 1&1, Aruba Cloud, and CloudSigma, where VMs did not demonstrate varying internal network throughput depending on size
and/or instance type, a screen session was created on each VM to listen for a TCP connection. Each category of VMs, which
contains 2 VMs, connected with each other to perform throughput testing; i.e., 1&1s 2 vCPU 4GB RAM and 2 vCPU 8GB RAM
virtual machines conducted network throughput tests across one another.
7.
A total of approximately 1.4 million data points were collected throughout the period of the study.
8.
At the end of each test iteration, results were uploaded into Cloud Spectators database through use of SQL Alchemy (Pythonsee 4c in
Process).
6
Tests Used
Processor & Memory Bandwidth: Geekbench 3
Geekbench 3, a licensable product created by Primate Labs, is a cross-platform processor benchmark that can measure single-core and multi-core
performance by simulating real-world workloads. The Geekbench 3 test suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 10
floating point workloads, and 4 memory-bandwidth tasks. While processor and memory bandwidth are both performance factors that contribute to the
final score provided by Geekbench 3, the test suite weighs processing performance much more heavily than memory bandwidth. Also, memory
bandwidth is not necessarily affected by the amount of memory available for the VM, so VMs with larger amounts of memory may not exhibit larger
bandwidth. For more information on Geekbench 3 and to see its individual workloads, please see http://www.primatelabs.com/geekbench/.
Geekbench 3 Tasks (Figure 3.1)
TEST
TOOL
TASK
DESCRIPTION
Separate CPU tests that are all aggregated into a final

AES, Twofish, SHA1, SHA2, BZip2 Compression, BZip2
Integer
Geekbench 3
score.
Decompression, JPEG Compression, JPEG Decompression,
SubtestsPNG
include:
Integer Math,
Point
PNG
Compression,
Decompression,
Sobel,Floating
Lua, Dijkstra
Math
Floating Point
Geekbench 3
Black Scholes, Mandelbrot, Sharpen Filter, Blur Filter,

SGEMM, DGEMM, SFFT, DFFT, N-Body, Ray Trace
Integer and Floating Point tasks together represent

vCPU performance. The performance of all applications
is highly dependent on the vCPU since the vCPU is
responsible for the processing and orchestration of all
applications.
While memory performance is not considered one of
Memory
Geekbench 3
STREAM Copy, STREAM Scale, STREAM Add, STREAM

Triad
the key bottlenecks in performance for many common

applications, a subset of applicationsparticularly HPC
and in-memory databasesis highly dependent on
large sustained memory bandwidth.
Sequential and Random Disk IOPS: fio

Fio is an open source I/O generator that spawns a number of threads and processes to conduct a particular type of I/O action specified. For the
purpose of this study, fio was used to measure disk IOPS by tracking direct I/O to the VMs network storage. 5 x 200mb files were created for
sequential operations testing, and 5 x 200 mb files were created for random operations testing. All operations were 50% read and 50% write. Each
test iteration used an 8kb block size. Each test iteration lasted 60 seconds.
Internal Network Throughput: Iperf

Iperf is an open source tool used to measure TCP or UDP network bandwidth performance. In this study, Cloud Spectator used Iperf to measure the
network throughput between VMs residing in the same region/availability zone. A screen session was created for Iperf as a server machine on each
appropriate VM (see Process 6). Each Iperf test iteration lasted 60 seconds, and data was transferred in one direction, from the test VM to the Iperf
server VM.
7
VM Configurations & Pricing

VM configurations were matched to standardize by virtual processors. 2 vCPUs, 4 vCPUs, and 8 vCPUs machines from each provider were used in
this study that most closely matched in size. While some providers may offer the option for local storage, none was used and all disk testing was
conducted on persistent SAN storage. The storage columns in Figures 3.3 A through C are reflective of the SAN storage provisioned for each VM.
Azure uses Blob Storage, which automatically provides the user with as-needed storage. Thus, only the space on the volume that is needed
(depending on the OS and installed applications) is given to the user. With Blob Storage, users cannot deploy volumes with a pre-defined amount of
storage, although the user can specify how large he or she expects the blob to grow. For more information, see this article:
https://msdn.microsoft.com/en-us/library/azure/ee691964.aspx.
2 vCPU Virtual Machines (Figure 3.3 A)
Provider
Instance
vCPU
RAM (GB)
Storage (GB)
Monthly ()
1&1
4GB
100
29.99
1&1
8GB
100
79.20
AWS
C4.large
3.75
100 (EBS Optimized)
95.04
AWS
M3.large
7.5
100 (EBS Optimized)
108.72
Aruba Cloud
4GB
100
64.80
Aruba Cloud
8GB
100
79.20
A2
3.5
Blob Storage
68.10
Azure
Azure
D2
Blob Storage
104.60
CloudSigma
4GB
100
49.33
CloudSigma
8GB
100
73.73
8
4 vCPU Virtual Machines (Figure 3.3 B)

Provider
Instance
vCPU
RAM (GB)
Storage (GB)
Monthly ()
1&1
8GB
400
93.60
1&1
15GB
15
400
144.00
AWS
C4.xlarge
7.5
400 (EBS Optimized)
208.08
AWS
M3.xlarge
15
400 (EBS Optimized)
236.16
Aruba Cloud
8GB
400
158.40
Aruba Cloud
15GB
15
400
183.60
Azure
A3
Blob Storage
143.65
Azure
D3
14
Blob Storage
216.59
CloudSigma
8GB
400
135.27
CloudSigma
15GB
15
400
177.97
8 vCPU Virtual Machines (Figure 3.3 C)

Provider
Instance
vCPU
RAM (GB)
Storage (GB)
Monthly ()
1&1
15GB
15
800 (2 x 400)
172.80
1&1
30GB
30
800 (2 x 400)
280.80
AWS
C4.2xlarge
15
800 (EBS Optimized)
416.88
AWS
M3.2xlarge
30
800 (EBS Optimized)
473.76
Aruba Cloud
15GB
15
800
313.20
Aruba Cloud
30GB
30
800
367.20
Azure
A4
14
Blob Storage
287.24
Azure
D4
28
Blob Storage
433.11
CloudSigma
15GB
15
800
276.63
CloudSigma
30GB
30
800
368.15
Conversion Rates:
1.00 = 1.40
$1.00 = 0.89

9
Understanding Perform ance Results

The virtual machines performance information was depicted using the minimum, 5th percentile, median, 95th percentile, and maximum scores
retrieved from all data points collected for each of the tests mentioned above during the testing period. The information was integrated into percentile
graphs and value tables designed to visualize performance variation captured while testing over time. An example of a performance percentile graph
is displayed below:
Legend
Sample Performance Graph
Maximum: highest score achieved on this VM over the duration of the
4000
testing.
Mpixels/sec
3500
95TH Percentile (High-Score Category): 95% of all scores on this VM
3000
achieved this score or lower.
2500
Median (Median-Score Category): The number separating the higher half

of the scores of that VM from the lower half. If the median is closer to the
2000
95th percentile, then more high scores were observed than low scores; vice
1500
versa.
1000
5TH Percentile (Low-Score Category): 5% of all scores on this provider

achieved this score or lower.
500
Minimum: lowest score achieved on this VM over the duration of the
0
AWS
Azure
Google
Rackspace
testing.
Softlayer
Understanding The CloudSpecs Score (Price-Perform ance)

Cloud Spectator uses the CloudSpecs ScoreTM as an indication of price-performance value for each resource of the VM, separated by 1) processor
& memory bandwidth, 2) disk IOPS, and 3) internal network throughput. By definition, the CloudSpecs ScoreTM provides information on how much
performance the user receives for each unit of cost. The CloudSpecs ScoreTM is an indexed, comparable score ranging from 0-100 indicative of
value based on a combination of cost and performance. The calculation of the CloudSpecs ScoreTM is:
price-performance_value = [VM performance score] / [VM cost]
best_VM_value = max{price-performance_values}
CloudSpecs ScoreTM = 100*price-performance_value / best_VM_value
CloudSpecs ScoreTM
In this report, Cloud Spectator uses the aggregated performance scores as
100
100
the [provider performance score] to calculate each machines CloudSpecs

ScoreTM.
80
60
44
41
41
40
25
20
The graph on the left is an example of how Cloud Spectators priceperformance analysis is visualized. The closer the score is to 100, the
higher price-performance value it indicates. The score 100 represents the
0
CSP1
CSP2
CSP3
CSP4
CSP5
best-value VM among all in the comparison. The value is scaled; e.g., the
Cloud Service Provider (CSP)

10
VM from Cloud Service Provider 1 (CSP1) with a score of 100 gives 4x the value of the VM from CSP5 with a score of 25.
The CloudSpecs ScoresTM of any VM can change depending on the participants in the comparison. For example, if the highest score in a
comparison changes, the price-performance value represented by score 100 will change accordingly, and so will the other CloudSpecs ScoreTM
values.
If you have questions regarding Cloud Spectators price-performance calculation, please contact us at contact@cloudspectator.com.
Key Considerations
Pricing used in this study (for price-performance comparisons) is up to date effective June 10, 2015. Pricing may change for the specified
VMs after the release of this report.
Testing was conducted on specific VM types for each provider. Different VM configurations may yield different comparative results
between the providers. AWS and Azure offered fixed VM configurations, while 1&1, Aruba Cloud, and CloudSigma offered independently
customizable resource configurations.
Users may experience different performance across different physical hosts. Factors such as user contention or malfunctions of the
physical hardware can cause suboptimal performance. Cloud Spectator terminated and created new VMs for each test iteration to increase
the likelihood of testing on different physical hosts.
VMs selected were the base offerings across providers; greater performance may be obtained on certain providers by paying for additional
features/services.

11
Detailed Performance Findings

Processor and Memory Bandwidth
Below are the results of processor and memory bandwidth testing on all providers. Because memory bandwidth is not affected by the amount of
provisioned RAM, the VMs with larger amounts of RAM do not necessarily have higher Indexed Scores. The tables on the right specify the scores
achieved by each providers VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in
each category is highlighted and bolded in green in the corresponding tables.
Provider
1&1
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
AWS
2 vCPUs
Indexed Score
Processor & Memory PERFORMANCE: 2vCPUs

Figure 4.1 A
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
CloudSigma
8GB
1&1
AWS
4 vCPUs
Indexed Score
Provider
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
Aruba
Cloud
Azure
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
15GB
CloudSigma
CloudSigma

Figure 4.1 C
Provider
1&1
18000
16000
14000
12000
10000
8000
6000
4000
2000
0
8 vCPUs
AWS
Aruba
Cloud
Azure
15GB
30GB
1&1
c4.2xlarge m3.2xlarge
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
Min
4575
4059
3901
3125
3731
2973
1781
2176
2457
1150
5TH
4619
4663
3934
3139
4017
3883
2132
3557
2599
1374
Median
4711
4708
3952
3155
4085
4084
2166
3621
2896
2826
95TH
4731
4732
3973
3173
4165
4167
2185
3693
2965
2981
Max
4746
4748
4002
3192
4203
4197
2304
3746
2986
3000
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
7993
8304
6932
5699
6872
5660
3648
4978
1149
1192
5TH
8493
8688
7672
6105
7459
7388
4096
6650
1454
1474
Median
8764
8852
7736
6261
7693
7663
4118
6677
3698
4538
95TH
9088
9101
7775
6286
7842
7840
4136
6700
5782
4927
Max
9132
9144
7808
6308
7868
7881
4161
6716
5888
5387
CloudSigma

Figure 4.1 B
Indexed Score
Aruba
Cloud
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
15GB
30GB
CloudSigma
CloudSigma
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
14218
14858
12718
10580
10739
10765
5541
8687
4019
1336
5TH
15648
15768
13257
11228
11848
11692
7418
12646
5076
1550
Median
16138
16097
13705
11519
13775
13874
7839
12739
7281
5900
95TH
16281
16767
14056
11588
14119
14206
7917
12793
7848
6803
Max
16349
16910
14140
11659
14337
14400
7963
12866
8056
7640
12
Observations: Processor & M em ory Bandwidth Perform ance

The scores in Figures 4.1 A C are indexed through a combination of floating point and integer performance on processors and memory bandwidth
on RAM. CloudSigmas performance varied the most across processors. Its coefficient of variation (CV), which is a percentage expressing the
relationship between the average and standard deviation (useful for determining variability in performance), ranges up to 43.7% (on the 8 vCPU,
30GB RAM VM). By contrast, the CVs on 1&1s tested VMs ranged up to 2.6% (on the 4 vCPU, 15GB RAM VM).
Additional Observations
1&1s VMs achieved the highest performance across processor and memory performance in the study.
The Indexed Score, which weighs heavily on processor performance, is similar between VMs of the same vCPU count for 1&1, Aruba
Cloud, and CloudSigma. All three providers offer independently customizable VMs, which are backed with the same hardware. AWS and
Azure exhibit differences in performance results. AWSs C4 Family and M3 Family are provisioned with different processors; Azures A
Series and D Series are provisioned with different processors as well.
Although AWS and Azure VMs processor and memory performance scored lower than 1&1 VMs, both providers also achieved similar
levels of performance stability in the 72-hour test period, with CVs lower than 3% on all VMs. Aruba Cloud VMs CVs ranged up to 9.3%.
CloudSigma VMs large variation in performance, seen in the processor and memory performance results, are present in disk IOPS and
network performance as well.
CloudSigma and Azure VMs continue to display the lowest processor and memory bandwidth performance in each category (see tables in
Figure 4.1 A through C).
13
SAN Disk IOPS: Sequential Operations

Below are the results of disk IOPS testing on all providers, specifically for sequential operations (50% read, 50% write). Disk IOPS was tested with
direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the
scores achieved by each providers VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest
score in each category is highlighted and bolded in green in the corresponding tables.
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
2 vCPUs
IOPS
Sequential Read/Write- 2vCPU (100GB)

Figure 4.2 A
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
8GB
CloudSigma
CloudSigma
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
4 vCPUs
IOPS
Sequential Read/Write- 4vCPU (400GB)

Figure 4.2 B
Aruba
Cloud
Azure
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
CloudSigma
15GB
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
8 vCPUs
AWS
Aruba
Cloud
Azure
15GB
30GB
1&1
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
Min
4477
4622
235
227
379
267
586
480
214
208
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
3052
2845
2093
2452
241
278
580
427
219
177
5TH
6138
6082
299
299
713
597
1056
1220
349
328
5TH
5437
6570
2099
3013
618
655
1069
1166
311
306
Median
7143
6996
299
299
1493
1372
1380
1382
753
749
Median
7675
7933
3015
3015
1186
1200
1380
1381
603
633
95TH
7947
7972
3064
3064
2494
4414
1384
1412
2170
2011
Max
8341
8358
3064
3064
4060
6204
1399
1436
4538
3621
95TH
9133
8804
3017
3015
2483
2228
1384
1385
1768
2155
Max
9437
9138
3017
3015
4496
3892
1421
1404
4191
3909
CloudSigma
Sequential Read/Write - 8vCPU (800GB)

Figure 4.2 C
IOPS
Aruba
Cloud
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
15GB
30GB
CloudSigma
CloudSigma
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
3198
3265
2541
2907
342
367
396
531
202
195
5TH
4821
5709
3003
3013
617
599
620
1195
316
306
Median
6755
7650
3016
3015
1221
1300
663
1385
646
596
95TH
8703
9214
3017
3015
2352
3000
687
1414
1852
2074
Max
9280
9577
3017
3015
4345
5843
712
1435
4724
4012
14
SAN Disk IOPS: Random Operations

Below are the results of disk IOPS testing on all providers, specifically for random operations (50% read, 50% write). Disk IOPS was tested with
direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the
scores achieved by each providers VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest
score in each category is highlighted and bolded in green in the corresponding tables.
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
2 vCPUs
IOPS
Random Read/Write - 2vCPU (100GB)

Figure 4.3 A
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
8GB
CloudSigma
CloudSigma
Provider
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
1&1
AWS
4 vCPUs
IOPS

Figure 4.3 B
Aruba
Cloud
Azure
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
15GB
CloudSigma
CloudSigma

Figure 4.3 C
Provider
1&1
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
AWS
8 vCPUs
IOPS
Aruba
Cloud
Aruba
Cloud
Azure
15GB
30GB
1&1
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
15GB
30GB
CloudSigma
CloudSigma
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
Min
4710
4772
576
630
165
115
163
149
243
210
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
2830
3868
2759
2834
117
150
167
113
232
115
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
2814
3087
2816
2797
0
155
164
83
114
168
5TH
6002
5943
757
819
291
307
1021
1127
325
351
5TH
5110
6232
2985
2984
296
305
963
1116
308
302
5TH
4557
5490
2975
2984
295
296
341
1085
314
319
Median
6620
6621
760
822
612
552
1383
1386
661
652
Median
7272
7505
2987
2985
532
537
1376
1376
561
558
Median
6463
7364
2987
2985
534
518
369
1385
651
611
95TH
7346
7314
3064
3064
952
1064
1430
1428
1741
1435
95TH
8605
8359
2988
2985
983
951
1424
1380
1674
1557
95TH
8042
8751
2988
2985
992
919
382
1401
2156
2151
Max
8234
7695
3065
3064
1181
1709
1438
1436
3224
2997
Max
8896
8667
2988
2986
1167
1173
1436
1436
3049
2778
Max
8911
9109
2988
2986
1629
1175
392
1432
3897
3474
15
Observations: Sequential and Random Disk IOPS Perform ance

Providers offer unique approaches to SAN disk, from hardware components (SSD vs. traditional magnetic drives) to performance throttling. Even the
similarities can be quite different; for example, while 1&1 and CloudSigma both offer SSD-backed storage volumes, the performance difference
between the two providers is noticeable (see Figure 4.2-4.3), with 1&1 producing more than 6,500 IOPS, while CloudSigma produces less than 700
in the median range.
Providers with SSD offerings exhibit little performance difference between random and sequential IOPS, which is expected for SSDs.
These providers include 1&1, AWS and CloudSigma. One exception is on the small VMs for AWS, which exhibited different IOPS
performance results due to throttling.
Although 1&1 and CloudSigma both offer SSDs, 1&1s volumes achieved 4.4 4.7x more IOPS than CloudSigmas SSDs when examining
95TH percentile figures.
1&1, Aruba Cloud, and CloudSigma express large performance variability across the 24 hours of testing. Despite the variability, the low
points of 1&1s disk IOPS in both sequential and random operations still surpass other providers for the majority of tests. Aruba Cloud and
CloudSigmas variability results in some of the lowest performance observed during the study.
Although AWS offers SSD technology on its block storage offering, EBS, the performance of that offering is more dependent on the size of
the block storage volume provisioned. Although the actual IOPS performance varies between sequential and random operations, the
pattern of performance remains similar based on the size of the provisioned storage (in this test scenario, 100GB, 400GB, and 800GB
sizes were provisioned on 2 vCPU, 4 vCPU, and 8 vCPU machines, respectively). For its General Purpose SSD volumes, AWS offers 3
IOPS per GB with burst up to 3000 IOPS. Figures 4.2 A and 4.3 A illustrate AWSs burst for a 100GB volume. While burst for the 100GB
volume never exceeded 3065 IOPS (see Max in the corresponding tables), the non-burst range, expressed by the median, stays at 299
IOPS for sequential operations, but is higher for random operations. The minimum guaranteed IOPS, 300 (100GB * 3 IOPS per GB), is
sustained for the most part, although minimum values showed lower IOPS, with dips to 227 IOPS. At 800 GB, AWS sustains a fairly stable
rate of approximately 2750 3000 IOPS. Burst credits are assigned to volumes, and as long as the volume still has credits, it can burst to
3000 IOPS. In the period of the study, for 4 vCPU and 8 vCPU machines on AWS, the 400GB and 800GB volumes sustained enough
credits for a continuous 24-hours of high performance; interestingly, the c4.xlarge VMs dropped in performance for IOPS on sequential
operations in all three 24-hour iterations, which suggests that credits for IOPS are being used up on the c4.xlarge VMs much faster than on
others.
Azure and Aruba Cloud do not offer SSD technology-backed storage. Therefore, both providers offer higher IOPS performance on
sequential operations when compared with random operations, which is common behavior for magnetic drives. Figures 4.2 and 4.3
illustrate this behavior for both providers. In one of the sequences for the 8 vCPU with 15GB RAM VM, Aruba Cloud did not successfully
complete a cycle of random read/write operations, and therefore achieved a minimum of 0 in the total 72-hour study.
1&1 achieved much higher IOPS than all other providers tested, with maximums exceeding 9500 IOPS for sequential operations. Even in
the median, 1&1s SolidFire-backed SSD volumes obtained a minimum of 6463 IOPS, which is 2.15x more IOPS than can be achieved per
800GB volume on AWS, unless a user decides to purchase provisioned IOPS at an additional charge.
For random IOPS, Azure and Aruba Cloud displayed the lowest results in most scenarios. Azure and Aruba Cloud, as mentioned before,
are the only providers in this study that use magnetic storage for their storage volumes.
16
Internal Network Throughput

Below are the results of internal network testing on all providers. Internal network was examined by running TCP connections between two servers
within the same data center/region/zone. The tables on the right specify the scores achieved by each providers VMs. The lowest score in each
category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the
corresponding tables.
Provider
1&1
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
AWS
2 vCPUs
Throughput (Mbit/s)
Internal Network Throughput PERFORMANCE: 2 vCPUs

Figure 4.4 A
Aruba
Cloud
Azure
4GB
8GB
1&1
c4.large
m3.large
AWS
4GB
8GB
A2
ArubaCloud
D2
Azure
4GB
8GB
CloudSigma
CloudSigma

Figure 4.4 B
Provider
1&1
18000
14000
AWS
12000
4 vCPUs
Throughput (Mbit/s)
16000
10000
8000
6000
Aruba
Cloud
Azure
4000
2000
0
8GB
15GB
1&1
c4.xlarge m3.xlarge
AWS
8GB
15GB
A3
ArubaCloud
D3
Azure
8GB
CloudSigma
15GB
Provider
1&1
5000
AWS
4000
8 vCPUs
Throughput (Mbit/s)
6000
3000
2000
1000
Aruba
Cloud
Azure
0
30GB
1&1
AWS
15GB
30GB
ArubaCloud
A4
D4
Azure
Min
700
774
473
583
792
794
193
640
914
968
VM
8GB
15GB
c4.xlarge
m3.xlarge
8GB
15GB
A3
D3
8GB
15GB
Min
619
661
723
684
779
297
570
1451
1123
1423
5TH
929
928
473
628
834
838
302
863
1466
1979
5TH
885
896
725
899
826
743
863
1566
1942
2517
Median
955
955
473
628
875
894
708
880
2760
2900
Median
956
956
725
906
857
853
888
1624
6526
6970
95TH
955
955
492
663
895
895
734
895
4359
3660
Max
956
956
492
664
895
895
746
909
4832
4361
95TH
956
956
754
957
882
881
910
1683
13265
11396
Max
956
956
754
957
895
895
930
1735
16152
12821
95TH
956
956
962
957
895
893
925
1649
4657
5105
Max
956
956
962
957
895
895
936
1684
5983
5792
CloudSigma

Figure 4.4 C
15GB
VM
4GB
8GB
c4.large
m3.large
4GB
8GB
A2
D2
4GB
8GB
15GB
30GB
CloudSigma
CloudSigma
VM
15GB
30GB
c4.2xlarge
m3.2xlarge
15GB
30GB
A4
D4
15GB
30GB
Min
696
709
902
886
724
658
123
1432
1344
1005
5TH
841
891
921
903
801
820
890
1529
2001
2191
Median
953
956
925
906
852
861
910
1590
3208
3493
17
Observations: Internal Network Throughput Perform ance

With regards to internal network throughput, three out of five providers exhibited fairly stable performance of less than 10% CV. Azures A2 and A4
VMs expressed higher variability at 33.3% and 12.1%, respectively. CloudSigmas variability ranged much higher, up to 45.9% on the 4vCPU 8GB
RAM VM. When examining the median value of the same VM size, CloudSigma achieves much higher throughput (up to almost 7 Gbits) than the
other VMs. Comparing with the same VMs minimum value, the almost 7 Gbits drops to 1.5 Gbits. Despite the sharp decrease in performance,
CloudSigmas unstable performance still achieved higher throughput, for the most part, than all other providers.
AWSs VMs deliver a specific network throughput depending on the size and family of VM. As the size of the VMs increase, the network
throughput increases as well. The difference in network performance between the C4 Family and M3 Family are comparable. By contrast,
Azures A Series provides less throughput than its D Series.
Azures VMs, which scale depending on size and Series, exceeds 1GB/s network throughput for D3 and D4s (see Figures 4.4B and C). All
other providers, with the exception of CloudSigma, never exceed 1GB/s, although 1&1 VMs and AWSs larger VMs come close.
1&1 provides a continuous and fairly steady throughput of slightly less than 1GB/s (956 GB/s max) regardless of VM size. Similarly, Aruba
Cloud provides a lower, continuous throughput at a little less than 900 MB/s, regardless of VM size as well.
While AWS displayed some of the lowest internal network throughput numbers for 2 vCPU and 4 vCPU VMs, Aruba Cloud, which does not
scale internal network throughput with increased VM size, displayed the lowest throughput numbers for 8 vCPU VMs.
18
Detailed Price-Performance Findings

Processor & Memory
Figures 5.1 A through F illustrate the price-performance results for processor and memory bandwidth performance across all tested providers. Rows
are organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
Processor & Memory Price-Performance: 2 vCPUs
Figure 5.1 A
Processor and Memory Price-Performance: 2 vCPUs

Figure 5.1 B
100
90
90
80
80
70
60
50
40
40
37
26
30
20
CloudSpecs Score
100
70
49
40
30
A2
4GB
8GB
m3.large
8GB
D2
8GB
0
4GB
10
c4.large
20
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.1 D
100
100
100
90
90
80
80
70
52
60
50
40
31
40
29
30
CloudSpecs Score
100
68
70
60
43
50
50
41
40
30
8GB
15GB
m3.xlarge
15GB
D3
15GB
0
A3
10
0
8GB
20
10
c4.xlarge
20
8GB
CloudSpecs Score
64
50
10

Figure 5.1 C
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.1 E

Figure 5.1 F
100
47
51
42
A4
15GB
30GB
m3.2xlarge
30GB
D4
30GB
28
15GB
28
66
c4.2xlarge
29
100
90
80
70
60
50
40
30
20
10
0
15GB
35
CloudSpecs Score
100
CloudSpecs Score
58
60
20
100
90
80
70
60
50
40
30
20
10
0
87
4GB
CloudSpecs Score
100
100
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
19
Observations: Processor & M em ory Price-Perform ance

In all cases illustrated from Figure 5.1 A through F, 1&1s VMs achieved the highest CloudSpecs ScoreTM in the test group, indicating the strongest
price-performance value for processor and memory bandwidth. AWSs m3.large VM comes closest to matching in value with a 1&1 VM counterpart,
at a CloudSpecs ScoreTM of 87 (see Figure 5.1 B).
For VMs with 2 vCPUs and approximately 8GB RAM (see Figure 5.1 B), all values are most evenly matched, compared with other VM
sizes.
As VM sizes scale up in processors and RAM, CloudSigmas CloudSpecs ScoreTM drops, and its rank falls in relativity to other tested
providers VMs, due largely to performance (see Figure 4.1).
Relatively, Azures D Series, which has an average CloudSpecs ScoreTM of 53 across all VM sizes, provides more relative priceperformance value than its A Series, which has an average CloudSpecs ScoreTM of 27. Because charts cannot be relatively compared,
though, the difference between the CloudSpecs ScoresTM does not equate to an almost 2x value on Azures D Series.
20
SAN Disk IOPS: Sequential Operations

Figures 5.2 A through F illustrate the price-performance results for sequential disk IOPS performance across all tested providers. Rows are
organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
Sequen9al Read/Write: 2 vCPUs (100GB)
Figure 5.2 A

Figure 5.2 B
100
100
90
90
CloudSpecs Score
CloudSpecs Score
100
100
80
70
60
50
40
70
60
50
40
30
30
20
2
10
4GB
A2
4GB
8GB
m3.large
8GB
D2
8GB
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.2 D
100
100
90
90
CloudSpecs Score
CloudSpecs Score
100
100
80
70
60
50
40
23
30
20
80
70
60
50
40
12
20
12
10
c4.xlarge
8GB
A3
8GB
15GB
m3.xlarge
15GB
D3
15GB
8GB
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.2 E

Figure 5.2 F
100
CloudSpecs Score
100
25
m3.2xlarge
30GB
D4
30GB
30GB
12
15GB
10
A4
15GB
30
c4.2xlarge
100
90
80
70
60
50
40
30
20
10
0
15GB
CloudSpecs Score
30
30
7
10
100
90
80
70
60
50
40
30
20
10
0
11
c4.large
15
10

Figure 5.2 C
14
20
4GB
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
21
SAN Disk IOPS: Random Operations

Figures 5.3 A through F illustrate the price-performance results for random disk IOPS performance across all tested providers. Rows are organized
by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
Random Read/Write: 2 vCPUs (100GB)
Figure 5.3 A

Figure 5.3 B
100
100
90
90
CloudSpecs Score
CloudSpecs Score
100
100
80
70
60
50
40
70
60
50
40
30
30
20
10
10
4GB
A2
4GB
8GB
m3.large
8GB
D2
8GB
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.3 D
100
100
90
90
CloudSpecs Score
CloudSpecs Score
100
80
70
60
50
40
24
30
10
80
70
60
50
31
40
30
12
20
20
10
8GB
A3
8GB
15GB
m3.xlarge
15GB
D3
15GB
c4.xlarge
12
8GB
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.3 E

Figure 5.3 F
100
CloudSpecs Score
100
25
m3.2xlarge
30GB
D4
30GB
30GB
12
15GB
A4
15GB
31
c4.2xlarge
100
90
80
70
60
50
40
30
20
10
0
15GB
CloudSpecs Score
11
1&1
100
100
90
80
70
60
50
40
30
20
10
0
16
6
c4.large

Figure 5.3 C
12
20
4GB
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
22
Observations: Sequential and Random Disk IOPS Price-Perform ance

For sequential and random operations across all VMs, 1&1 offers the highest value due to a combination of high-performance SSDs and low overall
cost of the virtual machine (see Figures 5.2 and 5.3). While CloudSigma also offers SSDs, the disk IOPS achieved on CloudSigma VMs display
much lower performance (see Figures 4.2 and 4.3). AWS, which also offers SSD-based storage (General Purpose) offering was used in this study,
throttles performance at a maximum of 3,000 IOPS unless users purchase additional provisioned IOPS; therefore, performance on AWSs SSDs did
not exceed that of 1&1 either, which had a median range of 6,500-7,500 IOPS.
1&1 achieved the highest CloudSpecs ScoresTM as well as most IOPS for each VM tested.
Aruba Cloud offered the lowest value for random IOPS on SAN disk on all tested VMs, due to its magnetic drives and cost of the VMs.
AWSs c4.large and m4.large (see Figure 5.2 A and B) offered lower price-performance value due to throttled IOPS after burst limits were
exceeded for the 100GB storage volume.
Although CloudSigma used SSD technology for its storage volume, the price-performance value of those storage volumes tested are
similar to Aruba Cloud and Azure volumes, which use magnetic disks.
23
Internal Network Throughput

Figures 5.4 A through F illustrate the price-performance results for internal network throughput performance across all tested providers. Rows are
organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
Internal Network Price-Performance: 2 vCPUs
Figure 5.4 A

Figure 5.4 B
100
100
90
90
CloudSpecs Score
CloudSpecs Score
100
100
80
70
60
57
50
40
30
13
20
16
70
60
50
40
31
30
19
21
20
21
20
10
10
c4.large
4GB
A2
4GB
8GB
m3.large
8GB
D2
8GB
0
4GB
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.4 C

Figure 5.4 D
100
100
90
90
CloudSpecs Score
CloudSpecs Score
100
100
80
70
60
50
40
30
13
60
50
40
20
17
19
13
10
10
c4.xlarge
8GB
A3
8GB
15GB
m3.xlarge
15GB
D3
15GB
0
8GB
70
30
21
20
80
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma

Figure 5.4 E

Figure 5.4 F
CloudSpecs Score
48
27
26
A4
15GB
30GB
m3.2xlarge
30GB
D4
30GB
19
15GB
18
39
36
c4.2xlarge
25
100
100
90
80
70
60
50
40
30
20
10
0
15GB
CloudSpecs Score
100
100
90
80
70
60
50
40
30
20
10
0
1&1
AWS
ArubaCloud
Azure
CloudSigma
1&1
AWS
ArubaCloud
Azure
CloudSigma
Observations: Internal Network Throughput Price-Perform ance

CloudSigma VMs deliver the highest available throughput out of all providers tested. Despite having higher prices than 1&1, the high
network throughput performance resulted in a better CloudSpecs ScoreTM for CloudSigma. Similar with Processor & Memory price-performance,
24
Azure also achieved a better CloudSpecs ScoreTM for its D Series in regards to internal network throughput. The D series offered approximately 1.7x
more network throughput on VMs tested in the study.
Although AWSs virtual machines increased throughput based on size and family, that increase did not surpass the price-performance ratio
of 1&1 VMs throughput and cost; therefore, 1&1 achieved a higher CloudSpecs ScoreTM on network throughput than AWS.
Aruba Cloud, which offered higher throughput than AWS on smaller VMs (see Figure 4.4 A), displayed higher internal network throughput
price-performance than AWS until VM sizes scale up. As machine sizes increased, AWSs throughput scaled while Aruba Clouds did not, which is
the reason AWS has higher CloudSpecs ScoresTM for the larger VMs. For these VMs, (see Figure 5.4 B and C), Aruba Cloud fell short of AWS, as
well as all other competitors in the study.
Azures D3 and D4 VMs exceeded a throughput of 1 GB/s (see Figures 4.4 B and C), giving it better price-performance value in internal
network than all other providers for those VM sizes with the exception of CloudSigma (see Figures 5.4 D and F).
In all other VM sets, 1&1 achieves the highest CloudSpecs ScoreTM with the exception of CloudSigma for internal network.
Aruba Cloud, which does not scale internal network throughput with the size of the VM (see Figure 4.4), drops in value ranking as VMs are
scaled up in size; other providers, such as AWS and Azure, which scale internal network throughput with the corresponding VM size and
family/series, surpass Aruba Cloud in value for internal network as the VMs are scaled up.
25
Conclusion
Selecting the right provider and virtual machines requires a thorough and accurate performance comparison. The study conducted for this
report offers a general understanding of performance and price-performance strengths and weaknesses across each included vendors VMs, and
should be considered a source of information to help guide readers in their own testing and analyses. The processor, memory bandwidth, network
storage, and internal network were all examined and results clearly show that no single provider can offer everything to fit everyones needs.
Therefore, in order to accurately select the right provider for a business or an application, performance and price-performance analysis is crucial.
Results from this study show that, 1&1s VMs displayed strong performance and price-performance. 1&1s VMs demonstrated high
performance and stability for processing and memory bandwidth both on the same VM, and also across VMs of the same size. Its network-attached
storage produced the largest amount of IOPS seen in the study as well, regardless of VM size and without the need for purchasing additional
volumes, sizing up, or provisioning IOPS. These high-performance results, combined with the low cost of the VMs, reflect on the value that 1&1s
VMs can deliver to potential users, as seen with the price-performance results. 1&1s performance and price-performance offers an excellent
alternative for high-performance environments such as distributed file systems and data analytics processing.
Performance in the industry cannot be assumed to be equal or even similar, as illustrated in this report. When it comes to processor and
memory bandwidth performance, tiered providers such as AWS and Azure may offer varying performance depending on the family/series of the VM,
despite having equivalent amounts of vCPUs and similar amounts of memory. For disk IOPS, although both 1&1 and CloudSigma advertise SSD
volumes, 1&1s SSDs achieved 4.4 4.7x more IOPS than their CloudSigma counterparts. Internal network performance on CloudSigma, though,
exceeds all providers examined in the study.
While this study was conducted in the manner of understanding a typical end user experience, it should not be assumed to be accurate for
all use cases. Stress testing was conducted to better understand fluctuation and theoretically sustained performance, and should be seen as a
general indication of provider performance. For more detailed analysis on any specific use case, please contact Cloud Spectator at
contact@cloudspectator.com or by phone at +1 (617) 300 0711.
About
About Cloud Spectator
Cloud Spectator is a cloud analyst agency focused on cloud Infrastructure-as-a-Service (IaaS) performance. The company actively monitors several
of the largest IaaS providers in the world, comparing VM performance (i.e., CPU, RAM, disk, internal network, and workloads) and pricing to achieve
transparency in the cloud market. The company helps cloud providers understand their market position and helps business make intelligent
decisions in selecting cloud providers and lowering total cost of ownership. The firm was founded in early 2011 and is located in Boston, MA.
For questions about this report, to request a custom report, or if you have general inquiries about our products and services, please contact Cloud
Spectator at +1 (617) 300-0711 or contact@cloudspectator.com.
For press/media related inquiries, please contact:
Ken Balazs
VP Sales & Marketing
kbalazs@cloudspectator.com
26

11cloud Spectator Report - Final PDF

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

11cloud Spectator Report - Final PDF

Hochgeladen von

Copyright:

Verfügbare Formate

Cloud IaaS Performance & Price-Performance

Comparing Linux Compute Performance of 1&1, Amazon AWS, Aruba Cloud,

Prepared for 1&1 on Behalf of SolidFire

Cloud Spectator Performance Report

2. Executive Sum m ary

Provider Data Center/Region Locations

VM Configurations and Pricing

Understanding Performance Results

Understanding The CloudSpecs Score (Price-Performance)

4. Detailed Perform ance Findings

Processor & Memory Bandwidth

Disk IOPS: Sequential and Random Operations

5. Detailed Price-Perform ance Findings

Processor & Memory Bandwidth

Disk IOPS: Sequential and Random Operations

Cloud Spectator Performance Report

Cloud Spectator Performance Report

CPU & MEMORY PERFORMANCE

Because most applications and all data reside

In a cloud environment, network performance

dependent on the CPU. The CPU is responsible for

on the disk, having fast disk performance is a

is a critical piece. Scalability, in many cases,

the processing and orchestration of all applications.

is dependent on the availability of additional

The relationship between CPU performance and RAM

performance in many cases.

is also observed by examining RAM bandwidth. While

VMs that must maintain a strong network

memory performance is not considered one of the key

Cloud Spectator Performance Report

vCPU & Memory Performance Key Findings:

bandwidth performance by benchmarking the VMs using Geekench

1&1s VMs achieved the highest performance across processor

3, a suite of benchmark tests that simulate tasks such as

and memory performance in the study.

the course of a 72-hour testing period. Pricing was examined in

group, indicating the strongest price-performance value for

conjunction with the performance tests.

processor and memory bandwidth.

Cloud Spectator Performance Report

Storage Perform ance Findings

Storage Performance Key Findings:

the sequential read/write and random read/write operations of

over a 72-hour test period. Persistent storage (offered as block

Although AWS offers SSD technology on its block storage offering,

storage or redundant storage) was used in all storage tests. Pricing

EBS, the performance of that offering is more dependent on the

was examined in conjunction with the performance tests.

size of the block storage volume provisioned.

Internal Network Perform ance Findings

Internal Network Performance Key Findings:

CloudSigma VMs achieved the highest internal network

throughput, although the high throughput is unstable and

hour test period. Pricing was examined in conjunction with the

fluctuates between less than 500Mbits to over 10Gbits.

CloudSigma displayed the best price-performance value for

Cloud Spectator Performance Report

The following dependencies were installed for testing:

Internal network testing was conducted in one of the following manners:

Cloud Spectator Performance Report

Separate CPU tests that are all aggregated into a final