Processor Analysis: Generational Performance of Intel® Processors in An Expedient Cloud

Processor Analysis: Generational Performance of Intel Processors in an Expedient Cloud
Benchmarking different Intel Processors in Expedient Generation 2, Generation 3, Generation 4 and Generation 5 Offerings
By Cloud Spectator November 2013
Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013
Table of Contents Introduction Executive Summary Processor Performance

Floating Point Math Integer Math Compression Encryption Physics
2 3 4
5 6 7 8 9
Read from cached RAM Read from uncached RAM RAMspeed SMP
RAM Performance
10
11 12 13 15 16 17 18 19 20 21 21 23
Local Copy Sequential Read Sequential Write Random Seek + R/W Disk Mark Appendix A: Methodology Appendix B: Terms & Definitions Appendix C: Test Descriptions About Cloud Spectator
Disk Performance
14
Copyright Cloud Spectator, LLC 2013. All rights reserved
Introduction
Performance Testing Cloud Spectator monitors the processor, RAM, storage, and internal network performance of over 20 of the worlds most well -known IaaS services to understand important aspects of virtual server performance. Tests are run at least three times per day, 365 days per year to capture var iability in addition to performance level. Tests are chosen based on reliability and practicality. The goal is to provide an indication of where certain providers perform well relative to others. This can give consumers an indication of which services would be best for their application(s) by understanding the performance of provider resources most critical to that application. Singular benchmarks alone should not be the only deciding factor in the provider selection process. Feature sets, configuration matches, pricing and ancillary services such as security, compliance, and disaster recovery should always factor into any vendor selection process. However, performance is a very important piece to the puzzle. The Comparison For the purpose of generating this document, Cloud Spectator measured the processor, RAM and disk performance of Expedients virtual machines, across 2nd, 3rd, 4th, and 5th.Generation offerings. Each offering uses different Intel processor models. The goal was to understand how Intel processor performance has increased over time, using Expedients offerings as a practical cloud environment to test in . In addition, RAM and disk performance was also examined to account for the overall performance of the server. Over a period of fifteen (15) days, Cloud Spectator ran the same benchmark tests across Expedients 2nd, 3rd, 4th, and 5th Generation offerings. Tests were run from October 12th through October 26th, 2013. Cloud Spectator accounted for performance capability and stability for each envir onment to understand the value each one delivers to its users. Tests were run on 8GB servers with 4 vCPUs (for more details, see Methodology section in Appendix A). Statistical measures used to illustrate improvement included: average, standard deviation, coefficient of variation (CV), 15-day high and 15-day low. Please see Appendix B for definitions of each statistical measure.
The goal was to understand how Intel processor performance has evolved over time, using Expedients offerings as a practical cloud environment to test in.
Performance testing and benchmarking of cloud computing platforms is a complex task, compounded by the differences between providers and the use cases of cloud infrastructure users. IaaS services are utilized by a large variety of industries, and performance metrics cannot be completely understood by simply representing cloud performance with a single value. When selecting a cloud computing provider, IT professionals consider many factors: feature-sets, cost, security, location and more. However, performance is a key issue that driv es many others including cost. Processors The processors below are used in each given offering: Expedient Generation 2: Intel Xeon Processor E5520 2.27GHz Expedient Generation 3: Intel Xeon Processor X5650 2.67GHz Expedient Generation 4: Intel Xeon Processor E5-2670 2.60GHz Expedient Generation 5: Intel Xeon Processor E5-2670 v2 2.50GHz
Executive Summary
Findings
Newer Generations Deliver Significantly More Processor, RAM and Disk Performance Generation 5 offerings performed better than Generation 2 offerings in the following areas: CPU performance improved from Generation 2 to 5 by 62% RAM performance improved from Generation 2 to 5 by 76% Generation 5 offerings performed better than Generation 3 offerings in the following areas: CPU performance improved from Generation 3 to 5 by 28% RAM performance improved from Generation 3 to 5 by 57% Generation 5 offerings performed better than Generation 4 offerings in the following areas CPU performance improved from Generation 4 to 5 by 13% RAM performance improved from Generation 4 to 5 by 9% Generation 4 offerings performed better than Generation 2 offerings in the following areas CPU performance improved from Generation 2 to 4 by 44% RAM performance improved from Generation 2 to 4 by 60% Disk performance improved from Generation 2 to 4 by 134% Generation 4 offerings performed better than Generation 3 offerings in the following areas CPU performance improved from Generation 3 to 5 by 13% RAM performance improved from Generation 3 to 5 by 43% Disk performance improved from Generation 3 to 5 by 161% Generation 3 offerings performed better than Generation 2 offerings in the following areas CPU performance improved from Generation 2 to 3 by 28% RAM performance improved from Generation 2 to 3 by 11% Newer Generations Have Improved Processor and RAM Stability Coefficients of variation (CVs), representing variability, decreased from Generation 2 to Generation 5 by an average of 72%. Generation 5 also decreased in variability by 69% and 88% from Generation 4 and Generation 3 respectively. Averages in each scenario are much closer to the highs than lows, meaning that performance levels are more often higher than the mean. This means that the lows are responsible for most of the variability. Customers should expect a high level of performance most of the time. CPU TESTS: Highs are an average of 1.5 standard deviations from the mean, while lows are an average of 3.4 standard deviations from the mean. RAM TESTS: Highs are an average of 2.1 standard deviations from the mean, while lows are an average of 3.6 standard deviations from the mean. DISK TESTS: Highs are an average of 1.4 standard deviations from the mean, while lows are an average of 4.4 standard deviations from the mean.
Processor Performance
Why Processor Performance Matters
In a public cloud environment, resources are shared across virtual machines within the same physical server. Hardware components such as the processor are vir tualized and usually shared across VMs. While modern-day physical processors may accomplish tasks at a desirable rate, virtualized processor s, which may or may not receive a dedicated cores performance, may be less powerful. As the cloud industry progresses, though, with the introduction of newer cloud offerings with the latest processor technology, users experience a significant performance benefit while paying a comparable price for the IaaS service. This improvement in performance is highlighted within the following tests:
Floating Point Math

Floating Point Math and Integer Math tests are synthetic benchmarks commonly used in gauging and comparing CPU performance because the operations used make up the basic operations in all computer software. The Floating Point Math test uses the same calculations as the Integer Math test, except using numbers with decimals, which are commonly used in modern-day applications. The floating point math performance level and variability showed a significant improvement from Generation 2 through 4. Generation 3 average performance improved by 22% relative to Generation 2. Generation 4 average performance improved on Generation 3 by 35%. Generation 5 average performance improved on Generation 4 by 9%. Overall, this represents a 79% performance increase from Generations 2 to 5. Variability also decreased by a significant amount, 96% , from Generation 2 to Generation 5. These results were in line with what was expected after Intels introduction of Advanced Vector Extensions (AVX) technology in Generation 4 (Intel Xeon Processor E5-2670 2.60GHz). Intel Advanced Vector Extensions (Intel AVX) is an instruction set extension designed for applications that are floating point intensive. Intel AVX results in better management of data, improving performance through wider vectors, ne w extensible syntax, and rich functionality.
CPU TEST - Floating Point Math

7000
6000
5000
MIPS
4000
Gen 2 Gen 3
Gen 4 Gen 5
3000
2000 1000
0 Oct 12 Oct 14 Oct 16 Oct 18 Oct 20 Oct 22

Oct 13 Oct 15 Oct 17 Oct 19 Oct 21 Oct 23
The graph above compares Intel CPU floating point math performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark floating point math benchmark test for each provider over a period of 15 days, with tw o (high and low) data points shown for each day. For more information on the PassMark floating point math test, please see Appendix C.
PROVIDER 2 nd Generation 3 rd Generation 4 th Generation 5 th Generation
AVERAGE 3208 3910 5260 5756
STANDARD DEVIATION 185 138 145 14
CV 5.8% 3.5% 2.8% 0.2%
15-DAY HIGH 3335 4143 5464 5785
15-DAY LOW 2430 3591 4675 5691
Note: numbers expressed in millions of operations per second.
Integer Math
Floating Point Math and Integer Math tests are synthetic benchmarks commonly used in gauging and comparing CPU performance be cause the operations used make up the basic operations in all computer software. The Integer Math test runs mathematical operations of addition, subtraction, multiplication, and div ision of whole numbers to giv e an indication of raw CPU throughput. Integer math performance improved by a large margin from Generation 2 to Generation 3 and by much smaller margins from Generation 3 to Generation 4 and Generation 4 to Generation 5. Generation 3 saw a significant performance improvement of 22% over Generation 2. Generation 4 average performance improved slightly, by 4%, over Generation 3. Generation 5 performance improved slightly more, by 5% over Generation 4. The total performance improvement from Generation 2 to Generation 5 was 34%. Performance variability for all four generations was very low with the highest CV being 1.9%.
CPU TEST - Integer Math

9000 8000
7000
6000
MIPS
5000
Gen 2 Gen 3
4000 3000
2000 1000
Gen 4 Gen 5
0 Oct 12 Oct 13 Oct 14 Oct 15 Oct 20 Oct 21 Oct 22 Oct 23

Oct 16 Oct 17 Oct 18 Oct 19
The graph above compares Intel CPU integer math performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark integer math benchmark test for each prov ider ov er a period of 15 days, with tw o (high and low ) data points shown for each day. For more information on the PassMark integer math test, please see Appendix C.
AVERAGE 5911 7208 7513 7909
CV 1.6% 1.9% 0.9% 0.2%
15-DAY HIGH 5976 7412 7615 7938
15-DAY LOW 5508 6930 7325 7849
Note: numbers expressed in millions of operations per second.
Compression
Compressing files into smaller blocks of data is a common task used in software applications like Email and file backups. Compressing the same file in the same manner across different machines reveals a 43% improvement for this CPU-intensive task from the Expedient Generation 2 offering with the Intel Xeon Processor E5520 to the Expedient Generation 5 offering with the Intel Xeon Processor E5-2670 v2. The compression results reveal a clear improvement from Generation 2 to 3, and from 3 to 4. The Expedient Generation 3 offering saw a 20% improvement in compression performance from Generation 2, with decreased performance variability. From Generation 2 to 3, the CV decreased by 32%, indicating more predictable and steady performance levels. Generation 4 displays further CPU compression performance over Generation 3. Performance increased by 12% from Generation 3 to 4, and variability decreased by 31%. The performance improvement from Generation 4 to 5 was less noticeable than the preceding generations, while stability of performance improved drastically. Performance increased by 7%, while variability decreased by 84%.
CPU TEST - Compression

8000 7000
6000 5000 Kb/sec
Gen 2
4000
Gen 3
Gen 4 Gen 5
3000 2000
1000 0
Oct 15
Oct 16
Oct 17
Oct 18
SAMPLE
The graph above compares Intel CPU compression performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark compression benchmark test for each prov ider ov er a period of 15 days, with tw o (high and low ) data points shown for each day. For more information on the PassMark compression test, please see Appendix C.
AVERAGE 5276 6306 7044 7528
CV 4.1% 2.8% 1.9% 0.3%
15-DAY HIGH 5422 6621 7232 7558
Oct 23
Oct 12
Oct 13
Oct 14
Oct 19
Oct 20
Oct 21
Oct 22
15-DAY LOW 4328 5805 6506 7471
Note: numbers expressed in kilobytes processed per second.
Encryption
This encryption test encrypts random blocks of data using different encryption techniques. The machine will also create a hash of the data, which ensures that the encrypted data is not compromised. From a performance view, encryption uses many of the algor ithms used in the Integer and Floating Point math test, as well as more complex ones such as to the power of functions. Encryption is used in many Intern et-based applications to ensure privacy, including Internet browsers and communications tools. Generation 3 shows a significant improvement in encryption performance over Generation 2. Generation 3 displays a 53% improvement in encryption performance over Generation 2. Generation 4 slightly decreased in average encryption performance, decreasing by 1% compared to Generation 3. Generation 5 improved by increasing performance by 16% over Generation 4, and a 74% improvement over Generation 2. Generation 3 variability decreased significantly, by 26%, from Generation 2. The Generation 4 CV decreased 28% compared to Generation 3. Generation 5 variability decreased 87% from Generation 4, which was part of a 93% decrease in CV from Generation 2 to Generation 5. These results were in line with what was expected after Intels introduction of Intel AES-NI technology present in Generations 3 to 5 (Intel Xeon Processor X5650 2.67GHz and Intel Xeon Processor E5-2670 2.60GHz, respectively). Intel Advanced Encryption Standard New Instructions (Intel AES-NI) introduces new instructions for enhanced security and speed while implanting some intensive sub-steps of the AES algorithm into the hardware for faster execution.
CPU TEST - Encryption

1200 1000 800 Mb/s Gen 2 600 400
200 0 Gen 3 Gen 4 Gen 5
Oct 12
Oct 13
Oct 14
Oct 15
Oct 20
Oct 21
Oct 22
The graph above compares Intel CPU encryption performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph SAMPLE shows the scores from the PassMark encryption benchmark test for each prov ider over a period of 15 days, with tw o (high and low ) data points shown for each day. For more information on the PassMark encryption test, please see Appendix C.
AVERAGE 627 956 944 1092
CV 3.9% 2.9% 2.1% 0.3%
15-DAY HIGH 648 1003 971 1096
Oct 23
Oct 16
Oct 17
Oct 18
Oct 19
15-DAY LOW 492 863 881 1085
Note: numbers expressed in megabytes transferred per second. Copyright Cloud Spectator, LLC 2013. All rights reserved
Physics
Using the Tokamak Physics Engine, the Physics benchmark performs collision calculations and visual rendering simultaneously. The Tokamak Physics Engine is a high performance, real-time physics library designed specifically for games and has a built-in collision functionality, which is used for the test. Average performance level in the Physics Test improved steadily with more modern generations. Generation 3 average performance improved by 23% over Generation 2 while Generation 4 improved by 18% over Generation 3, and Generation 5 improved by 26% over Generation 4. This brings the total physics performance improvement from Generation 2 to Generation 5 to 82%. Variability decreased by 92% from Generation 2 to Generation 5.
CPU TEST - Physics

600 500
400
FPS
Gen 2 300 200 100 Gen 3
Gen 4
Gen 5
0 Oct 12 Oct 13 Oct 14 Oct 15 Oct 16 Oct 17

Oct 18 Oct 19 Oct 20 Oct 21 Oct 22 Oct 23
The graph above compares Intel CPU physic s calculation performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark physics benchmark test for each prov ider over a period of 15 days, with tw o (high and low ) data points shown for each day. For more information on the PassMark physic s test, please see Appendix C.
AVERAGE 260 320 377 475
CV 4.7% 7.8% 4.7% 0.4%
15-DAY HIGH 284 360 411 481
15-DAY LOW 221 264 326 469
Note: numbers expressed in frames per second.
RAM Performance
Why RAM Performance Matters
RAM is an integral part of any server. The RAM, or memory, is the temporary storage that the CPU can use to access data quickly, compared to waiting for data to slowly be pulled from the hard driv e storage. Playing a key factor in a wide range of applications, RAM performance can be just as important as CPU performance. Applications are loaded onto the RAM for faster access and better user experience. RAM is greatly utilized in servers running databases, games or media. Many new application workloads are now taking advantage of hosted envir onments where a large amount of RAM can be accessed. Growth in in-memory databases, gaming platforms, and many other use cases continue to raise the value of RAM performance.
10
Read from cached RAM

This test measures the time taken to read a small block of memory that is small enough to be held entirely in cache. The test not only measures RAM, but also the CPU cache, which decreases the time needed for a CPU to retrieve data directly from the memory . For workloads that fully utilize the CPU, it is important to be able to process stored information as quickly as possible. The Intel Xeon E5-v2 processors have 25MB of CPU cache and 1866 MHz memory speeds to provide fast access to frequently used data. Performance improvements from Generation 2 to 3 and Generation 4 to 5 are relatively modest compared to the jump from Generation 3 to 4 as shown by the Read from cached RAM test. However, performance stability is the tradeoff for higher performance in Generation 3 and 4. Average performance increases by 19% from Generation 2 to 3. Generation 4 improves over Generation 3 by a significant 62% , while Generation 5 improves upon Generation 4 by a modest 22%. Overall performance increased by 136% from Generation 2 to 5. Performance is the most stable for Generation 2 and 5 with CVs of 0.7%, while variability decreases by 72% and 63% for Generation 3 and Generation 4 respectively.
RAM TEST - Read from cached RAM

25000
20000
15000
Mb/s
Gen 2 Gen 3
Gen 4 Gen 5
10000
5000
Oct 12
Oct 13
Oct 14
Oct 15
The graph above compares RAM performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Read from cached RAM benchmark test for each prov ider over a period of 15 days, with tw o (high and low ) data points shown for each day. For more information on the Read from cached RAM test, please see Appendix C.
AVERAGE 9006 10759 17404 21220
CV 0.7% 2.5% 1.9% 0.7%
15-DAY HIGH 9105 11119 18401 21444
Oct 26
Oct 16
Oct 17
Oct 18
Oct 19
Oct 20
Oct 21
Oct 22
Oct 23
Oct 24
Oct 25
15-DAY LOW 8552 10201 16718 20629
Note: numbers expressed in megabytes transferred per second.
11
Read from uncached RAM

This test measures the time taken to read a large block of memory. Measured in GB/s, larger values are better. The test will use a 256 MB block, too large for any CPU cache in Intel processors, measuring the efficiency of the CPU to read directly from RAM. The results of the Read from uncached RAM test reveal steady increases in average performance from Generation 2 to Generation 5. The improvement from Generation 2 to 3 was 14%, while the percentage of improvement nearly doubled for the increase from Generation 3 to 4, which was 27%. The increase of Generation 4 to 5 was a lower 17%, while the overall improvement of Generation 5 from Generation 2 was 69%. As shown in the graph below, Generation 3 performance varies from two different levels, with a period of lower performance from October 16th to 24th. As a result, Generation 3 performance variability increases over Generation 2 by 110%. The subsequent server generations decreased in variability to offer more stable performance. Generation 4 variability decreases by 83% over Generation 3, and Generation 5 variability decreases by 50% over Generation 4. The performance variability decreased from Generation 5 to 2 by 81%.
RAM TEST - Read from uncached RAM

10000
9000 8000
7000 6000
Mb/s 5000 4000 3000
Gen 2 Gen 3
Gen 4 Gen 5
2000
1000 0
Oct 12
Oct 13
Oct 14
Oct 15
The graph above compares RAM performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Read from uncached RAM benchmark test for each provider over a period of 15 days, with tw o (high and low) data points shown for each day. For more information on the Read from uncached RAM test, please see Appendix C.
AVERAGE 5100 5822 7394 8617
CV 5.0% 10.6% 1.9% 0.9%
15-DAY HIGH 5963 6737 7586 8799
Oct 26
Oct 16
Oct 17
Oct 18
Oct 19
Oct 20
Oct 21
Oct 22
Oct 23
Oct 24
Oct 25
15-DAY LOW 3873 4764 6836 8293
12
RAMspeed SMP
Aggregate of several tests that measure Copy, Scale, Add and Triad functions for both integer and floating point values. RAMspeeds SMP version adds functionality for multi-core systems to test the memory bandwidth. Server performance usually increases as newer technology is brought into the fold. According to the RAMspeed SMP test, the improvement from Generation 2 and 3 servers to Generation 4 and 5 servers is great. While Generation 3 improves upon Generation 2 with only a 7% increase, Generation 4 improves upon Generation 3 with a significant 49% increase. Generation 5 improves upon Generation 4 servers only slightly with a 6% increase in performance. As for performance variability, Generation 3 was the least stable for the RAM test again. Generation 3 servers were 213% more variable than Generation 2 servers. Generation 4 servers addressed the performance stability problem by decreasing variability ov er Generation 3 servers by 76%. Generation 5 servers decreased variability again, dropping 8% over Generation 4, and an overall 32% from Generation 5 over Generation 2.
RAM TEST - RAMspeed SMP

10000
9000 8000
7000 6000
Mb/s 5000 4000 3000
Gen 2 Gen 3
Gen 4 Gen 5
2000
1000 0
Oct 12
Oct 13
Oct 14
Oct 15
The graph above compares RAM performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the RAMspeed SMP benchmark test for each provider over a period of 15 days, w ith tw o (high and low) data points shown for each day. For more information on the RAMspeed SMP test, please see Appendix C.
AVERAGE 5467 5848 8701 9250
CV 2.4% 7.6% 1.8% 1.7%
15-DAY HIGH 6041 6602 8939 9509
Oct 26
Oct 16
Oct 17
Oct 18
Oct 19
Oct 20
Oct 21
Oct 22
Oct 23
Oct 24
Oct 25
15-DAY LOW 5087 5090 8129 8443
13
Disk Performance
Why Disk Performance Matters
The disk is where it all starts. All the data has to go somewhere, and the disk is where it is stored. Whether they are using traditional spinning hard drives, or newer solid-state drives (SSD), service vendors need to provide customers with reliable and efficient means of storing their data. The disk needs to be able to quickly and reliably read/write the data it receives or that is requested from the other components of th e server. The most common measurement for storage system performance is the input/output operations per second (IOPS) which measures how long the system has to wait as data is read from, or written to, the disk. As the quantity of data stored and analy zed scales upwards, every single IOPS a disk can handle will carry greater significance. Note: While the primary focus of the report is based upon the improvements in performance from newer Intel processors, the storage system is important to examine as well when addressing the general topic of server performance. The disk performance testing was limited to Generation 2, 3 and 4 because Generation 5 was tested on pre-production servers that did not have production level disk subsystems. When Generation 5 servers will be brought to into live production, the servers will be hosted on similar, if not better, storage systems as Generation 4, thus the Generation 4 disk results can also be anticipated for Generation 5.
14
Local Copy
The Local Copy test performs a simple copy of a 10 GB file on the same storage device. It is a combination of sequential read and write, which is common for file uploading, downloading, and sharing. Generation 2 and Generation 3 servers have similar levels of performance relative to the significantly improved performance of Generation 4 servers. The average performance of Generation 3 servers decreased from Generation 2 by 19%, while performance variability increased by 237%. Generation 4 performance improved over Generation 3 by 255%, and Generation 2 by 200%. Variability for Generation 4 was negligibly higher than Generation 3 by 2%, but higher than Generation 2 by 244%. The overall performance increase outweighs the increase in variability for Generation 4 compared to Generation 2 as the 15-day low of Generation 4 is still 55% higher than the 15-day high of Generation 2.
DISK TEST - Local Copy

250
200
150
Mb/s Gen 2 100 Gen 3
Gen 4 50
Oct 12
Oct 13
Oct 16
Oct 17
Oct 20
Oct 21
Oct 24
Oct 25
Oct 14
Oct 15
Oct 18
Oct 19
Oct 22
Oct 23
The graph above compares disk performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Local Copy benchmark test for each provider over a period of 15 days, with tw o (high and low) data points shown for each day. For more information on the Local Copy test, please see Appendix C.
PROVIDER 2 nd Generation 3 rd Generation 4 th Generation
AVERAGE 53 45 159
STANDARD DEVIATION 3 7 26
CV 4.8% 16.3% 16.6%
15-DAY HIGH 55 53 196
Oct 26
15-DAY LOW 40 6 62
15
Sequential Read
A large test file is created on the disk under test. During the test period the file is read several times sequentially. The Sequential Read test shows Generation 4 offering higher performance over the Generation 2 and 3 offerings, but also higher variability of performance. Generation 3 performance was 14% lower than Generation 2, while Generation 4 performance was 167% higher than Ge neration 3, and 129% higher than Generation 2. Variability increased for Generation 3 over Generation 2 by 104%, while Generation 4 variability increased over Generation 3 by a smaller 9%.
DISK TEST - Sequential Read

350
300 250 Mb/s
200
Gen 2 150 100
Gen 3 Gen 4
50
0
Oct 12
Oct 13
Oct 16
Oct 17
Oct 20
Oct 21
Oct 24
Oct 25
Oct 14
Oct 15
Oct 18
Oct 19
Oct 22
Oct 23
The graph above compares disk performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Passmark Sequential Read benchmark test for each provider over a period of 15 days, with tw o (high and low) data points shown for each day. For more information on the Passmark test, please see Appendix C.
AVERAGE 108 92 246
CV 7.5% 15.4% 16.8%
15-DAY HIGH 116 111 303
Oct 26
15-DAY LOW 72 43 102
16
Sequential Write
A large file is written to the disk under test. The 500MB file is written sequentially. The Sequential Write test shows Generation 4 offering significantly higher performance over the Generation 2 and 3 offerings. Generation 3 performance was 12% lower than Generation 2, while Generation 4 performance was 133% higher than Generation 3, and 104% higher than Generation 2. Performance variability increased for Generation 3 over Generation 2 by 163%. Generation 4 variability decreased over Generation 3 by 50%, but increased over Generation 2 by 31%. The performance of the disk write speed of Generation 4 compared to Generation 2 outweighs the variability of performance as the 15-day low of Generation 4 is 32% greater than the 15-day high of Generation 2.
DISK TEST - Sequential Write

200 180
160 140
120 Mb/s 100
Gen 2
Gen 3 Gen 4
80 60
40 20

Oct 14 Oct 15 Oct 18 Oct 19 Oct 22 Oct 23 Oct 26
The graph above compares disk performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Passmark Sequential Write benchmark test for each prov ider over a period of 15 days, with tw o (high and low) data points shown for each day. For more information on the Passmark test, please see Appendix C.
AVERAGE 82 72 167
CV 6.3% 16.5% 8.2%
15-DAY HIGH 87 86 185
15-DAY LOW 54 21 115
17
Random Seek + R/W

A large test file is created on the disk under test. The file is read randomly; a seek is performed to move the file pointer to a random position in the file, a 16KB block is read or written then another seek is performed. The amount of data actually transferred is highly dependent on the disk seek time. The results of the Random Seek + R/W test show Generation 4 servers more than doubling the performance of Generation 2 and Ge neration 3 servers. Generation 3 has an average performance slightly greater than Generation 2, by a negligible 2% increase. Generation 4 has a performance increase of 113% and 118% over Generation 3 and Generation 2 respectively . Performance variability is highest on Generation 3, with a CV near 20%. Generation 3 variability increases by 234% over Generation 2, while Generation 4 variability decreases by 40% over Generation 3. Generation 4 still has greater variability compared to Generation 2, increasing by 61%.
DISK TEST - Random Seek + R/W

250
200
150 Mb/s
Gen 2 100
Gen 3 Gen 4 50

Oct 14 Oct 15 Oct 18 Oct 19 Oct 22 Oct 23 Oct 26
The graph above compares disk performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Passmark Random Seek + R/W benchmark test for each provider over a period of 15 days, with tw o (high and low) data points show n for each day. For more information on the Passmark test, please see Appendix C.
AVERAGE 87 89 191
CV 5.9% 19.7% 7.6%
15-DAY HIGH 96 133 207
15-DAY LOW 59 39 110
18
Disk Mark
Average of the previous three benchmarks. The average is scaled to be indexed as a score on a scale created by Passmark to relationally compare disk performance of different systems. The Disk Mark test shows the overall disk performance of Generation 4 as a significant improvement over Generation 2 and Generation 3. Generation 3 performed lower than Generation 2 by 9%, while also 166% higher in variability. Generation 4 performed higher than Generation 3 by 139%, and Generation 2 by 118%. Generation 4 also had 40% decreased variability over Generation 3, but a 61% increase over Ge neration 2. However, the overall higher performance of Generation 4 should outweigh the increased variability compared to Generation 2. Examining the 15-day low of Generation 4 compared to the 15-day high of Generation 2, there is a 38% increase in performance.
DISK TEST - Disk Mark

3000 2500 2000
Score 1500 1000 500 0 Gen 2 Gen 3
Gen 4
Oct 12
Oct 13
Oct 14
Oct 15
Oct 21
Oct 22
Oct 23
Oct 24
Oct 25
Oct 16
Oct 17
Oct 18
Oct 19
Oct 20
The graph above compares disk performance on Windows servers betw een Expedients 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Passmark Dis k Mark benchmark test for each prov ider ov er a period of 15 days, with tw o (high and low ) data points shown for each day. For more information on the Passmark test, please see Appendix C.
AVERAGE 1000 915 2184
CV 5.6% 14.9% 9.0%
15-DAY HIGH 1060 1182 2462
Oct 26
15-DAY LOW 685 375 1468
Note: numbers expressed as calculated score.
19
Appendix
Appendix A Methodology Server Setup:
The following server configurations were set up on each environment. The Intel CPU model that each envir onment uses is also detailed. Expedient Generation 2
http://www.expedie nt.com/products/c loud-computing/public.php CPU: Intel Xeon Processor E5520 2.2 7GHz OS: Windows Server 2008 R2 vCPUs: 4 RAM: 8GB Disk: 50GB
Expedient Generation 3
http://www.expedie nt.com/products/c loud-computing/public.php CPU: Intel Xeon Processor X5650 2.67GHz OS: Windows Server 2008 R2 vCPUs: 4 RAM: 8GB Disk: 50GB
http://www.expedient.com/products/c loud-computing/public.php CPU: Intel Xeon Processor E52670 2.60GHz OS: Windows Server 2008 R2 vCPUs: 4 RAM: 8GB Disk: 50GB
http://www.expedient.com/products/c loud-computing/public.php CPU: Intel Xeon Processor E52670 v2 2.50GHz OS: Windows Server 2008 R2 vCPUs: 4 RAM: 8GB Disk: 50GB
Tests Used:
The benchmarks used to test the providers listed in this report are deriv ed from a number of sources. Most of the tests use benchmarks fr om the CPU and disk tests of PassMark Software. For descriptions of test used, please see the Test Descriptions below. CPU

RAM
Integer Math Floating Point Math Compression Phy sic s Encry ption
Storage
Read Cached Read Uncached Write Large Block RAMspeed SMP Disk Mark Random Seek & R/W Sequential Read Sequential Write Local Copy
For the descriptions and details of each test, please see Appendix C.
Timeframe:
The test period ranged from 10/12/2013 though 10/26/2013.
Data Collection:
Performance data for each test was collected 10 times per day, every day throughout the above testing periods. Data shown in the graphs only display the highest and lowest points of each day. Statistical measures in the charts were calculated using all data points to obtain more accurate estimations of variability. Cloud Spectator obtains cloud servers by purchasing the server space dir ectly from the providers as any user would. For certain providers, the client may reimburse Cloud Spectator for the server space needed for data collection relevant to that active project. Cloud Spectator collects and compiles the data into the CloudSpecs database and translates it into a visual display.
20
Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4 and 5 - November 2013
Appendix B Terms and Definitions

For the purpose of understanding the relational values of the data, all numbers within the tables below each graph are expressed as whole numbers except the percentages, which are expressed to the tenth decimal point. Percentages are expressed in that manner to account for instances when the coefficient of variation (CV), which is expressed as a percentage, falls below 1%, indicating a high degree of performance predictability.
Average:
When describing averages, Cloud Spectator refers to the average numerical value over a period of 15 days from October 12th until October 26th, 2013, taking into account each and every data point. Average scores can be found inside the tables underneath each graph within this document. The average is used to summarize the data in a simplified overview.
Standard Deviation:
The standard deviation is calculated over a period of 15 days from October 12th until October 26th 2013. The standard deviation can be found inside the tables underneath each graph within this document. The standard deviation is used to understand the amount of variation from the average benchmark score; i.e., how predictable a server s performance is for that test. The standard deviation can only be used to understand the amount of variation within a certain environment, and cannot be used to compare different environments because of performance differences.
Coefficient of Variation (CV):
The coefficient of variation is expressed as a percentage. The CV can be found inside the tables underneath each graph within this document. The CV is a measure of precision. It normalizes the standard deviation as a percentage of the average, which can be compared across providers. A lower CV means more stable performance.
[(Standard Deviation) / (Average)] * 100
15-Day Highs and Lows:
From the tested period between October 12th and October 26th 2013, Cloud Spectator extracts and presents the highest and lowest achieved scores by each provider in the tables underneath the graphs within this document.
Appendix C Test Descriptions

The following tests were taken from PassMark Software to measure the CPU performance in different Expedient envir onments: Compression Test: The Compression Test measures the speed that the CPU can compress blocks of data into smaller blocks of data without losing any of the original data. The result is reported in Kilobytes per Second. This test uses complex data structures and complex data manipulation techniques to per form a function that is very common in software applications, ranging from backup software to Email software. The compression test uses an Adaptive encoding algorithm based on a method described by from Ian H. Witten, Radford M. Neal, and John G. Cleary in an article calle d Arithmetic Coding for Data Compression. The system uses a model which maintains the probability of each symbol being the next encoded. It reports a compression rate of 363% for English text, which is slightly better than the classic Huffman method. This tests uses memory buffers totaling about 16kb per core. Encryption Test: The Encryption Test encrypts blocks of random data using several different encryption techniques, such that the resulting data can only be accessed by someone with the encryption key. It also tests the computer s ability to create a hash of the data, which is also a common cryptographic technique that can be used to ensure the contents of data are not tampered with. The methods used are TwoFish, AES, Salsa20 and SHA256. This test uses many of the techniques in the math tests, but also uses a large amount of binary data manipulation and CPU mathematical functions like 'to the power of'. Encryption is a very useful benchmark, as it is now very widely used in software applications, ranging from Intern et browsers, communications software and many different business applications. This tests uses memory buffers totaling about 1MB per core. Where available, the test will make use of specialized CPU instruction sets to accelerate performance, such as Intel AES-NI for the AES test.
Copyright Cloud Spectator, LLC 2013. All rights reserved.
Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4 and 5 - November 2013 Floating Point Math Test: The Floating Point Math Test performs the same operations as the Integer Math Test, however, with floating point numbers. A floating point number is a number with a fractional part (ie. 12.568). These kinds of numbers are handled quite differently in the CPU compared to Integer numbers as well as being quite commonly used, therefore they are tested separately. This tests uses memory buffers totaling about 240kb per core. Integer Math Test: The Integer Math Test aims to measure how fast the CPU can perform mathematical integer operations. An integer is a whole number with no fractional part. This is a basic operation in all computer software and provides a good indication of 'raw' CPU throughput. The test uses large sets of random 32-bit and 64-bit integers and adds, subtracts, multiplies and divides these numbers. This tests uses integer buffers totaling about 240kb per core. Physics Test: The Physics Test uses the Tokamak Physics Engine to perform a benchmark of how fast the CPU can calculate the physics interactions of several hundred objects colliding. This tests uses memory buffers totaling about 30MB per core. Read from cached RAM: This test measures the time taken to read a small block of memory. The block is small enough to be held entirely in cache. Read from uncached RAM: This test measures the time taken to read a large block of memory. Measured in GB/s, larger values are better. The test will uses a 256 MB block. The block is too large to be held in cache. Reads on 32-bit system uses DWORDS and QWORDS on 64-bit version of the software. Write large RAM blocks: This test measures the time taken to write information into memory. Measured in GB/s, larger values are better. Similar to the read uncached test, except writing is performance instead. The test also uses a 256 MB block. RAMspeed SMP: Aggregate of several tests that measure Copy, Scale, Add and Triad functions for both integer and floating point values. Copy Transfers data from one memory location to another (A = B). Scale Multiplies the data with a constant value before writing it (A = Bn). Add Reads results from two different locations, adds those results and writes then to the new location (A = B + C). Triad Merges Add and Scale. It reads data from the first memory location, scales it (multiplies it), then adds data from the second one and writes to the new location (A = Bn + C). Local Copy: Copy of a 10 GB file to the same disk Sequential Read: A large test file is created on the disk under test. During the test period the file is typically read several times sequentially. Sequential Write: A large file is written to the disk under test. The 500MB file is written sequentially from start to end. Test conditions are otherwise the same as the read test. Random Seek + R/W: A large test file is created on the disk under test. The file is read randomly; a seek is performed to move the file pointer to a random position in the file, a 16KB block is read or written then another seek is performed. The amount of data actually transferred is highly dependent on the disk seek time. Disk Mark: Average of the previous three benchmarks. The average is then scaled up by multiplying the average with a 'magic' number in order to make the number larger.
Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4 and 5 - November 2013
About Cloud Spectator

Cloud Spectator is the premier, international cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud Spectator has monitored the cloud infrastructure (IaaS) industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 3 times a day, 365 days a year with use of open source benchmark tests. Currently, the CloudSpecs system actively tracks 20 of the top IaaS providers around the world. Cloud Spectator 800 Boylston Street, 16th Floor Boston, MA 02199 Website: www.cloudspectator.com Phone: (USA) 1-617-300-0711
Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and other countries.

Processor Analysis: Generational Performance of Intel® Processors in An Expedient Cloud

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Processor Analysis: Generational Performance of Intel® Processors in An Expedient Cloud

Hochgeladen von

Copyright:

Verfügbare Formate

Processor Analysis: Generational Performance of Intel Processors in an Expedient Cloud

By Cloud Spectator November 2013

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

Table of Contents Introduction Executive Summary Processor Performance

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

Floating Point Math

CPU TEST - Floating Point Math

0 Oct 12 Oct 14 Oct 16 Oct 18 Oct 20 Oct 22

PROVIDER 2 nd Generation 3 rd Generation 4 th Generation 5 th Generation

AVERAGE 3208 3910 5260 5756

STANDARD DEVIATION 185 138 145 14

CV 5.8% 3.5% 2.8% 0.2%

15-DAY HIGH 3335 4143 5464 5785

15-DAY LOW 2430 3591 4675 5691

Note: numbers expressed in millions of operations per second.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

CPU TEST - Integer Math

0 Oct 12 Oct 13 Oct 14 Oct 15 Oct 20 Oct 21 Oct 22 Oct 23

PROVIDER 2 nd Generation 3 rd Generation 4 th Generation 5 th Generation

AVERAGE 5911 7208 7513 7909

STANDARD DEVIATION 94 134 65 13

CV 1.6% 1.9% 0.9% 0.2%

15-DAY HIGH 5976 7412 7615 7938

15-DAY LOW 5508 6930 7325 7849

Note: numbers expressed in millions of operations per second.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

CPU TEST - Compression

PROVIDER 2 nd Generation 3 rd Generation 4 th Generation 5 th Generation

AVERAGE 5276 6306 7044 7528

STANDARD DEVIATION 216 176 136 24

CV 4.1% 2.8% 1.9% 0.3%

15-DAY HIGH 5422 6621 7232 7558

15-DAY LOW 4328 5805 6506 7471

Note: numbers expressed in kilobytes processed per second.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

CPU TEST - Encryption

PROVIDER 2 nd Generation 3 rd Generation 4 th Generation 5 th Generation

AVERAGE 627 956 944 1092

CV 3.9% 2.9% 2.1% 0.3%

15-DAY HIGH 648 1003 971 1096

15-DAY LOW 492 863 881 1085

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

CPU TEST - Physics

Gen 2 300 200 100 Gen 3

0 Oct 12 Oct 13 Oct 14 Oct 15 Oct 16 Oct 17

PROVIDER 2 nd Generation 3 rd Generation 4 th Generation 5 th Generation

AVERAGE 260 320 377 475

CV 4.7% 7.8% 4.7% 0.4%

15-DAY HIGH 284 360 411 481

15-DAY LOW 221 264 326 469

Note: numbers expressed in frames per second.

Copyright Cloud Spectator, LLC 2013. All rights reserved

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013