The Bitsource conducted a review of the two cloud computing platforms, Rackspace Cloud Servers and Amazon Elastic Compute Cloud (EC2), to get a general idea of overall system performance. Included in the tests were pure computing power (CPU), and raw disk I/O throughput. Using a consistent testing methodology across most instance sizes over a two-month time span (a painstaking process requiring lots of patience) has resulted in the following comparison of CPU performance, disk performance and cost between the two platforms.

We hope that this evaluation will save the reader some effort by answering questions about how the two platforms compare in terms of time, power, efficiency and cost. The purpose of this article is to answer a simple question: How do the two platforms stack up against each other in terms of disk and CPU efficiency?

Test Methodology

Ten executions were run for each test type, at different times, with 5 tests being run one week on 5 different servers, and the following week on another 5, to distribute the tests over time. (The point of this technique was to ensure that, if the cloud platform was experiencing heavy usage at the time of the test, the cloud’s workload would not skew the test results.) Results for all test runs were averaged, graphed, and compared.

For example, for kernel compile tests, 5 virtual instances for each size server were spun up, the same test was executed against all five 5 servers, and then the results were recorded. The following week, the remaining 5 tests were executed in the same fashion. Finally, all results were averaged for each instance size, which constituted the final result.

The control variables for all tests were as follows: CentOS 5.2 x86_64 hardware, GCC version 4.1.2, and only local storage (no remotely-mounted volumes). In the case of EC2 Small and Medium instances, no image is available on EC2 for deploying the x86_64 version of CentOS to it, so a 32-bit CentOS 5.2 image was used in these specific cases.

When performance-testing cloud platforms, it is important to test multiple instances simultaneously, during different times of the day, and over multiple weeks to get a real measure of system performance for CPU and disk performance. The reason is that the capacity of each cloud platform could vary dramatically given different times of day and load on the platforms. Ultimately, good resource-distribution algorithms, system architecture, and capacity planning will shine through in the performance results between platforms. It should be noted that the amount and timing of samples is important to get a measure of overall system performance for cloud platforms.

How the CPU Tests Were Conducted

Typically it is best to test specific applications to get a true measure of performance, but in this case a generic measures of system performance were used to make a comparison between both platforms. SPEC CPU2006 benchmarks were considered, but their minimum requirements exceeded what was available on the systems used for this procedure. Thus, a test of compiling the Linux kernel was selected in order to determine a general measurement of overall system performance and measure the CPU usage to accomplish the specified task. The Linux kernel version used in all kernel compilation CPU tests was stable 2.6.31.5.

Compiling the Linux kernel is a compute-intensive task, which requires work from most of the CPUs. We matched the number of parallel compilation tasks to the number of cores on the machine, to demonstrate getting maximal efficiency out of the host tested.

All instance sizes on Cloud Servers have four CPU cores available; regardless of the instance size, there is no limit on the amount of compute power available to that specific virtual system based on size.

On Amazon EC2, different instance sizes (based on memory) have different configurations for the number and type of CPUs available.

To demonstrate the full capacity of a particular instance size, the number of parallel jobs executed for the compilation was matched to the number of CPU cores available to the instance. For example, if an instance had 4 CPU cores, the compilation command executed would be make –j4 bzImage to compile the Linux kernel image. This command creates 4 parallel jobs to compile the Linux source, resulting in gains in performance for the compilation task, and fully saturating the system’s CPUs for the task of compiling the Linux kernel.

How the Disk Tests Were Conducted

The Iozone (http://iozone.org) disk benchmark was selected as the tool for benchmarking file system performance. Version 3.327 was used for performing the tests. Iozone is an open source disk-performance utility that starts a timer, performs a transfer, and records throughput of the disk subsystem. Iozone was chosen for its flexibility and the fact that it is freely available for anyone who might want to try replicating the results.

Iozone was run using a fixed file size to exceed the operating system caches. Any file size of less than the amount of available memory on the system will typically only test the CPU and disk caches on the system; therefore, a file size greater than the system memory was chosen in order to exercise the physical I/O subsystem.

The global options for all tests began with the record length to be written on the test file; a record length of 32 kilobytes was selected. The tests run were write, read, reread, random read, and random write. The only specific variable was the actual size of the file, and it was set to a size greater than the system’s physical memory in order to ensure that we exercised the physical I/O subsystem for the disk I/O tests.

The Iozone command to perform disks tests was as follows:

where X is the only variable, consisting of the file size according to system memory, as mentioned earlier.

For instances with less than 4 gigabytes of system memory, a static file size of 6 gigabytes was used for the Iozone tests for this particular range of instance sizes.

For instances with a system memory size of 4 to 8 gigabytes, a static test file size of 10 gigabytes was used for the Iozone tests.

For servers with 8 to 15 gigabytes of memory, a static file size of 17 gigabytes was used for the Iozone tests.

Performance Results

Kernel Compilation Results

The time taken to compile the Linux kernel on all Cloud Server instances was consistent across all instance sizes. All Cloud Server instance types compiled the Linux kernel in approximately 3 ½ minutes, with the exception of the 256 MB instance, which took a little over 5 minutes to compile the kernel.

For the kernel compilation test on Amazon EC2, the results vary widely between instance sizes. The best case was the EC2 Extra Large High CPU instance, which came in under 2 minutes and did not reflect typical performance for EC2 instances on average. The worst case was the EC2 Small instance which took approximately 24 minutes to complete. On average, it takes nearly 8 ¼ minutes to compile across all EC2 instance sizes, which is greatly due to the EC2 small instance taking 24 minutes to complete.

Linux Kernel Compilation Time Results

Linux Kernel Compilation Time Platform Averages

On average, CPU user time used between Cloud Servers and EC2 on all instance sizes was about 45% on average and didn’t vary much between the two platforms. The CPU usage graphs during the Linux kernel compilation tests show that the Cloud Servers instances use about 6% more CPU system time on average for compilation of the Linux kernel.

Linux Kernel Compilation CPU Usage Results

Linux Kernel Compilation CPU Usage Platform Averages

Price versus Performance Comparison

Included is a spreadsheet demonstrating the price to compile the Linux kernel on the various instance sizes. The calculation was done with the following formula:
(Minutes to compile kernel) x (dollars per minute) * (100 Cents per Dollar) = Cost (in cents) to compile kernel.

Cost to Compile Linux Kernel on CloudServers and EC2 Instances

CloudServers presented a better advantage for cost of compilation weighing in at 1/13th of a cent to completion at a 256MB instance size, which was the most economical solution. EC2’s smallest offering was the Small Instance at 1.7 gigabytes of system memory, and the cost of compilation there was 3.5 cents, mainly due to the inability to run parallel jobs for the compilation, as it is a single core system. The most affordable EC2 solution was actually the higher cost-per-hour Medium High CPU instance. This is mainly because of the ability to run jobs in parallel using both cores that come with this instance size, reducing the time it took to complete the compilation task; therefore making the cost of compiling the kernel cheaper on the Medium instance versus the Small instance.

In the mid-range the 8 gigabyte CloudServers instance was slightly less expensive to completion than the corresponding EC2 Large instance with a total compilation cost of 2.62 cents versus 2.93; however, the EC2 Large instance was slightly cheaper at 1.67 cents versus the 8 gigabyte instance’s 2.62 cents. This is mainly due to the 8 CPU cores on the EC2 Extra Large instance versus the 4 CPU cores on the comparable 8 gigabytes instance.

The most expensive CloudServers solution in this test was the large instance at 5.21 Cents to completion. It’s corresponding instance on the EC2 side came in at a cost of 3.28 cents for the same task.

Disk I/O Test Results

Cloud Servers Disk Read Performance

Disk read I/O performance on Cloud Servers rated 128.5 megabytes per second at minimum, and 270.1 megabytes per second at maximum. The average read time rated 176.5 megabytes per second on average across all instance sizes.

Disk random read I/O performance on Cloud Servers rated 2.6 megabytes per second at minimum and 12.5 megabytes per second at maximum. The average random read time rated at 5.7 megabytes per second on average across all instance sizes.

EC2 Disk Read Performance

Disk read I/O performance on EC2 rated 82 megabytes per second at minimum, and 101.7 megabytes per second at maximum. The average read time rated 93.2 megabytes per second on average across all instance sizes.

Disk random read I/O performance on EC2 rated 4.3 megabytes per second at minimum, and 9.8 megabytes per second at maximum. The average random read time rated 6.1 megabytes per second on average across all instance sizes.

Cloud Servers Disk Write Performance

Disk write I/O performance on Cloud Servers rated 89.9 megabytes per second at minimum, and 162.7 megabytes per second at maximum. The average write time rated 129.5 megabytes per second on average across all instance sizes.

Disk random write I/O performance on Cloud Servers rated 12.3 megabytes per second at minimum, and 45.4 megabytes per second at maximum. The average random write time rated 24.2 megabytes per second on average across all instance sizes.

EC2 Disk Write Performance

Disk write I/O performance on EC2 rated 33.4 megabytes per second at minimum, and 51.7 megabytes per second at maximum. The average write time rated 43 megabytes per second on average across all instance sizes.

Disk random write I/O performance on EC2 rated 10.6 megabytes per second at minimum, and 13.4 megabytes per second at maximum. The average random write time rated 11.7 megabytes per second on average across all instance sizes.

Disk I/O Throughput Results

Disk I/O Throughput Platform Averages

Performance Analysis Summary

No solution is perfect in every case, and depending on the user’s requirements, one platform may be better suited based on its performance. This review is intended to give a general idea of the performance of both platforms, leaving the reader to conclude which platform best suits his or her needs. There are a few clear distinctions in performance for CPU usage and time spent to accomplish the task of compiling the Linux kernel. Disk I/O throughput also shows a clear winner. Many different cases were tested, with the end goal being to give the reader as much data as possible to make informed decisions about the available platforms.

In addition to the detailed results, we have summarized the performance results from a high level, to demonstrate a general analysis of performance at the overall platform level.

On average, Cloud Servers was more than twice as fast as Amazon EC2 at compiling the Linux kernel across all instance sizes. It should be noted that the EC2 small instance skews the average considerably, weighing in at 24 minutes to compile the Linux kernel. This difference comes heavily into play when paying by the hour for compute resources. Even when pricing is per hour, if completing a task such as compiling the Linux kernel takes longer, the pricing will increase in proportion to the amount of available compute resources.

The amount of available computing power on the Cloud Servers platform is independent of instance size and hourly price. This means that users can expect to receive the same amount of computing power, regardless of instance size or price. The primary difference between the instance sizes offered by Cloud Servers is the amount of system memory available, not computing power. With EC2 instances, on the other hand, the amount of available resources is closely tied to instance size and price per instance.

The cost of compilation on average was cheaper on CloudServers rather than EC2. CloudServers offers more smaller instance sizes, which can accomplish the same amount of work as the large instances at a much lower hourly rate. The cost of compilation on CloudServers at best was in fractions of a cent, whereas the most economical EC2 solution was 1.5 cents. For users with large memory requirements, such as those whom cannot spread their work over many small instances, EC2 was less expensive for compilation among extra large sizes comparable to CloudServers.
There are significant savings to be made when utilizing the smaller CloudServers, and there is a lower cost of entry on CloudServers that is not present on the standard Small and Medium EC2 instances.

Disk I/O results show that Cloud Servers consistently have much better write and random write performance than EC2 across most sizes. For read performance, Cloud Servers also have consistently higher read performance on average, except in the case of random read performance, in which case EC2 is about 0.4 megabytes per second faster on average. Overall, Cloud Servers have a higher disk throughput on average.

The Bitsource conducted an independent, third-party performance analysis of Rackspace CloudServers and Amazon EC2 on behalf of Rackspace Inc.  The results were not influenced in any way by Rackspace Inc. as presented using the methods in this analysis.