Who offers the best value for money?

Comparison of public cloud offerings
Cloud Benchmark 2021: Who offers the best value?



By Michael Herman *

providers on the subject

Infrastructure-as-a-Service (IaaS) determines the pulse of digitization: Powerful processors (vCPUs) and virtual machines (VMs) from the public cloud are an integral part of many companies’ IT strategy. Providers are constantly adapting the technologies and services. The performance of the cloud services varies accordingly.

CPU, network and storage technologies are constantly evolving, resulting in price / performance differences between the top four public cloud service providers.

(Image: Igor Link – stock.adobe.com)

As of 2020, Cloud Mercato reviewed a series of comparable 16 vCPU VMs and block storage offerings from Microsoft Azure, Amazon Web Services (AWS), Google Cloud and Open Telecom Cloud, implemented from European data centers. In addition to this basic test panel, analysts also examined additional VMs with hyperscales for specific use cases. The benchmark focused on key parameters such as computing power, memory, bandwidth, OpenSSL encryption and relational databases. The study was commissioned by T-Systems in July 2021.

Computer power: Geekbench shows differences in performance

MIPS (millions of instructions per second) or FLOPS (floating point operations per second) as cited by CPU vendors are insufficient to fully understand the performance of CPUs and their performance differences in real-world cloud workloads. Therefore, analysts used the benchmark suite Geekbench 5.

Most of the VMs in the base test panel with comparable 16 vCPU VMs delivered a performance value of around 8,000 for multi-threading, the simultaneous processing of multiple threads in one process. From the second panel with additional VMs, the AWS m6i achieved a performance boost with a score of more than 10,000, while Azure and Google Cloud scored just over 8,000. However, the Hyperscales’ World Cups were surpassed by all World Cups from Open Telekom Cloud: They came up with values ​​of over 12,000 and thus a performance increase of more than 50 percent compared to the average.

Local bandwidth in the network: Great variety

To measure the local bandwidth in the network, the analysts used the iPerf 3 tool with the TCP (Transmission Control Protocol). For the test setup, they each used two identical VMs in the same data center. The number of threads to be processed corresponded to the number of CPUs to generate maximum throughput.

Google Cloud showed a performance of more than 29 Gbit / s, but with high fluctuations, especially compared to the theoretical maximum bandwidth. The network-optimized VMs from AWS (c5n, m5n, r5n) provided a significant increase in performance over the general variants of AWS by more than doubling the network bandwidth from about 10 to about 24 Gbit / s. However, AWS was still below the standard level for the corresponding c4 variant from Open Telekom Cloud, which comes to around 24.5 Gbit / s. Azure VMs had the lowest network performance score.

Significantly different encryption speed

Data encryption makes a significant contribution to secure, compatible use of the cloud and affects the performance of services: How many bytes can be encrypted or decrypted with a 16 vCPU VM in one second? To find out, use the Cloud Mercato open source tool OpenSSL Speed, which analyzes encryption and decryption using various algorithms and block sizes.

The test results show that Open Telekom Cloud is well ahead of the three hyperscales for both AES-256-CBC and SHA-512 encryption – with about 60 percent more performance, regardless of algorithm and block size. The difference is particularly large with a block size of 16,384 bytes and AES-256-CBC encryption: Here the range of performance ranges from Azure’s standard E16 at around 220,000 encrypted bytes per second to the s3 variant of Open Telekom Cloud, which with approx. 2.7 million bytes, has a speed that is more than 10 times higher.

RAM performance and block storage with significant range ranges

To evaluate the performance of the main memory (RAM), the analysts tested the bandwidth between CPU and memory in read and write access with Sysbench. The AWS variant c5 showed the best writing performance of the test panel, while the E16-4s from Azure showed the lowest performance. With more than 71,000 devices, the c5 also delivered good reading performance – surpassed by Google’s n2 standard (72,000) and the m4 variant of the Open Telecom Cloud (75,000). None of the VMs from the additional set for specific use cases (see diagram) achieved these general VM values.

To measure sustained block storage, analysts used Flexible I / O Tests (FIOs) to determine the number of IOPS (input / output per second). Google topped with more than 50,000 IOPS, where the writing performance exceeded the reading performance by about 10,000 IOPS. In second place here is Open Telekom Cloud with 20,000 IOPS.

Relational database services

Cloud Mercato tested the performance of relational database services with the Sysbench OLTP benchmark tool. To explore different applications and scenarios, Cloud Mercato has scaled from 1 to 128 clients using read-only, read-only and read / write modes. Google performed well in writing mode, but showed significant performance drops at high scaling. Open Telecom Cloud showed the best reading performance – especially with a high number of clients – with around 195,000 read accesses per second with 128 clients, followed by AWS with almost 150,000.

Performance, price and utility: who does best?

To realistically assess the usefulness of cloud services, it is important to consider not only the performance but also the price: How much performance do users get for one penny? Cloud Mercato has set this price / performance value based on the provider prices on August 1, 2021.

As in the previous year, a World Cup from Open Telekom Cloud did best. At the time of testing, the s3 variant offered about 15 percent more value for money compared to the second-best offer on the market. Meanwhile, the hyperscales have caught up and improved their price-performance ratio compared to the Open Telecom Cloud.

Nevertheless, the analyzes show that additional VMs for specific use cases are not automatically the best choice. General purpose World Cups, which are suitable for all purposes, can also master specific workloads at a favorable price-performance ratio. As in the previous year, Cloud Mercato analysts came to the conclusion: “The offers from Open Telekom Cloud are well-balanced for all applications and are on a par with the hyperscales in terms of both performance and price-performance ratios – and even surpasses competition. for the areas of encryption and CPU performance. ” Users therefore do not have to worry about choosing the right VM variant: “Open Telekom Cloud s3 and c4 is a good and consistent choice for almost all cases, with s3 with a plus of 15 percent compared to the second best offer compared to value for money.”

You can download the complete Cloud Mercato survey with all measurement data for the surveyed offers and all price lists after registration.

Even with AI at eye level

Another study from Cloud Mercato shows that virtual machines from Open Telecom Cloud can also withstand artificial intelligence (AI) compared to hyperscalers. In their “Performance & Benchmark of AI IaaS Providers”, analysts examined offers from Google, Azure, AWS and Open Telecom Cloud. In many AI applications such as object recognition, the performance of graphics cards (GPUs) is crucial for image processing and analysis. Therefore, the focus of the study besides CPU and RAM was primarily on the virtual GPU machines.

Result: Benchmark shows that providers differ only slightly in terms of the performance of their respective GPUs. Open Telecom Cloud is on par with hyperscales that offer similar performance and a similar price-performance ratio. “With its focus on GDPR compliance and the ModelArts Suite, Open Telecom Cloud is a strong European alternative to the development, training and operation of AI,” concludes the Cloud Mercato study.

* Author Michael Hermann works as a freelance journalist and author and writes primarily on ICT topics on behalf of companies and agencies.

(ID: 48402660)

Leave a Comment