In this blog, Edco Wallet shares a summary of the Cloud Report 2020, in which the three best-known cloud providers are compared by CockroachLabs. Which one is the best?
The transition to the cloud is in full swing. Private, public, hybrid, IaaS, PaaS, DbaaS; the cloud has countless forms and applications. But which cloud provider and which machines match best with your environment, and especially with your database? CockroachLabs - the company behind CockroachDB - recently published their annual Cloud Report. They have conducted an extensive and thorough benchmark with the three most well-known cloud providers: AWS, Azure and GCP. The benchmark is about a database instance on a cloud server and not DbaaS. In this blog Edco Wallet shares a summary of the Cloud Report 2020.
Cloud database setup
To get straight to the point: there is not one best advice that applies to everyone. For every database configuration and setup there are different wishes and there is a story for each use case. But this report can help you choose your cloud provider and the machines you are going to use. It gives you tips on how to take a closer look at your current cloud setup. Although for CockroachDB itself Storage I/O is an important performance indicator, they have run the benchmark across the board, so you can distill your indicators from it for any type of database.
The report compares the performance of AWS, Azure and GCP (Google Cloud Platform) on a range of microbenchmarks and workloads to measure the performance of the three cloud providers with their available machine types. The benchmark focuses on IaaS, i.e. the purchase of a server in IaaS services from AWS, Azure or GCP on which you install your own database. The report is broadly representative of the performance of the most common relational databases (such as PostgreSQL, SQL Server, and CockroachDB itself) but obviously not 100% exhaustive for all usage situations. For example, machine learning focussed users would probably want to use a different set of benchmarks when comparing cloud provider performance.
How was the test performed?
The test consists of a thousand benchmark tests performed on 26 of the latest and best performing machine types available from AWS, Azure and GCP on three axes: CPU, Network and Storage I/O. After reviewing the microbenchmarks, the latest top class machine types from each cloud provider were selected and benchmarked on the industry standard TPC-C. CockroachLabs has made all test methods and scripts available to reproduce the benchmark yourself or adapt it for own testing. The source code is also available. The tools used include stress-ng, sysbench, iPerf and ping.
Results CPU test
The best performing Azure machines achieved significantly better results on the CPU microbenchmark. The difference between the CPUs of the cloud providers was that although all machines have 16 vCPUs, the best performing Azure machines contain 16 cores with one thread per core. The other cloud providers use hyperthreading across all instances and have eight cores with two threads per core to reach 16 vCPUs.
The network of GCP looks much better than that of AWS or Azure. Not only do their best-performing machines beat the best-performing machines on any other network, even their least-performing machine is consistent with the maximum network throughput of AWS, as shown in the tests. This is particularly impressive since AWS outperformed GCP in the network tests last year.
In addition to the throughput, the network latency has also been tested. Without testing for latency, significant delays can be overlooked because they are masked by the overall performance. Latency limits, for example, the performance of individual operations. When using Azure at two locations (such as a data center in Belgium and one in the Netherlands) it will regularly happen that data is not moved directly between the two locations, but goes back and forth across the ocean. This makes a considerable difference in performance and this type of latency can cause a lot of problems with, for example, replication.
Even the best machine on Azure is more than five times worse than on AWS or GCP. GCP has improved its network latency extremely well since last year, but AWS is still head and shoulders the best. On average, all AWS machine types offer low network latency and the top machines outperform their competitors.
The importance of storage I/O depends on the application. For CockroachDB, an application that always wants to read and write to persistent storage, this is crucial. For other stateless applications, storage performance does not make such a difference. There is a variety of different storage technologies available, from classic hard disks and modern SSDs to network attached storage and replicated storage. Even if you are running in the cloud, there are a number of choices that can be made with regard to the form of storage.
After comparing all three top machines from the cloud providers, AWS offers superior write performance with the type i3and machine.
Just like write performance, AWS wins with read performance in all categories with its i3 machine type.
TPC-C is a popular OLTP benchmark tool that simulates an e-commerce company with a (large) number of different warehouses processing multiple transactions simultaneously.
TPC-C is measured in two different ways. One is throughput per minute type C (tpmC, also known as the number of orders processed per minute). The other is the total number of supported warehouses.
Cost efficiency is just as important as performance. If top performance costs two to three times more than the average performance, it may not be worth it. For this reason, TPC-C is usually measured in terms of price per tpmC. This makes it possible to make a fair comparison both across and within the cloud providers. In this analysis the standard on-demand prices are used, to keep it simple.
All three cloud providers are close to each other with the price per rpmC. However, this year we see that the GCP n2-highcpu-16 offers the best performance per dollar. If money doesn't matter then AWS is definitely the best performer on throughput.
GCP shows a great improvement in the 2020 Cloud Report with AWS and Azure on price per performance of TPC-C, but is below the performance of AWS and Azure regarding max. tpmC on a three node cluster. If network latency is an important factor in your configuration then AWS is by far is the best choice right now.
These kind of tests and benchmarks are a snapshot and the performance can change from month to month because the cloud providers of course change and update the hardware to keep up with the race for performance. That’s why it is good to keep comparing regularly in relation to the desired or applied configuration.
CockroachLabs has shown to be able to present a very comprehensive, broad and thorough analysis. An absolute must to use this report as a guide in decision making or to form a better and objective picture. As OptimaData, we closely follow the developments of CockroachDB and appreciate the Cloudreport 2020 of CockroachLabs as one of the best benchmarks of the moment.
You can download the full report here.
Before publication of the Cloudreport 2020 CockRoachLabs did inform the cloudproviders. They all respond on the CloudReport 2020 and also gave comments and input for the next cloudReport testing. There is a discussion about using specific tuning and performance configurations but CockRoachLabs did use only default configurations. Mainly because the options to configure and tune your setup is broad, complex and differ. In this blogpost you can read their response and comments.
Want to know more?
Do you have a database environment in the cloud and would you like to have an assessment of how the configuration is performing or advice on what could be improved? Or do you have plans to migrate to a cloud setup? Feel free to contact us, we are happy to think along with you.