Direct naar content

Database performance cloud providers, CloudReport 2020

The transition to the cloud is in full swing. Private, public, hybrid, IaaS, PaaS, DbaaS; the cloud has countless forms and applications. But which cloud provider and which machines match best with your environment, and especially with your database? CockroachLabs – the company behind CockroachDB – recently published their annual Cloud Report. They have conducted an extensive and thorough benchmark with the three most well-known cloud providers: AWS, Azure and GCP. The benchmark is about a database instance on a cloud server and not DBaaS. In this blog Edco Wallet shares a summary of the Cloud Report 2020.

Edco Wallet

Co-Founder & eigenaar
Edco Wallet - Co-Founder & eigenaar

Cloud database setup

To get straight to the point: there is not one best advice that applies to everyone. For every database configuration and setup there are different wishes and there is a story for each use case. But this report can help you choose your cloud provider and the machines you are going to use. It gives you tips on how to take a closer look at your current cloud setup. Although for CockroachDB itself Storage I/O is an important performance indicator, they have run the benchmark across the board, so you can distill your indicators from it for any type of database.

Starting points

The report compares the performance of AWS, Azure and GCP (Google Cloud Platform) on a range of microbenchmarks and workloads to measure the performance of the three cloud providers with their available machine types. The benchmark focuses on IaaS, i.e. the purchase of a server in IaaS services from AWS, Azure or GCP on which you install your own database. The report is broadly representative of the performance of the most common relational databases (such as PostgreSQL, SQL Server, and CockroachDB itself) but obviously not 100% exhaustive for all usage situations. For example, machine learning focussed users would probably want to use a different set of benchmarks when comparing cloud provider performance.

How was the test performed?

The test consists of a thousand benchmark tests performed on 26 of the latest and best performing machine types available from AWS, Azure and GCP on three axes: CPU, Network and Storage I/O. After reviewing the microbenchmarks, the latest top class machine types from each cloud provider were selected and benchmarked on the industry standard TPC-C. CockroachLabs has made all test methods and scripts available to reproduce the benchmark yourself or adapt it for own testing. The source code is also available. The tools used include stress-ng, sysbench, iPerf and ping.

Results CPU test

The best performing Azure machines achieved significantly better results on the CPU microbenchmark. The difference between the CPUs of the cloud providers was that although all machines have 16 vCPUs, the best performing Azure machines contain 16 cores with one thread per core. The other cloud providers use hyperthreading across all instances and have eight cores with two threads per core to reach 16 vCPUs.

Results CPU test AWS, Azure en GCP

Network Throughput

The network of GCP looks much better than that of AWS or Azure. Not only do their best-performing machines beat the best-performing machines on any other network, even their least-performing machine is consistent with the maximum network throughput of AWS, as shown in the tests. This is particularly impressive since AWS outperformed GCP in the network tests last year.

Network Throughput AWS, Azure, GCP

Network latency

In addition to the throughput, the network latency has also been tested. Without testing for latency, significant delays can be overlooked because they are masked by the overall performance. Latency limits, for example, the performance of individual operations. When using Azure at two locations (such as a data center in Belgium and one in the Netherlands) it will regularly happen that data is not moved directly between the two locations, but goes back and forth across the ocean. This makes a considerable difference in performance and this type of latency can cause a lot of problems with, for example, replication.

Even the best machine on Azure is more than five times worse than on AWS or GCP. GCP has improved its network latency extremely well since last year, but AWS is still head and shoulders the best. On average, all AWS machine types offer low network latency and the top machines outperform their competitors.

Network latency AWS, Azure en GCP

Storage I/O

The importance of storage I/O depends on the application. For CockroachDB, an application that always wants to read and write to persistent storage, this is crucial. For other stateless applications, storage performance does not make such a difference. There is a variety of different storage technologies available, from classic hard disks and modern SSDs to network attached storage and replicated storage. Even if you are running in the cloud, there are a number of choices that can be made with regard to the form of storage.

After comparing all three top machines from the cloud providers, AWS offers superior write performance with the type i3and machine.

Storage I/O Maximum write throughput AWS, Azure and GCP

Storage I/O Write latency and iOPs AWS, Azure and GCP