In recent years, ‘someone else's computer in a different place’ is no longer a sufficient explanation of what the cloud actually is. Cloud-native has become a way of working. That doesn't just mean working ín the cloud, but it's about making full use of the technology that the cloud can offer. Issues such as scalability, efficiency, agility and cost saving motivate organizations to develop and apply more and more cloud-native technology. Martijn Wallet gave an interview at the KubeCon & CloudNativeCon conference in Valencia about how we at OptimaData apply cloud-native technology when working with clients.
From database consultants to database reliability engineers
Our customers have very different work methods. Some work with virtual machines, others use bare metal. Still others have chosen a more modern infrastructure such as Kubernetes. Thanks to our knowledge of the customer’s existing environment and our insight into the customer’s needs, we are able to create the best possible solution. This does require the necessary database experience and a true Devops mentality. Our database consultants are very experienced and do not shy away from a complex environment. They have thorough knowledge of and experience with multiple platforms. We think ‘database reliability engineers’ is a better term. After all, they ensure that our customers manage their database in the most efficient, consistent and cost-effective manner, including cloud databases and cloud native products.
Kubernetes’s flexible mechanism for implementing applications
The reason why customers choose Kubernetes is often because it allows them to implement and modify applications in an easy and maintenance-friendly manner. With the right helm charts and tooling, it provides a very flexible mechanism for implementing applications. Postgres is, in our experience, the most advanced open source database, especially if you run it in containers on Kubernetes. It has a small footprint and can be supplemented with PostGIS and TimescaleDB, for example. Some of our customers know about the possibility of running Postgres on containers, others are still hesitating and need to be convinced with strong arguments and a good Proof of Concept. With the improvements of ‘persistent storage’ in Kubernetes and the evolution of ‘orchestrators’ for all kinds of databases such as MySQL – with or without Galera – and Postgres, we can advise our customers optimally.
Kubernetes in production: it can be done!
People wonder if Kubernetes operators can be used in a production environment. We have done so successfully, for example at Royal Van Oord, an international maritime contractor. Royal Van Oord is a great party to work with. Royal Van Oord was already running on Crunchy Postgres containers, and the company was looking for extensive Postgres support. This was before the Crunchy operator was developed. In early 2019, the company asked us to support their Postgres containers and facilitate sound backup and recovery. We implemented this by using logical replication from the containers in Kubernetes to two Postgres instances on virtual machines.
The number of Postgres containers, or applications using a Postgres pod, increased from a handful to several dozen. The load varies between applications. Some are analytical, others involve online transaction processing (OLTP). The size can also vary greatly; some pods are hundreds of gigabytes, others are a few gigabytes. With this logical replication setup, we had a few challenges:
• The application life cycle (Postgres containers are restarted).
• The size of some Postgres containers, and initial data synchronization with logical replication takes a long time.
• Specific extensions like PostGIS and TimescaleDB used by Royal Van Oord's applications need some attention during initialization of logical replication.
Postgres Kubernetes Operator
As we worked together, and given the situation, we advised Royal Van Oord in early 2021 to deploy a Postgres Kubernetes Operator to deal with those challenges and the cumbersome maintenance. Before we did that, we wrote an opinion on the available operators at that time, in late 2020. There were three:
1. Zalando’s Postgres Kubernetes operator
2. The Crunchy data operator
3. EDB’s Could Native PostgreSQL operator.
Some of the factors we factored in were cost, support, convenience, flexibility, documentation, and, as far as can be determined, market share. We then arrived at EDB's Cloud Native PostgreSQL Operator.
Why EDB’s Cloud Native PostgreSQL Operator?
When selecting an operator, it is important to also look ahead to the future, not just at the current application. With the options available at the time, we felt strongly that in the long run, the EDB Cloud Native PostgreSQL (CloudNativePG) would be the most complete and future-proof fit for this project and for Royal Van Oord in general. The key factors in this decision were:
• EDB's Enterprise-grade support (and a large partner network).
• The well thought out operator.
• The support for bi-directional replication (the only operator with this capability).
• The operator also uses the Enterprise monitor. This provides a unified view of all clusters and nodes, both in k8s and on IaaS.
First, the very quick implementation of a cluster setup. It can literally be done in a few minutes. Also, deploying a pgbouncer setup is a matter of a few lines of code.
Second, thanks to the state fullness of the cluster, the operator ensures that the cluster is always displayed as defined, bringing the cluster closer to true High Availability with minimal human intervention.
Last but not least, the CloudNative Postgres Plugin, cnp, makes it very easy to manage a Postgres cluster on Kubernetes.
The major challenges in running databases in Kubernetes
These changes were not easy. It required good collaboration with the customer. It also called for a big change of mindset: from traditional implementations to an explanatory model, and from scripts to helm charts. Another challenge was that at that time there was no EDB-supported image with PostGIS and TimescaleDB extensions. Of course, there are the nodes, node pools, taint and affinity that can be configured. A good piece of advice here is to just experiment with it. There were also some challenges in terms of monitoring, alerts, and getting the right statistics on the dashboards. But the dashboards that come with the operator are a good starting point. You need to think about resource limits on nodes on which pods are run that are more resource intensive. That’s about it.
CloudNativePG has become open source
We have learned that EDB has chosen to open up Cloud Native Postgres as open-source software. We applaud that. When more people from the community are willing and able to contribute, the operator only gets stronger. From a purely financial business perspective, you might want to make a different choice, but in my opinion that should not stop a decision like this. I think we should be happy with a great new open-source product like the Cloud Native PostgreSQL operator, CloudnativePG.
Watch demo CloudNativePG and interview
Below you can watch the interview between Gabriele Bartolini from EDB and Martijn Wallet.
Through this link you can watch all of EDB’s session, starting with an introduction about the Postgres Kubernetes Operator, then a clear explanation and the open-source project, the interview with Martijn (from 25:55) and finally the demo.
Want to know more?
Curious if cloud-native technology can be useful for your organization and how it works exactly? Feel free to contact us without strings attached! We are happy to brainstorm with you.