Which Managed Kubernetes Is Right for Me?

Kubernetes helps with scaling, deploying, and managing containerized workloads, facilitating a faster deployment cycle and configuration management—all while providing improved access control. Kubernetes, the founding CNCF project, is at the base of cloud-native and can be easily deployed through any cloud provider.

This blog will compare on-premises, or self-hosted, Kubernetes clusters to managed ones. It will also outline your options for Kubernetes in the cloud. To do this, we’ll look at ease of use and set-up, custom node support, cost, release cycles, version support, and more.

Managed vs. Self-Hosted Kubernetes

Building and maintaining infrastructure requires both experienced engineers and domain experts. But not every organization can assemble such a dream team. Domain experts are rare, and almost all of them are already working with large software companies, tech giants, etc.

So, when choosing between managed and self-hosted Kubernetes, here are the main points you’ll need to take into consideration.

Cost of Resource Management

Self-managed Kubernetes means you’re running the Kubernetes installations, either in your data center or on virtual machines in the cloud. This entails a separate cost attached to the machines used to run your control plane, meaning you’ll have to plan for high availability and disaster recovery on your own. You’ll also have to set up automation to scale the nodes along with dependencies and provision your network for increased load.

Staffing

As noted above, self-managed Kubernetes requires a big team that understands the deployment and amangement of different Kubernetes components. Your team will need to be able to handle etcd, the control plane, nodes, the Container Network Interface (CNI), service mesh, and smaller components like RBAC.

With managed Kubernetes, you don’t have to manage etcd or control planes, and many managed Kubernetes oversee CNI and service mesh for you. This makes it a lot easier for a small team of three to five people to handle a Kubernetes cluster than if you went with self-managed clusters.

Upgrades and Updates

Upgrading clusters is a big undertaking and can take a lot of time if you’re handling it yourself. Tasks include researching the changes in the next Kubernetes version, as well as components or APIs that are deprecated.

Managed Kubernetes, on the other hand, can be easily upgraded in one or two steps. You don’t need to take care of an etcd backup or make sure you individually upgrade control plane nodes to maintain high availability. All of this is taken care of for you by your cloud provider.

That said, cloud providers tend to lag behind vanilla Kubernetes, for the reasons mentioned earlier. At the time of writing this, Kubernetes 1.29 has just been released (December 2023), while the major providers of managed Kubernetes are lagging behind, offering versions based on Kubernetes 1.25 and even 1.24. the upside of this is the stability of the versions on offer. However, if you are waiting for a specific feature or bug-fix, you might be waiting for a up to a year.

Cost of Changing Cloud Providers

Moving between cloud providers costs more than money. You might also incur performance and reliability costs. It’s important to consider which provider will best meet your needs so you can avoid these peripheral costs. If you’re managing your infrastructure as code, there will be changes required for this migration as well.

In-House Expertise

If you’re thinking about going with cloud-agnostic Kubernetes clusters, you may consider managing them yourself, as this will give you the flexibility to move clusters across clouds since you aren’t depending on any cloud resources except the underlying machine. You have to choose your CNI very carefully though, as not all CNIs work directly on all clouds.

Managing your own Kubernetes will require in-house Kubernetes experts who can dig deeper into issues and find are solution.

Given the above considerations, a managed Kubernetes cluster is preferable to a self-managed one if:

• You want to stay on one cloud
• You don’t need the latest Kubernetes releases
• You’re ready to offload patching vulnerabilities to your cloud provider

You’ll find managed Kubernetes are easier to upgrade and highly available and—most importantly—you’ll have support from your cloud provider. There are many managed Kubernetes options to choose from; we’ll take a look at the main ones below.

There is also a middle road. Those are firms that provide white-glove managed Kubernetes, which in effect become your network facing SRE team. They will give you benefits of in-house vanlilla Kubernetes, while taking away much of the toil.

Primary Managed Kubernetes Providers

• EKS: AWS launched their public Kubernetes offering, Elastic Kubernetes Service, in June of 2018.
• AKS: Microsoft Azure’s Kubernetes offering, Azure Kubernetes Service, was released in early 2019. AWS is known to be reliable and have more-performant machines.
• GKE: Google Kubernetes Engine was released in 2015 and is one of the most advanced Kubernetes offerings because Google works closely with the Kubernetes development community.
• OpenShift: Red Hat OpenShift is used to manage and orchestrate containers with Kubernetes running as its core. Red Hat is known for security and provides additional functionality, such as creating deployment pipelines and supporting a variety of features.
• VMware Tanzu: Tanzu was introduced as a preview in Aug 2019 and later released in 2020. A set of tools and products used to run and manage Kubernetes clusters, Tanzu comes with a UI (mission control) to manage the cluster and supports CI/CD integration with the concourse platform provided by VMware.

Considerations for Choosing a Provider

Below are the key factors you should consider when looking into managed Kubernetes clusters.

Ease of Use

In most cases, you’ll be using kubectl from the command line instead of the GUI. But for the most intuitive UI, Amazon EKS is very easy to use, giving you only the options you need. AKS and GKE, on the other hand, have a lot of options that you may not use. OpenShift and Tanzu come with a custom UI for a better developer experience.

Generally speaking, deploying from the UI is discouraged. You typically wrap your deployments with pipelines that are supposed to deploy on Kubernetes, so having a UI doesn’t add a lot of value.

GKE, is easier for managing Day-2 operations. It gives the unique option of config sync, allowing you to sync your Kubernetes cluster from a git repository.

Ease of Set-Up

All of the providers noted above are easy to set up.

Generating a kubeconfig file in GKE and getting started is especially easy. With AKS and EKS, you have to make changes in the IAM, then add the user to the cluster. Finally, OpenShift comes packed with its own components and has Istio as its service mesh. VMware Tanzu also has its own CNI and service mesh.

Connectivity to Other Cloud Resources

Each of the providers mentioned has great integration with their other cloud offerings, like RDS or storage, through an internal network of various resources. AWS, for example, provides the most reliable platforms, while Azure offers a lot of configuration options.

It’s also good to note that AWS uses layer 2 networking, while AKS and GKE use layer 3 networking. So, if you’re planning to use CNI, which uses BGP for routing, you may face issues in GKE and AKS.

Native Plugin Support

Service Mesh

• EKS: Has its own mesh known as App Mesh, which is simply a wrapper over Istio
• AKS: Offers an Open Service Mesh
• GKE: Officially supports Istio
• OpenShift: Supports Istio by default
• VMware Tanzu: Has its own service mesh built on its NSX platform

‍

CNI and Networking  

• EKS: Offers its own VPC CNI plugin. It can run in island mode with the help of security groups and egress by using NAT. All pods are connected to each other directly, just like nodes. You can allow access to pods from inside the VPC with security groups.
• AKS: Offers Azure VPC CNI and kubenet. Using NSGs will allow you to control the networking and separate the cluster from the external network. If not restricted by an NSG, all pods can be connected with the IPs, as these are assigned from the same VPC network space.
• GKE: Offers Cilium as a CNI. Each IP is assigned to pods from the VPC network space. Outbound traffic is passed through NAT, while internal traffic among pods and nodes can be controlled with network policies.
• OpenShift: The OpenShift SDN is deployed by default but has support for third-party CNIs out of the box, like flannel. It has its own ingress controller and uses ingress and egress with OpenShift Routes to control inbound and outbound traffic. The OpenShift router uses HAProxy to implement the routes.
• VMware Tanzu: Comes with Antrea, which is designed to be used with Open vSwitch. You can run it in island mode, and it features support for ingress through third-party controllers like nginx. Egress can be controlled with a network policy.

‍

Cost

Determining the absolute “most expensive” and “cheapest” managed Kubernetes offering is complex, as pricing can vary greatly based on numerous factors, including:

• Cluster size and configuration: Costs scale with the number of nodes, their specifications, and additional features like advanced networking or auto-scaling.
• Resource usage: Your actual usage of compute resources, storage, and network bandwidth significantly impacts the final bill.
• Location: Prices can differ in different regions depending on cloud provider infrastructure and local market factors.
• Discounts and commitments: Providers often offer sustained use discounts, committed use discounts, and free tiers with limitations that can significantly alter the effective cost.

Read more on Kubernetes security pricing structure.

Custom Worker Nodes

AKS does not support custom worker nodes and has no plans to include it. GKE and OpenShift don’t offer it either. In fact, such nodes are restricted in GKE, meaning you cannot create a node that has basic tooling or software, like installing antivirus and then using it as a Kubernetes node. On the other hand, EKS has very good support for custom worker nodes and VMware Tanzu supports them as well.

Potentially Most Expensive:

• GKE: Can be the priciest due to its per-minute billing for all resources, including the control plane, and potentially higher compute node pricing compared to some competitors.
• AWS EKS: Similar to GKE, EKS also charges for the control plane and has variable costs based on resource usage. However, its preemptible instances can offer significant cost savings.

Potentially Cheapest:

• Azure AKS: Offers a free tier for small clusters and competitive hourly rates for larger deployments. Their “spot” VMs further reduce costs through opportunistic pricing.

Ease of Upgrade

Node upgrades are automated in GKE and AKS, while you have to manually update the worker nodes in EKS. AKS also provides a manual option for node replacement. When you upgrade on GKE, you don’t have control over how, or in what order, nodes are upgraded, thus your workload will be affected.

If the workload is stateful, this can cause issues. Manual node upgrades give you control over your own node replacement, so you can move your workload before taking the VM down for an upgrade.

The table below provides a convenient reference for many aspects of the primary managed Kubernetes solutions.‍

Comparison of Managed Kubernetes Services

Additional Notes:

• OpenShift and VMware Tanzu are not strictly managed Kubernetes offerings, but rather platforms built on top of Kubernetes that include additional features and tooling.
• The listed features and capabilities may vary depending on the specific version or configuration of each service.
• This table is intended as a general overview and should not be used as the sole basis for selecting a managed Kubernetes service. It is important to carefully evaluate your specific needs and requirements before making a decision.

Summary

Kubernetes clusters take time and staff to set up and maintain. That might mean a few minutes when using a managed cloud service or hours in a self-hosted version of Kubernetes. Ultimately, your organization’s ability to keep up with the open-source projects you selected for your stack is the biggest factor to consider when choosing managed over DIY Kubernetes. If you choose to go it alone, you’ll need experts on your team who can handle such a large undertaking. Separately managing etcd, upgrades, high availability, and reliability requires far more expertise than running a managed Kubernetes cluster.

Handling self-managed Kubernetes is a huge task. When you combine it with demands like security and compliance, it grows exponentially harder to manage. Cloud providers augment their offerings with pre-defined constructs like security groups, firewalls, and subnet and VNet segregation. For security and auditing, you can deploy tools like Gatekeeper, Falco, and Wazuh.

You can also check out ARMO Platform, which provides more visibility and control while also taking care of security and compliance for you. Try ARMO today.