Sidecar containers in Kubernetes: a personal journey

I had always wanted to use sidecars with Istio or Splunk forwarder in production, but as a Kubernetes maintainer, I knew there was no reliable way of telling Kubernetes to ensure sidecar containers were kept running before and after the main application.

In this post I will share the twists and turns of my adventure in addressing this long-standing Kubernetes challenge. This journey took me from being a passionate observer to an active contributor, and through the intricate mazes of Kubernetes Enhancements and Special Interest Groups (SIGs). Join me as I recount the challenges, partnerships, and innovations that shaped my path.

I got personally involved in the Sidecar KEP in 2021 after watching a talk about Sidecars at Netflix, in which Rodrigo Campos Catelin asked for help and someone to take over for Joseph Irving and himself. I made a promise to Rodrigo to continue his work on this KEP,  and was determined to follow through.

One of the initial gripes with the original proposal was its complexity and the user API based on labels. Over the next few release cycles I tried multiple times to find the right balance between “too basic to bring value”, and “too complex to achieve within a cycle”. I finally hit a wall when  SIG node refused to make more changes to the redesigned kubelet lifecycle manager, until it was stabilized.

I decided to join a SIG node subgroup dedicated to bug fixing and CI maintenance to help bring this stability, and also make alliances with key SIG members. A few releases later, with a stable kubelet and my friend Sergey Kanzhelev, now a chair of SIG node, we were ready to tackle this task.

By Security standards, at DevOps pace.

Actionable, contextual, <br/> end-to-end <br/> Kubernetes-native security

See ARMO in Action

The final hurdle we faced was choosing the right API for this feature. Our team was split: some favored a comprehensive systemd-like dependency tree, while others wanted a straightforward method to mark a container as the primary application within the Pod. The breakthrough came during a casual chat with Tim Hockin on Slack one evening as I was watching a TV show. We both almost simultaneously realized that generalizing initContainers was the solution. In retrospect, they might have been better named as “infrastructureContainers” or “supportContainers”, and paired with a restartPolicy.

The implementation was delayed by one release because the approvers were occupied with the kubelet breaking as a result of Dynamic Resource Allocation, just days before the 1.27 release. I’m deeply grateful to my co-authors (Sergey Kanzhelev, Todd Neal and Gunju Kim) on the KEP; without their assistance, completing it wouldn’t have been possible.

Being a Kubernetes contributor has given me great professional satisfaction. It has also grown my professional network and has cemented my sense of belonging in the cloud-native community.

If you would like to join an open-source community that is in its growth phase, I encourage you to join the Kubescape community. We’re still small, always friendly and you can make an impact.

Unifying AppSec, CloudSec and DevSec

The only runtime-driven, open-source first, cloud security platform:

Continuously minimizes cloud attack surface

Secures your registries, clusters and images

Protects your on-prem and cloud workloads

Powered by Kubescape & eBPF

See ARMO in Action