Tackling the recent Kong ingress controller security incident with ARMO’s behavioral CADR

Imagine this situation: you recently updated one of your infrastructure software components. A few weeks later, you notice your AWS bill has gone up and you’re not sure why. After some digging, you find that the auto-scaler for this component is constantly scaled to the maximum. It takes days to realize that this change in behavior started right after the software update. You start asking around to see if anyone else is having the same issue with this new version. A few more days pass, and you finally discover that there’s a crypto miner hidden in the new version due to a malware injection in your supply chain!

You may think this sounds like a cybersecurity vendor’s pitch. Well, it’s not. This was the reality, this week, for some Kong ingress controller users.

Supply chain attacks, where attackers infiltrate the software delivery pipeline, are a critical threat. A recent incident involving the Kong Ingress Controller is a stark reminder that the software supply chain is an attack surface you cannot discount.

In this post, we will dissect the incident that affected the Kong Ingress Controller and demonstrate how ARMO’s Behavioral Cloud Application Detection and Response (CADR) solution can proactively identify and mitigate such supply-chain zero-day attacks. Let’s begin by reviewing the details of the breach.

The Kong Ingress Controller Incident: What Happened?

On December 29th, 2024, high CPU usage reports surfaced from users of the Kong Kubernetes Ingress Controller. This anomaly was linked to a malicious Docker image uploaded to DockerHub. Earlier this week, Dan Lorenc posted a detailed timeline and analysis of the incident on LinkedIn. Thanks Dan!

Compromised Access Token

The breach began with the compromise of a DockerHub Personal Access Token (PAT) used by the Kong project’s maintainers. This token allowed the attacker to upload images to the project’s DockerHub repository.

Malicious Image Uploaded

Prior to December 23rd, 2024, the attacker used the compromised PAT to upload a malicious version of the Kong Ingress Controller image (version 3.4.0) directly to DockerHub. This image included code designed to mine cryptocurrency and send the proceeds to a specific wallet controlled by the attacker.

User Reports

On December 29th, a user reported unusually high CPU usage on this issue. This is a phenomenon often associated with cryptojacking, since a system’s resources are hijacked for mining cryptocurrency.

Response by the project

The maintainers acted swiftly, removing the malicious images from DockerHub by January 2nd, 2025. They uploaded new, clean versions of the affected images and revoked the compromised PAT to prevent further misuse. As part of the remediation efforts, the keys used for uploading images were rotated. Additional security measures were likely implemented to prevent similar incidents.

Key Takeaways from the Incident

Impact of Compromised Credentials: The incident underscores how a single compromised credential can lead to widespread repercussions in the software supply chain.

Detection Delays: The time that elapsed between the compromise and user reports highlights the need for robust monitoring mechanisms.

In the next section, we will explore how ARMO’s CADR solution can detect and prevent such incidents. Thus, providing organizations with the tools they need to secure their Kubernetes environments against supply chain attacks.

How ARMO Behavioral CADR Detects and Mitigates Supply-chain Attacks

ARMO’s Behavioral Cloud and Application Detection and Response (CADR) leverages advanced eBPF monitoring, anomaly detection, and rule-based monitoring to identify and mitigate attacks in cloud runtime  environments (e.g. Kubernetes). Here’s how ARMO CADR detects this type of attack:

Detecting Cryptographic Calculations

ARMO monitors container activities at the kernel level using eBPF. In this way, it identifies cryptographic calculations often associated with cryptocurrency miners.

In this case – XMRIG currency requires extensive use of floating point calculations, which is highly unusual in the average cloud application. 

Monitoring External Communications

ARMO keeps track of communications initiated by containers, flagging those to domains used by cryptocurrency systems. This includes detecting connections to cryptocurrency wallets or mining pools providing early indicators of malicious activity.

As mentioned in the GitHub issue above, the miner software was contacting “pool.supportxmr.com” which is a known endpoint for Crypto related systems. Therefore ARMO CADR flagged it as highly suspicious activity.

Fileless Malware Detection

Attackers often use advanced techniques, such as running malicious processes directly from memory to avoid disk-based detection. ARMO detects these in-memory processes by identifying fileless malware.

In this case, the malware was using the “memfd_create” system call and used the related file descriptor to execute the new process from memory. ARMO CADR identified this and generated an alert.

Supply Chain Attack Detection

ARMO Behavioral CADR analyzes behavioral differences between software versions to identify unexpected changes. For example, this incident flagged the new version of the Kong Ingress Controller for initiating external communication with a low-reputation endpoint. This was a suspicious change in between versions that is indicative of a supply chain attack.

These capabilities ensure that organizations using ARMO Behavioral CADR have a robust defense against both known and emerging threats. Thus, offering a proactive approach to securing their Kubernetes environments.

ARMO’s Behavioral CADR safeguard your cloud environments from supply chain attacks. Book a demo to see how ARMO Platform’s advanced detection methods provide real-time insights and reduce false positives, empowering your security team to respond swiftly and effectively to modern threats.

Quickly ensure your Kubernetes is secured.

Follow this simple checklist and make sure your Kubernetes security is covered in just a few steps.

Read Now

Read Now