Kubernetes RBAC: Deep Dive into Security and Best Practices

What is RBAC in Kubernetes?

Kubernetes has revolutionized container orchestration, becoming the go-to platform for managing containerized workloads at scale. However, with its growing popularity, the complexity of managing role-based access control (RBAC) on Day 2 and especially in a multi-cluster environment has become a daunting task. DevOps, SRE, and Platform teams are responsible for multiple clusters and different teams. This article explores the challenges of Kubernetes RBAC implementation, best practices for managing RBAC in Kubernetes, and the innovative solution offered by ARMO Platform to simplify RBAC management and enhance Kubernetes security.

Understanding RBAC in Kubernetes

RBAC is a fundamental security mechanism in Kubernetes that governs access to resources within the default namespace or any other namespace in a cluster. RBAC is often compared to attribute-based access control (ABAC), which provides more granular, context-aware access policies. RBAC policies define which users or groups can access specific Kubernetes resources and the actions they are allowed to perform on those resources. The RBAC process involves three steps: authentication, authorization, and admission control.

Authentication

Authentication is the process of verifying the identity of a user or entity. There are three types of entities in Kubernetes: a user (human or service accounts), a group (a collection of users), and a service account (used by pods inside the cluster). Users must be created using X.509 certificates signed by the cluster CA, which can be done through CSRs and human access certificate issuing. While this procedure is outside the scope of this document, many online tutorials explain the process.

Pods running inside the cluster don’t need a certificate or OAuth tokens, as they can use service accounts and controllers for authentication. Initially, a default service account is automatically created  in every namespace when the namespace is created. This service account is named default and is used by pods that do not explicitly specify a service account. User accounts are also available for processes that run inside pods. Creating an X.509 certificate for the accounts is unnecessary; instead, link pod and user accounts are typically done in manifest files. For example:

apiVersion: v1

kind: ServiceAccount

metadata:

name: my-service-account

In the Kubernetes RBAC process, authentication ensures that the user or entity requesting access to the Kubernetes cluster is who it claims to be. Once the entity is authenticated, the next step is authorization.

Authorization

In the Kubernetes cluster, authorization establishes what actions users or entities can allow through granted permissions. Once a user or entity is authenticated, the RBAC system checks their permissions and RBAC roles to determine if they have the necessary privileges to perform the requested actions and access rights. This step ensures that only authorized users, such as those with an admin role, can perform specific operations within the cluster, and is important for maintaining consistent permissions for users. However, please remember the principle of least privilege (PoLP), and that general usage of the cluster admin role is discouraged.

Admission Control

Admission control is a mechanism in Kubernetes that intercepts and processes API requests before they are persisted in the cluster control plane. The API server handles these requests and the admission control process ensures they comply with the defined authorization policies. Admission control enforces policies and rules to validate and modify the requested resources, including those requiring admin access. In the RBAC process, admission control ensures that the requested actions comply with the defined authorization policies. It can reject or modify requests that do not meet the specified criteria.

It is important to note that RBAC is unnecessary for Kubernetes resources specified in manifest files. For example, creating a pod and a secret and granting the pod access to the secret. The pod specification in the manifest file specifies that the secret must be “mounted” one way or another (through environment variables or files mounted inside the pod). The API server processes these manifest file requests and enforces the necessary permissions. In this example, the pod can access the secret no matter the RBAC permissions assigned to the pod’s service account. This is because the mounting of such secrets does not depend on RBAC or metadata.

Anatomy of RBAC rules

A Kubernetes RBAC rule is made up of three elements which define the API access levels: the API group, a verb (i.e., an action), and a target (either a resource name(s) or an API URL). RBAC rules are specified in roles and cluster roles, and are essential for managing role assignments. The difference between the two is that roles are scoped to a given single namespace, while cluster scoped resources apply to the entire cluster, and often utilize predefined roles or custom roles with different levels of access. When no namespace is specified, resources are created in the default namespace. This is what an RBAC rule looks like, as part of a YAML file specifying a role, including any default roles:

- apiGroups: [“”]

verbs: [get, list]

resources: [secret]

resourceNames: [mysupersecret]

The API group identifies which API group to target, determining if it is a namespaced resource. This is necessary because different API groups can have the same verbs. Additionally, Kubernetes is highly extensible and allows for adding new APIs with verbs and resource names that clash with other APIs. In the manifest file, this is a list, although usually only one API group is specified here. The API group “” (empty string) represents the core Kubernetes API.

The verb indicates the action to take. For example: get, list, create, delete, update, etc. Again, this item in the manifest file is a list, which allows for the specification of more than one action, thus avoiding the creation of redundant rules.

The third part is the verb’s target. It can be an explicitly specified resource, including various types of resources. Examples of resource types are: pod, networkpolicy, service, etc. Additionally, further restrictions can be applied by using resource names or specific fields within those resources.

The second option for the verb’s target is to specify the URL path. This is the request’s URL path and can contain an ending wildcard, which can be used to give access to certain parts of the API. An RBAC rule must specify a target that is either a namespaced resource or a URL, but not both. Also, note that a URL path can be specified as a target only for cluster roles, as roles are scoped to a single namespace.

Examples of RBAC rules

RBAC with a resource target

Example YAML manifest file:

apiVersion: v1

kind: Role

metadata:

        name: test-role

        rules: - apiGroups: [“”]

        verbs: [get, list]

        resources: [secret]

        resourceNames: [mysupersecret]

The role in this file has only one RBAC rule, which targets the core API (the empty string in “apiGroups”). The rule allows for the actions “get” and “list” on a secret named “mysupersecret,” providing read only access.

RBAC manifest file with a URL path

Example of YAML manifest file:

apiVersion: v1

kind: ClusterRole

metadata:

        name: test-cluster-role

        rules: - apiGroups: [“rbac.authorization.k8s.io”]

  verbs: [get, list]

  nonResourceURLs: [/apis/rbac.authorization.k8s.io/v1alpha1]

In this case, the rule gives “get” and “list” access to any request made on the “/apis/rbac.authorization.k8s.io/v1alpha1” path. Please note that the role here is a ClusterRole, as only cluster roles can use path-based permissions.

Accessing cloud services

For a pod access to cloud services, Kubernetes RBAC is not enough because it only manages Kubernetes resources and their credentials. In AWS, for example, service account access must be managed by linking a Kubernetes service account with an IAM role. Achieving this is a somewhat complex process, and AWS has well-documented it. In short, it adds annotations to the service account manifest. These annotations should point to the IAM role the service account should assume for accessing cloud services. This will also help prevent attackers from bypassing RBAC mechanisms through node access. For example:

apiVersion: v1

kind: ServiceAccount

metadata:

    name: my-service-account

   annotations:

         - eks.amazonaws.com/role-arn: <ROLE_ARN>

However, a prerequisite is first to create an OIDC provider (one per cluster). Other cloud vendors require similar processes to access their resources.

RBAC best practices

Effective Kubernetes RBAC management requires adherence to best practices to ensure the security and integrity of the Kubernetes environment, including strong security controls for critical components. Here are some RBAC best practices:

Principle of Least Privilege

Apply the principle of least privilege to accounts to grant the minimum permissions required for each user or group to perform their tasks, which involves assigning roles to users. While applying the same principle to humans is ideal, it can be more challenging to achieve in practice. Reviewing permissions in the kube-system namespace is a key best practice, as this namespace contains critical system components.

RBAC Strategy and Planning

Create a well-defined RBAC strategy, starting with assessing the current state and defining clear goals, including establishing granular roles. Develop a plan to transition from the current state to the desired RBAC policies incorporating clear role definitions, considering the organization’s specific needs.

Infrastructure-as-Code and Templating

Use infrastructure-as-code (IAC) or templating tools like Helm charts to manage RBAC policies efficiently. Templating enables use of variables and change tracking, aiding in troubleshooting and auditing processes.

Comprehensive Testing

Regularly test RBAC permissions to ensure that entities can perform their necessary actions without being allowed to perform unauthorized actions while also reviewing user permissions. Thorough testing helps identify and rectify vulnerabilities in the system.

Periodic Review and Cleanup

Periodically review existing RBAC and account objects through audit reviews to ensure they are up-to-date and delete unused roles, users, or groups that are no longer needed. This minimizes the attack surface and streamlines RBAC management, especially when considering the kube-system namespace.

RBAC in Kubernetes: Day 2 operations

Kubernetes RBAC policies aren’t static. As mentioned above periodic review and cleanup is a best practice for RBAC. Ensuring no unauthorized access is a key first step in this process.

The most straightforward way to manage RBAC policies is by using the kubectl command-line tool to view, create, update, and delete them.

Here are a few examples of kubectl commands for viewing RBAC policies and related resources:

• kubectl get roles lists all the roles defined in the cluster, including their associated rules and scope.
• kubectl get rolebindings lists all the role bindings in the entire cluster, detailing how individual users and groups are mapped to specific roles.
• kubectl describe role/<role-name> provides detailed information about a specific role.
• kubectl auth can-i <verb> <resource> checks if the current user has the specified permissions (verb) for a specific resource.

In addition to these commands, there are others to effectively add, remove, and delete RBAC policies across the entire cluster.

Although kubectl provides an easy-to-use operational experience, there are some drawbacks, which we discuss below.

Cluster scope

Kubectl commands are executed for a single cluster only, offering fine-grained controls over access policies within the cluster. This helps to minimize the risk of accidental modifications to other clusters. However, when managing RBAC policies across multiple clusters, tracking which policies have been updated and which ones need to be updated for cluster-wide resources can be challenging.

Lack of holistic view

The commands mentioned above show the RBAC-related Kubernetes resources in a CLI format, such as a cluster role to watch deployments:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-watcher
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch"]

The output shows the definition of the cluster role with its rules. It lacks contextual information about the level of access granted by each rule, and who is using a role within a cluster. To find that out, it’s necessary to check cluster role bindings and role bindings for cluster-wide permissions and namespace-scoped access.

Using kubectl commands is insufficient to prevent or detect unauthorized access, as they only provide granular information. What is missing is a broader, contextual understanding of who has access to what resources.

Challenges of managing RBAC in a multi-cluster environment:

RBAC management becomes more complex as organizations adopt Kubernetes and deploy multiple clusters to manage their containerized workloads. Several challenges arise in managing RBAC across multiple clusters:

Role Explosion

Managing granular access control becomes challenging in a dynamic environment with numerous users and roles. Each user or group may require unique permissions, leading to a growing number of roles, making it harder for administrators to keep track of privileges.

Complexity and Scalability

As the number of roles and permissions increases, managing relationships between different roles and permissions within clusters becomes challenging. Without Kubernetes security tools for synchronization, administrators may struggle to comprehend the overall access control system.

Lack of Holistic View

Command-line tools like Kubectl provide granular information about individual roles and bindings but lack a comprehensive view of how different roles are used across the clusters.

Imperative Execution

Imperative RBAC management through kubectl commands lacks continuous synchronization and checks, making it prone to human errors and time-consuming updates.

Effective Kubernetes RBAC management with ARMO Platform

RBAC management is crucial to securing a Kubernetes cluster and ensuring compliance with regulations and industry standards. However, without the proper RBAC tools in place, managing RBAC can be a complex and error-prone manual process. This is why RBAC visualization is essential. Visualizing RBAC helps administrators keep track of all privileges in a systematic and organized manner. A visual representation of the relationships between roles and bindings allows administrators to understand the policies more easily and thus make informed decisions about policy changes. RBAC investigation is critical to securely managing Kubernetes in production and ensuring security and compliance.

ARMO Platform includes an RBAC visualizer allowing administrators to see which privileges are assigned to any given user. With ARMO Platform, administrators can take control of RBAC management and reduce the attack surface by conforming to the principle of least privilege. Try it free today!

Frequently Asked Questions (FAQ):

Q: What is the difference between role and ClusterRole in Kubernetes?
A: At a high level, roles and RoleBindings are confined to specific namespaces, granting access within those boundaries. In contrast, ClusterRoles and ClusterRoleBindings operate at the cluster-wide level, providing permissions across all namespaces.

Q: What is an example of RBAC?
A: A typical RBAC scenario involves controlling access for various user types such as administrators, specialists, or regular users based on their job roles.

Q: What are the three elements of role-based access control in Kubernetes?
A: The primary components of RBAC include:

• Role: Specifies permissions at the namespace level.
• ClusterRole: Defines permissions that span across the entire cluster or multiple namespaces.
• Subject: Represents a user, group, or service account to whom permissions are applied.

Q: How can one use Kubernetes RBAC with AKS without leveraging Azure AD?
A: The absence of an identity provider such as Azure AD typically limits certain functionalities. Although alternative approaches might exist, they may not be officially supported or advisable. Collaborating with the team managing identity services to integrate Azure AD for comprehensive RBAC support is recommended.

Q: How can one assign the same RBAC role to two different IAM roles to access a cluster in EKS?
A: To achieve this, specify distinct userNames for each IAM role when configuring the RBAC settings. This approach allows both IAM roles to be associated with the same RBAC role, granting them equivalent permissions within the cluster.