K8S Security — Handle with Care!

We can have K8S running in multi-cluster mode or single-cluster mode based on our use-case needs. For analogy, I’m considering a single-cluster mode where it is also possible to run multiple environments inside it.

Fundamentally, the same principles do apply to multi-cluster mode K8S strategy as well but needs to bring more broader perspective to implement security in that context. So, be mindful of that note.

For any given K8S cluster, we can break it in to 4 sections like below to understand Security

  • Control Plane
  • Nodes
  • Namespaces
  • Native Controls

Now, Let’s go more in details about these individual sections <I’m not covering what each component does in this blog. For basic introduction and understanding, please refer my another blog here>

CONTROL PLANE

It consists of many components but let’s focus on key objects which are crucial from security perspective.

Always ‘Upgrade’ the K8S to the latest version to avoid vulnerabilities for all the components including the API-Server

Kube-API Server — Controlling who can access the K8S cluster and what actions they can perform to the cluster is the very first defensive step to consider. As the K8S is totally API driven, this is the fundamental component to focus

  • Use TLS for all the API Traffic — All the communication inside the cluster must be encrypted by default with TLS and allow necessary certificates to the cluster components.
  • Authentication — All API clients must be authenticated <to understand ‘who’> such as service accounts or normal users. Please refer here for more details
  • Authorisation — Once authenticated, every API must pass through an authorisation through Role-Based Access control (RBAC) <to understand ‘Who can do What’>

etcdIf you are able to access it, that literally means you’ve root permission in the cluster, that’s the implication if you don’t handle etcd security properly. Ideally only the API Server should’ve the access to it. Setup the firewall rules properly or use the security features of etcd which depend on x509 PKI.

Ensure to set up strong authentication for etcd with restrictive access.

NODES

These run containerised applications in the cluster and have multiple components inside like Kubelet, Kube-Proxy, Container Run-time. Consider below items for making your worker nodes more secure:

  • Authentication and Authorisation — by default, kubelet access is un-authorised. So, setting up proper mechanism is the minimum requirement
  • Follow CIS benchmarks to secure and harden the container run-times.

NAMESPACES

Primarily these are virtual clusters which are intended to logically isolate applications across multiple teams or projects or users. For eg., dev, test, production etc. They provide isolation and consists of resources such as pods, services, replication controllers etc.

The namespaces helps to reduce the blast radius of poorly configured applications and also helps to make billing and other resource management easier.

NATIVE CONTROLS

  • RBAC — We already discussed about it earlier but to summarise, it is a way of granting users granular access to kubernetes API resources. It primarily uses ‘Roles’ and ‘Role Bindings’ objects
  • Network Policies — by default, pods are non-isolated i.e., pods can communicate to each other within the cluster without any issues which is a potential threat from security perspective.

NW policies, give us the flexibility to restrict the communication between the pods and *only allow the communication between the selective pods.

Use mandatory fields under ‘spec’ section of the .yaml file such as podSelector, polityTypes, ingress, egress to name a few

  • Secrets — Kubernetes Secrets store and manage sensitive information. Avoid having secretes in container images. Keep Secrets in memory for as little time as possible. By default, Secretes can be retrieved in plain text format with API access. In order to safely use Secrets, always enable ‘Encryption at Rest’ and Configure restrictive RBAC rules
  • Pod Security Policies — This defines set of conditions that a pod must meet before it gets accepted in to the cluster/system

Use Privileged containers

PodSecurityPolicy enables fine-grained authorisation of pod creation and updates

PodSecurityPolicy is deprecated as of Kubernetes v1.21 and will be removed in v1.25

Apart from the above precautions, enable ‘Audit Logging’ to help understand what is happening inside the application. But the native logging functionality provided by a container engine or run-time is not enough for a complete logging solution. For effective logging, use ‘Cluster-level logging’ architecture which requires a separate backend to store, analyse and query logs

For better Container Security,

Secure Golden Image <enforce through governance and policies>

  • Build gold master container image based on tested OS base containers
  • Integrate CI/CD pipeline to deliver applications and app updates consistently and securely
  • Scan for vulnerabilities at Run-time and At-rest as well

Low-hanging fruits

  • Disable anonymous access where possible
  • Block privilege escalation on privileged containers
  • Use ‘Requests and Resource Limits’ to mitigate excessive use in case of exploitation <It is key to note that the limits can never be lower than the request needs>

Kube-Scheduler uses these details to figure out where to run your pods

Keep a note that if we put in a value larger than the core count of our biggest node, the pod will never be scheduled. For eg., your Pod is looking for 4 cores whereas the your cluster of capacity dual-core

On the other side, someone might end up setting these limits and these can be controlled through ‘ResourceQuotas and LimitRanges’ at the namespace level

  • Consider ‘DevSecOps’ methodology
  • Patch promptly

Governance can be achieved with below examples <few listed here but can be explored further for more streamlined governance objectives>

  • Don’t allow container start by a user using a shell on the host or remote CLI
  • Affinity in needed cases but anti-affinity when no need
  • Deployments only through package managers like Helm
  • Central managed Secrets and encrypted
  • Least Privileges and use RBAC to scheduling
  • Dedicate namespaces for better isolation

That’s all for now. Each of these sections can be elaborated further very extensively but concluding the section here as the intention is to give high level overview about the K8S Security

Thanks for taking time to Read!

References Used:

Kubernetes

etcd

CNCF

‘Kubernetes Best Practices’ and ‘Youtube’