Kubernetes proxy mode: Ip tables IPVS and eBPF.
Written together with Noam Amrani, Based on GithubRepo.
Introduction
Effective communication inside the cluster is a critical aspect of Kubernetes deployments, yet it is often overlooked. Service communication in Kubernetes involves a variety of technologies, including a newer technology called eBPF that is becoming increasingly important. In this blog post, we will discuss Kubernetes proxy modes, including IP Tables, IPVS, and eBPF, and explore their implications for performance, scalability, and security in on prem clusters as well as in different cloud environments.
* As this is a time of AI generating every post we see online, I felt compelled to add my personal touch to this article and my explanations were sketched by my own hand. Although my artistic abilities may not be my greatest strength I hope you will find them useful.
Cluster traffic
So, I guess we are all familiar with the architecture explaining how traffic from outside the Kubernetes cluster is being handled. It goes through some kind of load balancer that directs the traffic to a service, which then directs it to the pods. There are several solutions available to address various use cases when dealing with incoming traffic from outside the cluster, such as Ingress controllers and Service Meshes.
However, I would like to shift the focus to traffic flow inside the cluster, specifically between pods and other pods using services.
Kube Proxy
Let’s take a closer look at the internal traffic within Kubernetes in a very high Level. We have the Kubernetes worker nodes and the Kubernetes control plane.
The services and pods are created on the worker nodes, and all API calls are sent to the control plane. The control plane maintains an internal representation of each service called an Endpoints object, which contains a list of healthy backend endpoints (Pod IP + port).
On the worker nodes, there is an agent that reads the endpoint objects, usually implemented using kube-proxy, with various competing implementations from third-party Kubernetes networking providers such as Cilium, Calico, kube-router, and others.
Kube proxy modes
Ip Tables
The default mode of the kube-proxy is iptables, which is a firewall component for Linux. It monitors traffic from and to your server using tables that contain sets of rules, called chains, to filter incoming and outgoing data packets. Iptables provides five tables (filter, nat, mangle, security, raw), with the filter and nat tables being the most commonly used. The tables are organized into chains, with five predefined chains: PREROUTING, POSTROUTING, INPUT, FORWARD, and OUTPUT.
To make a long story short, iptables were designed to be a firewall, not a routing table. Therefore, they have some disadvantages when it comes to routing. For example with a large number of services (1000+) there will be to many rows in the tables and that may slow down traffic additionally they do not support round-robin load balancing.
IPVS
IPVS is a kernel-level load balancer that can be used as a proxy mode in Kubernetes. It is available in Kubernetes 1.9 or later. IPVS uses a set of kernel modules to perform load balancing, which allows it to distribute traffic across a set of backend pods more efficiently than iptables.
IPVS can be used to set up load balancing for different types of workloads. However, it has some limitations, including complexity and the need for specialized kernel modules. It also requires more setup and configuration than IP Tables.
As Posted in article by tigeera attached below, IPVS can handle big scale traffic a lot faster the IP tables.
eBPF
eBPF is not actually a kube-proxy mode but a new technology that can be used to enhance Kubernetes service communication and performance. It allows for more efficient and fine-grained packet filtering and processing. This can be used to optimize load balancing and network security in Kubernetes.
eBPF is still a relatively new technology, and it requires specialized knowledge and tools to use effectively. However, it has the potential to improve performance and visibility in Kubernetes. The main network provider to implement eBPF is calico. You can implement calico almost in every cloud and on Prem environment.
Cloud Providers
Different cloud providers implement the various proxy modes in their Kubernetes offerings differently. For example, Amazon EKS uses IP Tables as the default proxy mode, while Google Kubernetes Engine (GKE) uses IPVS. Microsoft Azure Kubernetes Service (AKS) supports both IP Tables and IPVS, but recommends IPVS for larger clusters.
Summary
In conclusion, Kubernetes proxy modes are an important part of service communication and performance in Kubernetes. Each proxy mode has its own benefits and limitations, and it is important to understand these when choosing the best option for a given workload or deployment.