How Kubernetes Actually Works

Behind the buzzword lies a surprisingly elegant system of watchers, reconcilers, and declarative state. Here is what actually happens when you deploy a container.

The Buzzword That Runs the World

Kubernetes has become one of those words that managers say in meetings without understanding and engineers use daily without fully grasping. Developed at Google and released as open source in 2014, it descended from an internal system called Borg that had been orchestrating Google's containers for over a decade. The name comes from the Greek word for "helmsman," which is fitting: Kubernetes steers containers the way a pilot steers a ship, not by rowing but by setting course and letting systems do the work.

To understand Kubernetes, you need to understand one core idea: declarative configuration. You do not tell Kubernetes what to do step by step. Instead, you describe the state you want, and Kubernetes figures out how to get there. This is the difference between saying "start three copies of my web server" and saying "I want three copies of my web server running at all times." The second statement implies self-healing. If one copy crashes, Kubernetes notices the gap between desired state and actual state, then acts to close it.

The Control Plane: Where Decisions Happen

The control plane is the brain of a Kubernetes cluster. At its center sits etcd, a distributed key-value store that holds every piece of cluster state. Every pod, service, and configuration lives as a record in etcd. The API server sits in front of etcd, acting as the only gateway for reads and writes. Nothing touches etcd directly.

The scheduler watches for newly created pods that have no node assignment. When one appears, the scheduler evaluates available nodes based on resource requests, affinity rules, and constraints, then binds the pod to a suitable node. It does not launch the pod. It simply writes "this pod belongs on node X" back to the API server.

The controller manager runs dozens of control loops, each responsible for one aspect of the system. The ReplicaSet controller watches for differences between desired and actual replica counts. The Node controller monitors node health. The Endpoint controller populates service endpoints. Each loop follows the same pattern: observe current state, compare to desired state, take action to reconcile.

The Node: Where Work Happens

Every worker node runs a kubelet, a process that watches the API server for pods assigned to its node. When the kubelet sees a new assignment, it instructs the container runtime (containerd, CRI-O, or another compatible runtime) to pull the image and start the container. The kubelet then monitors the container's health through liveness and readiness probes, reporting status back to the API server.

Kube-proxy runs on each node as well, maintaining network rules that allow pods to communicate across the cluster. When a service is created, kube-proxy sets up iptables rules (or IPVS rules in newer configurations) so that traffic destined for the service's virtual IP gets distributed to healthy backend pods.

Networking: The Hard Part

Kubernetes networking follows a simple model with complex implementations. Every pod gets its own IP address. Pods can communicate with any other pod without NAT. This flat network model is elegant in theory but requires a Container Network Interface (CNI) plugin to implement. Calico, Cilium, Flannel, and Weave are popular choices, each with different approaches to routing and network policy enforcement.

Services provide stable network identities for groups of pods. A ClusterIP service gives you an internal virtual IP. A NodePort exposes the service on every node's IP at a specific port. A LoadBalancer service provisions an external load balancer in supported cloud environments. Ingress resources route external HTTP traffic to services based on hostname and path rules.

Why It Won

Kubernetes won the container orchestration wars not because it was simplest (Docker Swarm was simpler) or most mature (Mesos was older). It won because of its extensibility. Custom Resource Definitions let anyone extend the Kubernetes API with new object types. Operators encode domain-specific operational knowledge into controllers. The ecosystem grew explosively because Kubernetes provided a platform for building platforms.

The Cloud Native Computing Foundation (CNCF) provided governance and neutrality, preventing any single vendor from controlling the project. Major cloud providers built managed Kubernetes services: GKE, EKS, AKS. The result is a de facto standard for container orchestration that, for better or worse, has become the default answer to "how do we run this in production?"

Pods, Deployments, and Services: The Building Blocks

The Pod is Kubernetes' smallest deployable unit. A pod contains one or more containers that share a network namespace (they can reach each other on localhost) and storage volumes. Most pods contain a single application container, but the sidecar pattern (adding helper containers for logging, monitoring, or proxying) is common. Istio's service mesh, for example, injects a sidecar proxy into every pod to handle traffic routing and observability.

A Deployment manages a set of identical pods (a ReplicaSet). It handles rolling updates (gradually replacing old pods with new ones), rollbacks (reverting to a previous version if something goes wrong), and scaling (increasing or decreasing the number of replicas). When you update a Deployment's container image, Kubernetes creates new pods with the new image, waits for them to become healthy, then terminates old pods, all without downtime.

A Service provides a stable network endpoint for a set of pods. Pods are ephemeral; they can be created, destroyed, and moved between nodes at any time. Services abstract this volatility by providing a DNS name and virtual IP that routes to healthy pods regardless of where they are running. This decoupling of network identity from physical location is essential for building reliable distributed systems.

ConfigMaps and Secrets externalize configuration from container images. A ConfigMap stores non-sensitive key-value pairs. A Secret stores sensitive data (passwords, tokens, certificates) in base64 encoding (not encrypted by default, which is a common source of security misconfigurations). Both can be injected into pods as environment variables or mounted as files.

Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) provide storage that outlives individual pods. A PV represents a piece of storage (a cloud disk, an NFS share, a local SSD). A PVC is a request for storage that Kubernetes matches against available PVs. This abstraction lets applications request storage without knowing the underlying infrastructure.

Horizontal Pod Autoscaling and Resource Management

Kubernetes can automatically scale the number of pod replicas based on CPU utilization, memory usage, or custom metrics. The Horizontal Pod Autoscaler (HPA) watches a target metric and adjusts the replica count to keep the metric near a target value. If CPU usage exceeds 70%, the HPA adds more pods. If it drops below 50%, pods are removed.

Resource requests and limits are critical but often misunderstood. A request is the minimum amount of CPU and memory a pod needs. The scheduler uses requests to decide where to place pods. A limit is the maximum a pod is allowed to use. If a pod exceeds its memory limit, it is killed (OOMKilled). If it exceeds its CPU limit, it is throttled.

Getting these values right is an art. Requests that are too high waste resources (nodes appear full when they are not). Requests that are too low lead to overcommitment (nodes run out of resources and pods are evicted). Limits that are too tight cause unnecessary kills. Limits that are too loose allow noisy neighbors to starve other pods.

The Kubernetes Ecosystem

The Cloud Native Computing Foundation (CNCF) landscape contains over 1,000 projects, many of which extend or complement Kubernetes. Helm provides package management for Kubernetes (charts that template and bundle related resources). Prometheus provides metrics collection and alerting. Grafana provides visualization. Istio and Linkerd provide service mesh capabilities (traffic management, mutual TLS, observability).

ArgoCD and Flux implement GitOps, a pattern where the desired state of a Kubernetes cluster is stored in a Git repository, and a controller continuously reconciles the cluster to match the repository. Changes to the cluster are made by committing to Git, providing an audit trail and rollback capability.

Operators extend Kubernetes with domain-specific automation. A database operator, for example, might handle provisioning, backup, failover, and scaling of database instances, encoding operational knowledge that would otherwise require a human DBA. The Operator Framework, created by Red Hat, and kubebuilder, created by the Kubernetes project itself, provide scaffolding for building operators.

Kubernetes in Practice: Common Patterns and Pitfalls

The namespace boundary: Namespaces provide logical isolation within a cluster. Teams typically get their own namespace with resource quotas and network policies. However, namespaces do not provide strong security isolation. A compromised pod in one namespace can potentially attack pods in another unless network policies are explicitly configured.

The init container pattern: Init containers run before the main application container starts, handling tasks like database migrations, configuration file generation, or waiting for dependent services. They run to completion sequentially before the main container starts.

The health check trap: Liveness probes that are too aggressive (short timeout, frequent checks) can cause cascading restarts during temporary load spikes. A pod that is slow to respond under heavy load is killed and restarted, adding to the load and causing more pods to fail. The fix is to use startup probes for slow-starting applications and to set generous timeouts on liveness probes.

The resource estimation problem: Most teams underestimate how difficult it is to set correct resource requests and limits. Tools like Vertical Pod Autoscaler (VPA) and Goldilocks can help by observing actual resource usage and recommending values.

Pods, Deployments, and Services in Depth

The Pod is Kubernetes' smallest deployable unit. A pod contains one or more containers that share a network namespace (they can reach each other on localhost) and storage volumes. Most pods contain a single application container, but the sidecar pattern (adding helper containers for logging, monitoring, or proxying) is common. Istio's service mesh, for example, injects an Envoy sidecar proxy into every pod to handle traffic routing, mutual TLS, and observability.

A Deployment manages a set of identical pods through a ReplicaSet. It handles rolling updates (gradually replacing old pods with new ones), rollbacks (reverting to a previous version if something goes wrong), and scaling (increasing or decreasing the number of replicas). When you update a Deployment's container image, Kubernetes creates new pods with the new image, waits for them to become healthy, then terminates old pods, all without downtime.

ConfigMaps and Secrets externalize configuration from container images. A ConfigMap stores non-sensitive key-value pairs. A Secret stores sensitive data like passwords and tokens in base64 encoding (not encrypted by default, which is a common source of security misconfigurations). Both can be injected into pods as environment variables or mounted as files.

Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) provide storage that outlives individual pods. This abstraction lets applications request storage without knowing the underlying infrastructure, whether it is a cloud disk, an NFS share, or a local SSD.

Resource Management and Autoscaling

Resource requests and limits are critical but often misunderstood. A request is the minimum amount of CPU and memory a pod needs, used by the scheduler to place pods. A limit is the maximum a pod is allowed to use. If a pod exceeds its memory limit, it is killed (OOMKilled). If it exceeds its CPU limit, it is throttled.

Getting these values right is an art. Requests that are too high waste resources (nodes appear full when they are not). Requests that are too low lead to overcommitment. Limits that are too tight cause unnecessary kills. The Vertical Pod Autoscaler (VPA) and tools like Goldilocks can help by observing actual resource usage and recommending appropriate values.

The Kubernetes Ecosystem

The CNCF landscape contains over 1,000 projects extending Kubernetes. Helm provides package management (charts that template related resources). Prometheus provides metrics collection and alerting. Grafana provides visualization. Istio and Linkerd provide service mesh capabilities including traffic management, mutual TLS, and distributed tracing.

ArgoCD and Flux implement GitOps, a pattern where the desired state of a cluster is stored in a Git repository, and a controller continuously reconciles the cluster to match. Changes to the cluster are made by committing to Git, providing a complete audit trail and the ability to roll back any change.

Common Patterns and Pitfalls

The health check trap: Liveness probes that are too aggressive (short timeout, frequent checks) can cause cascading restarts during temporary load spikes. A pod that is slow to respond under heavy load is killed and restarted, adding to the load and causing more pods to fail. Use startup probes for slow-starting applications and set generous timeouts on liveness probes.

YAML fatigue: Kubernetes configuration is verbose. A simple deployment with a service, ingress, configmap, and HPA can easily require 200+ lines of YAML. Tools like Helm, Kustomize, and cdk8s help manage this complexity, but the underlying verbosity remains a common criticism.

Key Takeaways

Kubernetes descends from Google's internal Borg system and implements declarative, self-healing container orchestration.
The control plane (etcd, API server, scheduler, controller manager) makes decisions, while worker nodes (kubelet, kube-proxy, container runtime) execute them.
Pods, Deployments, Services, ConfigMaps, and PersistentVolumes are the core building blocks for running applications.
Resource requests and limits are critical for cluster stability but notoriously difficult to tune correctly.
The CNCF ecosystem (Helm, Prometheus, ArgoCD, Istio) extends Kubernetes with package management, observability, GitOps, and service mesh capabilities.
Kubernetes won the orchestration war through extensibility (CRDs, Operators) and vendor neutrality (CNCF governance, managed services from all major clouds).
For small teams, Kubernetes may be overkill. For organizations managing hundreds of services across multiple regions, nothing else matches its capabilities.

The cost of managed services: While GKE, EKS, and AKS simplify cluster operations, they introduce cloud vendor lock-in and ongoing costs that can be substantial at scale. Organizations must weigh the operational savings of managed Kubernetes against the recurring infrastructure expense and the risk of being dependent on a single cloud provider.

The Honest Assessment

Kubernetes is powerful, but it is not simple. Running your own cluster is an operational burden that many teams underestimate. For small teams with a handful of services, Kubernetes may be overkill. For organizations managing hundreds of microservices across multiple regions, it provides patterns and primitives that no other system matches. Understanding how it actually works, rather than treating it as a black box, is the difference between using Kubernetes effectively and fighting it constantly.