Why Your Kubernetes Pods Keep Getting Evicted: A Practical Node Pressure Guide for SMBs

The Event You Can’t See Coming: Pod Evictions and Node Pressure

Every SRE has a story that starts the same way: “The pods just disappeared.” One morning your monitoring shows a handful of replicas restarting for no apparent reason, users report intermittent 503s, and when you finally look, Kubernetes has been evicting your pods for minutes — because the node was under pressure. The loudest symptom is an eviction; the underlying cause is almost always a resource or storage problem you were not watching.

Pod evictions are a normal, even healthy, part of how Kubernetes maintains a stable node. The scheduler, kubelet, and the eviction manager exist to prevent a single node from melting down. The failure mode for SMB teams is that evictions happen invisibly — no alert, no postmortem, just degraded service that drains on-call energy. This article walks through the real causes, how to detect them early, and how to stop them from becoming a habit. If reliability is a core goal, this is the operational layer that sits directly under your error budgets and SLOs.

The Two Kinds of Eviction You Actually See

There are two main ways a pod gets killed that you need to distinguish, because the fixes are completely different.

1. Kubelet-driven evictions (node pressure)

When a node crosses a threshold — memory, disk, or inodes — the kubelet’s eviction manager steps in. It starts by evicting the pod that is using the most of the pressured resource. The thresholds are controlled by evictionHard y evictionSoft settings in the kubelet configuration. When you see the event Evicted with a reason like MemoryPressure or EphemeralStorageLimit, this is what happened.

2. Disruption-driven evictions (voluntary)

When you drain a node, when the cluster autoscaler scales a node down, or when a Descheduler pass moves workloads, pods are evicted voluntarily. These usually respect PodDisruptionBudgets (PDBs) — unless you forgot to define one, in which case everything goes at once.

The distinction matters because “voluntary, PDB-respecting” evictions are graceful, while “kubelet emergency eviction” is the kubelet saying it is out of patience. If your only alert is “pod restarted,” you cannot tell which one hit you.

How to Detect an Eviction Problem Before It Hurts Users

The single most valuable habit is to treat evictions as a first-class signal, not background noise. Kubernetes already records them as events and, when a PDB or toleration is involved, as Evicted-reason pods. Turn that raw data into an alert.

# See recent evictions across all namespaces
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -50

# Or via jsonpath to get them as clean lines
kubectl get events -A -o json | jq -r \
  '.items[] | select(.reason=="Evicted") | [.lastTimestamp,.involvedObject.namespace,.involvedObject.name,.message] | @tsv'

To make this actually useful, add a Prometheus recording rule that surfaces evictions per node and per namespace so they show up in your dashboard and alert rules:

# prometheus rules (promql snippet for a recording rule)
groups:
  - name: evictions
    rules:
      - record: node:evictions:rate5m
        expr: |-
          sum by (node) (rate(kube_pod_status_reason{reason="Evicted"}[5m])) > 0

Pair that with node condition metrics so you see the cause alongside the symptom:

# Alert when a node is under memory or disk pressure for 5 minutes
- alert: NodeMemoryPressure
  expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Node {{ $labels.node }} is under memory pressure"

With these in place, an eviction becomes a pager-worthy event you can investigate calmly, rather than a mystery you discover after support calls. This is exactly the kind of alerting hygiene that separates useful signal from noise.

The Real Fixes: Pressure, Requests, and PodDisruptionBudgets

Here is where the practical work happens. In my experience, SMB eviction problems almost always trace back to one of three root causes.

1. You are over-packing nodes

The most common cause of node pressure is asking a node to do too much. The fix is not necessarily more nodes — it is honest resource requests and limits. If every container requests 500m CPU but uses 50m, the scheduler packs the node assuming 500m and the node starves. Use a resource validator to catch requests that are wildly out of line with real usage:

# Quick scan for pods whose CPU request is way above a sane baseline
kubectl top pods -A --sort-by=cpu | head -20

# Right-size with the Vertical Pod Autoscaler recommender (dry run mode)
kubectl apply -f - <<'EOF'
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: rightsize-api
  namespace: default
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"
EOF
kubectl get vpa rightsize-api -o jsonpath='{.status.recommendations.containerRecommendations}'

The updateMode: Off flag means the VPA only recommends; it won’t restart your pods. You read the recommendations, sanity-check them, and apply them in a real deployment. That gives you the density benefit of tight requests without the “out of nowhere” restarts.

2. Ephemeral storage and inode pressure

Disk pressure is a sneakier killer than memory. A container that writes logs or temp files to the writable layer can grow a node’s /var/lib/kubelet until the eviction manager steps in. Cap each pod’s ephemeral storage and watch inode usage — inode exhaustion happens well before a full disk:

# Pod with a hard cap on scratch space
resources:
  limits:
    ephemeral-storage: 1Gi
  requests:
    ephemeral-storage: 256Mi

# Check inode usage per node filesystem
df -i /var/lib/kubelet

For logging-heavy workloads, redirect output to a bounded volume or a log aggregator instead of letting it pile up in the container layer. If your team hasn’t yet, this is also the moment to review your backup and storage strategy so that eviction-plus-lost-data never becomes a disaster.

3. Missing PodDisruptionBudgets

Voluntary evictions (drains, autoscaler scale-downs) are supposed to be safe. They are only safe if you define a PDB, otherwise the drainer can take down every replica at once. A one-line PDB converts “all replicas down” into “one downtime window I chose”:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Add a PDB to every workload that matters, and suddenly node drains and autoscaler events become boring, scheduled, and safe — which is precisely how a lean team avoids the on-call burnout that follows unexplained outages.

Turn Evictions from Mystery into a Managed Event

Evictions are not a bug in Kubernetes; they are the kubelet doing its job. The problem is purely that they often arrive unexpectedly because nobody was watching the pressure signals. The fix is threefold and cheap: alert on evictions and node pressure, right-size your requests so nodes aren’t secretly overloaded, and define PDBs so voluntary disruptions are graceful. Do those three things and the “pods just disappeared” story becomes a thing you read about, not a thing that wakes you up at 3 a.m.

If pod evictions, node pressure, or an unpredictable cluster are eating into your team’s nights, you don’t have to figure it out alone. Book a free session with our SRE team and we’ll audit your cluster’s eviction and resource-health setup and give you a concrete fix list.