PodDisruptionBudgets That Survive an EKS or GKE Node Upgrade

A PodDisruptionBudget (PDB) that both protects availability and lets a managed node upgrade finish needs maxUnavailable: 1 (or minAvailable set below your replica count), at least two replicas spread across nodes, and unhealthyPodEvictionPolicy: AlwaysAllow so a crash-looping pod cannot wedge the drain. Set minAvailable equal to the replica count, or maxUnavailable: 0, and you have told Kubernetes to never voluntarily evict, which means the node drain never completes. The rest depends on which cloud you run, because EKS and GKE apply different timeouts and different force behaviour when your budget refuses an eviction.

This is the failure mode behind most "the cluster upgrade hung overnight" incidents. The PDB is doing exactly what you asked. You asked for the wrong thing.

What a PodDisruptionBudget actually controls

A PDB caps how many pods of one application can be down at once from voluntary disruptions: node drains, autoscaler compaction, rolling node upgrades. It does nothing for involuntary disruptions like a hardware fault or a kernel OOM kill.

The mechanism is the Eviction API, not deletion. <cite index="0-1">Cluster managers and hosting providers should use tools which respect PodDisruptionBudgets by calling the Eviction API instead of directly deleting pods, and kubectl drain periodically retries failed eviction requests until all pods on the target node are terminated or a configurable timeout is reached.</cite> When an eviction would breach the budget, <cite index="5-1">the API returns 429 Too Many Requests because the eviction is not currently allowed by the configured PodDisruptionBudget.</cite> The drainer backs off and retries. That retry loop is where upgrades either succeed slowly or stall completely.

You specify the budget one of two ways, and only one per PDB. <cite index="1-3">You can specify only one of maxUnavailable and minAvailable in a single PodDisruptionBudget, and maxUnavailable can only be used to control the eviction of pods that all have the same associated controller managing them.</cite> For a Deployment or StatefulSet, maxUnavailable: 1 is usually the honest expression of intent: "take one at a time." A minimal example:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: payments
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: payments-api

The setting that guarantees a stuck drain

The single most common misconfiguration is a budget that permits zero disruption. <cite index="1-1">If you set maxUnavailable to 0% or 0, or you set minAvailable to 100% or the number of replicas, you are requiring zero voluntary evictions, and you cannot successfully drain a node running one of those pods: if you try, the drain never completes.</cite> The API server accepts this. It is a valid budget. It just makes the node undrainable, and a managed upgrade that cannot drain a node will either fail or force-kill your pods, depending on the cloud.

The corollary: a PDB is only meaningful if the workload has more replicas than the budget requires available. Two replicas with maxUnavailable: 1 works. One replica with any PDB is a contradiction, because evicting the only pod always breaches the budget.

What EKS does during a managed node group update

EKS managed node groups replace nodes, they do not upgrade them in place. The update runs in phases and the drain step has a hard timeout you cannot ignore.

<cite index="12-2">EKS cordons the old node as soon as a new node reaches Ready state, then drains the pods; if the pods don't leave the node within 15 minutes and there's no force flag, the upgrade phase fails with a PodEvictionFailure error, and you can apply the force flag with the update-nodegroup-version request to delete the pods.</cite> It also paces itself: <cite index="12-2">it waits 60 seconds after evicting every pod before sending a termination request to the Auto Scaling group.</cite>

Parallelism comes from updateConfig. <cite index="12-1">EKS determines the maximum quantity of nodes to upgrade in parallel using the updateConfig property; the maximum unavailable has a quota of 100 nodes and the default value is one node.</cite>

The trade-off is stark and it is worth stating plainly. Without --force, EKS respects your PDB but gives up after 15 minutes per node and fails the update. With --force, EKS deletes the pods regardless of the budget. So a too-strict PDB does not protect you on EKS: it converts into either a failed upgrade or, the moment someone adds --force to get unblocked, a hard delete that ignores availability entirely. The cost lands later, usually at 2am, when the person clearing the stuck upgrade reaches for --force without checking whether the workload can survive it.

# Respects PDBs; fails after 15 min per node if a budget blocks eviction
aws eks update-nodegroup-version \
  --cluster-name prod --nodegroup-name app-ng \
  --kubernetes-version 1.31

# Same, but deletes blocking pods instead of failing
aws eks update-nodegroup-version \
  --cluster-name prod --nodegroup-name app-ng \
  --kubernetes-version 1.31 --force-update-enabled

One more EKS asymmetry to internalise: upgrades honour budgets, scaling operations do not. <cite index="6-1">PodDisruptionBudgets aren't respected when terminating a node with AZRebalance or when reducing the desired node count; those actions try to evict pods, but if it takes more than 15 minutes the node is terminated regardless.</cite> If you rely on the cluster autoscaler scaling in, your PDB is advisory at best. To buy more drain time on scale-in, add an Auto Scaling group lifecycle hook.

What GKE does during a surge upgrade

GKE's default is gentler and its timeout is longer. <cite index="9-0">By default, node pools use the surge upgrade strategy with maxSurge=1 and maxUnavailable=0, so only one surge node is added at a time and only one node is upgraded at a time.</cite> Parallelism is the sum of the two knobs: <cite index="11-2">maxSurge controls the additional nodes added temporarily, maxUnavailable controls how many can be simultaneously unavailable, and (maxUnavailable + maxSurge) determines how many nodes are upgraded at the same time.</cite>

The drain behaviour is where GKE differs most from EKS. <cite index="14-0">GKE drains the existing node respecting PodDisruptionBudget and GracefulTerminationPeriod settings for up to one hour, and after one hour any remaining pods are forcefully evicted so the upgrade can proceed.</cite> So a restrictive PDB buys you an hour, not a failed job, and then GKE forces through anyway. <cite index="13-1">If your PodDisruptionBudget object is too restrictive, it can prevent a graceful eviction from ever succeeding</cite> within that window.

Two GKE-specific traps. Spot capacity ignores your surge settings: <cite index="9-1">for nodes that use Spot VMs, surge upgrade values are ignored because there is no availability guarantee, and old nodes are drained directly without waiting for surge nodes to be ready.</cite> And parallel drains break a common Service setting: GKE's own guidance is that speeding up upgrades with PDBs assumes <cite index="10-1">you are not using externalTrafficPolicy: Local, which does not work with parallel node drains.</cite>

# Default: safe and slow, one node at a time
gcloud container node-pools update app-pool \
  --cluster=prod --max-surge-upgrade=1 --max-unavailable-upgrade=0

# Faster, no extra capacity, relies on PDBs to bound disruption
gcloud container node-pools update app-pool \
  --cluster=prod --max-surge-upgrade=0 --max-unavailable-upgrade=1

How the three managed platforms compare

The AKS column reflects that <cite index="15-0">a PDB with minAvailable equal to the replica count (or maxUnavailable: 0) prevents any pod from being evicted, causing the upgrade to fail with a "Cannot evict pod as it would violate the pod's disruption budget" error.</cite> The lesson is identical across all three: a zero-disruption budget does not mean zero downtime, it means a broken upgrade.

The setting that keeps a bad pod from blocking everything

Even a correctly sized budget can jam if a pod is unhealthy, because of how the default policy counts. <cite index="3-1">The default behaviour when no unhealthyPodEvictionPolicy is specified corresponds to IfHealthyBudget, under which running-but-not-yet-healthy pods can be evicted only if the guarded application is not disrupted, which has negative implications for draining nodes that can be blocked by misbehaving applications, more specifically pods in CrashLoopBackOff or pods that fail to report Ready.</cite>

Picture a Deployment of three replicas with minAvailable: 2, where one pod is crash-looping. Only two are healthy, so the budget is already at its floor, so the API refuses to evict anything, including the broken pod. The drain sits there. This is why the upstream recommendation is explicit: <cite index="0-2">set the AlwaysAllow unhealthy pod eviction policy on your PodDisruptionBudgets to support eviction of misbehaving applications during a node drain, because the default behaviour waits for pods to become healthy before the drain can proceed.</cite> Add unhealthyPodEvictionPolicy: AlwaysAllow to every PDB unless you have a specific reason not to.

What to check before you touch a node

Do this audit before any node pool upgrade or cluster version bump. It takes ten minutes and prevents the stuck-drain incident.

• Find zero-disruption budgets. List every PDB and flag any with maxUnavailable: 0, minAvailable at 100%, or minAvailable equal to the workload's replica count. Those are the ones that stall drains.
• Check replica counts against budgets. kubectl get pdb -A shows ALLOWED DISRUPTIONS. If a PDB shows 0 allowed disruptions right now, a drain touching those pods will block. Fix the replica count or the budget before you start.
• Confirm spread. Two replicas mean nothing if both sit on the same node. Use topology spread constraints or anti-affinity so an eviction on one node does not take out the whole application.
• Set the eviction policy. Add unhealthyPodEvictionPolicy: AlwaysAllow to workloads that can crash-loop.
• Look for singletons. Any Deployment with one replica and a PDB is a guaranteed drain blocker. Either scale it up or accept it has no availability guarantee and remove the PDB.

# Budgets that currently allow no disruption
kubectl get pdb -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed \
  | awk '$3==0'

If you are moving existing workloads onto managed Kubernetes as part of a datacentre exit, PDB and topology hygiene is exactly the kind of detail that turns a clean cloud migration into a bad first on-call week, so bake it into the landing configuration rather than retrofitting it. And if your upgrades already hurt, the underlying question is usually observability and rollout design, which is the remit of our reliability and SRE practice rather than a YAML tweak. For teams standardising this across many clusters, encoding these budgets into the platform templates is a platform and DevOps concern.

Frequently asked questions

Does a PodDisruptionBudget prevent downtime during a node upgrade?

Not by itself. A PDB only limits how many pods are evicted at once during voluntary disruptions, so it needs enough healthy replicas spread across nodes to have anything to protect. It does nothing for involuntary disruptions like node crashes, and if set too strictly it blocks the drain rather than preventing downtime.

Why is my EKS node group update stuck or failing with PodEvictionFailure?

EKS drains each node and fails the phase if pods have not left within 15 minutes when no force flag is set. The usual cause is a PDB with maxUnavailable: 0, minAvailable at the replica count, or a workload with too few replicas to allow any eviction. Fix the budget or scale up; --force-update-enabled clears the block but deletes the pods regardless of availability.

What is the difference between minAvailable and maxUnavailable?

Both express the same budget from opposite directions, and you set only one per PDB. minAvailable states how many pods must stay up; maxUnavailable states how many can be down at once. For most Deployments maxUnavailable: 1 is clearest because it does not silently change meaning when you scale the replica count.

Should I set unhealthyPodEvictionPolicy to AlwaysAllow?

For most workloads, yes. The default IfHealthyBudget policy refuses to evict unhealthy pods when the budget is already at its floor, so a single crash-looping pod can block a node drain indefinitely. AlwaysAllow lets the drainer evict misbehaving pods so the upgrade proceeds, which is almost always what you want during maintenance.

How long does GKE wait for a PDB before forcing an upgrade?

Up to one hour per node. GKE respects the PodDisruptionBudget and graceful termination period during a surge upgrade, and after an hour it forcefully evicts any remaining pods so the upgrade can continue. A restrictive PDB therefore delays a GKE upgrade rather than failing it, but it does not stop the eventual disruption.