When using Cilium in ipam: cluster-pool mode, the Cilium operator manages a global address
pool itself and splits that pool into PodCIDR blocks assigned to each node through the
CiliumNode object. The mask applied to each block directly determines how many pods can run
on the node simultaneously. The default mask is /24, which allows 254 IPs per node.
On beefy nodes (lots of CPU/RAM) this limit is reached quickly: it becomes impossible to
schedule new pods until the existing ones have been rescheduled elsewhere. The way out is to
increase the size of the CIDR allocated per node (for instance switching to a /23 or a
/22) and to deploy the change live, node by node.
The mask applied by default to each node is configured via the Cilium Helm values, more
precisely through ipam.operator.clusterPoolIPv4Mask and the pool
ipam.operator.clusterPoolIPv4PodCIDRList. Here is the configuration I use to switch from a
/24 to a /22 (1022 IPs per node):
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4Mask: "22"
clusterPoolIPv4PodCIDRList:
- 172.16.0.0/16
If you have existing CiliumNode objects with a different mask allocated via
spec.ipam.ipamConfig, the operator will honor that override and won’t allocate a new block
with the default mask as long as the override remains. You therefore have to either remove
those overrides or set a per-node ipamConfig.podCIDR consistent with the new size.
Be aware that changing clusterPoolIPv4Mask does not automatically reallocate the CIDRs
already assigned to existing nodes: those nodes keep their old block. You therefore have to
perform a hot rollout, node by node, and force the PodCIDR reallocation for each node.
The idea is to force, for each node, the reallocation of its PodCIDR by removing the first
entry of the spec.ipam.podCIDRs array on the CiliumNode. The Cilium operator sees that no
CIDR is allocated anymore and reallocates a new one from the pool, sliced according to the
new mask configured.
This is precisely why the cluster-pool mode lends itself well to this operation: unlike
kubernetes mode where the CIDR is allocated by kube-controller-manager on the Node
object (and Cilium has no way to reallocate it itself), here everything is driven by the
Cilium operator through the CiliumNode, and the reallocation is immediate.
Once the new CIDR is allocated, the remaining work is to delete the pods that still hold an
IP in the old block (otherwise Cilium cannot reschedule them as long as they exist in its
state), and to restart the cilium pod itself so that it reloads the new IPAM configuration.
We therefore drain the node, patch the CiliumNode, delete the Cilium pod, delete the
residual pods still on the old block, then uncordon.
The operation is repeated for each node one by one, which keeps the cluster fully operational during the rollout.
I use the following script, run on a host with kubectl and jq configured. It takes the
node name as argument:
#!/usr/bin/env bash
set -uo pipefail
NODE="${1:-}"
if [[ -z "$NODE" ]]; then
echo "Usage: $0 <node-name>"
exit 1
fi
echo "========================================"
echo "Target node: ${NODE}"
echo "========================================"
#
# 1. CORDON
#
echo
echo "[1/6] Cordoning node..."
if ! kubectl cordon "$NODE"; then
echo "ERROR: Failed to cordon node ${NODE}"
exit 1
fi
#
# 2. DRAIN
#
echo
echo "[2/6] Draining node..."
if ! kubectl drain "$NODE" \
--ignore-daemonsets \
--delete-emptydir-data \
--timeout=5m
then
echo "ERROR: Failed to drain node ${NODE}"
exit 1
fi
#
# 3. REMOVE FIRST POD CIDR
#
echo
echo "[3/6] Updating CiliumNode/${NODE}..."
POD_CIDR=$(kubectl get ciliumnode "$NODE" \
-o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)
if [[ -z "$POD_CIDR" ]]; then
echo "ERROR: No podCIDR found in CiliumNode/${NODE}"
exit 1
fi
echo "Current first podCIDR: ${POD_CIDR}"
if ! kubectl patch ciliumnode "$NODE" \
--type='json' \
-p='[{"op":"remove","path":"/spec/ipam/podCIDRs/0"}]'
then
echo "ERROR: Failed to patch CiliumNode/${NODE}"
exit 1
fi
#
# 4. DISPLAY NEW POD CIDR
#
echo
echo "[4/6] Waiting 1 second..."
sleep 1
NEW_POD_CIDR=$(kubectl get ciliumnode "$NODE" \
-o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)
echo "New first podCIDR: ${NEW_POD_CIDR:-<none>}"
#
# 5. DELETE CILIUM POD
#
echo
echo "[5/6] Deleting Cilium pod on ${NODE}..."
CILIUM_POD=$(kubectl get pods \
-n kube-system \
--field-selector "spec.nodeName=${NODE}" \
-l k8s-app=cilium \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [[ -n "$CILIUM_POD" ]]; then
echo "Deleting Cilium pod: ${CILIUM_POD}"
if ! kubectl delete pod "$CILIUM_POD" -n kube-system; then
echo "WARNING: Failed to delete Cilium pod"
fi
else
echo "WARNING: No Cilium pod found on ${NODE}"
fi
#
# 6. DELETE PODS WITH 172.x.x.x IP
#
echo
echo "[6/6] Finding pods with IP starting with 172..."
PODS=$(kubectl get pods \
--all-namespaces \
--field-selector "spec.nodeName=${NODE}" \
-o json 2>/dev/null)
if [[ -z "$PODS" ]]; then
echo "ERROR: Failed to retrieve pods from node ${NODE}"
exit 1
fi
echo "$PODS" |
jq -r '
.items[]
| select(.status.podIP != null)
| select(.status.podIP | startswith("172"))
| [
.metadata.namespace,
.metadata.name,
.status.podIP
]
| @tsv
' |
while IFS=$'\t' read -r NAMESPACE POD IP; do
echo "Deleting ${NAMESPACE}/${POD} (${IP})"
if ! kubectl delete pod "$POD" -n "$NAMESPACE"; then
echo "WARNING: Failed to delete ${NAMESPACE}/${POD}"
fi
done
echo "Uncordon ${NODE}"
kubectl uncordon ${NODE}
echo
echo "========================================"
echo "Script completed"
echo "========================================"
Once the script is saved as /tmp/cilium_host_rollout.sh, just run it on each node, one
after the other:
for NODE in $(kubectl get node | awk '{print $1}'); do /tmp/cilium_host_rollout.sh ${NODE}; done
The loop processes one node at a time and only moves to the next once the script returns. You can of course remove a node from the list (control plane, nodes under maintenance, etc.) by filtering on the labels you want.
On a kube-node801 node, we observe the transition from an old /26 to the new /24
allocated by the operator:
========================================
Target node: kube-node801
========================================
[1/6] Cordoning node...
node/kube-node801 cordoned
[2/6] Draining node...
node/kube-node801 already cordoned
Warning: ignoring DaemonSet-managed Pods: monitoring/node-exporter-m86h7, kube-system/cilium-bs7j6, logging/fluent-bit-rbtv7
evicting pod database/postgres-vacuum-cron-29802720-n4r6m
evicting pod database/postgres-vacuum-cron-29782560-czmj5
evicting pod database/postgres-vacuum-cron-29792640-p887q
pod/postgres-vacuum-cron-29782560-czmj5 evicted
pod/postgres-vacuum-cron-29802720-n4r6m evicted
pod/postgres-vacuum-cron-29792640-p887q evicted
node/kube-node801 drained
[3/6] Updating CiliumNode/kube-node801...
Current first podCIDR: 172.20.84.192/26
ciliumnode.cilium.io/kube-node801 patched
[4/6] Waiting 1 second...
New first podCIDR: 172.18.28.0/24
[5/6] Deleting Cilium pod on kube-node801...
Deleting Cilium pod: cilium-bs7j6
pod "cilium-bs7j6" deleted from kube-system namespace
[6/6] Finding pods with IP starting with 172...
Deleting monitoring/node-exporter-m86h7 (172.20.84.227)
pod "node-exporter-m86h7" deleted from monitoring namespace
Deleting logging/fluent-bit-rbtv7 (172.20.84.194)
pod "fluent-bit-rbtv7" deleted from logging namespace
Uncordon kube-node801
node/kube-node801 uncordoned
The drain re-evicts the non-DaemonSet pods, the Cilium DaemonSet is then deleted manually
to force the IPAM state to be reloaded, and finally the residual pods (typically the other
DaemonSets like node-exporter or fluent-bit) whose IP still belongs to the old block are
deleted so that Cilium can reassign them an IP in the new block on the next scheduling.
This procedure lets you switch a Cilium cluster running in cluster-pool IPAM mode from one
PodCIDR mask to another with no global service interruption. By processing the nodes one by
one and using cordon/drain, only the node currently being migrated momentarily loses its
application pods, which is perfectly acceptable on most production clusters as long as you
have at least two replicas per critical workload.