A note on Kubernetes' disaster recovery, StatefulSets and Helm charts
Tuesday 27 October 2026 José Carrasquel Vera 71 views
Kubernetes Sysadmin Docker Deployment DevOps
https://cylab.be/blog/532/a-note-on-kubernetes-disaster-recovery-statefulsets-and-helm-charts
A couple weeks ago I gave a demo in Cylab about Kubernetes. One of the main goals was to show that it is not so complex and that, perhaps, it can be a good way to put spare hardware to work. I was proud to show how easily Zulip can be deployed with Helm charts.
But then I got a bit cocky and decided to show what happens to Zulip if a Kubernetes node died ungracefully. The quick auto recovery I was hinting never really happened. Instead, the application was down and never recovered.
Below is a small writeup about this and how I improved disaster recovery for Zulip.
Zulip
Zulip is a chat app relying on the following elements:
- Zulip server
- PostgreSQL database
- Redis
- RabbitMQ
- memcached
The problem
The developers maintain a Helm chart which deploys:
- A replica 1 StatefulSet for Zulip
- A replica 1 StatefulSet for PostgreSQL
- A replica 1 StatefulSet for Redis
- A replica 1 StatefulSet for RabbitMQ
- A replica 1 Deployment for Memcached
NAME READY STATUS RESTARTS AGE
pod/zulip-0 1/1 Running 0 3m32s
pod/zulip-memcached-5475b64f65-lzhg2 1/1 Running 0 17h
pod/zulip-postgresql-0 1/1 Running 0 4m33s
pod/zulip-rabbitmq-0 1/1 Running 0 17h
pod/zulip-redis-master-0 1/1 Running 0 17h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/zulip-memcached 1/1 1 1 17h
NAME READY AGE
statefulset.apps/zulip 1/1 3m33s
statefulset.apps/zulip-postgresql 1/1 4m34s
statefulset.apps/zulip-rabbitmq 1/1 17h
statefulset.apps/zulip-redis-master 1/1 17h
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-zulip-postgresql-0 Bound pvc-ae02f1d2-c9a5-404e-beab-85e0958d7ac2 8Gi RWO ceph-block <unset> 8m53s
zulip-data Bound pvc-0a71b7fb-33c5-429e-b26f-c7f775b477f0 3Gi RWO ceph-block <unset> 17h
This chart is just fine for simple (even production) Deployments. The main problem arises when you need high availability when a node goes down. In this case, Kubernetes will make sure that the pods bound to the persistentVolumes have really terminated before it fires up new ones. This can cause very long down times and may even require human intervention for recovery. Read Non graceful node shutdown or experiment yourself.
The Solution: Only Standard deployments and cloud native tools
The Zulip scalability documentation recommends:
Disaster recovery: One can easily run a warm spare application server and a warm spare database (using PostgreSQL warm standby replicas). Make sure the warm spare application server has copies of /etc/zulip and you’re either syncing LOCAL_UPLOADS_DIR or using the S3 file uploads backend.
Care must be taken when dividing traffic for a single Zulip realm between multiple Zulip application servers, which is why we recommend a hot spare over load-balancing for most installations desiring extra redundancy.
So the first step would be to replace the PostgreSQL StatefulSet for something more robust. Enter CloudNativePG:
CloudNativePG is the Kubernetes operator that covers the full lifecycle of a highly available PostgreSQL database cluster with a primary/standby architecture, using native streaming replication.
It lets us deploy PostgreSQL clusters with a simple declaration:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgresql-cluster
namespace: zulip
spec:
instances: 3
storage:
size: 1Gi
storageClass: openebs-hostpath
# No two pods in the same node.
affinity:
enablePodAntiAffinity: true
topologyKey: kubernetes.io/hostname
podAntiAffinityType: required
postgresql:
parameters:
synchronous_commit: "on"
Which deploys:
jose@debian-k8s0:~$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/postgresql-cluster-1 1/1 Running 0 17h
pod/postgresql-cluster-2 1/1 Running 0 17h
pod/postgresql-cluster-3 1/1 Running 0 17h
pod/zulip-0 1/1 Running 0 61m
pod/zulip-memcached-5475b64f65-lzhg2 1/1 Running 0 17h
pod/zulip-postgresql-0 1/1 Running 0 62m
pod/zulip-rabbitmq-0 1/1 Running 0 17h
pod/zulip-redis-master-0 1/1 Running 0 17h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/postgresql-cluster-r ClusterIP 10.97.165.48 <none> 5432/TCP 17h
service/postgresql-cluster-ro ClusterIP 10.97.230.9 <none> 5432/TCP 17h
service/postgresql-cluster-rw ClusterIP 10.111.100.72 <none> 5432/TCP 17h
service/zulip ClusterIP 10.98.97.34 <none> 80/TCP 17h
service/zulip-memcached ClusterIP 10.98.213.8 <none> 11211/TCP 17h
service/zulip-postgresql ClusterIP 10.109.204.154 <none> 5432/TCP 62m
service/zulip-postgresql-hl ClusterIP None <none> 5432/TCP 62m
service/zulip-rabbitmq ClusterIP 10.103.58.19 <none> 5672/TCP,4369/TCP,25672/TCP,15672/TCP 17h
service/zulip-rabbitmq-headless ClusterIP None <none> 4369/TCP,5672/TCP,25672/TCP,15672/TCP 17h
service/zulip-redis-headless ClusterIP None <none> 6379/TCP 17h
service/zulip-redis-master ClusterIP 10.108.28.24 <none> 6379/TCP 17h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/zulip-memcached 1/1 1 1 17h
NAME READY AGE
statefulset.apps/zulip 1/1 61m
statefulset.apps/zulip-postgresql 1/1 62m
statefulset.apps/zulip-rabbitmq 1/1 17h
statefulset.apps/zulip-redis-master 1/1 17h
jose@debian-k8s0:~$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-zulip-postgresql-0 Bound pvc-ae02f1d2-c9a5-404e-beab-85e0958d7ac2 8Gi RWO ceph-block <unset> 62m
postgresql-cluster-1 Bound pvc-c087548f-d6f7-434f-9fe9-d5e5e9dbcc4c 1Gi RWO openebs-hostpath <unset> 17h
postgresql-cluster-2 Bound pvc-173384d1-eb90-409e-87ce-0029cc3cbdbd 1Gi RWO openebs-hostpath <unset> 17h
postgresql-cluster-3 Bound pvc-08f23c31-6a05-402b-a8b3-28fabf53c91a 1Gi RWO openebs-hostpath <unset> 17h
zulip-data Bound pvc-0a71b7fb-33c5-429e-b26f-c7f775b477f0 3Gi RWX ceph-filesystem <unset> 17h
Here, we have changed the storageClass to use openEBS’ hostPath to avoid nested redundancy for storage: ceph-block is configured as replica 3, resulting in 9 replicas of the database’s storage. Observe that the postgresql-cluster pods are not governed by Deployments or StatefulSets. In fact they are controlled by the controller itself, see this.
Observe also that 3 services have been created for the PostgreSQL cluster.
jose@debian-k8s0:~$ kubectl get endpointslices.discovery.k8s.io
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
postgresql-cluster-r-f2vvs IPv4 5432 10.244.3.241,10.244.5.165,10.244.0.101 18h
postgresql-cluster-ro-k47ws IPv4 5432 10.244.5.165,10.244.0.101 18h
postgresql-cluster-rw-fkx5c IPv4 5432 10.244.3.241 18h
The controller takes care of putting the leader pod in the rw service and replacing it it case it becomes unavailable.
Now that we have an external PostgreSQL databse, we can adapt Zulip’s Helm chart values to according to this:
postgresql:
enabled: false
externalPostgresql:
host: "postgresql-cluster-rw"
port: 5432
database: "app"
user: "app"
## password supports both scalar values and valueFrom maps.
password:
valueFrom:
secretKeyRef:
name: postgresql-cluster-app
key: password
sslmode: ""
...
Observe that the postgresql-cluster-app contains the necessary information to connect to the PostgreSQL cluster.
$ kubectl describe secrets postgresql-cluster-app
Name: postgresql-cluster-app
Namespace: zulip
Labels: app.kubernetes.io/managed-by=cloudnative-pg
cnpg.io/cluster=postgresql-cluster
cnpg.io/reload=true
cnpg.io/userType=app
Annotations: cnpg.io/operatorVersion: 1.30.1
Type: kubernetes.io/basic-auth
Data
====
dbname: 3 bytes
fqdn-jdbc-uri: 155 bytes
fqdn-uri: 136 bytes
host: 21 bytes
jdbc-uri: 137 bytes
password: 64 bytes
pgpass: 100 bytes
port: 4 bytes
uri: 118 bytes
user: 3 bytes
username: 3 bytes
After deployment, we get the following error in the zulip pod.
Waiting for PostgreSQL database to accept connections...
zulip-postgresql:5432 - no response
Many hours of searching gave me no solution as to why the externalPostgresql configuration is not being followed. So instead I tweaked the CloudNativePG cluster definition above to add a second service for rw named zulip-postgresql
managed:
services:
additional:
- selectorType: rw
serviceTemplate:
metadata:
name: "zulip-postgresql"
labels:
test-label: "true"
annotations:
test-annotation: "true"
spec:
type: ClusterIP
Which solved the connection issues
$ kubectl get service postgresql-cluster-rw zulip-postgresql
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
postgresql-cluster-rw ClusterIP 10.111.100.72 <none> 5432/TCP 45h
zulip-postgresql ClusterIP 10.110.221.31 <none> 5432/TCP 11m
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
postgresql-cluster-1 1/1 Running 0 45h 10.244.3.241 debian-k8s5 <none> <none>
postgresql-cluster-2 1/1 Running 0 45h 10.244.5.165 debian-k8s1 <none> <none>
postgresql-cluster-3 1/1 Running 0 45h 10.244.0.101 debian-k8s0 <none> <none>
zulip-0 1/1 Running 6 (22m ago) 55m 10.244.4.206 debian-k8s3 <none> <none>
zulip-memcached-5475b64f65-5f9g2 1/1 Running 0 74m 10.244.3.32 debian-k8s5 <none> <none>
zulip-rabbitmq-0 1/1 Running 0 74m 10.244.4.251 debian-k8s3 <none> <none>
zulip-redis-master-0 1/1 Running 0 74m 10.244.4.241 debian-k8s3 <none> <none>
This tweak shows that perhaps the Zulip Helm chart is not mature enough and should be handled with care.
Disaster recovery test
Let’s see what happens with zulip when the node debian-k8s3, holding the zulip and redis and RabbitMQ pods, goes down.
$ kubectl get pods -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
postgresql-cluster-1 1/1 Running 0 45h 10.244.3.241 debian-k8s5 <none> <none>
postgresql-cluster-2 1/1 Running 0 45h 10.244.5.165 debian-k8s1 <none> <none>
postgresql-cluster-3 1/1 Running 0 45h 10.244.0.101 debian-k8s0 <none> <none>
zulip-0 1/1 Running 6 (27m ago) 60m 10.244.4.206 debian-k8s3 <none> <none>
zulip-memcached-5475b64f65-5f9g2 1/1 Running 0 78m 10.244.3.32 debian-k8s5 <none> <none>
zulip-rabbitmq-0 1/1 Running 0 78m 10.244.4.251 debian-k8s3 <none> <none>
zulip-redis-master-0 1/1 Running 0 78m 10.244.4.241 debian-k8s3 <none> <none>
zulip-0 1/1 Running 6 (32m ago) 65m 10.244.4.206 debian-k8s3 <none> <none>
zulip-redis-master-0 1/1 Running 0 83m 10.244.4.241 debian-k8s3 <none> <none>
zulip-rabbitmq-0 1/1 Running 0 83m 10.244.4.251 debian-k8s3 <none> <none>
zulip-0 1/1 Running 6 (37m ago) 70m 10.244.4.206 debian-k8s3 <none> <none>
zulip-redis-master-0 1/1 Running 0 88m 10.244.4.241 debian-k8s3 <none> <none>
zulip-rabbitmq-0 1/1 Running 0 88m 10.244.4.251 debian-k8s3 <none> <none>
zulip-redis-master-0 1/1 Terminating 0 88m 10.244.4.241 debian-k8s3 <none> <none>
zulip-0 1/1 Terminating 6 (37m ago) 70m 10.244.4.206 debian-k8s3 <none> <none>
zulip-rabbitmq-0 1/1 Terminating 0 88m 10.244.4.251 debian-k8s3 <none> <none>
As described in the Kubernets Non graceful node shutdown, the zulip-0 pods stays in Terminating state and doesn’t come back. This effectively leads to Zulip being down. The fact that we are using RWX PVC is not enough. Zulip’s pod should be backed by a Deployment, which is not supported in the Zulip’s Helm chart.
Because of this, we’ll ditch the Helm chart alltogether and build the whole stack with native deployments without relying on the Helm Chart. The main idea is to take the definitions of the Helm chart and transform them into deployments. Moreover we replace Redis by Valkey.
Second disaster ricovery test
We see that most of the pods are scheduled in the debian-k8s1 node. Let’s turn it off:
$ kubectl get pods -w -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
memcached-54b8b9fcfd-rtls8 1/1 Running 0 14h 10.244.5.84 debian-k8s1 <none> <none>
postgresql-cluster-1 1/1 Running 0 14h 10.244.5.161 debian-k8s1 <none> <none>
postgresql-cluster-2 1/1 Running 0 14h 10.244.3.172 debian-k8s5 <none> <none>
postgresql-cluster-3 1/1 Running 0 14h 10.244.0.131 debian-k8s0 <none> <none>
rabbitmq-845b89f7dc-vw9s4 1/1 Running 1 (14h ago) 14h 10.244.5.81 debian-k8s1 <none> <none>
valkey-664b7dc89f-c5jps 1/1 Running 0 14h 10.244.5.146 debian-k8s1 <none> <none>
zulip-9dd9f84fd-lfpt2 1/1 Running 0 13h 10.244.5.53 debian-k8s1 <none> <none>
rabbitmq-845b89f7dc-vw9s4 1/1 Terminating 1 (14h ago) 14h 10.244.5.81 debian-k8s1 <none> <none>
valkey-664b7dc89f-c5jps 1/1 Terminating 0 14h 10.244.5.146 debian-k8s1 <none> <none>
postgresql-cluster-1 1/1 Terminating 0 14h 10.244.5.161 debian-k8s1 <none> <none>
valkey-664b7dc89f-xhd2x 0/1 Pending 0 0s <none> <none> <none> <none>
rabbitmq-845b89f7dc-mnlps 0/1 Pending 0 0s <none> <none> <none> <none>
rabbitmq-845b89f7dc-mnlps 0/1 Pending 0 0s <none> debian-k8s3 <none> <none>
valkey-664b7dc89f-xhd2x 0/1 Pending 0 0s <none> debian-k8s3 <none> <none>
rabbitmq-845b89f7dc-mnlps 0/1 ContainerCreating 0 0s <none> debian-k8s3 <none> <none>
zulip-9dd9f84fd-lfpt2 1/1 Terminating 0 13h 10.244.5.53 debian-k8s1 <none> <none>
valkey-664b7dc89f-xhd2x 0/1 Init:0/1 0 0s <none> debian-k8s3 <none> <none>
zulip-9dd9f84fd-psggt 0/1 Pending 0 0s <none> <none> <none> <none>
zulip-9dd9f84fd-psggt 0/1 Pending 0 0s <none> debian-k8s3 <none> <none>
zulip-9dd9f84fd-psggt 0/1 ContainerCreating 0 0s <none> debian-k8s3 <none> <none>
memcached-54b8b9fcfd-rtls8 1/1 Terminating 0 14h 10.244.5.84 debian-k8s1 <none> <none>
memcached-54b8b9fcfd-cqxgm 0/1 Pending 0 0s <none> <none> <none> <none>
valkey-664b7dc89f-xhd2x 0/1 Init:0/1 0 1s <none> debian-k8s3 <none> <none>
rabbitmq-845b89f7dc-mnlps 0/1 ContainerCreating 0 1s <none> debian-k8s3 <none> <none>
memcached-54b8b9fcfd-cqxgm 0/1 ContainerCreating 0 1s <none> debian-k8s5 <none> <none>
zulip-9dd9f84fd-psggt 0/1 ContainerCreating 0 10s <none> debian-k8s3 <none> <none>
zulip-9dd9f84fd-psggt 0/1 Running 0 10s 10.244.4.186 debian-k8s3 <none> <none>
memcached-54b8b9fcfd-cqxgm 0/1 Running 0 19s 10.244.3.137 debian-k8s5 <none> <none>
memcached-54b8b9fcfd-cqxgm 1/1 Running 0 24s 10.244.3.137 debian-k8s5 <none> <none>
valkey-664b7dc89f-xhd2x 0/1 Init:0/1 0 27s 10.244.4.12 debian-k8s3 <none> <none>
valkey-664b7dc89f-xhd2x 0/1 PodInitializing 0 28s 10.244.4.12 debian-k8s3 <none> <none>
valkey-664b7dc89f-xhd2x 0/1 Running 0 30s 10.244.4.12 debian-k8s3 <none> <none>
valkey-664b7dc89f-xhd2x 1/1 Running 0 31s 10.244.4.12 debian-k8s3 <none> <none>
rabbitmq-845b89f7dc-mnlps 0/1 Running 0 55s 10.244.4.18 debian-k8s3 <none> <none>
rabbitmq-845b89f7dc-mnlps 1/1 Running 0 88s 10.244.4.18 debian-k8s3 <none> <none>
zulip-9dd9f84fd-psggt 0/1 Running 0 90s 10.244.4.186 debian-k8s3 <none> <none>
zulip-9dd9f84fd-psggt 1/1 Running 0 90s 10.244.4.186 debian-k8s3 <none> <none>
We see that the whole application eventually recovers and we see no real issue in the frontend (Zulip browser). However, the recreation of the new pods actually took too much time. The default for Kubernetes is 5 minutes before pod rescheduling. This is probably too much for us but we can fix it on a per-pod basis using taint based eviction.
Kubernetes automatically adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300, unless you, or a controller, set those tolerations explicitly. These automatically-added tolerations mean that Pods remain bound to Nodes for 5 minutes after one of these problems is detected.
In our case, we will add it to the template of all our Deployments, allowing for a downtime of only 5 seconds. We don’t want to modify the default behaviour since this wouldn’t be necessary for replicated Deployments with pod PodAntiAffinity.
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 5
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 5
As an example here is our memcached Deployment Jinja2 template:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: memcached
namespace: zulip
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: memcached
template:
metadata:
labels:
app: memcached
spec:
containers:
- name: memcached
image: "{{ zulip_memcached_image }}"
args:
- "-m"
- "256"
- "-I"
- "1m"
- "-t"
- "4"
- "-l"
- "0.0.0.0"
ports:
- containerPort: 11211
readinessProbe:
tcpSocket:
port: 11211
initialDelaySeconds: 3
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 11211
initialDelaySeconds: 5
periodSeconds: 15
resources:
requests:
cpu: "2m"
memory: "30Mi"
limits:
cpu: "250m"
memory: "256Mi"
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 5
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 5
With this, the recovery takes less than a minute, which is reasonable for our use case. To get no downtime a bit more work is needed. This would involve using highly available versions of Zulip dependencies and/or figuring out a way to set up hot-spare services in Kubernetes, which doesn’t seem to be supported out of the box.
Conclusion
Many Helm charts are extremely mature and well maintained. This is of course great but can be a bit misleading. Just because there is a Helm chart for an application, it doens’t necessarily mean that it fits the requirements.
The standard approach, especially if you want to really learn Kubernetes, is to try to deploy the app with handmade Kubernetes objects and only try the Helm charts if necessary, with the exception, of course, of very mature Helm charts.
This blog post is licensed under
CC BY-SA 4.0
