A note on Kubernetes' disaster recovery, StatefulSets and Helm charts

Tuesday 27 October 2026 José Carrasquel Vera 63 views

Kubernetes Sysadmin Docker Deployment DevOps

https://cylab.be/blog/532/a-note-on-kubernetes-disaster-recovery-statefulsets-and-helm-charts

A couple weeks ago I gave a demo in Cylab about Kubernetes. One of the main goals was to show that it is not so complex and that, perhaps, it can be a good way to put spare hardware to work. I was proud to show how easily Zulip can be deployed with Helm charts.

But then I got a bit cocky and decided to show what happens to Zulip if a Kubernetes node died ungracefully. The quick auto recovery I was hinting never really happened. Instead, the application was down and never recovered.

Below is a small writeup about this and how I improved disaster recovery for Zulip.

Zulip

Zulip is a chat app relying on the following elements:

Zulip server
PostgreSQL database
Redis
RabbitMQ
memcached

The problem

The developers maintain a Helm chart which deploys:

A replica 1 StatefulSet for Zulip
A replica 1 StatefulSet for PostgreSQL
A replica 1 StatefulSet for Redis
A replica 1 StatefulSet for RabbitMQ
A replica 1 Deployment for Memcached

NAME                                   READY   STATUS    RESTARTS   AGE
pod/zulip-0                            1/1     Running   0          3m32s
pod/zulip-memcached-5475b64f65-lzhg2   1/1     Running   0          17h
pod/zulip-postgresql-0                 1/1     Running   0          4m33s
pod/zulip-rabbitmq-0                   1/1     Running   0          17h
pod/zulip-redis-master-0               1/1     Running   0          17h


NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/zulip-memcached   1/1     1            1           17h

NAME                                  READY   AGE
statefulset.apps/zulip                1/1     3m33s
statefulset.apps/zulip-postgresql     1/1     4m34s
statefulset.apps/zulip-rabbitmq       1/1     17h
statefulset.apps/zulip-redis-master   1/1     17h

NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       VOLUMEATTRIBUTESCLASS   AGE
data-zulip-postgresql-0   Bound    pvc-ae02f1d2-c9a5-404e-beab-85e0958d7ac2   8Gi        RWO            ceph-block         <unset>                 8m53s
zulip-data                Bound    pvc-0a71b7fb-33c5-429e-b26f-c7f775b477f0   3Gi        RWO            ceph-block         <unset>                 17h

This chart is just fine for simple (even production) Deployments. The main problem arises when you need high availability when a node goes down. In this case, Kubernetes will make sure that the pods bound to the persistentVolumes have really terminated before it fires up new ones. This can cause very long down times and may even require human intervention for recovery. Read Non graceful node shutdown or experiment yourself.

The Solution: Only Standard deployments and cloud native tools

The Zulip scalability documentation recommends:

Disaster recovery: One can easily run a warm spare application server and a warm spare database (using PostgreSQL warm standby replicas). Make sure the warm spare application server has copies of /etc/zulip and you’re either syncing LOCAL_UPLOADS_DIR or using the S3 file uploads backend.

Care must be taken when dividing traffic for a single Zulip realm between multiple Zulip application servers, which is why we recommend a hot spare over load-balancing for most installations desiring extra redundancy.

So the first step would be to replace the PostgreSQL StatefulSet for something more robust. Enter CloudNativePG:

CloudNativePG is the Kubernetes operator that covers the full lifecycle of a highly available PostgreSQL database cluster with a primary/standby architecture, using native streaming replication.

It lets us deploy PostgreSQL clusters with a simple declaration:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgresql-cluster
  namespace: zulip
spec:
  instances: 3
  storage:
    size: 1Gi
    storageClass: openebs-hostpath
  # No two pods in the same node.
  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: required
  postgresql:
    parameters:
      synchronous_commit: "on"

Which deploys:

jose@debian-k8s0:~$ kubectl get all
NAME                                   READY   STATUS    RESTARTS   AGE
pod/postgresql-cluster-1               1/1     Running   0          17h
pod/postgresql-cluster-2               1/1     Running   0          17h
pod/postgresql-cluster-3               1/1     Running   0          17h
pod/zulip-0                            1/1     Running   0          61m
pod/zulip-memcached-5475b64f65-lzhg2   1/1     Running   0          17h
pod/zulip-postgresql-0                 1/1     Running   0          62m
pod/zulip-rabbitmq-0                   1/1     Running   0          17h
pod/zulip-redis-master-0               1/1     Running   0          17h

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
service/postgresql-cluster-r      ClusterIP   10.97.165.48     <none>        5432/TCP                                17h
service/postgresql-cluster-ro     ClusterIP   10.97.230.9      <none>        5432/TCP                                17h
service/postgresql-cluster-rw     ClusterIP   10.111.100.72    <none>        5432/TCP                                17h
service/zulip                     ClusterIP   10.98.97.34      <none>        80/TCP                                  17h
service/zulip-memcached           ClusterIP   10.98.213.8      <none>        11211/TCP                               17h
service/zulip-postgresql          ClusterIP   10.109.204.154   <none>        5432/TCP                                62m
service/zulip-postgresql-hl       ClusterIP   None             <none>        5432/TCP                                62m
service/zulip-rabbitmq            ClusterIP   10.103.58.19     <none>        5672/TCP,4369/TCP,25672/TCP,15672/TCP   17h
service/zulip-rabbitmq-headless   ClusterIP   None             <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   17h
service/zulip-redis-headless      ClusterIP   None             <none>        6379/TCP                                17h
service/zulip-redis-master        ClusterIP   10.108.28.24     <none>        6379/TCP                                17h

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/zulip-memcached   1/1     1            1           17h

NAME                                  READY   AGE
statefulset.apps/zulip                1/1     61m
statefulset.apps/zulip-postgresql     1/1     62m
statefulset.apps/zulip-rabbitmq       1/1     17h
statefulset.apps/zulip-redis-master   1/1     17h

jose@debian-k8s0:~$ kubectl get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       VOLUMEATTRIBUTESCLASS   AGE
data-zulip-postgresql-0   Bound    pvc-ae02f1d2-c9a5-404e-beab-85e0958d7ac2   8Gi        RWO            ceph-block         <unset>                 62m
postgresql-cluster-1      Bound    pvc-c087548f-d6f7-434f-9fe9-d5e5e9dbcc4c   1Gi        RWO            openebs-hostpath   <unset>                 17h
postgresql-cluster-2      Bound    pvc-173384d1-eb90-409e-87ce-0029cc3cbdbd   1Gi        RWO            openebs-hostpath   <unset>                 17h
postgresql-cluster-3      Bound    pvc-08f23c31-6a05-402b-a8b3-28fabf53c91a   1Gi        RWO            openebs-hostpath   <unset>                 17h
zulip-data                Bound    pvc-0a71b7fb-33c5-429e-b26f-c7f775b477f0   3Gi        RWX            ceph-filesystem    <unset>                 17h

Here, we have changed the storageClass to use openEBS’ hostPath to avoid nested redundancy for storage: ceph-block is configured as replica 3, resulting in 9 replicas of the database’s storage. Observe that the postgresql-cluster pods are not governed by Deployments or StatefulSets. In fact they are controlled by the controller itself, see this.

Observe also that 3 services have been created for the PostgreSQL cluster.

jose@debian-k8s0:~$ kubectl get endpointslices.discovery.k8s.io
NAME                            ADDRESSTYPE   PORTS                         ENDPOINTS                                AGE
postgresql-cluster-r-f2vvs      IPv4          5432                          10.244.3.241,10.244.5.165,10.244.0.101   18h
postgresql-cluster-ro-k47ws     IPv4          5432                          10.244.5.165,10.244.0.101                18h
postgresql-cluster-rw-fkx5c     IPv4          5432                          10.244.3.241                             18h

The controller takes care of putting the leader pod in the rw service and replacing it it case it becomes unavailable.

Now that we have an external PostgreSQL databse, we can adapt Zulip’s Helm chart values to according to this:

postgresql:
  enabled: false
externalPostgresql:
  host: "postgresql-cluster-rw"
  port: 5432
  database: "app"
  user: "app"
  ## password supports both scalar values and valueFrom maps.
  password:
    valueFrom:
      secretKeyRef:
          name: postgresql-cluster-app
          key: password
  sslmode: ""
...

Observe that the postgresql-cluster-app contains the necessary information to connect to the PostgreSQL cluster.

$ kubectl describe secrets postgresql-cluster-app
Name:         postgresql-cluster-app
Namespace:    zulip
Labels:       app.kubernetes.io/managed-by=cloudnative-pg
              cnpg.io/cluster=postgresql-cluster
              cnpg.io/reload=true
              cnpg.io/userType=app
Annotations:  cnpg.io/operatorVersion: 1.30.1

Type:  kubernetes.io/basic-auth

Data
====
dbname:         3 bytes
fqdn-jdbc-uri:  155 bytes
fqdn-uri:       136 bytes
host:           21 bytes
jdbc-uri:       137 bytes
password:       64 bytes
pgpass:         100 bytes
port:           4 bytes
uri:            118 bytes
user:           3 bytes
username:       3 bytes

After deployment, we get the following error in the zulip pod.

Waiting for PostgreSQL database to accept connections...
zulip-postgresql:5432 - no response

Many hours of searching gave me no solution as to why the externalPostgresql configuration is not being followed. So instead I tweaked the CloudNativePG cluster definition above to add a second service for rw named zulip-postgresql

  managed:
    services:
      additional:
        - selectorType: rw
          serviceTemplate:
            metadata:
              name: "zulip-postgresql"
              labels:
                test-label: "true"
              annotations:
                test-annotation: "true"
            spec:
              type: ClusterIP

Which solved the connection issues

$ kubectl get service postgresql-cluster-rw zulip-postgresql
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
postgresql-cluster-rw   ClusterIP   10.111.100.72   <none>        5432/TCP   45h
zulip-postgresql        ClusterIP   10.110.221.31   <none>        5432/TCP   11m
$ kubectl get pods -o wide
NAME                               READY   STATUS    RESTARTS      AGE   IP             NODE          NOMINATED NODE   READINESS GATES
postgresql-cluster-1               1/1     Running   0             45h   10.244.3.241   debian-k8s5   <none>           <none>
postgresql-cluster-2               1/1     Running   0             45h   10.244.5.165   debian-k8s1   <none>           <none>
postgresql-cluster-3               1/1     Running   0             45h   10.244.0.101   debian-k8s0   <none>           <none>
zulip-0                            1/1     Running   6 (22m ago)   55m   10.244.4.206   debian-k8s3   <none>           <none>
zulip-memcached-5475b64f65-5f9g2   1/1     Running   0             74m   10.244.3.32    debian-k8s5   <none>           <none>
zulip-rabbitmq-0                   1/1     Running   0             74m   10.244.4.251   debian-k8s3   <none>           <none>
zulip-redis-master-0               1/1     Running   0             74m   10.244.4.241   debian-k8s3   <none>           <none>

This tweak shows that perhaps the Zulip Helm chart is not mature enough and should be handled with care.

Disaster recovery test

Let’s see what happens with zulip when the node debian-k8s3, holding the zulip and redis and RabbitMQ pods, goes down.

$ kubectl get pods -o wide -w
NAME                               READY   STATUS    RESTARTS      AGE   IP             NODE          NOMINATED NODE   READINESS GATES
postgresql-cluster-1               1/1     Running   0             45h   10.244.3.241   debian-k8s5   <none>           <none>
postgresql-cluster-2               1/1     Running   0             45h   10.244.5.165   debian-k8s1   <none>           <none>
postgresql-cluster-3               1/1     Running   0             45h   10.244.0.101   debian-k8s0   <none>           <none>
zulip-0                            1/1     Running   6 (27m ago)   60m   10.244.4.206   debian-k8s3   <none>           <none>
zulip-memcached-5475b64f65-5f9g2   1/1     Running   0             78m   10.244.3.32    debian-k8s5   <none>           <none>
zulip-rabbitmq-0                   1/1     Running   0             78m   10.244.4.251   debian-k8s3   <none>           <none>
zulip-redis-master-0               1/1     Running   0             78m   10.244.4.241   debian-k8s3   <none>           <none>
zulip-0                            1/1     Running   6 (32m ago)   65m   10.244.4.206   debian-k8s3   <none>           <none>
zulip-redis-master-0               1/1     Running   0             83m   10.244.4.241   debian-k8s3   <none>           <none>
zulip-rabbitmq-0                   1/1     Running   0             83m   10.244.4.251   debian-k8s3   <none>           <none>
zulip-0                            1/1     Running            6 (37m ago)    70m   10.244.4.206   debian-k8s3   <none>           <none>
zulip-redis-master-0               1/1     Running            0              88m   10.244.4.241   debian-k8s3   <none>           <none>
zulip-rabbitmq-0                   1/1     Running            0              88m   10.244.4.251   debian-k8s3   <none>           <none>
zulip-redis-master-0               1/1     Terminating        0              88m   10.244.4.241   debian-k8s3   <none>           <none>
zulip-0                            1/1     Terminating        6 (37m ago)    70m   10.244.4.206   debian-k8s3   <none>           <none>
zulip-rabbitmq-0                   1/1     Terminating        0              88m   10.244.4.251   debian-k8s3   <none>           <none>

As described in the Kubernets Non graceful node shutdown, the zulip-0 pods stays in Terminating state and doesn’t come back. This effectively leads to Zulip being down. The fact that we are using RWX PVC is not enough. Zulip’s pod should be backed by a Deployment, which is not supported in the Zulip’s Helm chart.

Because of this, we’ll ditch the Helm chart alltogether and build the whole stack with native deployments without relying on the Helm Chart. The main idea is to take the definitions of the Helm chart and transform them into deployments. Moreover we replace Redis by Valkey.

Second disaster ricovery test

We see that most of the pods are scheduled in the debian-k8s1 node. Let’s turn it off:

$ kubectl get pods -w -o wide
NAME                         READY   STATUS    RESTARTS      AGE   IP             NODE          NOMINATED NODE   READINESS GATES
memcached-54b8b9fcfd-rtls8   1/1     Running   0             14h   10.244.5.84    debian-k8s1   <none>           <none>
postgresql-cluster-1         1/1     Running   0             14h   10.244.5.161   debian-k8s1   <none>           <none>
postgresql-cluster-2         1/1     Running   0             14h   10.244.3.172   debian-k8s5   <none>           <none>
postgresql-cluster-3         1/1     Running   0             14h   10.244.0.131   debian-k8s0   <none>           <none>
rabbitmq-845b89f7dc-vw9s4    1/1     Running   1 (14h ago)   14h   10.244.5.81    debian-k8s1   <none>           <none>
valkey-664b7dc89f-c5jps      1/1     Running   0             14h   10.244.5.146   debian-k8s1   <none>           <none>
zulip-9dd9f84fd-lfpt2        1/1     Running   0             13h   10.244.5.53    debian-k8s1   <none>           <none>
rabbitmq-845b89f7dc-vw9s4    1/1     Terminating   1 (14h ago)   14h   10.244.5.81    debian-k8s1   <none>           <none>
valkey-664b7dc89f-c5jps      1/1     Terminating   0             14h   10.244.5.146   debian-k8s1   <none>           <none>
postgresql-cluster-1         1/1     Terminating   0             14h   10.244.5.161   debian-k8s1   <none>           <none>
valkey-664b7dc89f-xhd2x      0/1     Pending       0             0s    <none>         <none>        <none>           <none>
rabbitmq-845b89f7dc-mnlps    0/1     Pending       0             0s    <none>         <none>        <none>           <none>
rabbitmq-845b89f7dc-mnlps    0/1     Pending       0             0s    <none>         debian-k8s3   <none>           <none>
valkey-664b7dc89f-xhd2x      0/1     Pending       0             0s    <none>         debian-k8s3   <none>           <none>
rabbitmq-845b89f7dc-mnlps    0/1     ContainerCreating   0             0s    <none>         debian-k8s3   <none>           <none>
zulip-9dd9f84fd-lfpt2        1/1     Terminating         0             13h   10.244.5.53    debian-k8s1   <none>           <none>
valkey-664b7dc89f-xhd2x      0/1     Init:0/1            0             0s    <none>         debian-k8s3   <none>           <none>
zulip-9dd9f84fd-psggt        0/1     Pending             0             0s    <none>         <none>        <none>           <none>
zulip-9dd9f84fd-psggt        0/1     Pending             0             0s    <none>         debian-k8s3   <none>           <none>
zulip-9dd9f84fd-psggt        0/1     ContainerCreating   0             0s    <none>         debian-k8s3   <none>           <none>
memcached-54b8b9fcfd-rtls8   1/1     Terminating         0             14h   10.244.5.84    debian-k8s1   <none>           <none>
memcached-54b8b9fcfd-cqxgm   0/1     Pending             0             0s    <none>         <none>        <none>           <none>
valkey-664b7dc89f-xhd2x      0/1     Init:0/1            0             1s    <none>         debian-k8s3   <none>           <none>
rabbitmq-845b89f7dc-mnlps    0/1     ContainerCreating   0             1s    <none>         debian-k8s3   <none>           <none>
memcached-54b8b9fcfd-cqxgm   0/1     ContainerCreating   0             1s    <none>         debian-k8s5   <none>           <none>
zulip-9dd9f84fd-psggt        0/1     ContainerCreating   0             10s   <none>         debian-k8s3   <none>           <none>
zulip-9dd9f84fd-psggt        0/1     Running             0             10s   10.244.4.186   debian-k8s3   <none>           <none>
memcached-54b8b9fcfd-cqxgm   0/1     Running             0             19s   10.244.3.137   debian-k8s5   <none>           <none>
memcached-54b8b9fcfd-cqxgm   1/1     Running             0             24s   10.244.3.137   debian-k8s5   <none>           <none>
valkey-664b7dc89f-xhd2x      0/1     Init:0/1            0             27s   10.244.4.12    debian-k8s3   <none>           <none>
valkey-664b7dc89f-xhd2x      0/1     PodInitializing     0             28s   10.244.4.12    debian-k8s3   <none>           <none>
valkey-664b7dc89f-xhd2x      0/1     Running             0             30s   10.244.4.12    debian-k8s3   <none>           <none>
valkey-664b7dc89f-xhd2x      1/1     Running             0             31s   10.244.4.12    debian-k8s3   <none>           <none>
rabbitmq-845b89f7dc-mnlps    0/1     Running             0             55s   10.244.4.18    debian-k8s3   <none>           <none>
rabbitmq-845b89f7dc-mnlps    1/1     Running             0             88s   10.244.4.18    debian-k8s3   <none>           <none>
zulip-9dd9f84fd-psggt        0/1     Running             0             90s   10.244.4.186   debian-k8s3   <none>           <none>
zulip-9dd9f84fd-psggt        1/1     Running             0             90s   10.244.4.186   debian-k8s3   <none>           <none>

We see that the whole application eventually recovers and we see no real issue in the frontend (Zulip browser). However, the recreation of the new pods actually took too much time. The default for Kubernetes is 5 minutes before pod rescheduling. This is probably too much for us but we can fix it on a per-pod basis using taint based eviction.

Kubernetes automatically adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300, unless you, or a controller, set those tolerations explicitly. These automatically-added tolerations mean that Pods remain bound to Nodes for 5 minutes after one of these problems is detected.

In our case, we will add it to the template of all our Deployments, allowing for a downtime of only 5 seconds. We don’t want to modify the default behaviour since this wouldn’t be necessary for replicated Deployments with pod PodAntiAffinity.

tolerations:
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 5
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 5

As an example here is our memcached Deployment Jinja2 template:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memcached
  namespace: zulip
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: memcached
  template:
    metadata:
      labels:
        app: memcached
    spec:
      containers:
        - name: memcached
          image: "{{ zulip_memcached_image }}"
          args:
            - "-m"
            - "256"
            - "-I"
            - "1m"
            - "-t"
            - "4"
            - "-l"
            - "0.0.0.0"
          ports:
            - containerPort: 11211
          readinessProbe:
            tcpSocket:
              port: 11211
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: 11211
            initialDelaySeconds: 5
            periodSeconds: 15
          resources:
            requests:
              cpu: "2m"
              memory: "30Mi"
            limits:
              cpu: "250m"
              memory: "256Mi"
      tolerations:
        - key: "node.kubernetes.io/unreachable"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 5
        - key: "node.kubernetes.io/not-ready"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 5

With this, the recovery takes less than a minute, which is reasonable for our use case. To get no downtime a bit more work is needed. This would involve using highly available versions of Zulip dependencies and/or figuring out a way to set up hot-spare services in Kubernetes, which doesn’t seem to be supported out of the box.

Conclusion

Many Helm charts are extremely mature and well maintained. This is of course great but can be a bit misleading. Just because there is a Helm chart for an application, it doens’t necessarily mean that it fits the requirements.

The standard approach, especially if you want to really learn Kubernetes, is to try to deploy the app with handmade Kubernetes objects and only try the Helm charts if necessary, with the exception, of course, of very mature Helm charts.

This blog post is licensed under CC BY-SA 4.0 creative commons attribution share-alike