Resize disk

🌇 Sunset Kubernetes deployments

This page covers our PostHog Kubernetes deployment, which we sunset and no longer support. We will continue to provide security updates for Kubernetes deployments until at least May 31, 2024.

For existing customers

We highly recommend migrating to PostHog Cloud (US or EU). Take a look at this guide for more information on the migration process.

Looking to continue self-hosting?

We still maintain our Open-source Docker Compose deployment. Instructions for deploying can be found here.

Requirements

You need to run a Kubernetes cluster with the Volume Expansion feature enabled. This feature is supported on the majority of volume types since Kubernetes version >= 1.11 (see docs).

kubectl get storageclass -o json | jq '.items[].allowVolumeExpansion'
true

DEFAULT_STORAGE_CLASS=$(kubectl get storageclass -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}')
kubectl patch storageclass "$DEFAULT_STORAGE_CLASS" -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched

How-to

1. Connect to the Postgresql container to verify the data directory filesystem size (in this example 10GB)

   Terminal

   kubectl -n posthog exec -it posthog-posthog-postgresql-0 -- /bin/bash
   I have no name!@posthog-posthog-postgresql-0:/$ df -h /bitnami/postgresql
   Filesystem                                                                Size  Used Avail Use% Mounted on
   /dev/disk/by-id/scsi-0DO_Volume_pvc-966716a8-cac6-407a-afb4-8cab52b0ad9b  9.8G  145M  9.2G   2% /bitnami/postgresql

2. Resize the underlying PVC (in this example we are resizing it to 20G)

   Terminal

   kubectl -n posthog patch pvc data-posthog-posthog-postgresql-0 -p '{ "spec": { "resources": { "requests": { "storage": "20Gi" }}}}'
   persistentvolumeclaim/data-posthog-posthog-postgresql-0 patched

   Note: while resizing the PVC you might get an error disk resize is only supported on Unattached disk, current disk state: Attached (see below for more details).

   In this specific case you need to temporary scale down the StatefulSet replica value to zero. This will briefly disrupt the Postgresql service availability and make the PostHog UI inaccessible. On newer versions of PostHog events will be queued and ingestion won't be impacted

   You can do that by running: kubectl -n posthog patch statefulset posthog-posthog-postgresql -p '{ "spec": { "replicas": 0 }}'

   After you successfully resized the PVC, you can restore the initial replica definition with: kubectl -n posthog patch statefulset posthog-posthog-postgresql -p '{ "spec": { "replicas": 1 }}'

3. Delete the StatefulSet definition but leave its pods online (this is to avoid an impact to using PostHog): kubectl -n posthog delete sts --cascade=orphan posthog-posthog-postgresql

4. In your Helm chart configuration, update the postgresql.persistence value in value.yaml to the target size (20G in this example)

5. Run a helm upgrade to recycle all the pods and re-deploy the StatefulSet definition

6. Connect to the Postgresql container to verify the new filesystem size

   Terminal

   kubectl -n posthog exec -it posthog-posthog-postgresql-0 -- /bin/bash
   I have no name!@posthog-posthog-postgresql-0:/$ df -h /bitnami/postgresql
   Filesystem                                                                Size  Used Avail Use% Mounted on
   /dev/disk/by-id/scsi-0DO_Volume_pvc-966716a8-cac6-407a-afb4-8cab52b0ad9b  20G   153M   19G   1% /bitnami/postgresql