Running & Scaling Databases on Kubernetes: Patterns and Considerations

Running stateful workloads like databases on Kubernetes was once considered challenging, even controversial. Kubernetes excels at managing stateless applications, but databases require stable network identities, persistent storage, and careful handling during updates and scaling. However, with the evolution of Kubernetes features like StatefulSets, the Container Storage Interface (CSI), and the rise of Kubernetes Operators, running and scaling databases effectively within Kubernetes is now a viable and increasingly common pattern.

This guide explores key considerations, strategies, and best practices for deploying, managing, and scaling databases on Kubernetes.

1. The Foundation: StatefulSets for Stable Identity & Storage

Unlike Deployments (designed for stateless apps), StatefulSets are the Kubernetes workload API object specifically designed for stateful applications like databases. They provide crucial guarantees:

• Stable, Unique Network Identifiers: Each Pod in a StatefulSet gets a persistent, ordinal hostname (e.g., postgres-0, postgres-1). This allows peers and clients to reliably address specific instances.
• Stable, Persistent Storage: Each Pod gets its own unique PersistentVolumeClaim (PVC) based on a volumeClaimTemplates definition. When a Pod is rescheduled, it reattaches to the same PersistentVolume (PV), ensuring data persistence.
• Ordered, Graceful Deployment and Scaling: Pods are created, updated, and deleted in a strict, predictable order (e.g., 0, 1, 2…). This is vital for database cluster bootstrapping, upgrades, and scaling operations where order matters (like configuring replication).
• Ordered, Graceful Deletion and Termination: Pods are terminated in reverse ordinal order, allowing for graceful shutdown procedures.

# Example StatefulSet for a simple PostgreSQL primary (single replica)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-db # Name of the StatefulSet
spec:
  # ServiceName links the StatefulSet to a Headless Service for stable network discovery
  serviceName: "postgres-headless"
  replicas: 1 # Start with one replica (primary)
  selector:
    matchLabels:
      app: postgres # Label selector to find Pods managed by this StatefulSet
  template: # Pod template
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 10 # Allow time for graceful shutdown
      containers:
      - name: postgres
        image: postgres:14 # Use a specific version
        ports:
        - containerPort: 5432
          name: postgresdb
        env:
          # IMPORTANT: Use Secrets for passwords in production!
          - name: POSTGRES_PASSWORD
            value: "mysecretpassword"
          - name: PGDATA # Define data directory within the volume
            value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: postgres-data # Mount the persistent volume
          mountPath: /var/lib/postgresql/data
        # Add readiness/liveness probes here
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 10
          periodSeconds: 5
  # Define the template for PersistentVolumeClaims
  volumeClaimTemplates:
  - metadata:
      name: postgres-data # Name of the PVC template
    spec:
      accessModes: [ "ReadWriteOnce" ] # Typical access mode for database volumes
      storageClassName: "standard-ssd" # Request a specific StorageClass (e.g., SSD)
      resources:
        requests:
          storage: 10Gi # Request 10 GiB of storage
  # Define update strategy (RollingUpdate is common for controlled updates)
  updateStrategy:
    type: RollingUpdate
    # rollingUpdate:
      # partition: 0 # Controls staged rollouts (update pods >= partition index)

Note: This is a basic example. Production deployments often involve Operators for managing replication, backups, etc.

2. Scaling Strategies for Databases on Kubernetes

Scaling databases involves handling increased load (reads/writes) and data volume. Common strategies include:

• Vertical Scaling (Scale-Up): Increase the resources (CPU, memory) allocated to the database Pod(s).

  • How: Modify the resources.requests and resources.limits in the StatefulSet template. Kubernetes will reschedule the Pod onto a node with sufficient resources (if available).
  • Pros: Simple to implement initially.
  • Cons: Limited by node size, requires Pod restart (downtime), doesn’t improve write contention on a single primary.

• Horizontal Scaling (Scale-Out) - Read Replicas: Offload read queries to one or more read-only replicas of the primary database.

  • How: Increase the replicas count in the StatefulSet (if using an Operator that manages replication) or deploy separate read-replica StatefulSets configured to replicate from the primary. A Kubernetes Service can load balance read traffic across replicas.
  • Pros: Improves read performance significantly, relatively common database feature.
  • Cons: Doesn’t scale write performance, introduces replication lag (RPO > 0), requires application logic to direct reads/writes appropriately (or use a proxy).

• Horizontal Scaling (Scale-Out) - Sharding: Partition the database horizontally, distributing subsets of data (shards) across multiple primary nodes. Each shard handles reads/writes for its data subset.

  • How: Requires database systems designed for sharding (e.g., MongoDB, Vitess, CockroachDB, Citus). Often managed via Operators or specific configurations. Applications need to be aware of the sharding key or use a proxy that handles routing.
  • Pros: Scales both read and write performance, distributes data volume.
  • Cons: Significantly increases operational complexity, application changes often required, potential for “hot shards,” complex cross-shard transactions.

• Connection Pooling: While not direct scaling, connection poolers (like PgBouncer for PostgreSQL, ProxySQL for MySQL, or built-in poolers) deployed as sidecars or separate services reduce the overhead of establishing database connections, improving performance under high connection load. They manage a pool of persistent connections to the database.

3. The Role of Kubernetes Operators

Managing stateful applications like databases (handling backups, failover, upgrades, replication setup, scaling) involves complex operational logic. Kubernetes Operators encode this operational knowledge into software running within the cluster.

• What they do: Operators use Custom Resource Definitions (CRDs) to represent the database cluster (e.g., kind: PostgresqlCluster). They watch these resources and automate tasks like:
  • Provisioning primary and replica StatefulSets.
  • Configuring replication.
  • Handling automated failover if a primary fails.
  • Managing backups and restores.
  • Orchestrating version upgrades.
  • Setting up monitoring endpoints.
• Benefits: Drastically simplifies deploying and managing complex database setups on Kubernetes, making them behave more like cloud-managed database services.
• Examples: Crunchy Data PostgreSQL Operator, Percona Operators (MySQL, MongoDB, PostgreSQL), Zalando Postgres Operator, KubeDB, CloudNativePG.

Using a mature Operator is often the recommended approach for running production databases on Kubernetes.

4. Persistent Storage: The Database’s Foundation

Databases require persistent storage that survives Pod restarts and rescheduling.

• PersistentVolumes (PVs) & PersistentVolumeClaims (PVCs): Kubernetes uses PVs (cluster resources representing physical storage) and PVCs (namespaced requests for storage) to decouple storage consumption from provisioning. StatefulSets automatically create PVCs based on volumeClaimTemplates.
• StorageClasses: Define different types of storage (e.g., standard-ssd, premium-io1, nfs) with varying performance characteristics and provisioning behavior. Specify a storageClassName in your PVC template to request the appropriate type.
• Container Storage Interface (CSI): The standard way for storage vendors (cloud providers, on-premises solutions) to provide storage drivers for Kubernetes. Ensure your cluster has the appropriate CSI driver installed for your desired storage type (e.g., AWS EBS CSI Driver, Azure Disk CSI Driver, GCP Persistent Disk CSI Driver).
• Performance Matters: Database performance is heavily dependent on storage I/O performance (IOPS, throughput, latency). Choose StorageClasses backed by high-performance storage (typically SSDs) for production databases. Monitor PV metrics (latency, IOPS) closely.
• Access Modes: Most databases require ReadWriteOnce (RWO) access mode, meaning the volume can only be mounted by a single node at a time. This aligns well with StatefulSet Pods typically running on one node. ReadWriteMany (RWX) might be needed for specific shared-storage scenarios but is less common for primary database volumes.
• Local Persistent Volumes: Offer the highest performance (using local NVMe SSDs on nodes) but tie a Pod to a specific node and data is lost if the node fails. Suitable only for specific use cases where data loss is acceptable or replication handles durability (e.g., some distributed databases).

5. Backup, Recovery, and High Availability

Protecting your data and ensuring availability is critical.

• Backup Strategy:
  • Database-Native Tools: Leverage built-in tools (e.g., pg_dump, mysqldump, Percona XtraBackup) often orchestrated by Operators or custom CronJobs within Kubernetes. Store backups externally (e.g., S3, Azure Blob Storage).
  • Volume Snapshots: Use CSI volume snapshot capabilities (if supported by your driver/StorageClass) for point-in-time backups of the underlying storage volume. Often faster than logical dumps but may require database quiescing for consistency.
  • Velero: A Kubernetes-native tool for backing up cluster resources (including PVCs using volume snapshots or filesystem backups via Restic integration) and application state. Excellent for cluster-level DR and application mobility.
  • Regular Testing: Crucially, regularly test your restore procedures to ensure backups are valid and meet your RTO.
• Disaster Recovery (DR):
  • Define RTO and RPO (see previous post on Cloud-Native DR).
  • Implement cross-region/cross-cluster replication using database features (streaming replication), Operator capabilities, or tools like Velero for cluster state.
  • Use IaC to provision the DR infrastructure.
• High Availability (HA):
  • Within a cluster, rely on Kubernetes rescheduling StatefulSet Pods on node failure (data persists via PV reattachment).
  • Use database replication (primary/replica setup) managed by an Operator for automated failover within the cluster if the primary Pod/node fails. This minimizes downtime compared to just relying on rescheduling.

6. Monitoring & Observability

Understand database performance and health within the Kubernetes context.

• Database Metrics: Expose database-specific metrics (connections, query latency, cache hit rate, replication lag, transaction rates) using dedicated Prometheus exporters (e.g., postgres_exporter, mysqld_exporter). Deploy these as sidecars or separate Deployments and configure Prometheus ServiceMonitors to scrape them.
• Resource Metrics: Monitor Pod resource usage (CPU, memory from cAdvisor) and PV usage/performance (disk I/O, capacity from kubelet and potentially CSI driver metrics).
• Logging: Aggregate database logs alongside application logs using a cluster-wide logging agent (Fluent Bit, etc.). Ensure logs capture slow queries, errors, and connection information.
• Alerting: Set up alerts in Prometheus/Alertmanager based on critical database metrics (high latency, low cache hit rate, high connection count, replication lag exceeding thresholds, low disk space on PVs).

7. Security Considerations

Protecting sensitive data is paramount.

• Network Policies: Use Kubernetes NetworkPolicy to strictly limit network access to the database Pods, allowing connections only from specific application Pods or namespaces. Deny all other ingress traffic.
• Secrets Management: Store database credentials (usernames, passwords) securely in Kubernetes Secrets. Use tools like External Secrets Operator or Vault integration to manage these secrets externally and sync them into the cluster, rather than committing plain secrets to Git. Operators often integrate with secret management.
• Authentication & Authorization: Configure strong database user authentication. Use TLS for client connections if possible (often handled by Operators or sidecars like pgbouncer with TLS). Limit database user permissions based on the principle of least privilege.
• Encryption: Use encrypted Persistent Volumes (via StorageClass configuration) for data-at-rest encryption. Ensure transport encryption (TLS/mTLS) is used for replication and client connections where appropriate.

Conclusion: Is Running Databases on K8s Right for You?

Running databases on Kubernetes offers benefits like unified orchestration, standardized deployment patterns, and potential cost savings compared to dedicated VMs. However, it introduces complexity around storage management, networking, stateful upgrades, and requires careful operational practices.

Considerations:

• Managed Services: Cloud provider managed database services (RDS, Azure SQL, Cloud SQL) often abstract away much of this operational burden and might be a better choice, especially for smaller teams or less complex needs.
• Operators: If running databases in Kubernetes, leveraging a mature Kubernetes Operator significantly simplifies deployment, scaling, HA, and lifecycle management.
• Expertise: Ensure your team has the necessary Kubernetes and database administration skills.

By carefully considering storage performance, implementing robust backup and HA strategies using StatefulSets and potentially Operators, securing access, and establishing thorough monitoring, you can successfully run and scale databases within your Kubernetes clusters.

References

1. Kubernetes Documentation - StatefulSets: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
2. Kubernetes Documentation - Persistent Volumes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
3. Kubernetes Operators: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
4. Velero (Kubernetes Backup/Restore): https://velero.io/
5. Awesome Kubernetes Operators List: https://github.com/operator-framework/awesome-operators