Skip to content

Tutorials

Kubernetes on BKS

Run the shop app from A three-tier web app on a BKS cluster: the cluster from OpenTofu, and the app and its database from Kubernetes manifests, behind a Gateway. Then break it, scale it, and rebuild the whole cluster from code.

Well-Architected

Pillar: Reliability. Kubernetes replaces a failed pod and moves work off a node you take away, and the database's data lives on a volume that outlasts its pod, its node and the cluster.

Also touches: Operational excellence (the cluster and the app both in files), Performance efficiency (more pods under load, fewer when it's quiet), and Security and sovereignty (an unrestricted credential used for one job, and a cluster you own).

What you'll build

internet
   |
Gateway (NGINX Gateway Fabric) + a load balancer with a public address
   |
shop-app Service
   |
shop-app pods, two or more, spread across the workers
   |
shop-db Service
   |
shop-db-0   PostgreSQL, data on a persistent volume

It's the same application as 4.4, built the Kubernetes way:

In 4.4 In this tutorial
Instances built from user data Pods, from container images and manifests
An internal load balancer with a health monitor A Service, and a readiness probe on /health
The public load balancer and its floating IP A Gateway, which gets a load balancer with a public address
The database's data volume A persistent volume claim, which BKS makes into a volume in your project
count = 2 replicas: 2, and an autoscaler that changes it with load

BKS creates the cluster's control plane and worker nodes as instances in your project. From then on the cluster is yours, control plane included, along with what you install on it: in this tutorial, NGINX Gateway Fabric and metrics-server. See Shared responsibility.

Before you start

  • Completed: A three-tier web app and Rebuild, don't repair.
  • OpenTofu 1.13.1, as in 4.1; kubectl; and Helm 3.8 or later. The commands run the same way from Windows, macOS and Linux unless a tab shows otherwise.
  • Quota for 3 instances, their boot volumes, a 10 GB volume and a load balancer. Each node is an instance with a 55 GB boot volume, sized by the cluster template's flavours.
  • Time: about an hour, much of it waiting for the cluster.
  • The code: every file in this tutorial is also in kubernetes-on-bks/ in learn-projects, the course's code repository, at release v1.2.0.

Step 1: An unrestricted application credential

Creating a cluster from OpenTofu needs an unrestricted application credential. BKS sets up credentials of its own for the cluster, so that its nodes can create volumes and load balancers in your project, and a restricted credential isn't allowed to do that.

What an unrestricted credential allows

An unrestricted credential can create and delete other application credentials in your name. Anyone who holds it can make themselves a new credential and keep access after you revoke this one. Use it only to create and delete the cluster, give it a short expiry, keep it out of every repository, and revoke it when you're done. Everything else in the course uses restricted credentials.

  1. Create a credential as in Application credentials and clouds.yaml, with Expiration set to 30 days and Unrestricted (dangerous) ticked.
  2. Add its entry to your clouds.yaml under its own name, beside your restricted entry, so you choose it on purpose.
  3. Set OS_CLOUD to that entry's name for this tutorial's OpenTofu commands:
$env:OS_CLOUD = "<cloud-name>"

export OS_CLOUD=<cloud-name>

Step 2: The cluster

Create a folder called kubernetes-on-bks with a manifests folder inside it. The cluster is one OpenTofu resource. versions.tf pins the provider:

terraform {
  required_version = ">= 1.13"
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 3.4"
    }
  }
}

provider "openstack" {}

variables.tf names the cluster and its template, and sets two workers, so the app has somewhere to go when one is taken away:

variable "name" {
  description = "The cluster's name"
  type        = string
  default     = "shop-cluster"
}

variable "cluster_template" {
  description = "The cluster template: the node image and Kubernetes version"
  type        = string
  default     = "debian-13-v1.36.1"
}

variable "worker_count" {
  description = "How many worker nodes"
  type        = number
  default     = 2
}

cluster.tf looks up the template and creates the cluster with one control plane node. For a cluster you depend on, use three: see Create a cluster for every option.

data "openstack_containerinfra_clustertemplate_v1" "k8s" {
  name = var.cluster_template
}

resource "openstack_containerinfra_cluster_v1" "shop" {
  name                = var.name
  cluster_template_id = data.openstack_containerinfra_clustertemplate_v1.k8s.id
  master_count        = 1
  node_count          = var.worker_count
}

outputs.tf returns what kubectl needs to connect. It holds an administrator's certificate for the cluster, so it's marked sensitive, and the OpenTofu state holds it too: keep the state as safe as the cluster.

output "kubeconfig" {
  description = "kubectl's configuration for the cluster, with an administrator's certificate"
  value       = openstack_containerinfra_cluster_v1.shop.kubeconfig.raw_config
  sensitive   = true
}

Build it:

tofu init
tofu apply

The plan shows 1 resource to add. Type yes. The cluster takes about 10 minutes.

Write the kubeconfig to a file and point kubectl at it:

tofu output -raw kubeconfig | Set-Content -Encoding ascii shop-cluster.kubeconfig
$env:KUBECONFIG = "$PWD\shop-cluster.kubeconfig"
kubectl get nodes

tofu output -raw kubeconfig > shop-cluster.kubeconfig
export KUBECONFIG=$PWD/shop-cluster.kubeconfig
kubectl get nodes

Three nodes show Ready: one with the role control-plane, and two workers.

Step 3: The database

Everything for the app goes in its own namespace, shop. manifests/namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: shop

manifests/db.yaml runs PostgreSQL as a StatefulSet with one replica. Its volumeClaimTemplates asks for a 10 GB persistent volume, which BKS creates as a volume in your project and attaches to whichever node runs the pod. The data lives on that volume, not in the pod, so it survives the pod being replaced or moved. The Service gives the database a name, shop-db, that the app connects to wherever the pod runs.

apiVersion: v1
kind: Service
metadata:
  name: shop-db
spec:
  clusterIP: None
  selector:
    app: shop-db
  ports:
    - port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: shop-db
spec:
  serviceName: shop-db
  replicas: 1
  selector:
    matchLabels:
      app: shop-db
  template:
    metadata:
      labels:
        app: shop-db
    spec:
      containers:
        - name: postgres
          image: postgres:17.6
          env:
            - name: POSTGRES_USER
              value: shop
            - name: POSTGRES_DB
              value: shop
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: shop-db
                  key: password
            # The volume's root holds lost+found, so the data goes in a folder below it
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - containerPort: 5432
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "shop"]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

The database password goes in a file that stays on your computer. Copy manifests/db.env.example to manifests/db.env and set a password of your own:

password=<db-password>

Step 4: The app

manifests/app.py is the 4.4 app, reading the database's address and password from its environment:

import os
import socket
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import psycopg2

DSN = f"host={os.environ['DB_HOST']} dbname=shop user=shop password={os.environ['DB_PASSWORD']} connect_timeout=3"


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            body = b"ok\n"
        else:
            with psycopg2.connect(DSN) as conn, conn.cursor() as cur:
                cur.execute("CREATE TABLE IF NOT EXISTS visits (id serial PRIMARY KEY, at timestamptz DEFAULT now())")
                cur.execute("INSERT INTO visits DEFAULT VALUES")
                cur.execute("SELECT count(*) FROM visits")
                count = cur.fetchone()[0]
            body = f"app: {socket.gethostname()}, visits: {count}\n".encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(body)


ThreadingHTTPServer(("", 8080), Handler).serve_forever()

manifests/app.yaml runs it as a Deployment with two replicas, spread across the workers. An init container installs the PostgreSQL driver before the app starts. The readiness probe keeps a pod out of the Service until /health answers, as 4.4's health monitor kept an instance out of its pool, and the liveness probe restarts a pod that stops answering. The CPU request is what the autoscaler measures against in step 7.

apiVersion: v1
kind: Service
metadata:
  name: shop-app
spec:
  selector:
    app: shop-app
  ports:
    - port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shop-app
  template:
    metadata:
      labels:
        app: shop-app
    spec:
      # Spread the replicas across the workers, so losing one node doesn't take them all
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: shop-app
      initContainers:
        - name: deps
          image: python:3.14.0-slim
          command: ["pip", "install", "--no-cache-dir", "--target", "/deps", "psycopg2-binary==2.9.12"]
          volumeMounts:
            - name: deps
              mountPath: /deps
      containers:
        - name: app
          image: python:3.14.0-slim
          command: ["python", "/app/app.py"]
          env:
            - name: PYTHONPATH
              value: /deps
            - name: DB_HOST
              value: shop-db
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: shop-db
                  key: password
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
          volumeMounts:
            - name: app
              mountPath: /app
            - name: deps
              mountPath: /deps
      volumes:
        - name: app
          configMap:
            name: shop-app
        - name: deps
          emptyDir: {}

manifests/kustomization.yaml ties the files together. It puts everything in the shop namespace, turns app.py into a ConfigMap and db.env into a Secret:

namespace: shop

resources:
  - namespace.yaml
  - db.yaml
  - app.yaml

configMapGenerator:
  - name: shop-app
    files:
      - app.py

secretGenerator:
  - name: shop-db
    envs:
      - db.env

Apply it, and wait for the pods:

kubectl apply -k manifests
kubectl get pods -n shop -o wide
kubectl get pvc -n shop

After a minute or two, shop-db-0 and both shop-app pods show Running and 1/1 ready, with the app pods on different workers. The claim data-shop-db-0 shows Bound, and its volume appears in the Breqwatr Cloud Portal (Strata) under Volumes, named after the claim's volume (pvc- and an ID).

Step 5: Let traffic in

The Gateway API routes traffic into the cluster. Install NGINX Gateway Fabric, the controller that runs it, as in Gateway API:

kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v2.8.0" | kubectl apply -f -
helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --version 2.8.0 --create-namespace -n nginx-gateway --wait

manifests/gateway.yaml creates a Gateway listening on port 80, and a route that sends everything to the shop-app Service:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shop
  namespace: shop
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: shop
  rules:
    - backendRefs:
        - name: shop-app
          port: 8080

kubectl apply -f manifests/gateway.yaml
kubectl get gateway shop -n shop

The controller creates a LoadBalancer Service for the Gateway, and BKS creates a load balancer for it in your project, under Load balancers in the portal. Once PROGRAMMED shows True, ADDRESS is its public address. Send it a few requests:

1..6 | ForEach-Object { (Invoke-WebRequest -UseBasicParsing http://<gateway-address>/).Content }

for i in 1 2 3 4 5 6; do curl -s http://<gateway-address>/; done

Each response names the pod that answered, and the visit count rises by one each time. Both app pods answer.

Step 6: Break it

Delete one of the app pods:

kubectl get pods -n shop -l app=shop-app
kubectl delete pod -n shop <shop-app-pod>
kubectl get pods -n shop -l app=shop-app

The Deployment asks for two replicas, so Kubernetes starts a new pod straight away, with a new name. Requests keep being answered while it does: the other pod serves them, and the new one joins once its readiness probe passes.

Now take a whole worker away. Find the node running the database, and drain it: Kubernetes moves everything off it and stops scheduling new pods there.

kubectl get pod shop-db-0 -n shop -o wide
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl get pods -n shop -o wide

shop-db-0 starts on the other worker, and BKS moves its volume there: the volume is detached from the drained node and attached to the new one, which takes a minute or two. Send requests again: the visit count carries on from where it was, because the data was on the volume. Then let the node take work again:

kubectl uncordon <node-name>

This is Rebuild, don't repair one level up: you don't fix a pod or a node, you let it be replaced, and keep the data somewhere that outlasts it.

Step 7: Scale it

The autoscaler reads CPU use from metrics-server, which a BKS cluster doesn't include. Install it as in Horizontal pod autoscaling:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml
kubectl -n kube-system patch deployment metrics-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

manifests/hpa.yaml keeps between 2 and 6 app pods, aiming for half their CPU request each:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shop-app
  namespace: shop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: shop-app
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

kubectl apply -f manifests/hpa.yaml

Put some load on the app from inside the cluster, and watch the autoscaler:

kubectl run load -n shop --image=busybox:1.37 --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://shop-app:8080/ > /dev/null; done"
kubectl get hpa shop-app -n shop --watch

Within a few minutes TARGETS climbs past 50% and REPLICAS goes up. Stop the load with Ctrl+C, then kubectl delete pod load -n shop: about 5 minutes later the autoscaler takes the app back to 2 pods.

More pods need room on the workers. To add a worker, change worker_count in variables.tf to 3 and run tofu apply; to have the cluster add and remove workers itself, see Cluster autoscaling.

Step 8: Rebuild the cluster

The cluster is disposable: everything in it comes from the files in your folder, except the data. Back the data up, delete the cluster, build it again from code, and put the data back.

Back up the database to your computer:

kubectl exec -n shop shop-db-0 -- pg_dump -U shop -f /tmp/shop.sql shop
kubectl cp shop/shop-db-0:/tmp/shop.sql shop.sql

Delete the app and the cluster. Deleting the Gateway removes its load balancer, and deleting the namespace removes the claim and its volume. Both belong to your project rather than the cluster, so deleting only the cluster would leave them behind.

kubectl delete -f manifests/gateway.yaml
kubectl delete namespace shop
tofu destroy

Build it again. Run tofu apply and write the kubeconfig as in step 2, install NGINX Gateway Fabric as in step 5, then apply the app:

kubectl apply -k manifests
kubectl wait -n shop --for=condition=Ready pod/shop-db-0 --timeout=300s

Restore the data before anything uses the app, then let traffic in:

kubectl cp shop.sql shop/shop-db-0:/tmp/shop.sql
kubectl exec -n shop shop-db-0 -- psql -U shop -d shop -f /tmp/shop.sql
kubectl apply -f manifests/gateway.yaml

The Gateway has a new address. Send it requests: the visit count carries on from the backup.

Check it worked

  • kubectl get nodes shows the control plane and two workers Ready.
  • kubectl get pods -n shop shows shop-db-0 and at least two shop-app pods Running.
  • Requests to the Gateway's address name more than one pod, and the visit count rises by one each time.
  • After the rebuild, the count continues from the backup.

Clean up

kubectl delete -f manifests/gateway.yaml
kubectl delete namespace shop
tofu destroy

Then:

  • Check Volumes and Load balancers in the portal: nothing from the cluster should be left.
  • Revoke the unrestricted application credential, and remove its entry from clouds.yaml.
  • Delete shop-cluster.kubeconfig and shop.sql.

What you learned

  • A BKS cluster is built from code like anything else, and once it exists it's yours to run; see Shared responsibility.
  • Kubernetes keeps the number of pods you ask for: it replaces a deleted pod and moves work off a drained node; see Reliability.
  • Data that must outlast pods and nodes goes on a persistent volume, which is a volume in your project; see Pets vs cattle.
  • An autoscaler changes the number of pods with load; see Performance efficiency.
  • Load balancers and volumes that the cluster creates belong to the project, so clean them up through Kubernetes before you delete the cluster.

Next