Skip to content

How-to

Horizontal pod autoscaling

Change the number of pods in a deployment with its CPU use, so it runs more replicas under load and fewer when it's quiet.

A HorizontalPodAutoscaler compares the deployment's average CPU use with a target, as a percentage of what each pod requests, and sets the number of replicas between a minimum and a maximum. It reads CPU use from metrics-server, which a BKS cluster doesn't include: you install it. When more pods need more nodes, cluster autoscaling adds them.

Before you start

  • A cluster, and kubectl connected to it: see Create a cluster.
  • A deployment whose containers have a CPU request. The target is a percentage of the request, so a pod without one can't be scaled on CPU.

Install metrics-server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml
kubectl -n kube-system patch deployment metrics-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

The patch is needed on BKS. Each node's kubelet serves its metrics with a self-signed certificate that doesn't name the node's address, so metrics-server can't verify it, and logs x509: cannot validate certificate. --kubelet-insecure-tls skips that check for the connection from metrics-server to the kubelets, inside the cluster.

After a minute, CPU and memory use show for each node:

kubectl top nodes

Create the autoscaler

This example runs a small web app that uses CPU for each request, with a CPU request of 200m, a fifth of a vCPU.

web.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.k8s.io/hpa-example
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 200m
            limits:
              cpu: 500m
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80

The autoscaler keeps average CPU use near 50% of the request, with between 1 and 5 replicas.

hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

kubectl apply -f web.yaml -f hpa.yaml
kubectl get hpa web

TARGETS shows the current use against the target, such as cpu: 0%/50%, once metrics-server has a reading.

Watch it scale

Send the app a steady stream of requests from a pod in the cluster:

kubectl run load --image=busybox:1.37 --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://web >/dev/null; done"
kubectl get hpa web --watch

As CPU use passes the target, REPLICAS rises, up to the maximum. Stop the load, and the autoscaler waits 5 minutes before it removes replicas, so a short lull doesn't scale the app down only to scale it up again:

kubectl delete pod load
kubectl get deployment web

Check it worked

  • Under load, kubectl get hpa web shows more replicas than the minimum, and kubectl get deployment web shows them ready.
  • After the load stops, the replicas drop back to the minimum.
  • kubectl delete -f hpa.yaml -f web.yaml removes the example.