Skip to content

Tutorials

Rebuild, don't repair

When an instance goes wrong, replace it from code instead of signing in to fix it. Using the application from A three-tier web app, replace an app instance, then the database instance, then every instance at once, and see the data come through each time.

Well-Architected

Pillar: Reliability. Recovery is a rehearsed rebuild from code with a known result, and the data lives on a volume that outlasts the instances.

Also touches: Operational excellence (no hand-made changes to drift from the code) and Security and sovereignty (a rebuilt instance starts from a current image, with nothing left behind from the old one).

What you'll do

Step What's replaced What survives
2 One app instance Everything else; the other app instance keeps serving
3 The database instance The data volume, and the database's address
4 Every instance The network, the security groups, the load balancers, the data volume

The visit count is the check throughout: if it carries on from where it was, the data survived.

Before you start

Run the commands in the three-tier folder, with OS_CLOUD set as before. They're the same in PowerShell and in a macOS or Linux shell.

Step 1: Protect the data

Rebuilding is safe only if the data can't go with the instances. In database.tf, add a lifecycle block to the data volume:

resource "openstack_blockstorage_volume_v3" "db_data" {
  name = "${var.prefix}-db-data"
  size = 10

  lifecycle {
    prevent_destroy = true
  }
}

tofu plan shows no changes: the setting only stops OpenTofu from deleting the volume. Any plan that would delete it now fails with "Resource instance cannot be destroyed" instead.

Note the current visit count from the url output; you'll compare with it after each step.

Step 2: Replace an app instance

Say shop-app-01 is misbehaving. Instead of signing in to investigate, replace it:

tofu apply -replace='openstack_compute_instance_v2.app[0]'

The plan replaces the instance and its pool member, because the new instance gets a new address. While OpenTofu works, the internal load balancer's health monitor takes shop-app-01 out of the pool, and shop-app-02 answers every request. Once the new instance has installed its software and passes its health checks, the load balancer sends it traffic again.

Request the url a few times: both app instances answer, and the count carries on.

Step 3: Replace the database instance

There's only one database instance, so the application can't record visits while it's replaced. Shut it down cleanly first, so the database stops writing before the instance goes:

openstack --os-cloud <cloud-name> server stop shop-db-01
tofu apply -replace=openstack_compute_instance_v2.db

OpenTofu deletes the instance and creates a new one from the image, on the same port, so its address doesn't change and the app tier finds it again. It detaches the data volume from the old instance and attaches it to the new one. At first boot, db.sh finds the volume already formatted and the database cluster already on it, and starts it as it was.

Wait a few minutes for the first boot, then request the url: the count continues from where it was before step 3.

The same happens whenever templates/db.sh changes: the first-boot script is part of the instance, so OpenTofu plans to replace the database instance, and the data comes through.

Step 4: Rebuild every instance

Replace all five instances in one apply:

tofu apply -replace=openstack_compute_instance_v2.db -replace='openstack_compute_instance_v2.app[0]' -replace='openstack_compute_instance_v2.app[1]' -replace='openstack_compute_instance_v2.web[0]' -replace='openstack_compute_instance_v2.web[1]'

Stop the database instance first, as in step 3. The network, the security groups, the load balancers, the floating IP and the data volume all stay. When the instances have finished their first boot, the application works as before, and the count carries on.

Check it worked

  • After each step, tofu plan shows no changes.
  • The visit count never goes back: each step's first request is one higher than the last request before it.
  • openstack --os-cloud <cloud-name> volume show shop-db-data shows the same volume ID throughout, attached to the newest database instance.

Clean up

prevent_destroy stops tofu destroy too. To delete everything, including the data, remove the lifecycle block from database.tf, then:

tofu destroy

Type yes when asked.

What you learned

  • Replacing an instance from code gives a known result in a known time; repairing by hand gives neither. See Pets vs cattle.
  • Data that must survive lives on a volume of its own, protected with prevent_destroy, and the instance's first-boot script picks it up again.
  • A fixed port keeps an address the same across rebuilds, so the tiers that depend on it don't need to change.
  • Two instances behind a load balancer keep a tier serving while one is replaced; a single instance can't. See Reliability.

Next