Background

Reliability¶

Reliability is a workload's ability to keep doing its job when parts of it fail, and to recover when they do. It starts with knowing what can fail, and ends with a restore you have actually tested.

What the pillar covers¶

Every component fails eventually: an instance, a disk, a host, a whole region. A reliable workload is built on the assumption that something will fail, and is designed so that the failures it has to survive don't stop it. Reliability covers where the workload runs, how many copies of it run, how its data is protected, and how quickly it can be brought back.

Two measures help set the target for each workload:

Measure	The question it answers
Recovery time objective (RTO)	How long can the workload be down before it matters?
Recovery point objective (RPO)	How much data, measured in time, can be lost?

A test environment might accept a day of downtime and losing yesterday's changes. A customer-facing service might accept minutes and no loss at all. The answers decide how much of the rest of this page a workload needs.

Design principles¶

Principle	What it means in practice
Know your failure domains	Decide which failures the workload must survive: an instance, a host, a region
No single instance	A service that matters runs on more than one instance, behind a load balancer
Detect and route around failure	Health checks find a failed instance and stop sending it traffic
Protect the data separately	Instances can be rebuilt; data needs snapshots, backups and copies kept apart from the instance
Test the restore	A backup is only as good as the last restore from it
Rebuild from code	An environment defined in code can be rebuilt after any failure, in the same region or the other one
Change carefully	Most incidents start with a change, so small, reversible changes are a reliability measure too

What it looks like on Breqwatr Cloud¶

Within a region¶

Run a workload on more than one instance, behind a load balancer. If one instance fails, the others keep serving through the load balancer's address while it's replaced. A load balancer's health monitor checks each instance and stops sending traffic to one that fails. Regions and failure domains explains the pattern.

Across regions¶

Breqwatr Cloud has two regions, ca-central in Ontario and ca-east in Quebec, and they are independent failure domains. A workload that has to survive the loss of a region runs in both, with its data copied between them and a way to send users to the region that's working. Not every workload needs this; it's the answer for the ones whose RTO and RPO demand it.

Protecting data¶

Tool	What it is	Good for
Volume snapshot	A point-in-time copy of a volume, kept on the same storage as the volume	Fast rollback before a risky change
Volume backup	A full copy of a volume, kept on separate storage	Recovery if the original storage is lost
Instance snapshot	An image that points at snapshots of the instance's volumes, kept on the same storage as the volumes; not an independent backup	Capturing an instance to launch copies of it
Copies in the other region	Backups or replicas of data held in the other region	Recovery from the loss of a region

Snapshots and backups capture a disk as it is. For a consistent copy of a running application, prepare it first:

	Windows	Linux
Flush and pause writes	Volume Shadow Copy Service (VSS), or the application's own backup mode	`sync`, or `fsfreeze` on the mounted filesystem
Databases	The engine's own backup, such as a SQL Server backup	The engine's own backup, such as `pg_dump` for PostgreSQL

Rebuilding from code¶

When the environment is defined in OpenTofu and configured with Ansible, recovery after a failure is a rebuild: run the code, restore the data, and point traffic at the result. The Rebuild, don't repair tutorial does exactly that.

Impact¶

The failure you're most likely to see, a single instance, doesn't stop the service.
Recovery is a rehearsed procedure with a known time, not an improvisation.
Each workload gets the level of protection its RTO and RPO call for, and no more.

Common anti-patterns¶

A single instance for a service that matters: it stops whenever that instance does.
Backups never restored: the first restore attempt happens during an outage, and fails.
Snapshots as the only protection: a volume snapshot or an instance snapshot sits on the same storage as the volume, so it doesn't help if that storage is lost.
Data on the root disk: if the data lives on a root volume that is deleted with the instance, a rebuild loses it.
Two regions, one copy of the data: instances in both regions, but the data in only one.