Skip to content

Background

Regions and failure domains

A failure domain is the set of things that fail together. Breqwatr Cloud has two regions, ca-central in Ontario and ca-east in Quebec, and each is an independent failure domain.

What a failure domain is

Every piece of infrastructure can fail, and some failures take other things down with them. A failure domain draws a line around everything that goes down together. Designing for reliability starts with choosing which of those failures a workload should survive.

When this fails It takes out What keeps running
An instance's operating system or application That instance Every other instance
A physical host Every instance on that host Instances on other hosts
A region Everything in that region Everything in the other region

On a traditional virtualization platform, a high availability cluster restarts virtual machines on another host after a host fails, with a short outage while each machine restarts. In a cloud, resilience comes from how a workload is built: more than one instance, so losing one doesn't stop the service, and a copy in another region for the largest failures.

Breqwatr Cloud's regions

Region Location Notes
ca-central Ontario The default region
ca-east Quebec

The two regions are independent failure domains: a failure in one does not take down the other. Each region runs its own instances, volumes and networks, and a resource created in one region stays there. One application credential works in both regions, and you choose the region per command or per tool configuration.

Resilience within a region

Within a region, a workload should run on more than one instance, behind a load balancer. If one instance fails, the others keep serving through the load balancer's address while it's replaced.

This works the same for Windows and Linux: two IIS instances on Windows, or two nginx instances on Linux, behind one load balancer.

It works in any region, so it applies whether a workload runs in one region or two. Some Breqwatr Cloud deployments have a single region, and for a workload there, this is how it keeps running when an instance fails.

Resilience across regions

To survive the loss of a whole region, run the workload in both ca-central and ca-east. That takes more than a second set of instances:

  • Data in both regions. Instances are easy to rebuild; data is not. A database replica, copied backups or object storage copies in the other region keep the data available when one region is not.
  • A way to send users to the working region. Traffic steering is part of the workload's design, for example a DNS name you control that can point at either region.
  • The same build in both places. Infrastructure as code makes this practical: the same definition, run with a different region, builds the same environment. See GUI, API and infrastructure as code.

The two combine: more than one instance behind a load balancer in each region, with the workload running in both.

On Breqwatr Cloud

  • Default region: ca-central. A downloaded clouds.yaml points at it and lists ca-east as the alternative.
  • Per-region networks: each project gets its own default network in each region it uses.
  • Load balancers sit within one region and spread traffic across instances there.

Impact

  • You decide which failures a workload survives, and the answer can differ between a test environment and a production one.
  • More than one instance behind a load balancer protects against the failure you're most likely to see, a single instance, with little extra work.
  • Running in both regions adds protection against a rarer but larger failure, at the price of more spend and more design.

Common anti-patterns

  • One instance, no load balancer: a service on a single instance stops whenever that instance does.
  • Instances in two regions, data in one: if the only copy of the data lives in one region, a second region's instances have nothing to serve.
  • Failover that's never been tried: a plan to move to the other region that has never been run tends to fail on first use.
  • A region written into everything: scripts and definitions with the region hard-coded can't build the same environment elsewhere.
  • Reliability: choosing failure domains is the core of reliability design.
  • Operational excellence: failover that is rehearsed and built from code.
  • Spend efficiency: each extra instance or region adds spend, so the choice is a trade-off for each workload.

Further reading