Background

Operational excellence¶

Operational excellence is the ability to change and run a workload safely: knowing what's running, changing it in small steps you can undo, and finding out quickly when something goes wrong.

What the pillar covers¶

Most outages start with a change. Operational excellence is about making change routine rather than risky. It covers how a workload is built, how changes reach it, how you know it's healthy, and what people do when it isn't.

On a traditional platform, much of this lives in people's heads and in change tickets. In a cloud, the same API that builds a workload can also record, repeat and review every change to it, which makes a more disciplined way of working practical.

Design principles¶

Principle	What it means in practice
Operations as code	Infrastructure, configuration and routine tasks live in files under version control, not in a sequence of clicks
Small, frequent, reversible changes	Each change does one thing and can be rolled back or rolled forward quickly
Observe the workload	Collect metrics and logs, and alert on the signs of trouble before users notice
Write runbooks, then rehearse them	Common tasks and known failures have written steps, and the steps have been tried
Learn from every failure	After an incident, find what made it possible and change that, rather than who was involved
Keep environments alike	Test and production are built from the same code, so a change behaves the same in both

Operations as code¶

OpenTofu describes what infrastructure exists, Ansible describes how instances are configured, and both live in a repository. A change becomes a reviewed commit rather than a remembered set of steps. GUI, API and infrastructure as code covers the tools.

Small, reversible changes¶

A change that touches one thing is easy to review, easy to test and easy to undo. With infrastructure as code, undoing a change is often a matter of applying the previous version of the files. With replaceable instances, rolling forward can mean building new instances from a fixed definition and retiring the old ones. See Pets vs cattle.

Observability¶

You can't operate what you can't see. Observability means collecting enough information to answer "is it working, and if not, why?" without logging in to each instance. The building blocks are the same on Windows and Linux; the tools differ.

Signal	Windows	Linux
Logs	Windows Event Log	journald and files under `/var/log`
Metrics	Performance Monitor counters	CPU, memory and disk figures from tools such as `vmstat` and `iostat`
Shipping them off the instance	An agent that forwards events and counters to a central system	An agent that forwards logs and metrics to a central system

Keep logs and metrics off the instance itself. When an instance is replaced, its local logs go with it.

Runbooks¶

A runbook is a written procedure for one task: restoring a volume from a snapshot, rotating an application credential, failing over to the other region. A runbook that has never been run is a guess. Rehearse each one, ideally in a test project, and update it when the workload changes.

What it looks like on Breqwatr Cloud¶

One API for every tool. The Breqwatr Cloud Portal (Strata), the OpenStack CLI, OpenTofu and Ansible all drive the same platform, so you can look in the portal and change through code.
Projects separate environments. An organisation can hold several projects, for example one for test and one for production, each with its own resources.
A credential per tool. Each pipeline or automation tool can have its own application credential, so its access can be revoked on its own.
The audit log. Your organisation's activity log records changes to members, roles, projects and organisation settings, and when Breqwatr support was granted access and signed in. It shows who did what, and when.
Instance console and boot log. The portal shows each instance's console and its boot log, which helps when an instance won't start or can't be reached over the network.
Platform changes. Breqwatr changes the platform through regular, scheduled releases. See the release model on docs.

Impact¶

Changes become routine: reviewed, small and quick to undo.
Problems are found from metrics and logs, often before users report them.
Knowledge of how the workload runs lives in code and runbooks, not only in people.

Common anti-patterns¶

Changes made by hand in production: the next rebuild from code quietly undoes them, or the code stops describing what's really there.
Big-bang changes: many changes deployed together, so when something breaks nobody knows which one caused it.
Logs only on the instance: when the instance is replaced or won't boot, the evidence is gone.
Runbooks nobody has run: the procedure fails the first time it's needed, during an incident.
Test unlike production: a test environment built differently passes changes that then fail in production.