Cloud Fault Tolerance, High Availability and Disaster Recovery

Last time we looked at scalability, elasticity and agility: the ways the cloud helps a system grow and shrink and move quickly. This time we are looking at what happens when things go wrong, because they will. The honest starting point for this whole topic is that every system can fail. Servers fail. Disks fail. Networks fail. Hardware components fail. Even well-managed data centres have bad days.

The interesting question is not how to stop that from happening, because you cannot. It is how building on the cloud can keep your software running anyway. Let's look at some key concepts.

Fault tolerance

Fault tolerance is the ability of a system to continue running when part of it fails.

In a cloud platform, services are usually designed with some level of fault tolerance built in. Although users interact with the cloud as a virtual service, it is still backed by real physical equipment in data centres, often spread across large server farms. This includes servers, disks, networking devices, and power systems. Data may be stored across multiple disks, so if one disk fails, another copy is still available. If a physical server or storage device in the server farm starts failing, becomes degraded, or shows signs of instability, the platform may move the service away from it automatically. The user does not usually have to manage that process directly, and may not even see it happening, which is rather the point.

A useful way to think about fault tolerance is to imagine a car with a spare tyre. The spare tyre does not stop a puncture from happening, but it gives you a way to keep going when something goes wrong. You are not promising the road will be smooth. You are making sure one bad moment does not leave you stranded.

These fault-tolerant features are often built into cloud systems. Although they are usually abstracted away from the user, it is useful to know they are happening.

Disaster recovery

Fault tolerance usually deals with smaller, local failures. A single disk, a single server, a component quietly giving up. Disaster recovery deals with bigger events.

A disaster might be a major power failure, a flood, a fire, a large network outage, a serious human error, or a problem affecting an entire region. These events are less common, but their impact can be much larger. A spare tyre will not help you if the whole garage has flooded!

Disaster recovery is the ability of a system to recover after a major failure. A common approach is to run or replicate an application across more than one geographical regions. If one region becomes unavailable, traffic can be redirected to another region where a working copy of the application exists.

This is not always simple, and disaster recovery requires planning. You need to think about what data should be replicated, how quickly the system needs to recover, how much data loss is acceptable, and what the failover process should actually look like when it runs. It is also worth testing that process before a real disaster forces you to find out whether it works.

Not every system needs the same level of disaster recovery. A personal blog and a hospital system do not carry the same risk. A small internal tool and a banking platform do not need the same recovery plan. The more critical the system, the more carefully disaster recovery needs to be designed, and the more it tends to cost. There is no prize for over-engineering the recovery plan for a side project.

High availability

Availability measures how much of the time a system is accessible and working. If users can access the system, it is up. If they cannot, it is down. There is not much grey area from the user's point of view.

High availability means designing a system so that it stays available for long periods with very little downtime. This matters because downtime has consequences. If an online shop is unavailable, customers cannot buy. If a booking system is unavailable, people cannot make reservations. If a business-critical application is unavailable, employees may not be able to do their jobs.

Availability is often described using percentages, and this is where the numbers get more interesting than they first appear. Ninety-nine per cent availability sounds high. Across a full year, though, it still allows the better part of four days of downtime. As the percentage increases, the allowed downtime shrinks quickly. Here is roughly how the common figures work out over a year:

See how those extra nines are doing a lot of work. Each one cuts the allowed downtime by roughly a factor of ten, and each one tends to be harder and more expensive to reach than the last. But downtime has a cost too. For many services, even a single day offline could mean unhappy users and lost revenue.

That level of reliability does not happen by accident. It usually requires careful design, redundancy, monitoring, and the right choice of services. Many cloud services provide strong availability features by default, but you still need to understand what each one actually offers and whether it suits what you are building. A high number on a marketing page is not the same as a high number in your own design.

The important thing is to match the design to the importance of the system. Not everything needs extreme availability. Most things genuinely do not. But when a system is critical, availability becomes one of the most important design goals you have, and it is worth spending real effort on.

Tying it back together

These ideas are closely related to the scalability we covered last time, and they tend to share the same machinery. The same ability to spread a workload across many servers, and to add or remove capacity as needed, is also what lets a system survive a failed component or shift its traffic to another region.

Scaling helps you cope with demand. Fault tolerance, disaster recovery and high availability help you cope with failure. In practice you are usually designing for both at once.

None of this means chasing the highest possible reliability for everything. As with so much in the cloud, the useful question is not how reliable can I make this but how reliable does this particular system actually need to be. A blog and a banking platform deserve very different answers, and pretending otherwise just wastes time and money.

Next we will look at economies of scale, and why running things at the size cloud providers operate at changes the costs in ways that are hard to match on your own.