Everyone wants a service that never fails. Google’s SRE book says that this is the wrong goal: “100% is probably never the right reliability target”. A higher target costs much more. Users often cannot see the difference. And the effort to reach it slows down every other change.
So the real questions are different. How available must this service be? How do you measure it? What do you do when it fails too often? This lesson answers them with numbers.
What availability means
Availability is the share of time, or the share of requests, in which a service works. People write it as a number of nines: 99% is “two nines”, and 99.999% is “five nines”.
There are two ways to measure it:
- By time: uptime ÷ (uptime + downtime). This works for one machine that is either up or down.
- By requests: successful requests ÷ all requests. Google uses this method for large services. In the SRE book’s words, with servers in many places a service is “at least partially up at all times”. So a share of successful requests describes it better.
The Calculus of Service Availability gives a third form, which shows how to improve the number: availability = MTTF ÷ (MTTF + MTTR). MTTF is the mean time to failure, so how often failures happen. MTTR is the mean time to repair, so how long each one lasts. To improve availability, make failures rarer, or make them shorter.
What a nine allows
Each extra nine divides the allowed downtime by 10. This is part of the availability table from the SRE book:
| Availability | Per year | Per 30 days | Per day |
|---|---|---|---|
| 99% | 3.65 days | 7.2 hours | 14.4 minutes |
| 99.9% | 8.76 hours | 43.2 minutes | 1.44 minutes |
| 99.95% | 4.38 hours | 21.6 minutes | 43.2 seconds |
| 99.99% | 52.6 minutes | 4.32 minutes | 8.64 seconds |
| 99.999% | 5.26 minutes | 25.9 seconds | 0.87 seconds |
Try your own target and request rate:
What a number of nines allows
Your service has an SLO of 99.95%. It gets 1,000 requests each second. How much downtime and how many failed requests does the SLO allow?
Assumptions
- Downtime means a full outage: every request fails during that time.
- A month is 30 days here, as in the Google SRE availability table.
Make your own guess first. Then show the steps one at a time.
Look at five nines: about 26 seconds in a month. A person cannot react that fast. An alert must be sent, a person must see it, and the person must act. So at that level, recovery must be automatic. The SRE workbook says it plainly: the only way to defend such a target is “to design the system so that the chance of a 100% outage is extremely low”.
SLI, SLO and SLA
Three terms describe a reliability target. The SRE book defines them precisely, and people often mix them up:
| Term | What it is | Example |
|---|---|---|
| SLI: service level indicator | A measurement of the service | The share of requests that succeed in under 300 ms |
| SLO: service level objective | A target for an SLI | 99.9% of requests succeed in under 300 ms, over 30 days |
| SLA: service level agreement | A contract with a cost if the SLO is missed | If availability is under 99.9% in a month, the customer gets a refund |
The SRE book gives a simple test for an SLA: ask “what happens if the SLOs aren’t met?” If there is no explicit consequence, it is an SLO, not an SLA. Most engineers work with SLOs. Lawyers and business teams write SLAs.
Good SLOs
The SRE book gives advice for choosing SLOs:
- Use percentiles, not averages. An average hides slow requests. The book shows a system where “a typical request is served in about 50 ms”, but “5% of requests are 20 times slower”.
- Start from what users care about, not from what is easy to measure.
- Write the full condition. For example: “99% of Get RPC calls will complete in less than 100 ms.” Say the window and where you measure.
- Have few SLOs. The book says to choose “just enough SLOs to provide good coverage”.
- Do not make it much better than the SLO. Users depend on the reliability that they get, not on the number that you publish. Google once had a lock service, Chubby, that was so reliable that other services did not plan for its failure. So Google started to take it down on purpose, a few times each quarter, to keep it near its SLO.
Dependencies cost nines
A service cannot be more available than the services it needs. The Calculus of Service Availability puts it this way: “A service cannot be more available than the intersection of all its critical dependencies.” A critical dependency is a service whose failure makes your service fail.
When a request needs several services in sequence, multiply their availabilities. When you have independent copies, the service fails only if all copies fail. Try both:
Availability of a chain and of copies
A request passes through 5 services in sequence, each available 99.9% of the time. How available is the whole path? And what if you run 2 independent copies of a 99% service instead of one?
Assumptions
- Failures are independent. In real systems, shared causes (one bad config, one region) make failures happen together, so the true result is worse.
- Two copies help only if a client can use either one, and the failover is instant.
Make your own guess first. Then show the steps one at a time.
From this, Google uses the rule of the extra 9: each critical dependency must have about one more nine than your service. A 99.99% service needs dependencies at about 99.999%. If a dependency cannot give that, you must protect your service from it. Examples are a cache of its answers, a default answer, or a way to work without it for a short time.
The same paper gives a second rule. Availability also depends on how often failures happen and how long they last. For example, three full outages of 20 minutes each in a year make 60 minutes. That already breaks a 99.99% target, which allows about 53 minutes a year. The paper lists three ways to improve:
- Fewer outages: safer releases, more testing, design reviews.
- Smaller outages: split the service so that one failure affects only some users.
- Faster recovery: good monitoring, and one-step actions such as a rollback.
The error budget
The error budget is 100% minus the SLO. A 99.9% SLO gives a budget of 0.1%. The budget is not a failure. It is an amount of risk that the team agrees to accept.
The SRE book explains how teams use it. Product teams want to release often. Operations teams want stability. The error budget gives both teams one number. While budget remains, the team can release new versions. When the budget is used up, releases stop, and the team works on reliability. Step through one month:
An error budget over 30 days
The SLO is 99.9% over 30 days. So 0.1% of the time may fail. 0.1% of 30 days is 43.2 minutes. That is the error budget.
Two points from the walkthrough. First, every failure uses the same budget, including failures in another team’s service. Second, the rule for an empty budget must be agreed before the budget is empty. Google writes it as an error budget policy.
Burn rate
The SRE workbook adds one more idea for alerts. The burn rate is how fast the service uses its budget. A burn rate of 1 uses exactly all of the budget by the end of the window. With a 99.9% SLO over 30 days:
| Burn rate | Error rate | Budget is gone in |
|---|---|---|
| 1 | 0.1% | 30 days |
| 2 | 0.2% | 15 days |
| 10 | 1% | 3 days |
| 1,000 | 100% | 43 minutes |
The workbook recommends to page a person when the burn rate stays at 14.4 for one hour, which uses 2% of the budget. It recommends a ticket when the burn rate stays at 1 for three days, which uses 10% of the budget. So a large failure sends a page fast, and a small, steady one goes into the normal work queue.
Recap
- Availability is the share of time or of requests in which the service works. Each extra nine divides the allowed downtime by 10.
- 99.9% allows about 43 minutes in 30 days. 99.99% allows about 4 minutes. 99.999% allows about 26 seconds, which needs automatic recovery.
- An SLI is a measurement. An SLO is a target for it. An SLA is a contract with a consequence.
- Critical dependencies multiply. Each one needs about one more nine than your service.
- Availability = MTTF ÷ (MTTF + MTTR). Improve it with fewer, smaller or shorter outages.
- The error budget is 100% minus the SLO. While it lasts, release. When it is gone, fix reliability. The burn rate says how fast it is being used.
Check what you learned
Question 1 of 8A service has an SLO of 99.9% over 30 days. About how much full downtime does that allow?