An estimate answers one question: is this a problem? Is 1 server enough, or 1,000? Does the data fit in memory, on one disk, or on a fleet of disks? You do not need an exact answer. You need the right order of magnitude, which means the right power of ten.
This lesson gives you a method, the numbers to remember, and three practice drills. Do each drill yourself before you show the steps.
Why estimate
The SRE Workbook explains the purpose: “examples of sound reasoning and assumption making are more important than any final values.” An estimate is a way to think. It shows which part of a design is hard, before you spend time on the wrong part.
An estimate also stops bad designs early. In the SRE Workbook example, an estimate shows that one machine would need 2,500 disks to keep up with the writes. The design changes before anyone builds it.
The method
Use the same four steps each time:
- Write the assumptions. Say each number and where it comes from. Others must be able to check it.
- Round to powers of ten. Keep one digit and a power: 86,400 becomes 10⁵, and 200 million becomes 2×10⁸.
- Multiply the digits, add the powers. 2×10⁸ × 2×10¹ = 4×10⁹. To divide, divide the digits and subtract the powers.
- Check the result. Compare it with something you know. Does the answer have the correct units? Is it possible?
Step through the method on a real example:
The estimation method, one step at a time
Start with the numbers that you know or can assume. Say each assumption out loud, so that others can check it.
The SRE Workbook uses this style for its calculations. For example: (5 × 10⁵ queries/sec) × (8.64 × 10⁴ seconds/day) × (2 × 10³ bytes) = 86.4 TB/day. With powers of ten, it is hard to lose a zero.
Numbers to remember
You need only a few numbers. Learn this table:
| Quantity | Value | Quick value |
|---|---|---|
| Seconds in a day | 86,400 | 10⁵ |
| Seconds in a month | about 2.6 million | 2.5×10⁶ |
| Seconds in a year | about 31.5 million | 3×10⁷ |
| 1 million each day | about 12 each second | divide by 10⁵ |
| 1 billion each day | about 12,000 each second | |
| KB, MB, GB, TB, PB | 10³, 10⁶, 10⁹, 10¹², 10¹⁵ bytes | use powers of 1,000 |
| 1 Gbps | 125 MB each second | divide by 8 |
| 10 Gbps | 1.25 GB each second |
Also remember the times from the previous lesson. A memory read is about 100 ns, and an SSD read is about 16 µs. A round trip is about 500 µs in a datacenter and 150 ms across an ocean.
Drill 1: requests each second
Most designs start here. Turn users and actions into requests each second, then find the peak. Make your own answer first, then show the steps:
Users to requests each second
A social app has 200 million daily active users. Each user opens the app and loads the feed 20 times a day. How many feed requests each second must the system handle at the peak?
Assumptions
- Each feed load is one request to the feed service.
- The peak hour has about 2 times the average traffic. This is common for a service that is used around the world.
- 1 day = 86,400 seconds. For quick work, use 100,000 (10⁵).
Make your own guess first. Then show the steps one at a time.
Two rules from this drill:
- Daily numbers divide by 10⁵. 1 million a day is about 10 each second. With the exact value of 86,400 it is about 12.
- Always use the peak. The peak is often 2 to 10 times the average. The ratio depends on where the users are. A service in one country has a large peak in the evening. A service used around the world is flatter.
Drill 2: storage
Storage grows every day, and it often decides the design. Multiply the size of each item by the number of items, then by the time you keep them, then by the number of copies:
Storage for photos over five years
Users upload 100 million photos a day. Each photo is stored in three sizes. How much storage does the service need after five years, with copies for safety?
Assumptions
- One photo has an original of 2 MB, a medium size of 200 KB and a thumbnail of 20 KB.
- Each size is stored 3 times. Erasure coding could use about 1.5 times instead. That is a later topic.
- Uploads stay the same for five years. In a real plan, add growth.
Make your own guess first. Then show the steps one at a time.
Two rules from this drill:
- Find the large term, ignore the small ones. The original photo is 90% of the bytes. The thumbnails almost do not count. In an estimate, you can drop a term that is less than 10% of the total.
- Copies multiply everything. Three copies triple the cost. That is why systems at this size use erasure coding, which gives the same safety with fewer extra bytes. A later topic explains it.
Drill 3: servers, with Little’s law
To count servers, you need one more tool. Little’s law links three averages for any stable system:
L = λ × W
Here L is the average number of items in the system. λ (lambda) is the average arrival rate. W is the average time that each item spends in the system. John Little proved it in 1961. The law holds for any order of service, first come or not, and for any pattern of arrivals.
For a service, L is the number of requests in progress at the same time. Each request in progress uses a worker, a thread or a connection. So Little’s law tells you how many of those you need:
Servers for a load, with Little's law
A service gets 50,000 requests each second at the peak. Each request takes 40 ms, and one worker handles one request at a time. How many servers do you need?
Assumptions
- Little's law: the average number of requests in the system = arrival rate × time in the system.
- Each server runs 64 workers, for example one for each thread in a pool.
- Run servers at 60% of their full capacity. This leaves room for spikes and for the loss of a server.
Make your own guess first. Then show the steps one at a time.
Little’s law has an important result: latency costs capacity. If each request takes twice as long, twice as many requests are in progress at the same time, so you need twice as many workers. A slow dependency can therefore exhaust a service that has spare CPU.
Bandwidth
To find bandwidth, multiply requests each second by the bytes in each request. Then compare it with the network link.
For example, the SRE Workbook joins 10,000 ad clicks each second, and each click is about 2 KB. So (10⁴ clicks/sec) × (2 × 10³ bytes) = 2 × 10⁷ bytes each second. That is 20 MB each second, or 160 Mbps. That fits easily on one network link. But the same calculation for 500,000 queries each second gives 1 GB each second, which fills a 10 Gbps link to 80%.
Remember the factor of 8: a link speed is in bits each second, and data size is in bytes.
Common mistakes
- Mixing bits and bytes. A 10 Gbps link moves 1.25 GB each second, not 10 GB.
- Using the average. Always estimate the peak.
- Forgetting copies. Storage and write bandwidth multiply by the number of copies.
- Too much precision. An answer of “46,296 requests each second” suggests a precision that the assumptions do not have. Say “about 50,000”.
- Hidden assumptions. If you do not say an assumption, nobody can correct it.
Recap
- An estimate tells you if something is a problem. You need the order of magnitude, not the exact value.
- Method: write the assumptions, round to powers of ten, multiply the digits and add the powers, then check the result.
- Remember: a day is about 10⁵ seconds. 1 million a day is about 12 each second. A link speed in bits divides by 8 to give bytes.
- Storage = item size × items each day × days kept × copies. Find the large term.
- Little’s law: requests in progress = arrival rate × time for each request. Slow requests use capacity.
Check what you learned
Question 1 of 8A service gets 500 million requests a day. About how many requests each second is that, on average?