“Make it faster” and “make it scale” sound like the same request. They are not. A system can be very fast for one user and fail with one thousand. Another system can handle a million users, but each request is slow. This lesson separates the two ideas. It then shows why adding servers gives less than you expect, and how to predict how much less.
Two different questions
- Performance asks: how fast is one request? You measure it with latency, the time for one request. You can also measure work per second on one machine.
- Scalability asks: what happens when the load grows? Werner Vogels, the CTO of Amazon, gives a precise definition: a service is scalable if, when you add resources, its performance increases “in a manner proportional to resources added”.
So a scalable system gives about twice the throughput with twice the servers. A system that gives 1.2 times the throughput with twice the servers does not scale well, even if each request is fast.
Throughput is the number of requests that the system completes each second. Latency and throughput are related, but they are not the same. In the SRE book, Google describes two kinds of storage cluster. One kind keeps its queues empty for low latency. The other kind keeps its queues full for high throughput, and it costs “perhaps as little as 10–50%” as much. You cannot have the best of both on one cluster.
Scaling up and scaling out
There are two ways to add capacity:
| Scale up (vertical) | Scale out (horizontal) | |
|---|---|---|
| What you do | Use a larger machine: more cores, memory, disk | Add more machines of normal size |
| Code changes | Usually none | The work must be split across machines |
| Limit | The largest machine you can buy | Shared state, and coordination between machines |
| Cost | Grows faster than the capacity at the high end | Grows about in line with the capacity |
| Failure | One machine is one point of failure | One failure removes only part of the capacity |
Most systems do both. Scaling up is the simple first step, and it is often enough for a long time. Scaling out is necessary when one machine is too small, too expensive or too risky. Step through a typical path:
Scaling up, then scaling out
One server and one database. While the load is small, this is the simplest and cheapest design. Each request is fast, so performance is good.
The last step of the walkthrough shows the central lesson. Servers that hold no state are easy to scale out. State is hard to scale. A stateless server keeps nothing between requests, so any server can handle any request. All the difficult parts of scaling are in the parts that hold data: databases, caches and queues. Most of the later tracks on this site are about those parts.
Why more servers give less
If one server handles 1,000 requests each second, do 10 servers handle 10,000? Almost never. Two things stop it.
Shared work: Amdahl’s law
Gene Amdahl described the first limit in 1967. Some part of the work cannot run in parallel. For a computer of his time, he estimated that this serial part, which he called “housekeeping”, was about 40% of the instructions. That alone “would then place an upper limit on throughput of five to seven times” the speed of one processor, however many processors you add.
The rule that later took his name says this. A share p of the work can run in parallel. Then N machines give a speedup of
speedup = 1 ÷ ((1 − p) + p ÷ N)
With 95% parallel work, 32 machines give about 12.5 times the speed, not 32. And no number of machines can give more than 1 ÷ 0.05 = 20 times. The serial 5% sets the limit.
In a real service, the serial part is any work that all requests must do one at a time. Examples are a single lock, a counter in one row of a database, or one leader that orders all writes.
Keeping copies in agreement: the USL
Amdahl’s law says that more machines always help, a little. Real systems are often worse: after some point, more machines make the system slower. Neil Gunther’s Universal Scalability Law (USL) explains this with a second term:
capacity(N) = N ÷ (1 + σ(N − 1) + κ·N(N − 1))
Here σ (sigma) is contention: the share of time spent waiting for shared resources. That is the Amdahl term. κ (kappa) is coherency: the cost of keeping copies of data in agreement. Examples are cache invalidation, a lock that moves between machines, and nodes that exchange state.
The coherency term grows with N², because each machine may need to agree with each other machine. So at some point it grows faster than the gain. Gunther proves that the capacity has a maximum at N* = √((1 − σ) ÷ κ). After that, in his words, “the throughput becomes retrograde”. It goes down as you add machines.
Change the values below. Set κ to 0 to see Amdahl’s law, and raise it to see the peak:
How much faster are more servers?
A service runs on 1 server. You add more. Each request spends part of its time on shared work, for example a lock or a shared counter. The servers also spend some time keeping their data in agreement. How much more throughput does each number of servers give?
Assumptions
- Amdahl's law: speedup = 1 ÷ ((1 − p) + p ÷ N), where p is the part of the work that can run in parallel.
- Universal Scalability Law (USL): capacity = N ÷ (1 + σ(N − 1) + κN(N − 1)).
- σ (sigma) is contention for shared resources. κ (kappa) is the cost of keeping copies of data in agreement.
- The default σ = 3% and κ = 0.05% are example values. Measure your own system to find its values.
Make your own guess first. Then show the steps one at a time.
What makes a design scale
The two laws point to the same rules:
- Make servers stateless. Keep sessions and data in a shared store, so any server can handle any request. Then you can add and remove servers freely.
- Reduce shared work (σ). Avoid one lock, one counter or one leader on the path of every request. Split hot data into parts that do not need the same lock.
- Reduce agreement between machines (κ). Prefer designs where each machine owns its own part of the data, so machines rarely need to agree. Partitioning, which a later track covers, is the main tool.
- Measure at more than one size. A test with one server tells you about performance. A test at several sizes tells you about scalability.
Kleppmann adds an important point in Designing Data-Intensive Applications: there is no design that scales for every load. A system for 100,000 small requests each second is different from one for 3 requests each minute that are each 2 GB. So scalability always means scalable for a given kind of load. Describe that load in numbers first, as the previous lesson showed.
Scale up
- Simple: no code changes, one machine to run
- No coordination between machines, so no coherency cost
- Often the cheapest choice until the machine is very large
Scale out
- The work must be split, and shared state becomes the hard part
- Coordination adds latency and new kinds of failure
- Necessary when one machine is too small, too costly or too risky
Recap
- Performance is how fast one request is. Scalability is how performance changes as you add resources for more load.
- Latency is the time for one request. Throughput is requests completed each second.
- Scale up with a larger machine first. Scale out with more machines when you must.
- A stateless server is easy to scale out. State is the hard part.
- Amdahl’s law: the serial share of the work limits the speedup to 1 ÷ (serial share).
- The USL adds a coherency cost that grows with N². Throughput reaches a peak at √((1 − σ) ÷ κ) machines, then falls.
Check what you learned
Question 1 of 8A service answers each request in 5 ms with 10 users. With 1,000 users, requests start to time out. What kind of problem is this?