Latency vs throughput vs bandwidth
SystemsHigh priority~12 minHow long one request takes, how many complete per second, and the ceiling throughput never quite reaches — three numbers that move against each other, and why the one that matters is quoted at a percentile.
Definition
Three questions about the same system, and an answer to one tells you little about the other two. Latency is how long a single request takes, end to end. Throughput is how much work finishes per unit of time — requests per second, bytes per second. Bandwidth is the ceiling on throughput: the maximum rate a path could carry with nothing else in the way. Throughput is what you measure; bandwidth is what you were sold.
The cost model is that you rarely move one without paying in another.
Little's Law (Little, 1961) is the constraint underneath: L = λW, where the average number of requests inside the system equals the arrival rate times the time each one spends there. Read it as concurrency = throughput × latency and the coupling is explicit. At fixed concurrency, doubling throughput means halving the time any request is allowed to take.
Run it in the direction an interviewer asks for: 50 concurrent workers at 10 ms a request is 50 ÷ 0.01 s = 5,000 requests/second. One worker at the same latency is 100. Nothing about the code changed — only how many requests are allowed to be in flight at once.
When to use
The cue is a requirement that says fast or handles N users without saying which one it means. Split it before you draw anything. A p99 target is a latency requirement and it drives the shape of the request path — regions, hops, fan-out. A QPS number is a throughput requirement and it drives capacity. Answering a QPS question with a caching strategy is the tell that the two got merged.
Amazon's Dynamo paper shows the well-formed version of the requirement: a response within 300ms for 99.9% of its requests for a peak client load of 500 requests per second. Latency, percentile and load, in one sentence.
Techniques
Latency decomposes into four delays, and which one dominates decides which fixes are even available. Propagation is distance divided by signal speed. Transmission is bytes divided by link bandwidth — the point where bandwidth enters latency. Processing is the work the machine does. Queuing is time spent waiting behind other requests. Grigorik's High Performance Browser Networking uses this split and carries the figure that makes it concrete: light in fibre travels at roughly two-thirds of c. A 6,000 km transatlantic hop therefore costs about 30 ms one way, 60 ms round trip, before anyone does any work.
That is why add a cache is not a universal answer. A cache beside the origin does nothing for a propagation delay set by distance; only moving the data closer does, which is what a CDN is.
Latency is a distribution, not a number, so it is quoted at a percentile. The Google SRE book's reading: p99 is the value 99% of requests come in under, so p50 describes the typical request while p99 and p99.9 show a plausible worst case. Two consequence follows: measure where the user is. The same book notes that client-side latency is often the more user-relevant metric, since server-side timing excludes DNS, TLS and the last mile.
Bandwidth bounds throughput without delivering it. The gap between them is protocol overhead, queuing and the slowest hop on the path — and inside the system the same asymmetry holds: throughput is set by the slowest stage, so widening anything else changes nothing. Over a fat, long link the binding constraint is the bandwidth-delay product — bandwidth × round-trip time, the bytes that must be in flight to keep the pipe busy; what the transport does about it belongs to TCP vs UDP.
| Latency | Throughput | Bandwidth | |
|---|---|---|---|
| Measures | Time for one request | Completed work per unit time | Maximum possible rate |
| Unit | ms, at a percentile | requests/s, bytes/s | bits/s |
| Improved by | A shorter path and less work per request — caching, CDNs | More parallelism and more capacity — load balancers, queues | A bigger pipe, which is a purchase rather than a design |
| Does not buy you | More capacity | A faster single request | Throughput you actually observe |
| Where it bites | User-perceived responsiveness | Peak-hour saturation | Bulk transfer, cross-region replication |
Worked examples
A product page over 20 services
Take a product page assembled from 20 backend services in parallel, each with a p99 of 50 ms. Both figures are assumptions a candidate would propose and confirm, not measurements.
Treat the slow events as independent — an assumption that flatters the design, since real interference is correlated. The page is fast only when all 20 dependencies are fast, and 0.99²⁰ = 0.818. So 18% of page loads touch at least one dependency in its slowest 1%. The page's p99 is nowhere near 50 ms, because a per-service p99 target was never a page-level target.
| Quantity | Value | Derivation |
|---|---|---|
| Server with 10 ms typical latency and a p99 of 1 s, one server per request | 1 request in 100 is slow | The paper's own worked scenario. |
| Same servers, request fanned out to 100 of them | 63% of user requests exceed 1 s | 1 − 0.99¹⁰⁰ = 1 − 0.366; the point marked “x” on the paper's figure. |
| 1-in-10,000 slow at a single server, fan-out of 2,000 | Almost 1 in 5 exceed 1 s | 1 − 0.9999²⁰⁰⁰ = 1 − 0.819 ≈ 0.18; the point marked “o”. |
| Real Google service — p99 for one random leaf, timed at the root | 10 ms | Table 1 of the paper. |
| Same service — p99 for 95% of leaves finishing | 70 ms | Table 1. |
| Same service — p99 for every leaf finishing | 140 ms | Table 1. Waiting on the slowest 5% costs 140 − 70 = 70 ms, half the total. |
| BigTable read of 1,000 keys across 100 servers, hedged after 10 ms | p99.9 falls from 1,800 ms to 74 ms | The paper's benchmark; the hedge costs roughly 2% more requests. |
Three moves are open on the 20-service page, and each is priced. A hedged request sends a second copy to another replica after a short delay and takes whichever answer lands first — the remedy the row above measures, paid for in duplicated work. A tight timeout with a fallback caps the tail at the cost of a degraded page. Reducing fan-out removes tails entirely and buys back coupling between services.
None of the three is make each service faster. The paper's own account of where the tail comes from explains why: shared resources, background daemons, queueing and maintenance activity. The slow request is usually interference, not a slow request.
Tradeoffs
| What you give up | When it bites | |
|---|---|---|
| Batching for throughput | The first item in a batch waits for the window to close, so the exchange rate is the window length | A write pipeline sharing a store with an interactive read path |
| Concurrency past saturation | Queuing delay climbs non-linearly while throughput flattens — Little's Law with concurrency pinned | Load tests that report only the mean at peak |
| Replicating for capacity | Capacity, not speed — a replica shortens the queue or the distance, and does nothing to the work itself | A design promising lower latency from read replicas alone |
| Fanning out for parallelism | Every added dependency multiplies its own tail into the composed response | Any request that touches many services, which is most of them |
None of these rows is an argument against batching, replicating or fanning out. They are the price to name while proposing it, and the exchange rate is the part worth saying out loud: the batch window, the saturation point, the percentile you are trading away. Targets themselves — which percentile, at what value, over what window — belong to Availability.
Things to look out for
- Quoting an average. The SRE book's Figure 4-1 shows a service with a ~50 ms typical request where 5% of requests are 20 times slower — a mean hides exactly the tail a target exists to bound. It bites the moment average latency 100 ms is offered as a requirement.
- Averaging percentiles. The mean of two servers' p99 values is not the fleet's p99. Aggregate from raw distributions or histograms, or the number you report is not a percentile of anything.
- Measuring only server-side. Server timing excludes DNS, TLS and the last mile, which is most of what the user waits for. The dashboard stays green while the product feels slow.
- Assuming replicas cut latency. They add capacity. A replica only shortens a request if it shortens the queue or the distance — say which one you are claiming (Scalability owns the capacity half).
In the interview
- You want p99 under 200 ms and the request fans out to 20 services — is that achievable? Do the arithmetic aloud. If each dependency is itself at 200 ms p99, the composed response is far worse, because the page waits for the slowest of the twenty. Either the per-service budget is much tighter than the page budget, or the fan-out shrinks.
- Is bandwidth just throughput by another name? No, and the distinction is the page: bandwidth is bought, throughput is achieved, and latency is neither. Reach for the bandwidth-delay product when the prompt replicates across a WAN — a 1 Gbps link at 100 ms RTT holds 12.5 MB in flight, and nothing moves faster until that pipe is full.
- The depth signal is naming fixes that attack variance rather than the mean: hedged requests, a timeout with a degraded fallback, fewer services on the critical path. The BigTable hedging result above — a p99.9 of 1,800 ms cut to 74 ms for about 2% more load — is the strongest single number available here.