📘 Capacity Planning Is Harder Than It Looks

Why “we’re under 50% CPU” is a dangerous lie

Capacity planning is not about how fast your system is.
It’s about how it fails under load.

1️⃣ The Most Common (Wrong) Mental Model

❌ What teams think

“Our service can handle 10k RPS.”

✅ Reality

Capacity is work per second, not requests per second.

Two requests are never equal.

2️⃣ The First Big Mistake — RPS ≠ Load

Example

Request	CPU Cost
GET /profile	1 ms
GET /feed	40 ms
POST /checkout	120 ms

At 1,000 RPS:

• All profiles → fine

• All checkout → system dead

Traffic shape matters more than traffic volume.

3️⃣ The Real Unit of Capacity: CPU-Seconds

Key Rule

A system has N CPU-seconds per second.

Example:

• 8 cores

• Each core ≈ 1 CPU-second / second

Total budget:

8 CPU-seconds / second

If one request costs:

50 ms CPU = 0.05 CPU-seconds

Max sustainable throughput:

8 / 0.05 = 160 RPS

No amount of async changes this.

4️⃣ Little’s Law (The Law Everyone Ignores)

Concurrency = Throughput × Latency

If:

• Throughput = 200 RPS

• Latency = 500 ms

Then:

Concurrency = 100 in-flight requests

Increase latency → concurrency explodes → memory explodes.

5️⃣ Why “Headroom” Doesn’t Save You

❌ Common rule

“Keep CPU under 70%”

Why this fails

• GC pauses

• Kernel scheduling

• Cache misses

• Lock contention

At ~70–80% CPU:

• Context switching skyrockets

• Tail latency spikes

• Throughput drops

Systems don’t degrade linearly.

6️⃣ The Knee of the Curve (Critical Insight)

Every system has a knee point:

• Before knee → stable

• After knee → latency explodes

• Slight load increase → collapse

7️⃣ Capacity Planning Mistake #2 — Ignoring Variance

Even if average load is safe:

• P95 requests

• Cache misses

• Cold shards

• Slow disks

…will dominate P99.

Capacity planning must consider worst-case paths, not averages.

8️⃣ Async, Queues, and the Capacity Illusion

❌ “We added async + a queue”

What actually happened:

• Requests stopped blocking

• In-flight count increased

• Memory usage spiked

• Tail latency worsened

9️⃣ The Only Capacity That Matters: Bottlenecks

Your system’s capacity is the minimum of:

• CPU

• Memory

• DB connections

• Disk IOPS

• Network

• External dependencies

You don’t scale a system.
You scale its tightest bottleneck.

1️⃣0️⃣ Practical Capacity Model (Simple & Useful)

For each critical dependency, compute:

Capacity = (Resource Budget) / (Cost per request)

Example:

• DB connections = 100

• Avg query time = 50 ms

Max DB RPS:

100 / 0.05 = 2000 RPS

Now apply:

• Cache miss rate

• Retries

• Fanout

Real capacity is much lower.

1️⃣1️⃣ Why Load Tests Lie

Load tests often:

• Use uniform traffic

• Ignore cache warmup

• Ignore retries

• Ignore long tails

Result:

“It handled 5k RPS in staging!”

1️⃣2️⃣ What Good Capacity Planning Actually Looks Like

✅ Principles

1. Plan for P99 cost, not average

2. Assume partial failure

3. Model fanout

4. Enforce hard limits

5. Fail fast beyond capacity

1️⃣3️⃣ Capacity Without Limits Is Fiction

❌ No limit

app.get("/data", async (req, res) => {
  res.send(await work());
});

✅ Capacity-aware

if (inFlight > MAX) {
  return res.status(503).send("Over capacity");
}

Capacity only exists if it’s enforced.

1️⃣4️⃣ The Dark Truth

Capacity planning is not about preventing overload.
It’s about deciding how you fail.

Fail modes:

• Slow failure ❌

• Cascading failure ❌

• Fast, bounded failure ✅