📘 Load Is Not Traffic

Why “10k RPS” tells you almost nothing

Systems don’t fail because of traffic.
They fail because of work.

1️⃣ The Most Dangerous Metric in System Design

RPS (Requests Per Second)

It’s easy to measure.
It’s easy to graph.
It’s deeply misleading.

Two systems at 1k RPS can differ by 10× or 100× in actual load.

2️⃣ Traffic vs Load

Traffic

• Number of incoming requests

• Measured in RPS

• External view

Load

• CPU time

• Memory pressure

• DB queries

• Network I/O

• Lock contention

Load is internal. Traffic is external.

3️⃣ Why This Distinction Matters

You can have:

• Low traffic

• High load

Or:

• High traffic

• Low load

Only load determines failure.

4️⃣ Example 1 — Same Traffic, Different Load

Two endpoints

Endpoint	CPU Cost
/ping	0.2 ms
/feed	50 ms

At 1,000 RPS:

/ping  → 0.2 CPU-seconds/sec
/feed  → 50 CPU-seconds/sec

One melts the system.
One barely registers.

5️⃣ Why Autoscaling Fails Here

Autoscaling reacts to traffic spikes or CPU usage, not work distribution.

If traffic shifts from /ping to /feed:

• RPS unchanged

• CPU explodes

• Autoscaling reacts too late

6️⃣ Load Is Multi-Dimensional

Load is not one thing.

Resource	Load Example
CPU	JSON serialization
Memory	Large in-flight buffers
DB	Cache misses
Network	Large responses
Locks	Contended mutex

You overload the weakest dimension first.

7️⃣ Example 2 — DB Load Hiding Behind Low Traffic

app.get("/stats", async (req, res) => {
  const users = await db.query("SELECT * FROM users");
  res.send(users);
});

Traffic:

10 RPS

DB Load:

Full table scan × 10/sec

System dies quietly.

8️⃣ The Real Unit of Load: Work Units

Senior systems measure load in:

• CPU-milliseconds

• DB queries/sec

• Rows scanned/sec

• Bytes moved/sec

Not RPS.

9️⃣ Capacity Planning Without Load Awareness Fails

❌ Naive capacity plan

“We handle 5k RPS.”

✅ Real capacity plan

“We handle 200 CPU-ms/sec per core, with 5 DB queries per request.”

1️⃣0️⃣ Load Is Variable Per Request

Even the same endpoint varies:

• Cache hit vs miss

• Hot shard vs cold shard

• Small vs large payload

Planning for averages guarantees tail failures.

1️⃣1️⃣ Code Smell — Treating Traffic as Load ❌

if (rps > 5000) {
  reject();
}

This protects nothing.

1️⃣2️⃣ Correct Pattern — Load-Based Admission Control

let cpuBudget = 8000; // ms/sec
let used = 0;

app.get("/data", async (req, res) => {
  const cost = estimateCost(req);

  if (used + cost > cpuBudget) {
    return res.status(503).send("Overloaded");
  }

  used += cost;
  try {
    res.send(await work());
  } finally {
    used -= cost;
  }
});

Control work, not requests.

1️⃣3️⃣ Why Tail Latency Depends on Load, Not Traffic

• Low traffic + heavy requests → high P99

• High traffic + light requests → low P99

Tail latency tracks resource contention, not request count.