Inject the Latency You're Afraid Of

Almost everything we build assumes the network is fast and reliable, because on our machines it is. Then it ships to a place where a link is congested, a peer is across a continent, or a switch is quietly dropping one packet in a hundred — and behavior we never tested kicks in. Timeouts fire, retries stampede, a “brief” call blocks a request thread for thirty seconds. The fix isn’t to hope it degrades gracefully. It’s to cause the bad network yourself, in a lab, and watch what happens before production runs the experiment for you.

You already have a latency machine

You don’t need a fancy chaos platform for this. The Linux kernel’s traffic-control subsystem (tc, with the netem discipline) will add artificial delay, jitter, packet loss, duplication, and reordering to an interface’s traffic with a couple of commands. Add 250ms of delay to everything headed for a subnet. Add 500ms with a bit of jitter so it’s not a clean constant. Drop 2% of packets. Then run your system through its normal motions and see what it does when the network stops being polite.

The cheapest way to find out how your code handles a slow network is to make the network slow. It’s a couple of commands, and you can take it right back off.

It’s reversible — remove the rule and the interface is normal again — which is what makes it safe to experiment with. It’s also targeted: you can scope the impairment to a specific destination or port instead of nuking the whole box, so you degrade exactly the path you care about.

The bugs that only wake up under latency

A fast network hides an entire category of design decisions you didn’t know you’d made. Inject some delay and they come out:

• Timeouts you never tuned. The default was fine at sub-millisecond latency and is a disaster at 300ms. Now you find out what your timeout actually is, and whether it’s the same everywhere it should be.
• Retries with no backoff. A call gets slow, the client gives up and retries, the retries pile onto an already-struggling peer, and a slowdown becomes an outage. Latency injection is how you catch a retry storm on a bench instead of in an incident.
• Serial calls you assumed were free. Ten sequential requests at 1ms each is nothing; at 200ms each it’s two seconds of a user staring at a spinner. Latency turns “chatty” from a style nitpick into a visible failure.
• Failovers that never trigger — or trigger too eagerly. Loss and delay are how you actually exercise the health-check-and-fail-over path, instead of trusting it works because it did the one time you pulled a cable.

None of those show up in a unit test or a fast-network staging run. They only appear when the packets are slow, and the only reliable way to make the packets slow on demand is to make them slow on purpose.

Injecting the fault beats extrapolating from a clean number

The alternative to injection is guessing — “it’s fine at 1ms, so it’s probably fine at 200ms.” That extrapolation is exactly the move I distrust in stop extrapolating, build the rehearsal: systems don’t degrade linearly, and the interesting behavior is nonlinear and emergent. Latency injection is the rehearsal for the network dimension. Instead of assuming graceful degradation, you produce the adverse condition and observe the real response, which is usually humbler and more useful than the assumption.

And it pairs with load, it doesn’t replace it. A throughput benchmark tells you how fast the happy path goes; injected latency tells you how the whole thing behaves when the path stops being happy. Real confidence comes from running both — fast-and-loaded, and slow-and-lossy — because production will eventually serve you both.

Do it deliberately, and clean up after

A few guardrails, because you’re modifying a live box’s networking:

• Do it in a lab, not production. The point is to break things safely; a real environment is the wrong place to find out your retry logic is a stampede.
• Scope it. Impair the specific destination or port under test, not all traffic — or you’ll lock yourself out of the box you’re testing.
• Remove it when you’re done. An injected rule left behind is a self-inflicted gremlin that’ll waste someone’s afternoon later. Tear it down as deliberately as you set it up.
• Change one impairment at a time. Delay, then loss, then jitter — so you can attribute the behavior you see to the condition you added, same as any clean experiment.

The mindset shift is small and worth a lot: stop treating a bad network as an unlucky thing that happens to your system, and start treating it as an input you can supply. The network will be slow and lossy someday regardless. The only choice is whether you meet that behavior for the first time on a bench, where you can fix it, or in production, where it fixes you. If you’ve built latency injection into your own testing and found something ugly hiding behind a fast link, I’d like to hear what it was.