Worked example: design an API rate limiter

This walkthrough shows one strong design and the trade-offs it makes. It is not the only good answer: where another choice would work as well, we say so. The numbers are assumptions for a fictional company, chosen to make the reasoning concrete.

Pin down what is being limited

Before drawing anything, settle what "limit" means here. Good questions to ask out loud:

• Limited by what? Per API key, as the question says. Not per IP address, since many customers share offices and proxies.
• Over what window? Per minute is the stated example. Ask whether short bursts are acceptable, such as 20 requests in one second from a key whose limit is 100 a minute.
• How strict? Does a key that sends 103 requests in a minute need exactly three refusals, or is "about 100" good enough?
• What does a refusal look like? An HTTP 429 Too Many Requests response with a Retry-After header tells a well-behaved client when to come back.

For this example, assume bursts are fine as long as the average stays within the limit, and "about right" is acceptable. Those two answers shape everything that follows.

Get a rough sense of scale

Assume 20,000 requests a second at peak, spread across 50 API servers, and 100,000 active keys. Each key needs only a few numbers of state, so all the counts together fit comfortably in memory on one machine. The hard part is not storage. It is keeping 50 servers in agreement without slowing every request down.

Choose how to count

Common approaches, each defensible:

• Fixed window: count requests per calendar minute. Simple, but a key can send 100 requests at 12:00:59 and another 100 at 12:01:00.
• Sliding log: keep a timestamp for every request. Exact, but memory grows with traffic.
• Sliding window counter: blend this minute's count with last minute's. Close to exact for little memory.
• Token bucket: each key has a bucket that refills at a steady rate. A request takes a token or is refused.

This design uses a token bucket, because it allows the short bursts we agreed were fine while holding the average to the limit. It stores two numbers per key: tokens left and when the bucket was last refilled. A sliding window counter would meet the same requirements; say which one you chose and why.

Keep the counts in one place

If each API server counted on its own, a key could get its limit 50 times over, once per server. So the counts live in a shared in-memory store, such as Redis, next to the API servers.

The check and the update must happen together. Two servers reading "1 token left" at the same moment would both let a request through. Running the refill-and-take step as one atomic operation in the store avoids that race.

Keep the check off the slow path

The limiter runs as middleware in each API server, before any real work. One round trip to a store in the same data centre is usually well under a millisecond. Each server also caches the limits themselves, refreshed every few seconds, so changing a customer's limit needs no deploy and adds no extra lookup per request.

Decide what happens when the limiter fails

If the counter store goes down, the API can fail open (let requests through) or fail closed (refuse them). Refusing every customer because the limiter broke is worse than briefly not limiting, so this design fails open. As a backstop, each server keeps a rough local limit per key, set well above normal traffic, and the outage raises an alert.

The design

Client ──▶ Load balancer ──▶ API server ──▶ API handlers
                               │  limiter middleware
                               ▼
                         Counter store (token buckets)

Limits database ──▶ cached in each API server, refreshed every few seconds

Trade-offs

• Counts are shared, so every request makes one extra round trip. That is the price of a limit that holds across 50 servers.
• Token buckets allow bursts. A customer whose backend cannot absorb them would need a smaller bucket.
• Failing open means a broken store briefly removes protection. The local backstop limits how bad that can get.

What the grading looks for

The question's criteria name concerns, not components. This design meets each of them in one way: clarifying what is limited and how, choosing a counting approach and explaining it, keeping counts consistent across servers, keeping the check off the slow path, and deciding what happens when the limiter fails. A design that used a sliding window counter, or failed closed with a clear reason, could meet the same criteria.