How to Scale a Node.js API from 100 to 1M Requests

Most "how to scale Node.js" advice is a list of technologies. That is the wrong shape for the problem. Scaling is not a stack — it is a sequence of bottlenecks, each of which becomes the limiting factor only after the previous one is gone.

This article walks the sequence: what actually breaks at each tier, the specific change that removes it, and what that change costs. The Scale Lab on this site is the interactive version of the same progression.

#The rule that matters more than any of the tiers

Build for the load you have plus roughly one order of magnitude. Not three.

Architecture designed for a million users on day one has a predictable failure mode: it runs out of money before user one hundred. Distributed systems are not free. Every component added is a thing to deploy, monitor, patch, pay for, and reason about at 2am.

The corollary is equally important: leave the seams in place. Making a service stateless costs almost nothing on day one and is what makes tier two possible at all. Coupling your session storage to process memory costs nothing on day one either — and costs a refactor later.

#Tier 1 — around 100 users: do the boring thing

One API process, one managed database, a static frontend on a CDN.

Client
  ↓
API  (single instance)
  ↓
Database  (managed, daily backup)

No cache. No queue. No load balancer. The temptation to add them is strong and should be resisted, because at this tier nothing is a bottleneck yet.

Two things are worth doing early, because they are cheap now and expensive later:

• Test a restore. Not "configure backups" — actually restore one into a scratch database and confirm the data is there. An untested backup is a belief, not a capability.
• Keep the process stateless. No in-memory sessions, no local file uploads, no per-process caches holding authoritative data. This single discipline is what makes the next tier a config change rather than a rewrite.

#Tier 2 — around 1,000 users: stop being a single point of failure

The bottleneck here is not capacity. It is that a single process is simultaneously your capacity ceiling and a restart-shaped outage. Every deploy is downtime.

Three changes, in this order:

1. 01Move session state to Redis. Do this first. Adding a second instance before this produces users who get logged out every other request, which looks like a mysterious auth bug.
2. 02Add a second instance behind a load balancer with a real health check — one that verifies the database connection, not one that returns 200 OK from a handler that can never fail.
3. 03Cache hot reads with explicit TTLs.

A health check that does not check anything is worse than no health check, because the balancer will confidently route traffic to a process that cannot serve it:

// Useless: this returns 200 from a process whose database is gone.
app.get('/health', (_req, res) => res.send('ok'));

// Useful: liveness and readiness are different questions.
app.get('/health/live', (_req, res) => res.send('ok')); // am I running?

app.get('/health/ready', async (_req, res) => {         // can I serve?
  try {
    await Promise.all([
      db.query('SELECT 1'),
      redis.ping(),
    ]);
    res.send('ok');
  } catch {
    res.status(503).send('not ready');
  }
});

What it costs: cache invalidation now exists as a category of bug. Every cached value needs an owner, a TTL, and an answer to "what happens if this is thirty seconds stale". Answer that per value, at the point you add the cache — not during an incident.

#Tier 3 — around 10,000 users: get slow work off the request path

Now the bottleneck moves. Requests are being held open by work that the user is not waiting for: sending an email, generating a report, calling a payment provider, embedding a document. Each of those holds a request thread and, worse, a database connection.

Connection exhaustion arrives long before CPU exhaustion. A Node process with a pool of ten connections and a three-second third-party call has a hard ceiling of roughly three such requests per second, no matter how much CPU is idle.

BEFORE                          AFTER

POST /orders                    POST /orders
  ↓                               ↓
create order    (30ms)          create order      (30ms)
send email      (900ms)         enqueue email      (2ms)
call provider  (2400ms)         enqueue provider   (2ms)
generate PDF   (1800ms)         enqueue PDF        (2ms)
  ↓                               ↓
respond        (5130ms)         respond           (36ms)

The response no longer describes completed work — it describes accepted work. That is a product decision as much as a technical one, and the UI has to reflect it: a pending state the user can see, not a spinner that lies.

// The request does only what the user is actually waiting for.
const order = await orders.create(input);

await queue.addBulk([
  { name: 'email.confirmation', data: { orderId: order.id }, opts: { jobId: `email:${order.id}` } },
  { name: 'payment.capture',    data: { orderId: order.id }, opts: { jobId: `pay:${order.id}` } },
  { name: 'invoice.render',     data: { orderId: order.id }, opts: { jobId: `pdf:${order.id}` } },
]);

return reply.code(202).send({ id: order.id, status: 'pending' });

Note the explicit jobId on every job. Queues deliver at least once, which means every consumer will eventually process the same message twice — after a worker crash, a network blip, or a retry. Keying the job by something derived from its content makes the duplicate a no-op instead of a second charge on someone's card.

Also at this tier: a read replica for reporting queries, and a connection pooler in front of the primary.

What it costs: eventual consistency becomes visible to users. Replica lag and job delay are now things your product has to express. "Read your own writes" stops being free — a confirmation screen immediately after a write must read from the primary.

#Tier 4 — around 100,000 users: isolate failure domains

One deployable means one failure domain. A slow AI endpoint, a heavy export, and a login all share an event loop, a connection pool, and a deploy. A single expensive feature can degrade the entire product.

The split that works is by load shape, not by org chart:

Priority lanes in the queue matter as much as the service split. A single FIFO queue means a ten-thousand-row import sits in front of every password-reset email behind it.

#Tier 5 — 1M+: shard, log, and shed

At this point a single database primary is a hard ceiling on write throughput and a single blast radius.

• Shard by tenant. Routing lives in a layer the services never see.
• An event log becomes the integration backbone. Services publish facts instead of calling each other synchronously, which removes the tight coupling that makes a partial outage total.
• Analytics moves to a columnar store. Nobody runs reports against the transactional database.
• Load shedding becomes a tested feature, not an emergency measure. Under saturation, the system should get slower for unimportant work and stay fast for important work.

// Under saturation, refuse cheaply and tell the client when to come back.
// Event-loop lag is the honest saturation signal for Node — CPU% is not,
// because a blocked loop can look idle.
const LAG_BUDGET_MS = 120;

export function shed(priority: 'low' | 'normal' | 'high') {
  return (req: Request, res: Response, next: NextFunction) => {
    if (priority === 'low' && loopLag() > LAG_BUDGET_MS) {
      res.setHeader('Retry-After', '5');
      return res.status(429).json({ error: 'shedding_load' });
    }
    next();
  };
}

Refusing a request in two milliseconds is a service. Accepting it and timing out after thirty seconds is a denial of service you performed on yourself.

#What actually breaks, in the order it breaks

If you take one thing from this article, take the ordering — it is far more transferable than any specific technology:

1. 01The process restarts → you need more than one, which needs statelessness.
2. 02The database is doing repeated identical work → you need a cache, which needs an invalidation story.
3. 03Requests are held open by work nobody is waiting for → you need a queue, which needs idempotency.
4. 04One slow feature degrades everything → you need isolation, which needs service boundaries.
5. 05One primary cannot take the writes → you need sharding, which needs a routing layer.

Each step is a response to evidence. If you cannot name the measurement that justifies the next tier, you are not ready for it.

#Where to go next

The Scale Lab lets you step through these tiers interactively and see what each one adds. If you want to see what happens when a component in one of these topologies fails rather than saturates, the Reliability console simulates that directly.