What Happens When Your Database Goes Down?

The database going down is the failure people plan for least and fear most. It is also the one where the difference between preparation and improvisation is most visible to users.

Without a plan, every request returns a 500 and the product is simply gone. With one, most of it keeps working, writes are held rather than lost, and recovery is a promotion rather than an archaeology project.

#The first question: how do you find out?

Badly implemented health checks are why teams learn about database outages from customers.

// Checks that the process is running. Says nothing about whether it can serve.
app.get('/health', (_req, res) => res.send('ok'));

A TCP connection check is barely better — a database can accept connections while being unable to answer queries, which happens during failover, during a long recovery, and when the connection pool is exhausted but the server is fine.

Check by doing the smallest real thing:

const READY_TIMEOUT_MS = 1_000;

app.get('/health/ready', async (_req, res) => {
  try {
    // A real round trip through the pool, with its own budget.
    await withTimeout(db.query('SELECT 1'), READY_TIMEOUT_MS);
    res.send('ok');
  } catch {
    res.status(503).send('database unavailable');
  }
});

Two consecutive failures is a reasonable trigger — fast enough to react before a user-visible timeout, slow enough not to flap on a single slow query.

#The response, in order

FAILURE ──► DETECTION ──► FALLBACK ──► RECOVERY ──► STABLE
  0.0s         0.9s          1.9s        3.2s         4.4s

  primary      pool check    read-only   replica      buffered
  stops        fails ×2      mode;       promoted     writes
  accepting    circuit       writes                   replayed
  connections  opens         buffered                 idempotently

#1. Open a circuit, immediately

Once the check fails twice, stop trying. Every request that queues behind a dead database consumes a request thread and a pool slot, so a database outage becomes a memory problem in the API within a minute or two.

An open circuit fails in microseconds. That speed is what keeps the process healthy enough to serve everything that does not need the database.

#2. Flip to read-only, do not flip to error

This is the part that decides whether users see "degraded" or "broken", and it depends entirely on work you did before the incident.

The pattern that matters is serving stale on purpose, and saying so:

export async function readReport(id: string) {
  try {
    return { data: await db.reports.find(id), stale: false };
  } catch (err) {
    if (!isUnavailable(err)) throw err;

    // The cache exists for latency on a normal day. On a bad day it is the
    // difference between a degraded product and no product.
    const cached = await redis.get(`report:${id}`);
    if (!cached) throw err;

    return { data: JSON.parse(cached), stale: true, asOf: await redis.get(`report:${id}:at`) };
  }
}

The UI then shows a quiet marker — "as of 14:02" — rather than pretending the number is live. Users tolerate stale data they were told about. They do not tolerate discovering it later.

#3. Buffer writes instead of rejecting them

A rejected write is lost work and, usually, a user who has to retype something. A buffered write is a delay.

// Every buffered write carries a key derived from its content, so replaying
// the buffer after recovery cannot produce duplicates.
await queue.add(
  'write.replay',
  { table: 'comment', payload },
  { jobId: `comment:${payload.clientId}` },
);

return reply.code(202).send({ status: 'pending', id: payload.clientId });

Two conditions make this safe, and both are non-negotiable:

• Idempotency, so replay is not duplication.
• Ordering where it matters. If two buffered writes touch the same row, they must replay in the order they were accepted. A per-entity queue key does this; a global queue does not.

#4. Failover, then verify before trusting

Managed Postgres will promote a replica automatically. Self-managed needs an orchestrator; do not plan to do it by hand at 3am.

Either way, recovery is not "the host answered". Before closing the circuit, confirm the new primary actually accepts a write — a promoted replica still in read-only mode will happily answer SELECT 1 and then reject everything real.

#5. Replay, and watch the replay

Drain the buffer with concurrency limits. A recovered database receiving the entire backlog at full speed is a second outage.

#The unglamorous preparation

None of the above helps if the fundamentals are missing. In rough order of how much they matter:

1. 01A restore you have actually performed. Not "backups are configured" — a real restore into a scratch database, with someone confirming the data is there. Time it, so you know your recovery time objective instead of guessing.
2. 02A replica that is not just for reads. If it exists only as a performance tier, nobody has tested promoting it.
3. 03Connection pooling with sane limits. Connection exhaustion produces the same symptoms as an outage, and is far more common.
4. 04Statement timeouts. One runaway query holding locks can take the database down as effectively as a hardware failure.
5. 05A runbook. Written before the incident, by someone who is not panicking, and readable by whoever is on call.

#What this costs

Honestly: a read-only mode is real work. It means every write path has a defined behaviour when the database is unavailable, every cached read has a staleness story, and someone has decided — in advance — which operations may be optimistically accepted.

That is perhaps a week of engineering on a mature product, and it converts a total outage into a degraded one. Whether that trade is worth it depends on what an hour of downtime costs you, which is a number worth knowing before you need it.

#See it run

Open the Reliability console and inject "Database down". The health grid, incident timeline and console output all follow the sequence described here. The Failure Lab breaks down the same scenario as a written playbook — detection, blast radius, what the user sees, and how recovery proceeds.