How to Handle Third-Party API Failures

Your payment provider will have an incident. Your email service will rate-limit you. Your model provider will time out. None of this is hypothetical — it is the ordinary operating condition of software that talks to other software.

The question is not whether these happen. It is whether they take your product down with them.

#Every call needs four decisions

Before you write the integration, answer these. Writing them down takes ten minutes and saves the argument during the incident.

1. 01How long do I wait? A timeout, in milliseconds, chosen deliberately.
2. 02Do I retry? Only if the operation is safe to repeat.
3. 03When do I stop trying? A circuit breaker threshold.
4. 04What do I do instead? The defined degraded behaviour.

Most integrations answer none of these, which means the answers become: forever, yes indefinitely, never, and crash.

#1. Timeouts, because the default is "forever"

Node's HTTP client has no default timeout. Neither does fetch without an AbortSignal. A hung connection to a service that stopped responding — but never closed the socket — holds a request thread, a connection pool slot, and whatever memory the request context carries.

// Two different clocks. A slow-but-progressing streaming response should not
// be killed by the same budget as a connection that never answered at all.
const CONNECT_MS = 2_000;
const TOTAL_MS = 8_000;

export async function call(url: string, init: RequestInit = {}) {
  const signal = AbortSignal.timeout(TOTAL_MS);
  const res = await fetch(url, { ...init, signal });

  if (!res.ok) throw new UpstreamError(res.status, url);
  return res;
}

Set the timeout from the dependency's actual behaviour, not from a round number. If a provider's p99 is 900ms, a 2-second timeout catches genuine failure and tolerates normal variance. A 30-second timeout catches nothing and holds resources for half a minute.

#2. Retry only what is safe

Retrying is not universally correct. It depends entirely on whether repeating the operation is harmless.

Almost every serious provider supports idempotency keys. Use them. They are what converts "unsafe to retry" into "safe to retry", and they are the difference between a timeout being an inconvenience and being a double charge.

// The key is derived from the thing being paid for, so a retry — even from a
// different process, even minutes later — is recognised as the same charge.
await payments.charge(
  { amount, currency, customer },
  { idempotencyKey: `order:${orderId}:charge` },
);

And when you do retry: exponential backoff with jitter. Without jitter, every client that failed at the same moment retries at the same moment, and the recovering service is knocked over by the recovery.

async function withRetry<T>(fn: () => Promise<T>, attempts = 4): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      // 4xx means you sent something wrong. Retrying will not fix it.
      if (attempt >= attempts - 1 || !isRetryable(err)) throw err;

      const base = 2 ** attempt * 250;
      await sleep(base * (0.5 + Math.random()));   // jittered
    }
  }
}

Retrying a 400 is a bug wearing a resilience costume. Retry on 429, 5xx, and network errors — and honour Retry-After when the provider sends one, because it is better information than your formula.

#3. Circuit breakers stop you making it worse

Retries help with a blip. They actively harm during a sustained outage: you pile load onto a service that is already failing, and you burn your own capacity waiting for calls that will not succeed.

A circuit breaker tracks the failure rate and, past a threshold, stops calling entirely for a cooling-off period.

CLOSED ──── failures exceed threshold ────► OPEN
  ▲                                          │
  │                                          │ after cooldown
  │                                          ▼
  └──────── probe succeeds ──────────── HALF_OPEN
                                             │
                                             │ probe fails
                                             ▼
                                           OPEN

Three states, and the middle one is the important one. HALF_OPEN lets exactly one request through to test the water. If it succeeds, the circuit closes and normal traffic resumes. If it fails, the cooldown restarts. Without a half-open state you are either hammering a dead service or guessing when to resume.

The failure it prevents is subtle: when the circuit is open, your calls fail in microseconds instead of after an eight-second timeout. That is what keeps your own request threads free and your own service responsive while somebody else's is not.

#4. Define what "degraded" means, per dependency

This is the part that gets skipped, and it is the part your users experience.

Notice they are all different. "Retry three times then 500" is not a strategy — it is the absence of one. The right behaviour depends entirely on whether the dependency is load-bearing for the operation the user is performing.

The general shape for non-load-bearing work is: accept the action, mark it pending, queue the outbound call, and show the pending state honestly.

// The user's action succeeds. The side effect becomes the queue's problem.
await notifications.create({ id, status: 'pending' });
await queue.add('notify.send', { id }, {
  jobId: `notify:${id}`,
  attempts: 6,
  backoff: { type: 'exponential', delay: 1_000 },
});

return { id, status: 'pending' };

#Make the failure visible before a customer reports it

Instrument every outbound dependency with the same four signals — success rate, latency percentiles, timeout count, breaker state — and alert on breaker transitions. A circuit opening is the earliest unambiguous signal that a dependency is in trouble, and it is far more actionable than a general error-rate alert.

Log the correlation identifier the provider returns. When you eventually open a support ticket, "your request req_8f2a… failed at 14:02 UTC" gets a real answer. "Your API seems slow" does not.

#The uncomfortable part

All of this is work you do for a failure that has not happened yet, on someone else's schedule, and it is the first thing cut when a deadline gets close.

The trade is straightforward though: a timeout and a fallback take an afternoon. An outage caused by a dependency you had no plan for takes a day, plus the customer conversations afterwards, plus the trust.

The Failure Lab walks through the third-party failure scenario end to end — detection, fallback, recovery — and the Reliability console lets you inject it and watch the system respond.