Exponential Backoff and Retries for HTTP Clients

Retry only errors that can change on their own (rate limits, 503s, timeouts). Wait with capped exponential backoff plus jitter, let a Retry-After header override your own delay, and stop after a fixed budget. Exponential backoff means the wait doubles after every failed attempt (1 s, 2 s, 4 s...) up to a maximum, so a struggling server gets room to recover instead of a faster stream of requests. The rest of this post covers which errors to retry, the jitter variants, and working code for Python, Node and Go.

Here is the shortest useful version, a Python requests retry config that backs off, adds jitter and respects the header:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=4,                                    # 4 retries = 5 tries
    status_forcelist=[408, 429, 500, 502, 503, 504],
    backoff_factor=0.5,                         # waits 0, 1, 2, 4... s
    backoff_max=30,                             # cap
    backoff_jitter=0.5,                         # adds up to 0.5 s of random noise
    raise_on_status=False,                      # hand back the last response
)
s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=retry))
r = s.get("https://api.example.com/data", timeout=10)

Last updated: October 25, 2026. I ran every client below against a scripted server I wrote, bound to 127.0.0.1 (Python 3.11.2, requests 2.34.2, urllib3 2.8.0, tenacity 9.2.1, Node v26.12.1, axios 1.20.0, axios-retry 4.5.0, Go 1.26.5, macOS). I also ran my own jitter simulation against a local limiter. I did not test any third-party API, so nothing here describes how a real service behaves.

What Exponential Backoff Is and Why Plain Retries Hurt

The formula is short:

delay = min(cap, base * 2^attempt)

With a base of 1 second and a cap of 30 seconds, attempts 0 to 6 wait 1, 2, 4, 8, 16, 30 and 30 seconds. The cap matters. Without it the fifth retry waits 16 seconds and the tenth waits about 8.5 minutes.

Why not retry at once? A server that returns errors is usually busy or broken. If 500 clients retry instantly, the server gets 500 more requests in the next second, which is the worst moment for it. Backoff spreads those retries over time. Jitter (random noise on the delay) stops all clients from retrying at the same instant, and I measure that below.

Backoff and a circuit breaker solve different problems. Backoff slows one client down between attempts. A circuit breaker stops sending requests altogether after N failures in a row, then probes again after a cool-down. For most scripts, backoff plus a retry budget is enough. Add a breaker when many workers share a flaky dependency.

Which HTTP Errors Are Worth Retrying

Retry a status when the same request could succeed a moment later. Do not retry when the answer is a decision.

I sent 400, 403, 404 and 407 through all six clients. Every one sent the request exactly once, except one: Node's built-in fetch (v26.12.1) does not return a 407 at all. It throws TypeError: fetch failed with no useful cause, and my helper treats thrown errors as network errors, so it retried the 407 five times. If your code retries on every exception, a 407 will burn your whole budget. Check the status first where you can.

Network failures follow the same rule. Connection resets, timeouts and DNS hiccups are worth retrying. A TLS certificate error is not: the next attempt sees the same certificate.

One more caveat on 500 and timeouts. A timeout does not tell you whether the server did the work. RFC 9110 says idempotent methods can be repeated automatically after a communication failure (section 9.2.2). GET, PUT and DELETE qualify. POST does not, unless you send an idempotency key (more on that below).

Not every error should be retried. A 407 means the proxy wants credentials, so waiting will not fix it; see 407 Proxy Authentication Required.

Honor the Retry-After Header

When a server tells you how long to wait, believe it. Retry-After holds either a number of seconds or an HTTP-date, and RFC 9110 allows both (section 10.2.3):

Retry-After: 120
Retry-After: Fri, 31 Dec 1999 23:59:59 GMT

My rule: the server hint replaces the computed delay, still capped by a sane maximum, with a little extra jitter on top so a thousand clients don't wake up on the same millisecond. The same rule applies whichever status carries the Retry-After header, a rate limit or a 503, and you can read more about the rate limit status itself in the glossary entry.

Two edge cases need code. A date can be in the past (clock skew), so clamp the result to 0. And a missing or garbled header means you fall back to your own backoff. I added a case with Retry-After: soon for that: five of the six clients fell back and retried, and urllib3 2.8.0 raised InvalidHeader and did not retry at all.

I scripted three variants on the local server (two failures, then a 200) and measured the gaps between arrivals as the server saw them:

With Retry-After: 1, every client waited between 1.0 and 1.2 seconds, both times. With an HTTP-date, waits fell between 1.1 and 2.3 seconds (the date has one-second resolution, so my server's value was 1 to 2 seconds ahead). With a date in the past, nobody waited for it: the hint clamped to 0, and the clients retried after 0 to 240 ms. Without any header, the clients waited between 0 and about 260 ms, because I scaled the delays down (base 50 ms, cap 1 s) to keep the test fast.

Two library details came out of this run:

• urllib3 2.8.0 with Retry-After present sleeps exactly the header value and adds no backoff on top (1010 ms and 1008 ms for a 1 s hint). It reads the header only on 413, 429 and 503, per its changelog. Without the header, the first retry has almost no delay (4 ms and 127 ms in my run). The docs say so: backoff applies "after the second try". With backoff_factor=0.5 the schedule was 0, 1, 2, 4, 8, 16, 32 seconds after failures 1 to 7.
• axios-retry 4.5.0 honors the header, and its README says it waits for "the biggest value between the Retry-After header and the configured retryDelay".

Exponential Backoff With Jitter: Full, Equal, Decorrelated

The AWS Architecture Blog post Exponential Backoff And Jitter describes three variants, and I implemented them like this:

full:         sleep = random(0, min(cap, base * 2^attempt))
equal:        t = min(cap, base * 2^attempt); sleep = t/2 + random(0, t/2)
decorrelated: sleep = min(cap, random(base, previous_sleep * 3))

The post (read October 25, 2026) reports that full and equal jitter make about the same number of calls, decorrelated makes more, and equal jitter comes out weakest of the three. Its numbers come from a simulator of optimistic locking on a database, not an HTTP limiter, so I ran my own.

The setup. A real local server (127.0.0.1) with a fixed-window limiter: 10 requests per 200 ms window, 429 above that, no Retry-After. N clients start together on one persistent connection each. Base 50 ms, cap 1.6 s, up to 10 retries (11 tries) per client, then the client gives up. Five seeds (1 to 5), results are medians. A fixed window is a simplification: real limiters are often sliding windows or token buckets. Delays are scaled down, so the "busiest 50 ms bucket" depends on that scale and is only useful as a relative number.

Results for 200 clients (the fastest possible finish is 4.0 s):

And for 50 clients (fastest possible finish 1.0 s):

What I take from it, and no more:

• With 200 clients, unjittered backoff kept the clients synchronized. They hit the limiter in waves, and 90 of 200 ran out of retries. All three jittered variants cut rejected requests by roughly 25% to 33% and spread the late retries out (the busiest bucket dropped from 169 to between 51 and 93).
• With 50 clients, jitter did not help. Plain backoff finished at about the same time with fewer requests than full jitter. A small crowd does not need much spreading.
• Decorrelated sent the fewest requests but 7 clients hit the retry cap, which says more about my cap of 10 retries than about the formula. A longer budget might change that, and I did not test one.
• The ranking of equal, full and decorrelated is specific to this limiter, this window and these seeds. I would not claim any of them is better in general. Full jitter is what I use by default because the formula is one line.

This is one synthetic limiter. It shows that jitter spreads load. It does not show how any real API behaves.

Retry Budgets, Idempotency and Crawler Pipelines

Backoff alone does not stop a loop. Three limits do:

1. Attempt cap. Five tries is a fine default. My "503 forever" case stopped after exactly 5 tries in all six clients and returned the last 503.
2. Total time. Cap the whole job, not only each request. A timeout of 30 s with 5 tries and a 60 s hint can block a worker for minutes.
3. A per-job budget. In a nightly refresh of a knowledge base over thousands of URLs, I would give each job a retry budget (say 5% of its requests) and stop retrying when it is spent. Surface the error and move on. Ten thousand URLs times five tries is a lot of load on one host.

Idempotency decides whether a retry is safe. A GET is. A POST that creates something is not, because a timed-out request may have succeeded. The common convention is an Idempotency-Key header (a working-group draft, draft-ietf-httpapi-idempotency-key-header, not a finished standard, and the server has to support it). My helpers retry POST and PATCH only when the request carries that header. In my POST case every retry carried the same key and the same 10-byte body, and all six clients delivered it intact. Without the key, all six sent the POST once and returned the 503. urllib3's Retry skips POST by default: with the default allowed_methods, my 503 came back after the first try, where the retried version reached 200 on the third.

For crawlers, lower the concurrency per host before adding retries. Retries stack on top of normal traffic, so 50 parallel workers retrying a slow host make it slower. Log the attempt count with every result, so the cost of retrying is visible. And separate "try again later" from "this URL is blocked". A 403 will not clear with waiting, and the fixes are in the 403 Forbidden post. Rotating proxies is a different tool from backoff, covered in proxies for web scraping.

Working Retry Code in Python, Node and Go

All snippets share the same shape: retry the statuses from the table, cap the delay, stop after 5 tries. Each one is a small retry with backoff loop or config. The tests ran with scaled-down delays (base 50 ms, cap 1 s) and the snippets show production-sized values. Per-try timeouts are in the code because a hung connection is also a failure.

Python requests retry: urllib3 Retry, tenacity, or a hand-rolled function

The built-in option is the urllib3.util.Retry config from the top of this post, mounted on a requests session. Retry-After handling is on by default (respect_retry_after_header=True). Three details from my run of urllib3 2.8.0:

• backoff_jitter (added in urllib3 2.0.0, per the changelog) adds up to that many random seconds on top of the delay. It is not full jitter.
• The default allowed_methods leaves out POST. To retry it, add allowed_methods=Retry.DEFAULT_ALLOWED_METHODS | {"POST"}, and only for requests that carry an Idempotency-Key.
• A garbled Retry-After value raised InvalidHeader in my test instead of falling back to backoff.

With tenacity (9.2.1) the exponential-with-jitter piece is wait_random_exponential. Retry-After needs a small custom wait function, because tenacity does not read headers by itself:

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
import requests
from tenacity import Retrying, retry_if_exception_type, retry_if_result, stop_after_attempt, wait_random_exponential

RETRY_STATUS = {408, 429, 500, 502, 503, 504}
fallback = wait_random_exponential(multiplier=0.5, max=30)   # full jitter, capped

def retry_after(resp):
    v = resp.headers.get("Retry-After")
    if v is None:
        return None
    if v.strip().isdigit():
        return float(v)
    try:
        delta = (parsedate_to_datetime(v) - datetime.now(timezone.utc)).total_seconds()
    except (TypeError, ValueError):
        return None                                          # garbled header: use computed backoff
    return max(0.0, delta)                                   # a past date means 0

def wait(state):
    if not state.outcome.failed:
        hint = retry_after(state.outcome.result())
        if hint is not None:
            return min(hint, 60)                             # the server hint wins, but is capped
    return fallback(state)

retrying = Retrying(
    retry=retry_if_result(lambda r: r.status_code in RETRY_STATUS)
          | retry_if_exception_type((requests.ConnectionError, requests.Timeout)),
    wait=wait,
    stop=stop_after_attempt(5),
    retry_error_callback=lambda s: s.outcome.result(),       # return the last response
)
r = retrying(requests.get, "https://api.example.com/data", timeout=10)

If you want no dependency, the hand-rolled version is about 15 lines. This is the core of my tested file (it reuses retry_after from above):

import random, time

IDEMPOTENT = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE", "TRACE"}

def request_with_retry(method, url, *, tries=5, base=0.5, cap=30.0, **kw):
    # POST and PATCH are retried only when the request carries an Idempotency-Key
    if method.upper() not in IDEMPOTENT and "Idempotency-Key" not in kw.get("headers", {}):
        tries = 1
    for attempt in range(tries):
        try:
            resp = requests.request(method, url, timeout=10, **kw)
            if resp.status_code not in RETRY_STATUS or attempt == tries - 1:
                return resp
            hint = retry_after(resp)
        except (requests.ConnectionError, requests.Timeout):
            if attempt == tries - 1:
                raise
            hint = None
        delay = random.uniform(0, min(cap, base * 2 ** attempt))   # full jitter
        if hint is not None:
            delay = min(60, hint) + random.uniform(0, 0.1 * hint)  # hint wins, plus a little spread
        time.sleep(delay)

All three Python versions passed the same cases (the garbled-header case aside for urllib3): retried statuses reached 200 in 3 tries, the timeout case in 2, and 400, 403, 404 and 407 were sent once.

Node 18+: a fetch helper and axios retry (axios-retry)

Native fetch has no retry, so the helper is small. AbortSignal.timeout handles hung requests, and the body is drained before a retry so the connection can be reused:

const RETRY_STATUS = new Set([408, 429, 500, 502, 503, 504]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

function retryAfterMs(res) {
  const v = res.headers.get("retry-after");
  if (v === null) return null;
  if (/^\d+$/.test(v.trim())) return Number(v) * 1000;
  const t = Date.parse(v); // HTTP-date
  return Number.isNaN(t) ? null : Math.max(0, t - Date.now()); // past date = 0
}

const IDEMPOTENT = new Set(["GET", "HEAD", "OPTIONS", "PUT", "DELETE", "TRACE"]);

export async function fetchWithRetry(url, init = {}, { tries = 5, baseMs = 500, capMs = 30000, timeoutMs = 10000 } = {}) {
  // POST and PATCH are retried only when the request carries an Idempotency-Key
  const method = (init.method ?? "GET").toUpperCase();
  if (!IDEMPOTENT.has(method) && !new Headers(init.headers).has("idempotency-key")) tries = 1;
  for (let attempt = 0; attempt < tries; attempt++) {
    const last = attempt === tries - 1;
    let hint = null;
    try {
      const res = await fetch(url, { ...init, signal: AbortSignal.timeout(timeoutMs) });
      if (!RETRY_STATUS.has(res.status) || last) return res;
      hint = retryAfterMs(res);
      await res.arrayBuffer(); // drain the body
    } catch (err) {
      if (last) throw err; // network error or timeout
    }
    let delay = Math.random() * Math.min(capMs, baseMs * 2 ** attempt); // full jitter
    if (hint !== null) delay = Math.min(60000, hint) + Math.random() * 0.1 * hint;
    await sleep(delay);
  }
}

For axios-retry (4.5.0), a few defaults surprised me. The default condition retries network errors and 5xx on idempotent methods only. So a 429 is not retried, a 503 on a POST is not retried, and a timeout (ECONNABORTED) is not treated as a network error. In my first run each of these failed for that reason: 1 try and a 429 where I expected 3. Also, the request timeout is global across retries unless shouldResetTimeout is set, so a 1 s Retry-After outlasted my 400 ms timeout and the retry was skipped. This config passed all 17 cases:

import axios from "axios";
import axiosRetry, { isNetworkOrIdempotentRequestError } from "axios-retry";

const client = axios.create({ timeout: 10000 });
axiosRetry(client, {
  retries: 4,
  retryDelay: (n, err) => axiosRetry.exponentialDelay(n, err, 500), // 2^n * 500 ms + up to 20% jitter
  shouldResetTimeout: true,                                         // per-try timeout
  // POST is retried only when the request carries an Idempotency-Key
  retryCondition: (err) => {
    const method = (err.config?.method ?? "get").toUpperCase();
    const safe = ["GET", "HEAD", "OPTIONS", "PUT", "DELETE"].includes(method) ||
      Boolean(err.config?.headers?.["Idempotency-Key"]);
    return safe && (
      isNetworkOrIdempotentRequestError(err) ||
      err.code === "ECONNABORTED" ||
      [429, 503].includes(err.response?.status));
  },
});

Its exponentialDelay is 2^n times the factor plus up to 20% noise, which is closer to "backoff with a bit of jitter" than to full jitter. With a factor of 50 ms I measured waits of 114, 205, 431 and 829 ms in one run.

Go: net/http with a small helper

Go's standard library has no retry loop either. The part that bites is the request body: once a body is read, it is gone. http.NewRequestWithContext fills GetBody for *bytes.Buffer, *bytes.Reader and *strings.Reader bodies (net/http docs, read October 25, 2026), so each attempt can ask for a fresh copy. The other rule is to drain and close the response body before a retry, or the connection is not reused:

func doWithRetry(client *http.Client, req *http.Request, tries int, base, capD, perTry time.Duration) (*http.Response, error) {
	var lastErr error
	if req.Method != "GET" && req.Method != "HEAD" && req.Method != "PUT" && req.Method != "DELETE" &&
		req.Header.Get("Idempotency-Key") == "" {
		tries = 1 // POST and PATCH are retried only with an Idempotency-Key
	}
	for attempt := 0; attempt < tries; attempt++ {
		last := attempt == tries-1
		ctx, cancel := context.WithTimeout(req.Context(), perTry)
		r := req.Clone(ctx)
		if req.GetBody != nil { // replay the body for POST/PUT
			body, _ := req.GetBody()
			r.Body = body
		}
		resp, err := client.Do(r)
		var hint time.Duration
		hasHint := false
		if err == nil {
			if !retryStatus[resp.StatusCode] || last {
				resp.Body = &cancelOnClose{resp.Body, cancel}
				return resp, nil
			}
			hint, hasHint = retryAfter(resp.Header)
			hint = min(hint, 60*time.Second) // cap the server hint
			io.Copy(io.Discard, resp.Body) // drain, then close
			resp.Body.Close()
		} else if lastErr = err; last {
			cancel()
			return nil, err
		}
		cancel()
		// full jitter: random between 0 and min(cap, base * 2^attempt)
		delay := time.Duration(rand.Float64() * float64(min(capD, base<<attempt)))
		if hasHint { // the server hint wins, plus up to 10% spread
			delay = hint + time.Duration(rand.Float64()*0.1*float64(hint))
		}
		time.Sleep(delay)
	}
	return nil, lastErr
}

The full file (retryAfter, retryStatus and the small cancelOnClose wrapper) is about 125 lines, and it uses math/rand/v2, so there is no seeding. In the POST test every retry arrived with the same 10-byte body. I did not test third-party Go libraries such as go-retryablehttp.

What WebCrawlerAPI Does for You

I run WebCrawlerAPI, so this is the vendor view. The web scraping API page says the page "is rendered in a browser on our side, with retries, proxies and anti-bot handling, then converted and returned in the same response". Results differ by target site, so test on your own URLs.

The same page says "Retries, proxies and anti-bot handling run on our side." Retries in your own code for calls to the API are still your job, so the rules above apply there. The limits on parallel scraping per account and per website are listed in the rate limits docs, and I do not restate them here. For bigger batches, POST /v2/scrape with async=true returns a job ID immediately and you fetch the result later with a GET (see async requests), so the client does not have to wait for the response.

FAQ

What is exponential backoff?

A retry strategy where the wait doubles after each failed attempt, usually up to a cap. It gives an overloaded server time to recover.

Why retry with exponential backoff?

Immediate retries add load at the moment a server is struggling. Waiting longer each time reduces that load, and jitter stops many clients from retrying together.

What does exponential backoff with jitter mean?

Random noise is added to each delay. In full jitter, the wait is a random value between 0 and the capped exponential delay. In my 200-client simulation, jitter cut rejected requests by roughly a quarter to a third compared with plain backoff. With 50 clients, it did not help.

What is the formula for exponential backoff?

delay = min(cap, base * 2^attempt). With full jitter: random(0, min(cap, base * 2^attempt)).

How do I implement exponential backoff in Python?

Mount urllib3.util.Retry on a requests session, use tenacity's wait_random_exponential, or write the 15-line loop above. See the Python section.

Should I retry a rate limit response, and how long should I wait?

Yes, if the request is safe to repeat. Use the Retry-After value when present. Without it, start at about 1 second, double each time, add jitter, and give up after about 5 tries.

Exponential backoff or a circuit breaker?

They combine. Backoff paces one client's retries, and a circuit breaker stops requests across workers after repeated failures.