--- title: "Bash Retry Function with Exponential Backoff and Jitter" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/bash-retry-function-exponential-backoff --- ![Blog post image for Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff - A reusable Bash function that retries any failing command with configurable attempts and exponential backoff. Ideal for wrapping flaky network calls, AWS CLI commands, or deployment scripts in CI/CD pipelines.](/_astro/hero.Dap62rVN_RyOaK.webp) [Home](/)›[Codesnippets](/codesnippets)›[All Categories](/codesnippets/categories)›[Devops](/codesnippets/categories/devops) Codesnippets [Prev in DevopsAWS EC2 Instance Management with Boto3: Start, Stop, and Query Instances](/codesnippets/post/aws-ec2-instance-management-boto3-python)[Next in DevopsCheck S3 Bucket Existence](/codesnippets/post/bash-s3-bucket-exists) [Devops](/codesnippets/categories/devops)[Shell scripting](/codesnippets/categories/shell-scripting)[Automation](/codesnippets/categories/automation)[Bash](/codesnippets/bash) # Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 20 Jul 202605 Mins read06 Mins listen [Markdown for AI(opens in a new tab)](/post/bash-retry-function-exponential-backoff/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR A reusable Bash function that retries any failing command with configurable attempts and exponential backoff. Ideal for wrapping flaky network calls, AWS CLI commands, or deployment scripts in CI/CD pipelines. Series [Bash Snippets](/series/bash-snippets)1/1 All posts in this series (1) Code Snippets1 1. [Bash Retry Function: Automatically Retry Failing Commands with Exponential BackoffYou are here](/codesnippets/post/bash-retry-function-exponential-backoff) ### Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff Contents [The Problem](#the-problem)[Why blind retries make it worse](#why-blind-retries-make-it-worse)[The real-world impact](#the-real-world-impact)[The Solution](#the-solution)[Script Implementation](#script-implementation)[Setup and defaults](#setup-and-defaults)[The retry function](#the-retry-function)[Why the jitter matters](#why-the-jitter-matters)[Entry point](#entry-point)[Usage and Benefits](#usage-and-benefits)[Real invocations](#real-invocations)[Sourcing it in CI/CD](#sourcing-it-in-cicd)[Comparison](#comparison)[Frequently Asked Questions](#frequently-asked-questions)[References](#references) **Quick Tip** Wrap any flaky command in one reusable `retry` function and stop re-running red pipelines by hand. ## [The Problem](#the-problem) **The Problem** Some commands fail for reasons that have nothing to do with your code. A registry hiccups, an API rate-limits you for a second, a DNS lookup blips, an AWS endpoint returns a `503`. Run the exact same command again and it works. That’s a transient failure, and it’s everywhere in scripts that touch the network. ### [Why blind retries make it worse](#why-blind-retries-make-it-worse) The naive fix is a quick loop that retries immediately, but that often makes things worse. If a service is already struggling, a tight retry loop hammers it harder. Worse, if a hundred CI jobs all fail at the same moment and all retry at the same moment, they come back in a synchronized wave. That’s the thundering herd, and it can keep a recovering service down. ### [The real-world impact](#the-real-world-impact) The everyday cost is a pipeline that goes red for no real reason, so someone re-runs it by hand and loses ten minutes and their focus. The worse cost is a deploy script that gives up on the first blip and leaves a release half-applied. Both come from treating a temporary failure as a permanent one. ## [The Solution](#the-solution) **The Fix** Retry the command a few times, but wait longer between each attempt, and add a little randomness so retries spread out instead of stacking up. That’s exponential backoff with jitter, and it fits in one small Bash function you can wrap around anything. **TL;DR** - Retry any command with a configurable number of attempts. - Back off exponentially (1s, 2s, 4s, 8s) so you stop hammering a struggling service. - Add jitter so a fleet of scripts doesn’t retry in lockstep. ## [Script Implementation](#script-implementation) ### [Setup and defaults](#setup-and-defaults) Start with a safe shell, then read the knobs from the environment so the same function works in a laptop shell and a CI job without editing the code. retry.sh ``` 1#!/usr/bin/env bash2# retry.sh - retry a command with exponential backoff and jitter.3set -euo pipefail4 5# Tunables (override via environment):6: "${RETRY_MAX_ATTEMPTS:=5}" # how many tries before giving up7: "${RETRY_BASE_DELAY:=1}" # first backoff, in seconds8: "${RETRY_MAX_DELAY:=60}" # cap so backoff never runs away ``` ### [The retry function](#the-retry-function) The function takes the command and its arguments as `"$@"`, so it wraps anything without quoting gymnastics. It returns the command’s own exit code on success, and the last exit code if it runs out of attempts. Terminal window ``` 1retry() {2 local attempt=13 while true; do4 # Run the command exactly as passed. On success, we're done.5 if "$@"; then6 return 07 fi8 local exit_code=$?9 10 # Out of attempts: surface the failure clearly and pass the code up.11 if ((attempt >= RETRY_MAX_ATTEMPTS)); then12 echo "retry: '$*' failed after ${attempt} attempts (exit ${exit_code})" >&213 return "$exit_code"14 fi15 16 # Exponential backoff: base * 2^(attempt-1), capped at max delay.17 local delay=$((RETRY_BASE_DELAY * 2 ** (attempt - 1)))18 ((delay > RETRY_MAX_DELAY)) && delay=$RETRY_MAX_DELAY19 20 # Full jitter: sleep a random amount between 0 and delay.21 local wait=$((RANDOM % (delay + 1)))22 echo "retry: attempt ${attempt}/${RETRY_MAX_ATTEMPTS} failed (exit ${exit_code}); waiting ${wait}s" >&223 sleep "$wait"24 ((attempt++))25 done26} ``` ### [Why the jitter matters](#why-the-jitter-matters) The backoff line grows the wait each round, and the cap stops it from turning a fifth attempt into a multi-minute stall. The jitter line is the part people skip, and it’s the one that prevents the thundering herd. Instead of every caller waiting exactly 4 seconds and retrying at the same instant, each one waits a random slice of that window, so the load spreads out and the recovering service gets room to breathe. ### [Entry point](#entry-point) If you run the file directly it retries whatever you pass on the command line. If you source it, you just get the `retry` function. Terminal window ``` 1# Run directly: retry.sh 2if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then3 if (($# == 0)); then4 echo "usage: retry.sh [args...]" >&25 exit 646 fi7 retry "$@"8fi ``` ## [Usage and Benefits](#usage-and-benefits) **Why This Helps** One function covers every flaky command in your scripts, so you stop copy-pasting `sleep` loops and you get consistent logging for free. Because it wraps `"$@"`, it does not care whether the command is `aws`, `curl`, `kubectl`, or a script of your own. ### [Real invocations](#real-invocations) Terminal window ``` 1# Wrap an AWS CLI upload that sometimes hits throttling.2retry aws s3 cp ./build.tar.gz s3://artifacts/build.tar.gz3 4# Wrap a health check that races a service starting up.5retry curl -fsS https://api.example.com/health6 7# Tune it inline for a slow, important step.8RETRY_MAX_ATTEMPTS=8 RETRY_BASE_DELAY=2 retry ./deploy.sh ``` ### [Sourcing it in CI/CD](#sourcing-it-in-cicd) Keep `retry.sh` in a shared scripts library and source it at the top of your pipeline steps, so every job gets the same behavior. Terminal window ``` 1source ./scripts/retry.sh2retry terraform apply -auto-approve ``` ## [Comparison](#comparison) How the function compares to the other common options. Approach Backoff Jitter Works for any command `retry()` function Yes, exponential Yes Yes Tight `until` loop No No Yes `curl --retry N` Yes (curl 7.66+) Only with `--retry-all-errors` extras No, curl only AWS CLI built-in retries Yes (adaptive mode) Yes No, AWS CLI only Tool-specific retries are great when you’re already in that tool, but the function is the one thing that wraps all of them the same way. ## [Frequently Asked Questions](#frequently-asked-questions) A fixed delay treats a one-second blip and a thirty-second outage the same way. Exponential backoff waits a little after the first failure and progressively longer after each one, so you recover fast from a quick hiccup but stop hammering a service that’s genuinely down. The cap (`RETRY_MAX_DELAY`) keeps the later waits from becoming a stall. Jitter is deliberate randomness added to the wait time. Without it, every script that failed at the same moment retries at the same moment, hitting the recovering service in a synchronized wave. With full jitter, each caller sleeps a random amount between zero and the backoff window, so the retries spread out and the load smooths. Wrap the command in a small function that maps the failures you care about to a non-zero exit and everything else to zero. For example, treat a `curl` exit of 22 (HTTP error) or 28 (timeout) as retryable and return success for the rest, then pass that wrapper to `retry`. That keeps the retry loop generic while you decide what “failure” means. It uses Bash features: `RANDOM`, `local`, `((...))` arithmetic with `**`, and `BASH_SOURCE`. In a strict POSIX `sh` you’d swap `RANDOM` for something like `awk` or `/dev/urandom`, replace `local`, and compute the power manually. If you can run `#!/usr/bin/env bash`, keep it as is; it’s simpler and clearer. Those are excellent inside their own tool. `curl --retry` retries HTTP calls; the AWS CLI has adaptive retry modes for API throttling. The Bash function is the layer above: it retries anything, including your own scripts and combinations of commands, with one consistent policy and one consistent log format. Either lower `RETRY_MAX_ATTEMPTS` and `RETRY_MAX_DELAY` so the worst case is bounded, or wrap the call in GNU `timeout`, for example `timeout 120 retry ./deploy.sh`. The `timeout` approach gives you a hard ceiling regardless of how the backoff math works out. ## [References](#references) - [Bash Reference Manual: shell arithmetic](https://www.gnu.org/software/bash/manual/bash.html#Shell-Arithmetic) - [Bash Reference Manual: RANDOM and special parameters](https://www.gnu.org/software/bash/manual/bash.html#Bash-Variables) - [AWS Architecture Blog: exponential backoff and jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) - [AWS CLI: retries](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-retries.html) - [curl manual: —retry](https://curl.se/docs/manpage.html#--retry) - [GNU coreutils: timeout](https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html) Was this useful? ## Tags [#Bash](/codesnippets/tags/bash)[#Retry](/codesnippets/tags/retry)[#Exponential Backoff](/codesnippets/tags/exponential-backoff)[#CI/CD](/codesnippets/tags/cicd)[#Shell](/codesnippets/tags/shell)[#DevOps](/codesnippets/tags/devops) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Bash%20Retry%20Function%3A%20Automatically%20Retry%20Failing%20Commands%20with%20Exponential%20Backoff&url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff&title=Bash%20Retry%20Function%3A%20Automatically%20Retry%20Failing%20Commands%20with%20Exponential%20Backoff&summary=A%20reusable%20Bash%20function%20that%20retries%20any%20failing%20command%20with%20configurable%20attempts%20and%20exponential%20backoff.%20Ideal%20for%20wrapping%20flaky%20network%20calls%2C%20AWS%20CLI%20commands%2C%20or%20deployment%20scripts%20in%20CI%2FCD%20pipelines.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Bash%20Retry%20Function%3A%20Automatically%20Retry%20Failing%20Commands%20with%20Exponential%20Backoff%20https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff&text=Bash%20Retry%20Function%3A%20Automatically%20Retry%20Failing%20Commands%20with%20Exponential%20Backoff "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff&title=Bash%20Retry%20Function%3A%20Automatically%20Retry%20Failing%20Commands%20with%20Exponential%20Backoff "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff&t=Bash%20Retry%20Function%3A%20Automatically%20Retry%20Failing%20Commands%20with%20Exponential%20Backoff "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fbash-retry-function-exponential-backoff&media=&description=A%20reusable%20Bash%20function%20that%20retries%20any%20failing%20command%20with%20configurable%20attempts%20and%20exponential%20backoff.%20Ideal%20for%20wrapping%20flaky%20network%20calls%2C%20AWS%20CLI%20commands%2C%20or%20deployment%20scripts%20in%20CI%2FCD%20pipelines. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Check S3 Bucket Existence](/_astro/hero.CWiO4GWV_1YUYTb.webp)](/codesnippets/post/bash-s3-bucket-exists) ## [Check S3 Bucket Existence](/codesnippets/post/bash-s3-bucket-exists) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Aws](/codesnippets/categories/aws) - [Shell scripting](/codesnippets/categories/shell-scripting) - [Devops](/codesnippets/categories/devops) Quick Tip Don't let your deployment blow up because of a missing S3 bucket. This Bash script lets you check if a bucket exists before anything fails. The Problem Missing bucket failure [#Bash](/codesnippets/tags/bash)[#AWS](/codesnippets/tags/aws)[#DevOps](/codesnippets/tags/devops)+3 tags [read more](/codesnippets/post/bash-s3-bucket-exists) [![Essential Bash Variables for Every Script](/_astro/hero.B4bDowyY_YQpD7.webp)](/codesnippets/post/essential-bash-variables) ## [Essential Bash Variables for Every Script](/codesnippets/post/essential-bash-variables) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Shell scripting](/codesnippets/categories/shell-scripting) - [Devops](/codesnippets/categories/devops) Overview Quick Tip You know what's worse than writing scripts? Writing scripts that break every time you move them to a different machine. Built-in Bash variables fix that. The problem wi [#Bash](/codesnippets/tags/bash)[#Shell Scripting](/codesnippets/tags/shell-scripting)[#Linux](/codesnippets/tags/linux)+4 tags [read more](/codesnippets/post/essential-bash-variables) [![AWS EC2 Instance Management with Boto3: Start, Stop, and Query Instances](/_astro/hero.PnHlvJay_ZkFqWG.webp)](/codesnippets/post/aws-ec2-instance-management-boto3-python) ## [AWS EC2 Instance Management with Boto3: Start, Stop, and Query Instances](/codesnippets/post/aws-ec2-instance-management-boto3-python) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud](/codesnippets/categories/cloud) - [Aws](/codesnippets/categories/aws) - [Devops](/codesnippets/categories/devops) - [Automation](/codesnippets/categories/automation) If you've ever spent 20 minutes clicking through the AWS Console just to stop a handful of dev instances, you already know the pain. It's tedious, it doesn't scale, and one wrong click can ruin your a [#AWS](/codesnippets/tags/aws)[#EC2](/codesnippets/tags/ec2)[#Boto3](/codesnippets/tags/boto3)+6 tags [read more](/codesnippets/post/aws-ec2-instance-management-boto3-python) [![Bash Script Locking: Prevent Concurrent Runs with a PID File](/_astro/hero.CF-paSMK_vzFHW.webp)](/codesnippets/post/bash-script-locking-pid-file) ## [Bash Script Locking: Prevent Concurrent Runs with a PID File](/codesnippets/post/bash-script-locking-pid-file) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Bash](/codesnippets/categories/bash) - [Automation](/codesnippets/categories/automation) - [Devops](/codesnippets/categories/devops) Quick Tip Wrap any script that must not run twice at once in a PID-file lock, and a second copy simply exits instead of corrupting your data. The Problem The Problem Cron does not c [#Bash](/codesnippets/tags/bash)[#Pid file](/codesnippets/tags/pid-file)[#Locking](/codesnippets/tags/locking)+3 tags [read more](/codesnippets/post/bash-script-locking-pid-file) [![Why printf Beats echo in Linux Scripts](/_astro/hero.Dl3YkIwZ_Z1w9hux.webp)](/codesnippets/post/printf-beats-echo-linux-scripts) ## [Why printf Beats echo in Linux Scripts](/codesnippets/post/printf-beats-echo-linux-scripts) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Shell scripting](/codesnippets/categories/shell-scripting) - [Devops](/codesnippets/categories/devops) - [Linux](/codesnippets/categories/linux) Scripting Tip A script that works on your machine can produce different output on another system. The output command is often the reason. printf behaves the same way across shells, and echo d [#Bash](/codesnippets/tags/bash)[#Shell Scripting](/codesnippets/tags/shell-scripting)[#Printf](/codesnippets/tags/printf)+5 tags [read more](/codesnippets/post/printf-beats-echo-linux-scripts) [![Multi-Environment Secret Management with HashiCorp Vault](/_astro/hero.B4H3tk7I_1BuPOM.webp)](/codesnippets/post/hashicorp-vault-multi-environment-secrets) ## [Multi-Environment Secret Management with HashiCorp Vault](/codesnippets/post/hashicorp-vault-multi-environment-secrets) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Security](/codesnippets/categories/security) - [Devops](/codesnippets/categories/devops) Managing secrets safely across multiple environments with HashiCorp Vault Storing secrets in .env files, hardcoding them, or even using separate secret managers per environment creates security [#Vault](/codesnippets/tags/vault)[#SecretsManagement](/codesnippets/tags/secretsmanagement)[#MultiEnvironment](/codesnippets/tags/multienvironment)+3 tags [read more](/codesnippets/post/hashicorp-vault-multi-environment-secrets) 6 related posts