Skip to content

Safety and resilience

This page covers how redress behaves under failure and how to configure it for production.

Hook failure isolation

on_metric, on_log, on_attempt_start, and on_attempt_end are best-effort. Exceptions derived from Exception are swallowed at the hook boundary, so observer failures do not change operation results or retry decisions. This also means hook failures are silent unless you handle them. Cancellation, KeyboardInterrupt, and SystemExit still propagate.

Attempt hook isolation applies to sync and async execution, call() and execute(), and policies without a retry component. A failing attempt-end hook cannot cause a successful operation to run again. See Attempt lifecycle hooks for the full contract.

Recommended handling:

  • Keep hooks fast and side-effect light; avoid blocking I/O or retries inside the hook.
  • If a hook calls a networked backend, set short timeouts and consider queueing/buffering.
  • Wrap hooks and log failures so you can detect when observability breaks.

Example wrapper:

def safe_metric_hook(hook, logger):
    def _hook(event, attempt, sleep_s, tags):
        try:
            hook(event, attempt, sleep_s, tags)
        except Exception as exc:
            logger.warning("redress_metric_hook_failed", error=str(exc))
    return _hook

If you need hook failures to affect control flow, enforce that outside of hooks; hooks are intentionally isolated.

Backoff jitter and thundering herds

When many workers fail at once, fixed or purely exponential delays can line up retries and create a thundering herd. Jitter spreads attempts over time so upstreams can recover.

Guidance:

  • Prefer jittered strategies like decorrelated_jitter for defaults.
  • Use larger base_s and max_s for RATE_LIMIT to widen the retry window.
  • Keep max_s high enough to spread bursts; too-small caps still synchronize retries.
  • Use per-class strategies and caps (per_class_max_attempts, max_attempts) to limit coordinated retries.
  • For high fan-out callers, add concurrency limits (queues, semaphores) in addition to jitter.

See Retry strategies for details on built-ins.

Production checklist

  • Timeouts: set per-attempt timeouts shorter than deadline_s so there is room for retries.
  • Deadline: tune deadline_s to your upstream SLA; it bounds retry decisions and sleeps but does not interrupt an in-flight operation. See Performance tuning for timeout limitations.
  • Max attempts: set max_attempts and max_unknown_attempts to prevent runaway loops.
  • Per-class caps: use per_class_max_attempts for noisy classes like RATE_LIMIT/SERVER_ERROR.
  • Budgets: use Budget(max_retries, window_s) to cap aggregate retry volume (backpressure).
  • Backpressure: start with max_retries near peak concurrency and a window_s that matches your upstream recovery window; alert on BUDGET_EXHAUSTED.
  • Backoff: use jittered strategies with max_s sized to the recovery window.
  • Sleeper hooks: if you need leases/locks or external schedulers, use before_sleep or sleeper.
  • Adaptive backoff: use adaptive() only when you want retry volume to react to failure spikes.
  • Observability: set operation, attach on_metric or on_log, keep tags low-cardinality.
  • Key metrics: watch retry, success, deadline_exceeded, max_attempts_exceeded, max_unknown_attempts_exceeded, permanent_fail, and distributions of sleep_s.

See Observability for hook patterns and alerting ideas.

Use Performance tuning to measure latency and retry load, or Troubleshooting to diagnose unexpected behavior.