Fix Sidekiq::RetryExhausted: Job retries exhausted after 25 attempts in Sidekiq

This error occurs when a Sidekiq job has failed and been retried the maximum number of times without succeeding. After exhausting retries, the job moves to the Dead Job queue. Investigate the root cause of the failure, fix the underlying issue, and re-enqueue the job from the Dead queue or programmatically using the Sidekiq API.

Reading the Stack Trace

Here's what each line means:

• sidekiq (7.2.1) lib/sidekiq/job_retry.rb:145:in `retries_exhausted': Sidekiq's retry handler determines the job has exceeded its maximum retry count.
• app/jobs/email_notification_job.rb:12:in `perform': The email notification job at line 12 has been failing consistently across all retry attempts.
• sidekiq (7.2.1) lib/sidekiq/processor.rb:160:in `execute_job': The processor executes the job which fails again on the final retry attempt.

Common Causes

class EmailNotificationJob
  include Sidekiq::Job

  def perform(user_id, template)
    user = User.find(user_id)
    ExternalEmailService.send(to: user.email, template: template)
    # ExternalEmailService has been returning 503 for days
  end
end

class OrderProcessingJob
  include Sidekiq::Job

  def perform(order_id)
    order = Order.find(order_id)  # Raises RecordNotFound if deleted
    order.process!
  end
end

class PaymentJob
  include Sidekiq::Job

  def perform(payment_id)
    payment = Payment.find(payment_id)
    gateway_response = PaymentGateway.charge(payment)
    # Does not handle declined cards or network errors
    payment.update!(status: 'completed')
  end
end

The Fix

Add a sidekiq_retries_exhausted callback to handle the case when all retries fail. Use find_by to gracefully handle deleted records. After a threshold of failures, fall back to an alternative email delivery method instead of continuing to retry.

class EmailNotificationJob
  include Sidekiq::Job

  def perform(user_id, template)
    user = User.find(user_id)
    ExternalEmailService.send(to: user.email, template: template)
  end
end

class EmailNotificationJob
  include Sidekiq::Job
  sidekiq_options retry: 10

  sidekiq_retries_exhausted do |job, exception|
    Rails.logger.error("Email job #{job['jid']} exhausted retries: #{exception.message}")
    FailedJobNotifier.alert(job, exception)
  end

  def perform(user_id, template)
    user = User.find_by(id: user_id)
    return if user.nil?

    ExternalEmailService.send(to: user.email, template: template)
  rescue ExternalEmailService::ServiceUnavailable => e
    raise if executions < 5
    FallbackMailer.send_email(user, template).deliver_later
  end
end

Testing the Fix

require 'rails_helper'
require 'sidekiq/testing'

RSpec.describe EmailNotificationJob do
  before { Sidekiq::Testing.inline! }

  it 'sends email for existing user' do
    user = create(:user)
    expect(ExternalEmailService).to receive(:send)
    EmailNotificationJob.perform_async(user.id, 'welcome')
  end

  it 'handles deleted user gracefully' do
    expect {
      EmailNotificationJob.perform_async(999999, 'welcome')
    }.not_to raise_error
  end

  it 'falls back on persistent failure' do
    user = create(:user)
    allow(ExternalEmailService).to receive(:send).and_raise(
      ExternalEmailService::ServiceUnavailable
    )
    expect(FallbackMailer).to receive_message_chain(:send_email, :deliver_later)
    job = EmailNotificationJob.new
    allow(job).to receive(:executions).and_return(5)
    job.perform(user.id, 'welcome')
  end
end

Run your tests:

bundle exec rspec spec/jobs/email_notification_job_spec.rb

Pushing Through CI/CD

git checkout -b fix/sidekiq-retry-exhausted,git add app/jobs/email_notification_job.rb,git commit -m "fix: add retry exhaustion handler and fallback for email jobs",git push origin fix/sidekiq-retry-exhausted

Your CI config should look something like this:

name: CI
on:
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports: ['5432:5432']
      redis:
        image: redis:7
        ports: ['6379:6379']
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.3'
          bundler-cache: true
      - run: bin/rails db:setup
      - run: bundle exec rspec

The Full Manual Process: 18 Steps

Here's every step you just went through to fix this one bug:

1. Notice the error alert or see it in your monitoring tool
2. Open the error dashboard and read the stack trace
3. Identify the file and line number from the stack trace
4. Open your IDE and navigate to the file
5. Read the surrounding code to understand context
6. Reproduce the error locally
7. Identify the root cause
8. Write the fix
9. Run the test suite locally
10. Fix any failing tests
11. Write new tests covering the edge case
12. Run the full test suite again
13. Create a new git branch
14. Commit and push your changes
15. Open a pull request
16. Wait for code review
17. Merge and deploy to production
18. Monitor production to confirm the error is resolved

Total time: 30-60 minutes. For one bug.

gem install bugstack

require 'bugstack'

Bugstack.init(api_key: ENV['BUGSTACK_API_KEY'])

Deploying the Fix (Manual Path)

1. Add sidekiq_retries_exhausted callback for alerting.
2. Handle edge cases like deleted records.
3. Add fallback strategies for critical jobs.
4. Open a pull request.
5. Merge and monitor the dead job queue in staging.

Frequently Asked Questions