Fix Sidekiq::DeadSet::Error: Job moved to dead set after exhausting all retries in Sidekiq

This error indicates a job was moved to Sidekiq's Dead Set after exhausting all retry attempts. Dead jobs remain in Redis for six months by default. Investigate the original error, fix the root cause, then re-enqueue the dead jobs using the Sidekiq Web UI or the DeadSet API programmatically.

Reading the Stack Trace

Here's what each line means:

• sidekiq (7.2.1) lib/sidekiq/job_retry.rb:185:in `send_to_morgue': Sidekiq moves the failed job to the dead set (morgue) after all retries are exhausted.
• sidekiq (7.2.1) lib/sidekiq/job_retry.rb:145:in `retries_exhausted': The retry handler determines the job has exceeded its maximum retry count.
• app/jobs/payment_processing_job.rb:18:in `perform': The payment processing job failed on its final retry attempt.

Common Causes

class PaymentProcessingJob
  include Sidekiq::Job

  def perform(payment_id)
    payment = Payment.find(payment_id)
    gateway = PaymentGateway.new
    gateway.charge(payment.amount, payment.card_token)
    # Gateway has been down for 21 days (entire retry window)
  end
end

class InvoiceJob
  include Sidekiq::Job

  def perform(invoice_id)
    invoice = Invoice.find(invoice_id)  # Record was deleted
    InvoiceMailer.send(invoice).deliver_now
  end
end

class S3UploadJob
  include Sidekiq::Job

  def perform(file_path)
    Aws::S3::Client.new(region: 'wrong-region')
      .put_object(bucket: 'my-bucket', key: file_path)
    # Wrong region causes auth failure every time
  end
end

The Fix

Add a sidekiq_retries_exhausted callback that marks the payment as failed and sends a notification. Handle non-retryable errors like card declines immediately without retrying. This ensures users and admins are informed when payments fail permanently.

class PaymentProcessingJob
  include Sidekiq::Job

  def perform(payment_id)
    payment = Payment.find(payment_id)
    PaymentGateway.new.charge(payment.amount, payment.card_token)
  end
end

class PaymentProcessingJob
  include Sidekiq::Job
  sidekiq_options retry: 15, dead: true

  sidekiq_retries_exhausted do |job, exception|
    payment = Payment.find_by(id: job['args'].first)
    if payment
      payment.update!(status: 'failed', failure_reason: exception.message)
      PaymentFailureNotifier.notify(payment, exception)
    end
    Rails.logger.error("Payment job #{job['jid']} dead: #{exception.message}")
  end

  def perform(payment_id)
    payment = Payment.find(payment_id)
    result = PaymentGateway.new.charge(payment.amount, payment.card_token)
    payment.update!(status: 'completed', transaction_id: result.id)
  rescue PaymentGateway::CardDeclined => e
    payment.update!(status: 'declined', failure_reason: e.message)
    # Do not retry on declined cards
  end
end

Testing the Fix

require 'rails_helper'
require 'sidekiq/testing'

RSpec.describe PaymentProcessingJob do
  before { Sidekiq::Testing.inline! }

  let(:payment) { create(:payment, status: 'pending') }

  it 'marks payment as completed on success' do
    allow(PaymentGateway).to receive_message_chain(:new, :charge)
      .and_return(OpenStruct.new(id: 'txn_123'))
    PaymentProcessingJob.perform_async(payment.id)
    expect(payment.reload.status).to eq('completed')
  end

  it 'marks payment as declined without retrying' do
    allow(PaymentGateway).to receive_message_chain(:new, :charge)
      .and_raise(PaymentGateway::CardDeclined)
    expect {
      PaymentProcessingJob.perform_async(payment.id)
    }.not_to raise_error
    expect(payment.reload.status).to eq('declined')
  end
end

Run your tests:

bundle exec rspec spec/jobs/payment_processing_job_spec.rb

Pushing Through CI/CD

git checkout -b fix/sidekiq-dead-job-handling,git add app/jobs/payment_processing_job.rb,git commit -m "fix: add retry exhaustion handler for payment job dead jobs",git push origin fix/sidekiq-dead-job-handling

Your CI config should look something like this:

name: CI
on:
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports: ['5432:5432']
      redis:
        image: redis:7
        ports: ['6379:6379']
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.3'
          bundler-cache: true
      - run: bin/rails db:setup
      - run: bundle exec rspec

The Full Manual Process: 18 Steps

Here's every step you just went through to fix this one bug:

1. Notice the error alert or see it in your monitoring tool
2. Open the error dashboard and read the stack trace
3. Identify the file and line number from the stack trace
4. Open your IDE and navigate to the file
5. Read the surrounding code to understand context
6. Reproduce the error locally
7. Identify the root cause
8. Write the fix
9. Run the test suite locally
10. Fix any failing tests
11. Write new tests covering the edge case
12. Run the full test suite again
13. Create a new git branch
14. Commit and push your changes
15. Open a pull request
16. Wait for code review
17. Merge and deploy to production
18. Monitor production to confirm the error is resolved

Total time: 30-60 minutes. For one bug.

gem install bugstack

require 'bugstack'

Bugstack.init(api_key: ENV['BUGSTACK_API_KEY'])

Deploying the Fix (Manual Path)

1. Add sidekiq_retries_exhausted callbacks to critical jobs.
2. Handle non-retryable errors explicitly.
3. Set up monitoring for the dead job queue.
4. Open a pull request.
5. Merge and verify dead job handling in staging.

Frequently Asked Questions