Fix JSON::GeneratorError: source sequence is illegal/malformed (JSON::GeneratorError) in Sidekiq

This error occurs when Sidekiq tries to serialize job arguments to JSON but encounters data that cannot be converted, such as binary strings, circular references, or non-UTF-8 encoded text. Ensure all job arguments are JSON-serializable primitive types and convert binary data to Base64 or store it externally before passing to the job.

Reading the Stack Trace

Here's what each line means:

• json (2.7.1) lib/json/common.rb:224:in `generate': Ruby's JSON generator encounters bytes that cannot be encoded as valid JSON/UTF-8.
• sidekiq (7.2.1) lib/sidekiq/client.rb:58:in `normalize_item': Sidekiq normalizes the job payload by serializing it to JSON before pushing to Redis.
• app/controllers/uploads_controller.rb:18:in `create': The controller passes binary file data directly as a job argument.

Common Causes

class FileProcessingJob
  include Sidekiq::Job

  def perform(file_content)
    # Process the file
  end
end

# Controller
FileProcessingJob.perform_async(params[:file].read)  # Binary data

data = some_api_response.force_encoding('ASCII-8BIT')
ProcessingJob.perform_async(data)  # Not valid UTF-8

user = User.find(1)
NotificationJob.perform_async(user)  # Should pass user.id, not the object

The Fix

Store the file using ActiveStorage and pass only the blob ID to the job. The job then retrieves and processes the file. This keeps job arguments small, JSON-serializable, and avoids passing binary data through Redis.

# Controller
def create
  file = params[:file]
  FileProcessingJob.perform_async(file.read)
end

# Controller
def create
  file = params[:file]
  blob = ActiveStorage::Blob.create_and_upload!(
    io: file,
    filename: file.original_filename
  )
  FileProcessingJob.perform_async(blob.id)
end

class FileProcessingJob
  include Sidekiq::Job

  def perform(blob_id)
    blob = ActiveStorage::Blob.find(blob_id)
    blob.open do |tempfile|
      # Process the file from tempfile
    end
  end
end

Testing the Fix

require 'rails_helper'
require 'sidekiq/testing'

RSpec.describe FileProcessingJob do
  before { Sidekiq::Testing.fake! }

  it 'enqueues with a blob ID' do
    blob = ActiveStorage::Blob.create_and_upload!(
      io: StringIO.new('test content'),
      filename: 'test.txt'
    )
    expect {
      FileProcessingJob.perform_async(blob.id)
    }.to change(FileProcessingJob.jobs, :size).by(1)
  end

  it 'processes the file from the blob' do
    Sidekiq::Testing.inline!
    blob = ActiveStorage::Blob.create_and_upload!(
      io: StringIO.new('test content'),
      filename: 'test.txt'
    )
    expect { FileProcessingJob.perform_async(blob.id) }.not_to raise_error
  end
end

Run your tests:

bundle exec rspec spec/jobs/file_processing_job_spec.rb

Pushing Through CI/CD

git checkout -b fix/sidekiq-serialization,git add app/jobs/file_processing_job.rb app/controllers/uploads_controller.rb,git commit -m "fix: store file in ActiveStorage and pass blob ID to Sidekiq job",git push origin fix/sidekiq-serialization

Your CI config should look something like this:

name: CI
on:
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports: ['5432:5432']
      redis:
        image: redis:7
        ports: ['6379:6379']
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.3'
          bundler-cache: true
      - run: bin/rails db:setup
      - run: bundle exec rspec

The Full Manual Process: 18 Steps

Here's every step you just went through to fix this one bug:

1. Notice the error alert or see it in your monitoring tool
2. Open the error dashboard and read the stack trace
3. Identify the file and line number from the stack trace
4. Open your IDE and navigate to the file
5. Read the surrounding code to understand context
6. Reproduce the error locally
7. Identify the root cause
8. Write the fix
9. Run the test suite locally
10. Fix any failing tests
11. Write new tests covering the edge case
12. Run the full test suite again
13. Create a new git branch
14. Commit and push your changes
15. Open a pull request
16. Wait for code review
17. Merge and deploy to production
18. Monitor production to confirm the error is resolved

Total time: 30-60 minutes. For one bug.

gem install bugstack

require 'bugstack'

Bugstack.init(api_key: ENV['BUGSTACK_API_KEY'])

Deploying the Fix (Manual Path)

1. Replace binary arguments with references like IDs or URLs.
2. Store files in ActiveStorage or S3 before enqueuing jobs.
3. Run job specs verifying serialization.
4. Open a pull request.
5. Merge and verify job processing in staging.

Frequently Asked Questions