Fix WorkerLostError: celery.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) Job: 42 in Celery

This error occurs when a Celery worker process is killed by the OS, usually because it exceeded memory limits (OOM killer) or hit the hard time limit. The task cannot catch this because SIGKILL terminates the process immediately. Fix it by reducing memory usage in tasks, increasing container memory limits, or setting task_acks_late so tasks are redelivered to a healthy worker.

Reading the Stack Trace

Here's what each line means:

• WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL) Job: 42'): Signal 9 (SIGKILL) indicates the OS forcefully terminated the worker — likely due to memory exhaustion or the hard time limit.
• File "/app/venv/lib/python3.12/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost: The billiard process pool detects the worker child process has exited and marks the task as lost.
• File "/app/venv/lib/python3.12/site-packages/celery/concurrency/prefork.py", line 83, in on_hard_timeout: The prefork pool's timeout handler fired, indicating the task exceeded the hard time limit.

Common Causes

@celery.task
def process_images(image_paths):
    images = [Image.open(p) for p in image_paths]  # loads all images into memory
    for img in images:
        process(img)

# docker-compose.yml
worker:
  mem_limit: 256m  # too small for image processing

@celery.task(time_limit=30)  # hard kill after 30 seconds
def long_running_task():
    process_large_dataset()  # takes 5+ minutes

The Fix

Process images one at a time using a context manager to release memory after each one. Set worker_max_memory_per_child to automatically restart workers before they accumulate too much memory. Adjust time limits to realistic values and always set soft_time_limit below time_limit to allow graceful cleanup.

@celery.task
def process_images(image_paths):
    images = [Image.open(p) for p in image_paths]
    for img in images:
        process(img)

@celery.task(bind=True, time_limit=600, soft_time_limit=540)
def process_images(self, image_paths):
    for path in image_paths:
        with Image.open(path) as img:
            process(img)
        # Process one at a time, releasing memory between iterations

# Also set: worker_max_memory_per_child = 200000  # restart worker after 200 MB

Testing the Fix

import pytest
import tempfile
from PIL import Image
from app.tasks import process_images
from app import create_app

@pytest.fixture
def app():
    app = create_app()
    app.config['CELERY_ALWAYS_EAGER'] = True
    app.config['TESTING'] = True
    return app

def test_process_single_image(app, tmp_path):
    img_path = tmp_path / 'test.png'
    Image.new('RGB', (100, 100)).save(img_path)
    with app.app_context():
        result = process_images([str(img_path)])
        assert result is not None

def test_process_empty_list(app):
    with app.app_context():
        result = process_images([])
        assert result is not None

def test_process_multiple_images(app, tmp_path):
    paths = []
    for i in range(5):
        p = tmp_path / f'test_{i}.png'
        Image.new('RGB', (100, 100)).save(p)
        paths.append(str(p))
    with app.app_context():
        result = process_images(paths)

Run your tests:

pytest tests/ -v

Pushing Through CI/CD

git checkout -b fix/celery-worker-memory,git add app/tasks.py celeryconfig.py,git commit -m "fix: process images sequentially and set memory limits for workers",git push origin fix/celery-worker-memory

Your CI config should look something like this:

name: CI
on:
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis:7-alpine
        ports:
          - 6379:6379
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: 'pip'
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v --tb=short
        env:
          CELERY_BROKER_URL: redis://localhost:6379/0

The Full Manual Process: 18 Steps

Here's every step you just went through to fix this one bug:

1. Notice the error alert or see it in your monitoring tool
2. Open the error dashboard and read the stack trace
3. Identify the file and line number from the stack trace
4. Open your IDE and navigate to the file
5. Read the surrounding code to understand context
6. Reproduce the error locally
7. Identify the root cause
8. Write the fix
9. Run the test suite locally
10. Fix any failing tests
11. Write new tests covering the edge case
12. Run the full test suite again
13. Create a new git branch
14. Commit and push your changes
15. Open a pull request
16. Wait for code review
17. Merge and deploy to production
18. Monitor production to confirm the error is resolved

Total time: 30-60 minutes. For one bug.

pip install bugstack

import bugstack

bugstack.init(api_key=os.environ["BUGSTACK_API_KEY"])

Deploying the Fix (Manual Path)

1. Run pytest locally to verify images are processed without memory spikes.
2. Open a pull request with the memory optimization and config changes.
3. Wait for CI checks to pass on the PR.
4. Have a teammate review and approve the PR.
5. Merge to main and monitor worker memory usage in staging.

Frequently Asked Questions