Fix RangeError: Array buffer allocation failed in Node.js

This error occurs when Node.js tries to allocate a buffer larger than available memory or the V8 heap limit. Common causes include reading huge files into memory at once or concatenating unbounded data. Fix it by using streams for large data, setting appropriate memory limits, and validating input sizes.

Reading the Stack Trace

Here's what each line means:

• at Buffer.allocUnsafe (node:buffer:394:18): Node tried to allocate a buffer large enough to hold the entire file contents but exceeded available memory.
• at readFileSync (node:fs:471:20): fs.readFileSync loads the entire file into memory at once, which fails for very large files.
• at processUpload (src/handlers/upload.js:15:28): Your upload handler on line 15 reads the uploaded file synchronously into a single buffer.

Common Causes

const fs = require('fs');
function processUpload(filePath) {
  const data = fs.readFileSync(filePath); // File could be 4GB+
  return processData(data);
}

let allData = Buffer.alloc(0);
stream.on('data', (chunk) => {
  allData = Buffer.concat([allData, chunk]); // No size limit
});

// Processing large dataset without streaming
const records = JSON.parse(fs.readFileSync('huge-dataset.json', 'utf8'));

The Fix

Check the file size before reading and reject files exceeding a limit. Use streaming with createReadStream and pipeline to process data in chunks instead of loading everything into memory at once.

const fs = require('fs');

function processUpload(filePath) {
  const data = fs.readFileSync(filePath);
  return processData(data);
}

const fs = require('fs');
const { pipeline } = require('stream/promises');
const { Transform } = require('stream');

const MAX_FILE_SIZE = 100 * 1024 * 1024; // 100MB

async function processUpload(filePath) {
  const stats = fs.statSync(filePath);
  if (stats.size > MAX_FILE_SIZE) {
    throw new Error(`File too large: ${stats.size} bytes exceeds ${MAX_FILE_SIZE} byte limit`);
  }

  const readStream = fs.createReadStream(filePath);
  const transform = new Transform({
    transform(chunk, encoding, callback) {
      callback(null, processChunk(chunk));
    },
  });

  await pipeline(readStream, transform, outputStream);
}

Testing the Fix

const fs = require('fs');
const { processUpload } = require('./upload');

describe('processUpload', () => {
  it('rejects files exceeding the size limit', async () => {
    jest.spyOn(fs, 'statSync').mockReturnValue({ size: 200 * 1024 * 1024 });
    await expect(processUpload('/tmp/huge.bin')).rejects.toThrow('File too large');
  });

  it('processes files under the size limit via streaming', async () => {
    jest.spyOn(fs, 'statSync').mockReturnValue({ size: 1024 });
    const mockStream = { pipe: jest.fn().mockReturnThis(), on: jest.fn() };
    jest.spyOn(fs, 'createReadStream').mockReturnValue(mockStream);
    expect(fs.createReadStream).toBeDefined();
  });
});

Run your tests:

npm test

Pushing Through CI/CD

git checkout -b fix/nodejs-buffer-overflow,git add src/handlers/upload.js src/handlers/__tests__/upload.test.js,git commit -m "fix: use streaming for large file uploads to prevent buffer overflow",git push origin fix/nodejs-buffer-overflow

Your CI config should look something like this:

name: CI
on:
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm test -- --coverage
      - run: npm run lint

The Full Manual Process: 18 Steps

Here's every step you just went through to fix this one bug:

1. Notice the error alert or see it in your monitoring tool
2. Open the error dashboard and read the stack trace
3. Identify the file and line number from the stack trace
4. Open your IDE and navigate to the file
5. Read the surrounding code to understand context
6. Reproduce the error locally
7. Identify the root cause
8. Write the fix
9. Run the test suite locally
10. Fix any failing tests
11. Write new tests covering the edge case
12. Run the full test suite again
13. Create a new git branch
14. Commit and push your changes
15. Open a pull request
16. Wait for code review
17. Merge and deploy to production
18. Monitor production to confirm the error is resolved

Total time: 30-60 minutes. For one bug.

npm install bugstack-sdk

const { initBugStack } = require('bugstack-sdk')

initBugStack({ apiKey: process.env.BUGSTACK_API_KEY })

Deploying the Fix (Manual Path)

1. Replace synchronous file reads with streaming for all large file operations.
2. Add file size validation before processing.
3. Run tests locally to confirm the fix.
4. Open a PR and wait for CI checks.
5. Merge and monitor memory usage in staging.

Frequently Asked Questions