Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating one PDF is easy. Generating ten thousand on a deadline is an entirely different engineering problem. Here are the patterns that actually hold up in production.

pdfbatch processingnode.jsqueuesperformance

Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating a single PDF is a solved problem. You pick a library, feed it some HTML or a template, and out comes a file. But the moment your product needs to send 50,000 monthly statements at midnight on the first of the month, or export 200,000 shipping labels before a warehouse cutoff, things break in interesting ways: memory spikes, timeouts, rate limits, partial failures, and the dreaded "it worked in staging" surprise.

This post walks through the patterns that hold up when you need to render thousands or millions of PDFs reliably.

Start by separating the request from the work

The first mistake is generating PDFs inside the HTTP request that triggered them. Synchronous batch generation will eat your worker pool, time out behind load balancers, and leave you with no way to retry.

Decouple immediately. Treat batch generation as a job pipeline:

txt
[API request] → [enqueue job] → [worker pool] → [storage] → [notify user]

A minimal version with a job queue like BullMQ:

ts
import { Queue } from "bullmq";

const pdfQueue = new Queue("pdf-render", {
  connection: { host: "redis", port: 6379 },
});

app.post("/exports/statements", async (req, res) => {
  const { customerIds, month } = req.body;

  const jobs = customerIds.map((id) => ({
    name: "render-statement",
    data: { customerId: id, month },
    opts: { attempts: 3, backoff: { type: "exponential", delay: 2000 } },
  }));

  await pdfQueue.addBulk(jobs);
  res.status(202).json({ enqueued: jobs.length });
});

The API returns in milliseconds. Workers render in the background. Each job retries independently.

Fan out, then fan in

For very large batches, a single "render this batch" job is fragile — if it crashes at item 8,000 of 10,000, you've lost everything in between. Instead, fan out at the individual document level.

A common pattern:

  1. A planner job enumerates the work and enqueues one render job per document.
  2. Render workers process each document in parallel.
  3. A coordinator tracks completion (counter in Redis, row in a batches table) and triggers a final step — zipping, uploading, emailing — when the count hits zero.
ts
async function onRenderComplete(batchId: string) {
  const remaining = await redis.decr(`batch:${batchId}:pending`);
  if (remaining === 0) {
    await finalizeQueue.add("finalize", { batchId });
  }
}

This pattern survives crashes, scales horizontally, and gives you per-document retry semantics.

Cap concurrency, or you'll DoS yourself

Workers will happily render PDFs as fast as the queue feeds them. That's exactly the problem. Headless Chromium instances are heavy — usually 100–300MB each — and rendering pipelines tend to share resources (database, template store, asset CDN, fonts).

Set concurrency per worker explicitly:

ts
import { Worker } from "bullmq";

new Worker(
  "pdf-render",
  async (job) => renderPdf(job.data),
  {
    connection,
    concurrency: 4, // tune based on memory, not CPU
  }
);

A good starting heuristic: concurrency = floor(available_memory_mb / 400). Then measure under real load and adjust. CPU is rarely the bottleneck; memory and I/O usually are.

Reuse the renderer

Spinning up a fresh Puppeteer or Playwright instance per document is the single biggest performance trap in self-hosted setups. Browser startup easily costs 300–800ms — more than the actual render.

Keep one browser per worker process and create fresh pages per job:

ts
import puppeteer, { Browser } from "puppeteer";

let browser: Browser | null = null;

async function getBrowser() {
  if (!browser) {
    browser = await puppeteer.launch({ args: ["--no-sandbox"] });
  }
  return browser;
}

export async function renderPdf(data: unknown) {
  const b = await getBrowser();
  const page = await b.newPage();
  try {
    await page.setContent(renderTemplate(data), { waitUntil: "networkidle0" });
    return await page.pdf({ format: "A4", printBackground: true });
  } finally {
    await page.close();
  }
}

Recycle the browser every N jobs to dodge memory leaks. Numbers between 500 and 2,000 are typical.

Stream output to object storage

Holding thousands of PDF buffers in memory while you wait to zip them is how OOM kills happen. Stream each rendered PDF straight to S3 (or R2, GCS, etc.) with a deterministic key:

ts
await s3.send(new PutObjectCommand({
  Bucket: "exports",
  Key: `batches/${batchId}/${customerId}.pdf`,
  Body: pdfBuffer,
  ContentType: "application/pdf",
}));

When the batch finalizes, generate a manifest or zip on demand using S3 multipart uploads — no PDF ever needs to live in the coordinator's memory.

Make jobs idempotent

Retries will happen. Network blips, deploys, OOM kills — your job will run twice eventually. Use a deterministic key for storage so duplicate renders overwrite rather than duplicate:

ts
const key = `batches/${batchId}/${documentId}.pdf`;

And skip work that's already done:

ts
const exists = await s3HeadObject(key).catch(() => null);
if (exists) return { skipped: true };

When self-