Batch PDF generation: patterns for rendering thousands of documents

Generating a single PDF is a solved problem. You pick a library, pass it some HTML or a template, and get bytes back. The trouble starts when the product team says "we need to email every customer their monthly statement" and suddenly you need to render 80,000 documents in a tight window without taking down the rest of your system.

This post walks through the patterns that hold up at scale: concurrency control, chunking, retries, storage, and a few traps worth avoiding.

Why naive batch generation breaks

The obvious implementation looks like this:

ts
for (const user of users) {
  const pdf = await renderStatement(user);
  await s3.upload(pdf);
}

It works until it doesn't. Three problems show up fast:

1. Serial throughput. If each render takes 400ms, 50,000 documents takes over five hours.
2. Unbounded concurrency. The fix most people reach for — Promise.all(users.map(render)) — opens 50,000 concurrent renders, exhausts memory, and OOM-kills the process.
3. No recovery. If job 47,000 fails, you either restart from zero or build ad-hoc tracking.

You need a pipeline, not a loop.

Pattern 1: bounded concurrency with a worker pool

The simplest meaningful improvement is a concurrency limiter. p-limit is a one-liner, but the idea generalizes:

ts
import pLimit from "p-limit";

const limit = pLimit(10); // tune per environment

await Promise.all(
  users.map((user) =>
    limit(async () => {
      const pdf = await renderStatement(user);
      await s3.upload(`statements/${user.id}.pdf`, pdf);
    })
  )
);

Pick the concurrency number empirically. CPU-bound renderers (headless Chromium, wkhtmltopdf) usually want roughly one worker per core. API-based renderers can go much higher because the CPU work happens elsewhere.

Pattern 2: chunk the input, stream the output

Loading 80,000 user records into memory at once is wasteful. Stream them from the database in chunks:

ts
async function* loadUsersInBatches(batchSize = 500) {
  let cursor = 0;
  while (true) {
    const rows = await db.users.findMany({
      where: { id: { gt: cursor } },
      orderBy: { id: "asc" },
      take: batchSize,
    });
    if (rows.length === 0) return;
    yield rows;
    cursor = rows[rows.length - 1].id;
  }
}

for await (const batch of loadUsersInBatches()) {
  await Promise.all(batch.map((u) => limit(() => renderAndUpload(u))));
}

Keyset pagination (not OFFSET) matters here. On a 10M-row table, OFFSET 500000 gets painful.

Pattern 3: use a queue, not a long-running process

For anything over a few thousand documents, push the work onto a durable queue. BullMQ, SQS, Cloud Tasks — pick your poison. Each job becomes one document (or a small chunk):

ts
import { Queue } from "bullmq";

const queue = new Queue("statements");

for await (const batch of loadUsersInBatches()) {
  await queue.addBulk(
    batch.map((user) => ({
      name: "render-statement",
      data: { userId: user.id, period: "2025-01" },
      opts: { attempts: 3, backoff: { type: "exponential", delay: 2000 } },
    }))
  );
}

The worker handles retries, dead-letter queues, and horizontal scaling for free. Spin up more worker containers when the queue is deep, scale down when it drains.

Pattern 4: idempotency keys

Retries will happen. Make them safe. An idempotency key per document means a retried job doesn't produce a duplicate PDF or re-send an email:

ts
async function renderAndUpload({ userId, period }) {
  const key = `statement:${userId}:${period}`;
  const existing = await s3.headObject(`statements/${key}.pdf`).catch(() => null);
  if (existing) return;

  const pdf = await renderStatement(userId, period);
  await s3.putObject(`statements/${key}.pdf`, pdf, {
    Metadata: { idempotencyKey: key },
  });
}

If your renderer is an API, pass the key through. Kamy, for example, accepts an idempotencyKey on render requests and will return the previous result instead of re-rendering:

ts
await kamy.documents.render({
  template: "statement",
  data: { userId, period },
  idempotencyKey: `statement:${userId}:${period}`,
});

Pattern 5: offload rendering entirely

Running headless Chromium at scale is possible but miserable. You end up managing browser pools, zombie processes, font installation, memory leaks, and the fact that one bad template can hang a worker for 30 seconds.

For large batches, a rendering API is usually cheaper than the engineering time to maintain your own farm. The pattern becomes:

ts
async function renderAndUpload({ userId, period }) {
  const { url } = await kamy.documents.render({
    template: "monthly-statement",
    data: await buildStatementData(userId, period),
    idempotencyKey: `statement:${userId}:${period}`,
  });
  await db.statements.update({
    where: { userId_period: { userId, period } },
    data: { pdfUrl: url },
  });
}

Your workers do almost no CPU work, so you can crank concurrency way up and let the rendering service handle the hard parts.

Pattern 6: progress tracking and observability

For