Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating a single PDF is a solved problem. Generating fifty thousand monthly statements before 9 AM, or rendering a million tax documents in a regulatory window, is a different kind of problem. The bottleneck stops being "how do I render HTML to PDF" and starts being "how do I orchestrate work, manage memory, handle failures, and not melt my infrastructure."

This post walks through the patterns that hold up in production when batch sizes go from hundreds to hundreds of thousands.

The naive approach (and why it breaks)

Most batch PDF code starts looking like this:

ts
for (const customer of customers) {
  const pdf = await renderPdf(customer);
  await s3.putObject({ Bucket: "statements", Key: `${customer.id}.pdf`, Body: pdf });
}

This works for 50 documents. At 50,000 it has three problems:

1. Serial execution wastes CPU and network.
2. One failure kills the run. A single bad record at index 12,481 means restarting.
3. No observability. You don't know where you are or how long it'll take.

Let's fix each of these.

Pattern 1: Bounded concurrency

The first instinct is Promise.all. Don't. Firing 50,000 PDF renders simultaneously will exhaust file descriptors, blow up memory, or rate-limit your renderer.

Use a concurrency limiter. p-limit works fine, but for batch jobs I prefer a manual worker pool because it composes better with streaming inputs:

ts
async function runPool<T>(
  items: AsyncIterable<T>,
  concurrency: number,
  worker: (item: T) => Promise<void>
) {
  const iterator = items[Symbol.asyncIterator]();
  const workers = Array.from({ length: concurrency }, async () => {
    while (true) {
      const { value, done } = await iterator.next();
      if (done) return;
      await worker(value);
    }
  });
  await Promise.all(workers);
}

Concurrency should match your renderer's characteristics. For local Puppeteer/Chromium, 4–8 is usually the sweet spot per machine. For an HTTP-based PDF API like Kamy, you can go much higher (50–200) since the heavy lifting happens server-side.

Pattern 2: Stream the input

Loading all 50,000 records into memory before starting is a recipe for OOM kills. Stream from your database:

ts
async function* streamCustomers(db: Pool) {
  const cursor = db.query(new Cursor("SELECT id, data FROM customers WHERE active = true"));
  while (true) {
    const rows = await cursor.read(500);
    if (rows.length === 0) break;
    for (const row of rows) yield row;
  }
}

Pair this with the worker pool above and your memory footprint stays flat regardless of batch size.

Pattern 3: Queue-based fan-out

For anything over ~10,000 documents, run a single process is the wrong architecture. Move to a queue.

The pattern: a planner enumerates work and pushes jobs onto a queue. Workers pull jobs, render, and upload. This gives you horizontal scaling, retries, and dead-letter handling for free if you're using something like BullMQ, SQS, or Cloud Tasks.

ts
// Planner
for await (const customer of streamCustomers(db)) {
  await queue.add("render-statement", {
    customerId: customer.id,
    period: "2025-01",
  }, {
    jobId: `stmt-${customer.id}-202501`, // idempotency
    attempts: 3,
    backoff: { type: "exponential", delay: 5000 },
  });
}

ts
// Worker
queue.process("render-statement", 20, async (job) => {
  const { customerId, period } = job.data;
  const data = await loadStatementData(customerId, period);

  const pdf = await kamy.documents.render({
    template: "monthly-statement",
    data,
    output: "buffer",
  });

  await s3.putObject({
    Bucket: "statements",
    Key: `${period}/${customerId}.pdf`,
    Body: pdf,
  });
});

The jobId is critical: it makes the whole pipeline idempotent. Re-running the planner won't create duplicate work.

Pattern 4: Offload the rendering

The single biggest reason batch PDF jobs are slow is that teams run a Chromium instance per worker. It's RAM-hungry, leaks file descriptors, and crashes in interesting ways at 3 AM.

If you're not in the business of operating headless browsers, don't. A hosted API like Kamy lets you treat rendering as a stateless HTTP call:

ts
import { kamy } from "@kamydev/sdk";

const pdf = await kamy.documents.render({
  template: "monthly-statement",
  data: statementData,
  output: "url", // signed URL, valid 1 hour
});

For batch jobs specifically, request URLs instead of buffers. Your worker never holds 50MB of PDF in memory — it just hands the URL to your storage layer or downstream consumer.

Kamy also exposes a batch endpoint that's worth using when you have a homogeneous run:

http
POST /v1/documents/batch
Content-Type: application/json

{
  "template": "monthly-statement",
  "items": [
    { "key": "cust_001", "data": { "name": "Alice", "balance": 120.50 } },
    { "key": "cust_002", "data": { "name": "Bob", "balance": 89.10 } }
  ],
  "webhook": "https://api.example.com/hooks/pdf-done"