Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating one PDF is easy. Generating fifty thousand invoices before the end-of-month billing run is a different problem. Here are the patterns that actually work in production.

pdfbatch processingnode.jsqueuesbackendperformance

Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating one PDF is easy. Generating fifty thousand invoices before the end-of-month billing run is a different problem. The naive for loop that worked for a demo will melt your server, exhaust memory, and hit every rate limit you have. This post walks through the patterns we see backend teams use when moving from "occasional PDF" to "PDF factory."

The core problem

PDF rendering is CPU-bound and memory-hungry. A single render with a headless browser can spike to 200–400 MB of RSS. Multiply by concurrent workers and you're one bad batch away from an OOM kill. On top of that, most batch jobs involve:

  • Fetching per-document data (often from multiple sources)
  • Rendering HTML or a template to PDF
  • Uploading the result somewhere (S3, GCS, an email attachment)
  • Recording state so failed items can be retried

Any of these can be the bottleneck, and they rarely fail in the same way.

Pattern 1: Never render inline with the request

The first rule: if a user clicks "Export 5,000 invoices," do not render on the request thread. Enqueue a job and return a job ID immediately.

ts
// POST /exports
app.post("/exports", async (req, res) => {
  const jobId = crypto.randomUUID();
  await queue.add("batch-export", {
    jobId,
    userId: req.user.id,
    filters: req.body.filters,
  });
  res.json({ jobId, status: "queued" });
});

The client polls /exports/:jobId or subscribes to a websocket for progress. This is non-negotiable for anything over a few dozen documents.

Pattern 2: Fan out, don't loop

A single worker processing 10,000 documents sequentially is slow and fragile — one crash and you restart from zero. Split the batch into per-document jobs.

ts
// Coordinator job
async function handleBatchExport({ jobId, filters }) {
  const ids = await db.invoices.findIds(filters);
  await db.jobs.insert({ jobId, total: ids.length, done: 0 });

  await queue.addBulk(
    ids.map((invoiceId) => ({
      name: "render-invoice",
      data: { jobId, invoiceId },
      opts: { attempts: 3, backoff: { type: "exponential", delay: 2000 } },
    })),
  );
}

Now each document is an independent unit of work. Retries are cheap. Progress is a simple counter increment. Backpressure is handled by the queue's concurrency setting.

Pattern 3: Bound your concurrency

The temptation is to crank concurrency to the number of CPU cores and let it rip. Don't. You need a concurrency ceiling based on the slowest resource in your pipeline — usually memory or a downstream API.

With BullMQ, that looks like this:

ts
new Worker("render-invoice", handler, {
  connection: redis,
  concurrency: 8,
  limiter: { max: 100, duration: 1000 }, // 100 jobs/sec cap
});

If you're calling a rendering service, respect its rate limits here. If you're rendering in-process with Puppeteer, concurrency should roughly match cores - 1 and no more.

Pattern 4: Offload rendering entirely

Running headless Chrome at scale is a full-time job: zombie processes, font management, Chromium updates, sandbox permissions in containers. A lot of teams eventually move rendering out of their app entirely.

This is where a service like Kamy fits. You define a template once, then submit render jobs by ID:

ts
import { kamy } from "@kamydev/sdk";

async function renderInvoice({ invoiceId }) {
  const invoice = await db.invoices.findById(invoiceId);

  const pdf = await kamy.documents.render({
    template: "invoice-v2",
    data: invoice,
    output: {
      destination: "s3",
      bucket: "acme-invoices",
      key: `invoices/${invoice.id}.pdf`,
    },
  });

  return pdf.url;
}

The tradeoff: you get predictable per-render latency and no infrastructure to babysit, but you're now bound by network I/O and API quotas. For high volumes, use the batch endpoint so the service can parallelize on its side:

ts
const batch = await kamy.batches.create({
  template: "invoice-v2",
  items: invoices.map((inv) => ({ id: inv.id, data: inv })),
  webhook: "https://api.acme.com/hooks/kamy",
});

Then handle completion via webhook instead of polling.

Pattern 5: Idempotency and partial failure

At 10,000 documents, something will fail. A DB timeout, a corrupted image, a template edge case. Design for it:

  • Idempotency keys: use the document ID plus a batch version as your render key so retries don't create duplicates.
  • Per-item status: track state per document, not just per batch.
  • Dead-letter queue: after N retries, move to DLQ and continue. Don't fail the whole batch.
ts
async function renderInvoice({ jobId, invoiceId }) {
  const key = `${jobId}:${invoiceId}`;
  const existing = await db.renders.findOne({ key });
  if (existing?.status === "done") return existing.url;

  try {
    const url = await doRender(invoiceId);
    await db.renders.upsert({ key, status: "done", url });
    await db.jobs.increment(jobId, "done");
    return url;
  } catch (err) {
    await db.renders.upsert({ key, status: "failed", error: err.message });
    throw