Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating a single PDF is a solved problem. Generating 50,000 of them for a monthly billing run — without timeouts, OOM errors, or angry ops engineers — is where things get interesting. This post covers the patterns that actually hold up in production when you need to render documents at scale.

The naive approach (and why it breaks)

Most batch PDF code starts life looking like this:

ts
for (const invoice of invoices) {
  const pdf = await renderInvoice(invoice);
  await uploadToS3(pdf);
}

It works fine for 10 invoices. At 10,000 it has three problems:

1. Sequential latency. If each render takes 300ms, 10,000 documents take 50 minutes.
2. Memory pressure. Headless Chrome or wkhtmltopdf instances pile up if you parallelize naively.
3. No recovery. One bad row at index 7,432 crashes the whole job.

You need concurrency control, isolation, and resumability.

Pattern 1: Bounded concurrency

The simplest upgrade is a worker pool. Don't Promise.all over 10,000 promises — you'll spawn 10,000 render contexts simultaneously. Use a concurrency limiter:

ts
import pLimit from "p-limit";

const limit = pLimit(20); // tune to your CPU/memory budget

const results = await Promise.allSettled(
  invoices.map((invoice) =>
    limit(async () => {
      const pdf = await renderInvoice(invoice);
      await uploadToS3(pdf, `invoices/${invoice.id}.pdf`);
      return invoice.id;
    })
  )
);

const failed = results
  .filter((r) => r.status === "rejected")
  .map((r) => (r as PromiseRejectedResult).reason);

Use allSettled, not all. One failed render shouldn't kill 9,999 successful ones. Log the failures, retry them separately.

The right concurrency number depends on your renderer. For headless Chrome, 4–8 per CPU core is a reasonable starting point. For lighter HTML-to-PDF engines, you can push higher.

Pattern 2: Queue-backed jobs

For anything over a few thousand documents, run the batch through a queue. This gives you retries, visibility, and horizontal scaling for free.

ts
import { Queue } from "bullmq";

const pdfQueue = new Queue("pdf-render", { connection: redis });

async function enqueueBatch(invoices: Invoice[], batchId: string) {
  await pdfQueue.addBulk(
    invoices.map((invoice) => ({
      name: "render-invoice",
      data: { invoiceId: invoice.id, batchId },
      opts: {
        attempts: 3,
        backoff: { type: "exponential", delay: 2000 },
        removeOnComplete: 1000,
      },
    }))
  );
}

The worker stays small and focused:

ts
import { Worker } from "bullmq";

new Worker(
  "pdf-render",
  async (job) => {
    const invoice = await db.invoice.findUnique({
      where: { id: job.data.invoiceId },
    });
    const pdf = await renderInvoice(invoice);
    const url = await uploadToS3(pdf, `invoices/${invoice.id}.pdf`);
    await db.invoice.update({
      where: { id: invoice.id },
      data: { pdfUrl: url, pdfGeneratedAt: new Date() },
    });
  },
  { connection: redis, concurrency: 10 }
);

Now you can scale by adding worker processes, and Redis tracks progress.

Pattern 3: Offload the rendering itself

Running headless Chrome at scale is its own ops problem. Browser crashes, font installation, memory leaks, sandbox flags on certain kernels — it adds up. A common refactor is to push the actual rendering to a service that does only that.

With a managed API like Kamy, your worker becomes a thin HTTP client:

ts
import { kamy } from "@kamydev/sdk";

async function renderInvoice(invoice: Invoice) {
  const { url } = await kamy.documents.render({
    template: "invoice-v2",
    data: invoice,
    output: "url",
  });
  return url;
}

For true batch workloads, you can also submit jobs in bulk and poll for completion rather than holding HTTP connections open:

ts
const batch = await kamy.batches.create({
  template: "invoice-v2",
  documents: invoices.map((inv) => ({
    key: inv.id,
    data: inv,
  })),
  webhook: "https://api.example.com/hooks/kamy",
});

The webhook fires per document (or per batch, depending on config), and you update your database from there. This decouples your worker fleet from the cost of running browsers.

Pattern 4: Idempotency and resumability

Long batches fail. Always design so re-running is safe.

ts
async function renderInvoiceIdempotent(invoice: Invoice, batchId: string) {
  const existing = await db.pdfRender.findUnique({
    where: { invoiceId_batchId: { invoiceId: invoice.id, batchId } },
  });
  if (existing?.status === "completed") return existing.url;

  const url = await renderInvoice(invoice);

  await db.pdfRender.upsert({
    where: { invoiceId_batchId: { invoiceId: invoice.id, batchId } },
    create: { invoiceId: invoice.id, batchId, url, status: "completed" },
    update: { url, status: "completed" },
  });

  return url;
}