Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating one PDF is easy. Generating fifty thousand invoices before the billing cycle closes is a different problem. Here are the patterns that actually hold up.

pdfbatch processingnode.jsqueuesperformance

Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating a single PDF is a solved problem. Generating fifty thousand of them in a two-hour billing window, without blowing up your database, your memory, or your S3 bill, is where things get interesting. This post walks through the patterns we see teams use when they move from "render on demand" to "render a lot, fast."

Start by decoupling generation from the request

The first mistake most teams make is generating PDFs inside a synchronous HTTP handler. It works for one document. It falls over the moment a user clicks "export all invoices."

The baseline pattern is simple: an API endpoint enqueues a job, a worker fleet consumes it, and the result is stored somewhere durable (usually S3 or R2). Clients either poll or receive a webhook.

ts
// api/exports.ts
app.post("/exports", async (req, res) => {
  const jobId = crypto.randomUUID();
  await queue.enqueue("render-pdf-batch", {
    jobId,
    userId: req.user.id,
    invoiceIds: req.body.invoiceIds,
  });
  res.json({ jobId, status: "queued" });
});

This gives you two things for free: retries and backpressure. Both matter a lot when you're doing thousands of renders.

Fan-out: one job per document

For batches larger than a few dozen, don't process the whole batch in one worker. Fan it out.

ts
// worker/batch-dispatcher.ts
export async function handleBatch(job: BatchJob) {
  const chunks = chunk(job.invoiceIds, 50);
  for (const ids of chunks) {
    await queue.enqueue("render-single", {
      parentJobId: job.jobId,
      invoiceIds: ids,
    });
  }
  await db.jobs.update(job.jobId, { totalChunks: chunks.length });
}

Each child job renders a chunk (say 50 PDFs), writes them to storage, and increments a counter. When the counter hits totalChunks, the parent job is complete. This pattern:

  • Isolates failures. One bad invoice doesn't kill 10,000 others.
  • Lets you scale workers horizontally without coordination.
  • Gives you accurate progress reporting for the UI.

Concurrency: the number that matters most

If you're rendering with headless Chrome, each browser instance eats 150–300 MB of RAM. You cannot spin up 100 of them per worker. The right concurrency depends on your renderer:

  • Headless Chromium: 2–5 pages per worker, reuse the browser.
  • wkhtmltopdf / weasyprint: 1–2 processes per CPU core.
  • Native libraries (pdf-lib, PDFKit): mostly CPU-bound, so match to core count.

Reuse the browser aggressively. Launching Chromium takes 300–800ms; opening a new page takes 20–50ms.

ts
const browser = await puppeteer.launch();

export async function renderPdf(html: string) {
  const page = await browser.newPage();
  try {
    await page.setContent(html, { waitUntil: "networkidle0" });
    return await page.pdf({ format: "A4" });
  } finally {
    await page.close();
  }
}

Restart the browser every N renders (say 500) to avoid memory creep.

Stream to storage, don't buffer

A single 20-page PDF might be 2 MB. Multiply by a batch of 5,000 and you've got 10 GB you don't want sitting in memory or on a temp disk.

Stream directly from the renderer to object storage:

ts
import { Upload } from "@aws-sdk/lib-storage";

const upload = new Upload({
  client: s3,
  params: {
    Bucket: "exports",
    Key: `${jobId}/${invoiceId}.pdf`,
    Body: pdfStream,
    ContentType: "application/pdf",
  },
});

await upload.done();

For the final deliverable, most users want one ZIP, not 5,000 URLs. Generate the ZIP as a separate finalization job that streams from S3 back into S3 using an archive stream. Never assemble it in memory.

Idempotency and retries

Queues will re-deliver messages. Your renderer must be safe to run twice.

The easiest approach: derive the storage key from stable inputs.

ts
const key = `${jobId}/${invoiceId}-v${templateVersion}.pdf`;
const exists = await s3.headObject({ Bucket, Key: key }).catch(() => null);
if (exists) return { key, cached: true };

If the object already exists, skip the render. This turns retries from a liability into a cheap correctness guarantee.

Rate limits and external data

Templates often need to fetch data — logos, user avatars, product images, currency rates. When you scale to thousands of concurrent renders, you can accidentally DDoS your own APIs.

Two rules:

  1. Hydrate all data before enqueuing the render job. Pass the fully-resolved payload.
  2. Cache remote assets (images, fonts) locally or in a CDN. Don't re-fetch them per document.

When to offload rendering entirely

Running Chromium at scale is operationally painful: sandbox flags, font packages, zombie processes, memory tuning, CVE patching. If PDF generation isn't your core product, a managed API is usually cheaper than the engineering time.

Kamy exposes a batch endpoint designed for this pattern:

ts
import { kamy } from "@kamydev/sdk";

const batch = await kamy.batches.create({
  template: "invoice",
  items: invoices.map((inv) => ({
    key: inv.id,
    data: inv,
  })),
  webhook: "https://