Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating one PDF is easy. Generating fifty thousand invoices before Monday morning is a different problem. Here are the patterns that actually hold up under load.

pdfbatch-processingnode.jsqueuesbackend

Batch PDF Generation: Patterns for Rendering Thousands of Documents

Generating a single PDF is a solved problem. Generating fifty thousand of them on the first of the month, without blowing up memory, timing out your web server, or getting rate-limited by your rendering service — that's the interesting problem. This post walks through the patterns we've seen work well for high-volume PDF generation, whether you're rolling your own renderer or using an API like Kamy.

Don't do it inline

The first rule: never generate PDFs in the same process that handles HTTP requests, at least not for batch workloads. Rendering is CPU- and memory-heavy. A Chromium-based renderer can easily use 200–500MB per document. If a user triggers a "download all invoices" action and you render synchronously, you'll either OOM your API pod or block it long enough that your load balancer kills the connection.

The correct shape is almost always:

  1. Accept the batch request, persist a job record, return a job ID.
  2. Push work onto a queue.
  3. Workers render, store to object storage, update job state.
  4. Client polls or receives a webhook when done.
ts
// API handler — fast, non-blocking
app.post("/batches", async (req, res) => {
  const batch = await db.batches.insert({
    userId: req.user.id,
    status: "queued",
    total: req.body.documents.length,
  });

  await queue.addBulk(
    req.body.documents.map((doc) => ({
      name: "render-pdf",
      data: { batchId: batch.id, doc },
    })),
  );

  res.status(202).json({ batchId: batch.id });
});

Chunking and concurrency

Once work is on a queue, the temptation is to spin up a hundred workers and let them rip. Don't. Rendering is expensive, and downstream services (your DB, your storage, your PDF API) have limits.

Pick a concurrency number based on measured throughput per worker, not vibes. If one worker sustains 3 PDFs per second and your target is 300 PDFs/sec, you need ~100 concurrent slots. Split those across multiple worker instances so one crash doesn't halt everything.

For BullMQ, this looks like:

ts
import { Worker } from "bullmq";

new Worker(
  "pdf",
  async (job) => {
    return await renderAndStore(job.data);
  },
  {
    concurrency: 8,
    limiter: { max: 50, duration: 1000 },
  },
);

The limiter here is important — it caps how fast you hit downstream services, which prevents thundering-herd behavior when a large batch arrives.

Idempotency and retries

At scale, things fail. Networks blip, containers get evicted, upstream APIs return 502s. Your worker must be safe to retry.

The pattern: derive a deterministic key for each document (batch ID + document ID, hashed if needed) and use it as the object storage key. If a job runs twice, the second run overwrites the same object — no duplicates, no orphaned files.

ts
async function renderAndStore({ batchId, doc }) {
  const key = `batches/${batchId}/${doc.id}.pdf`;

  if (await storage.exists(key)) {
    return { key, cached: true };
  }

  const pdf = await kamy.documents.render({
    template: "invoice",
    data: doc,
    idempotencyKey: `${batchId}:${doc.id}`,
  });

  await storage.put(key, pdf.buffer);
  return { key, cached: false };
}

The idempotencyKey on the render call means even the PDF service won't double-bill you if the worker retries after a network timeout on the response.

Batch APIs beat per-document calls

If your rendering backend supports it, submit documents in batches rather than one-per-request. You cut HTTP overhead, and the service can pipeline work internally. Kamy's batch endpoint, for example, accepts up to 500 documents per call and returns a batch ID you can poll.

ts
const { batchId } = await kamy.batches.create({
  template: "invoice",
  documents: chunk.map((doc) => ({
    id: doc.id,
    data: doc,
  })),
  webhookUrl: "https://api.example.com/hooks/kamy",
});

Then chunk your worker's input into groups of a few hundred instead of firing one job per document. Fewer queue messages, fewer HTTP round-trips, less coordination overhead.

Backpressure and progress

Users want progress bars. Give them one. Every worker should update a counter when it finishes a document:

ts
await db.batches.updateOne(
  { id: batchId },
  { $inc: { completed: 1 } },
);

Expose a GET /batches/:id endpoint that returns { total, completed, failed, status }. If you're feeling generous, stream updates over SSE or WebSocket — but polling every 2 seconds is fine for most cases and much simpler to operate.

For very large batches (100k+), consider partitioning: split the batch into sub-batches of ~1000 documents each, and track completion per sub-batch. This keeps DB update contention manageable and gives you natural checkpoints if something goes wrong.

Delivery

When the batch is done, you usually need to hand the user something. Options:

  • ZIP archive: fine up to a few hundred MB. Stream files from object storage into a ZIP writer to avoid loading everything into memory.
  • Merged PDF: use pdf-lib or similar to concatenate. Watch memory — merge in chunks of ~50 at a time.
  • Signed URL list: skip packaging entirely. Return a JSON manifest of pre-signed URLs. Fastest, and lets the user's browser or CDN handle downloads.
ts
const urls = await