Webhook-Driven PDF Workflows: Async Rendering at Scale

Synchronous PDF generation breaks down under load. Here's how to design webhook-driven async rendering pipelines that handle bursts, retries, and failures without blocking your API.

pdfwebhooksasyncnode.jsscalingapi-design

Webhook-Driven PDF Workflows: Async Rendering at Scale

The first time you generate a PDF in a web app, it's tempting to do it inline: user clicks "Download Invoice," your handler renders the document, streams it back, done. That works until it doesn't. A heavy template, a batch of 500 invoices, or a flaky font server will tie up your request worker for seconds — sometimes minutes. Timeouts cascade, load balancers kill connections, and users refresh until your queue is on fire.

The fix is to decouple PDF generation from the request lifecycle. You submit a render job, your worker produces the file, and a webhook notifies your service when it's ready. This post walks through the design patterns that make this reliable at scale.

Why synchronous rendering doesn't scale

A single PDF render typically takes 200ms to 10 seconds depending on complexity: HTML/CSS evaluation, font loading, image fetching, and PDF serialization. Three things tend to break when you keep this synchronous:

  1. Request timeouts. Most load balancers cap requests at 30–60 seconds. A multi-page report with charts can blow past that.
  2. Worker starvation. PDF rendering is CPU-heavy. A handful of concurrent renders will saturate your API workers and tank unrelated endpoints.
  3. Retry amplification. If the client retries a failed request, you re-render the entire document, doubling the load.

Async rendering with webhooks fixes all three. Your API returns immediately with a job ID, rendering happens on dedicated workers, and the result is pushed to a URL you control.

The basic flow

Here's the shape of a webhook-driven render:

txt
Client → POST /render → API returns { jobId }
                           ↓
                     Render worker
                           ↓
                   PDF stored in S3
                           ↓
         POST https://yourapp.com/webhooks/pdf
         { jobId, status: "completed", url }

With Kamy, you kick off an async render by passing a webhookUrl:

ts
import { kamy } from "@kamydev/sdk";

const job = await kamy.documents.renderAsync({
  template: "monthly-report",
  data: { userId: "u_123", month: "2025-01" },
  webhookUrl: "https://yourapp.com/webhooks/pdf",
  metadata: { userId: "u_123", reportType: "monthly" },
});

console.log(job.id); // "job_01HV..."

The API returns in under 100ms. Your worker gets the webhook when the PDF is ready — usually within a few seconds, but your API thread never blocks.

Handling the webhook

Your webhook endpoint has three jobs: verify the signature, acknowledge quickly, and queue the actual work.

ts
import express from "express";
import crypto from "crypto";

const app = express();

app.post(
  "/webhooks/pdf",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    const signature = req.header("Kamy-Signature") ?? "";
    const expected = crypto
      .createHmac("sha256", process.env.KAMY_WEBHOOK_SECRET!)
      .update(req.body)
      .digest("hex");

    if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
      return res.status(401).send("invalid signature");
    }

    const event = JSON.parse(req.body.toString());
    await jobQueue.enqueue("pdf-ready", event);

    res.status(200).send("ok");
  },
);

Two things to notice:

  • Verify on the raw body, not the parsed JSON. Any serialization change will invalidate the HMAC.
  • Acknowledge before processing. Return 200 immediately and push downstream work (email delivery, database updates) into a background queue. Webhook senders typically retry if you don't respond within 5–10 seconds.

Idempotency is non-negotiable

Webhook providers retry on 5xx responses or timeouts. That means your handler will occasionally receive the same event twice. Store the event ID and dedupe:

ts
async function handlePdfReady(event: PdfReadyEvent) {
  const inserted = await db.webhookEvents.insert({
    id: event.id,
    jobId: event.jobId,
    receivedAt: new Date(),
  }).onConflict("id").ignore();

  if (inserted.rowCount === 0) {
    return; // already processed
  }

  await deliverPdfToUser(event.jobId, event.url);
}

The unique constraint on event.id turns duplicate deliveries into no-ops without needing distributed locks.

Handling failures

Not every render succeeds. Fonts fail to load, data is malformed, a page overflows. Your webhook will receive failure events too:

json
{
  "id": "evt_01HV...",
  "jobId": "job_01HV...",
  "status": "failed",
  "error": {
    "code": "template_error",
    "message": "Undefined variable: invoice.customer.vatNumber"
  },
  "metadata": { "userId": "u_123" }
}

Branch on status and route failures appropriately — usually to an alerting channel plus a user-facing notification. Don't retry blindly; a template_error won't fix itself.

For transient issues (network, upstream API hiccups), the render service should retry with backoff before giving up. Kamy retries internally with exponential backoff and only fires a failed webhook once the job is terminally dead.

Scaling the worker side

If you're running your own renderer behind this pattern, a few things matter:

  • Separate the queue from the API. Use SQS, Redis Streams, or similar