Building a Document Automation Pipeline with a REST API

Most engineering teams eventually need to generate documents at scale: invoices, contracts, shipping labels, reports, certificates. The naive approach — spin up Puppeteer in your main app and render HTML to PDF on request — works until it doesn't. It blocks your API process, consumes memory unpredictably, and turns every template change into a redeploy.

A document automation pipeline fixes this by decoupling rendering from your business logic. In this post we'll design one around a REST API, with queueing, templating, storage, and delivery as separate concerns.

The shape of the pipeline

At a high level, every document automation pipeline has four stages:

txt
Trigger  →  Data assembly  →  Rendering  →  Delivery/Storage

• Trigger: an event in your system (order placed, subscription renewed, cron job fires).
• Data assembly: pull together everything the document needs — line items, customer info, tax rules.
• Rendering: turn structured data + a template into a PDF.
• Delivery: store the file, email it, push it to S3, or return a signed URL.

The trick is to make each stage independently scalable and independently replaceable.

Step 1: Define a document contract

Before writing any code, lock down the shape of your document request. This becomes the stable interface between producers (your app) and the renderer.

json
{
  "template": "invoice-v2",
  "data": {
    "invoiceNumber": "INV-10294",
    "issuedAt": "2025-01-14",
    "customer": {
      "name": "Acme Corp",
      "email": "[email protected]"
    },
    "lineItems": [
      { "description": "API usage", "qty": 1, "amount": 4900 }
    ],
    "currency": "USD"
  },
  "metadata": {
    "accountId": "acc_123",
    "idempotencyKey": "inv-10294-v1"
  }
}

Two things matter here. First, template is versioned (invoice-v2) — you want to change layouts without breaking in-flight jobs. Second, idempotencyKey lets you retry safely without generating duplicate PDFs.

Step 2: Enqueue, don't block

Rendering a PDF can take anywhere from 200ms to several seconds depending on complexity. You don't want that latency on your request path. Push the job to a queue instead.

ts
import { Queue } from "bullmq";

const documentQueue = new Queue("documents", {
  connection: { host: "redis", port: 6379 },
});

export async function requestInvoice(orderId: string) {
  const order = await db.orders.findById(orderId);

  await documentQueue.add("render", {
    template: "invoice-v2",
    data: buildInvoiceData(order),
    metadata: {
      accountId: order.accountId,
      idempotencyKey: `invoice-${order.id}-${order.version}`,
    },
  }, {
    attempts: 5,
    backoff: { type: "exponential", delay: 2000 },
  });
}

Your HTTP handler returns in a few milliseconds. The actual rendering happens on a worker that can scale horizontally.

Step 3: Call the rendering API

In the worker, call a dedicated PDF service via REST. You can run your own Puppeteer cluster if you want to operate browsers in production, or offload it to something like Kamy, which exposes a stateless HTTP API for template-based rendering.

ts
import { Worker } from "bullmq";

new Worker("documents", async (job) => {
  const res = await fetch("https://api.kamy.dev/v1/documents", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.KAMY_API_KEY}`,
      "Content-Type": "application/json",
      "Idempotency-Key": job.data.metadata.idempotencyKey,
    },
    body: JSON.stringify({
      template: job.data.template,
      data: job.data.data,
    }),
  });

  if (!res.ok) {
    throw new Error(`Render failed: ${res.status} ${await res.text()}`);
  }

  const { documentId, url } = await res.json();
  await persistDocumentRef(job.data.metadata.accountId, documentId, url);
}, {
  connection: { host: "redis", port: 6379 },
  concurrency: 10,
});

The Idempotency-Key header is important: if the worker crashes after the API call succeeds but before the DB write, the retry won't produce a duplicate PDF.

Step 4: Store the reference, not the file

The PDF binary belongs in object storage — S3, R2, GCS. Your database should store a reference: document ID, URL (or key), checksum, and status.

ts
async function persistDocumentRef(
  accountId: string,
  documentId: string,
  url: string,
) {
  await db.documents.insert({
    accountId,
    externalId: documentId,
    url,
    status: "ready",
    createdAt: new Date(),
  });
}

If your rendering provider returns a signed URL with a short TTL, either re-fetch on demand or copy the file into your own bucket. For long-term archival (think tax documents), the second option is safer.

Step 5: Delivery

Delivery is a separate concern. Once a document is marked ready, downstream processes can act on it: email it, notify a webhook, make it available in a customer portal.

ts
documentQueue.on("completed", async (job) => {
  if (job.data.metadata.deliver === "email") {
    await emailQueue.add("send-invoice", {
      to: job.data.data.customer.email,
      documentUrl: job.returnvalue.url,
    });
  }
});

Keeping delivery out of the render wor