Building a Document Automation Pipeline with a REST API

Most products eventually need to generate documents: invoices, contracts, receipts, reports, shipping labels. The first version is usually a one-off function that mashes HTML together and calls a PDF library. The second version is a mess of edge cases, retries, and timeouts. By the third version, you're building a proper pipeline whether you planned to or not.

This post walks through the architecture of a document automation pipeline backed by a REST API, the kinds of failure modes you'll hit, and how to keep the whole thing maintainable.

The shape of the pipeline

A document pipeline almost always has the same five stages:

1. Trigger — an event causes a document to be generated (order placed, contract signed, monthly cron).
2. Data assembly — gather the data needed to fill the template.
3. Render — turn template + data into a PDF (or DOCX, etc.).
4. Store — persist the output to durable storage.
5. Deliver — email it, return a signed URL, push to a webhook.

The mistake is treating these as one synchronous function call. Rendering can take seconds, storage can fail, email providers throttle. You want each stage to be independently retryable.

Designing the API surface

Whether you're calling an external service like Kamy or building your own internal one, the rendering step should look like a simple REST call. Something like this:

http
POST /v1/documents/render
Content-Type: application/json
Authorization: Bearer sk_live_...

{
  "template": "invoice",
  "data": {
    "invoiceNumber": "INV-2025-0142",
    "customer": { "name": "Acme Corp", "email": "[email protected]" },
    "lineItems": [
      { "description": "API credits", "quantity": 1000, "unitPrice": 0.02 }
    ],
    "total": 20.00
  },
  "options": { "format": "pdf", "async": true }
}

Two things worth highlighting:

• template is a name, not a payload. Templates live server-side, versioned. Sending HTML on every request is fine for prototypes but turns into a maintenance nightmare.
• async: true is the difference between a healthy pipeline and a fragile one. For anything non-trivial, render asynchronously and poll or receive a webhook.

A response from an async render:

json
{
  "id": "doc_01HXYZ...",
  "status": "queued",
  "statusUrl": "https://api.kamy.dev/v1/documents/doc_01HXYZ"
}

The worker side

Triggering generation from your app should put a job on a queue, not block the request. Here's a sketch using BullMQ:

ts
import { Queue } from "bullmq";

const docQueue = new Queue("documents", { connection: redis });

export async function enqueueInvoice(orderId: string) {
  await docQueue.add(
    "invoice",
    { orderId },
    {
      attempts: 5,
      backoff: { type: "exponential", delay: 2000 },
      removeOnComplete: 1000,
    },
  );
}

The worker handles data assembly and the render call:

ts
import { Worker } from "bullmq";
import { kamy } from "@kamydev/sdk";

new Worker(
  "documents",
  async (job) => {
    const order = await db.orders.findById(job.data.orderId);
    if (!order) throw new Error(`order ${job.data.orderId} not found`);

    const doc = await kamy.documents.render({
      template: "invoice",
      data: buildInvoicePayload(order),
    });

    await db.documents.insert({
      orderId: order.id,
      kamyId: doc.id,
      url: doc.url,
      createdAt: new Date(),
    });

    return doc.id;
  },
  { connection: redis, concurrency: 10 },
);

Concurrency, retries, and backoff are now properties of the queue — not something you have to hand-roll inside the render function.

Webhooks vs polling

If your render API supports webhooks, use them. Polling works but wastes calls and adds latency. A webhook handler is typically a few lines:

ts
app.post("/webhooks/kamy", express.raw({ type: "*/*" }), (req, res) => {
  const sig = req.header("Kamy-Signature");
  if (!verifySignature(req.body, sig, process.env.KAMY_WEBHOOK_SECRET)) {
    return res.status(401).end();
  }

  const event = JSON.parse(req.body.toString());
  if (event.type === "document.completed") {
    deliveryQueue.add("email", {
      documentId: event.data.id,
      url: event.data.url,
    });
  }
  res.status(200).end();
});

Always verify the signature. Always respond quickly — push the actual work to another queue.

Storage and delivery

Most rendering APIs (Kamy included) give you a temporary signed URL when a document is ready. That's fine for immediate download, but you usually want long-term storage you control. A simple pattern:

ts
async function archiveDocument(documentId: string, sourceUrl: string) {
  const res = await fetch(sourceUrl);
  if (!res.ok) throw new Error(`download failed: ${res.status}`);
  const body = await res.arrayBuffer();

  const key = `documents/${documentId}.pdf`;
  await s3.putObject({
    Bucket: "my-app-documents",
    Key: key,
    Body: Buffer.from(body),
    ContentType: "application/pdf",
  });

  return `s3://my-app-documents/${key}`;
}

Now your application owns the file. Delivery (email, in-app download, customer portal) reads from your bucket and generates short-lived signed URLs as needed.

Idempotency