Building a Document Automation Pipeline with a REST API

Generating one PDF is easy. Generating thousands of them — reliably, on schedule, with audit trails and retries — is a systems problem. Invoices, contracts, shipping labels, statements: every business app eventually needs a real document pipeline, not a one-off script.

This post walks through the architecture of a document automation pipeline built around a REST API. We'll cover the moving parts, the failure modes you'll actually hit, and a working example using Kamy as the rendering backend.

The shape of the pipeline

A document automation pipeline usually has five stages:

1. Trigger — an event (cron, webhook, queue message) decides a document needs to exist.
2. Data assembly — gather the data from your DB, third-party APIs, or upstream services.
3. Rendering — convert data + template into a final document (PDF, usually).
4. Storage — put the artifact somewhere durable (S3, GCS, your own object store).
5. Delivery — email it, push it to a portal, hand back a signed URL.

Each stage should be independently retriable. If rendering fails, you don't want to re-run the data assembly. If delivery fails, you don't want to re-render.

Stage 1: The trigger

Use a queue. Even if you start with a simple cron job, push the actual work onto a queue so workers can scale horizontally and failures don't block the scheduler.

ts
// scheduler.ts
import { Queue } from "bullmq";

const docQueue = new Queue("documents");

export async function scheduleInvoice(invoiceId: string) {
  await docQueue.add(
    "render-invoice",
    { invoiceId },
    { attempts: 5, backoff: { type: "exponential", delay: 2000 } }
  );
}

Idempotency matters here. Always pass a stable job ID (e.g. the invoice ID plus a version) so retries don't double-generate documents.

Stage 2: Data assembly

Keep this step pure: input is an ID, output is a JSON payload that fully describes the document. No side effects, no rendering logic, no formatting decisions buried in SQL.

ts
// assemble.ts
export async function buildInvoicePayload(invoiceId: string) {
  const invoice = await db.invoice.findUniqueOrThrow({
    where: { id: invoiceId },
    include: { customer: true, lineItems: true },
  });

  return {
    invoiceNumber: invoice.number,
    issuedAt: invoice.issuedAt.toISOString(),
    customer: {
      name: invoice.customer.name,
      address: invoice.customer.address,
    },
    items: invoice.lineItems.map((li) => ({
      description: li.description,
      quantity: li.quantity,
      unitPrice: li.unitPrice,
      total: li.quantity * li.unitPrice,
    })),
    total: invoice.lineItems.reduce(
      (sum, li) => sum + li.quantity * li.unitPrice,
      0
    ),
  };
}

The payload is also what you should log or store if you ever need to reproduce a document later.

Stage 3: Rendering via REST

This is where most teams get into trouble. Spinning up headless Chromium, managing fonts, dealing with concurrency, page breaks, and PDF/A compliance is a full-time job. A REST-based renderer like Kamy lets you POST data and a template reference and get back a PDF.

http
POST /v1/documents/render HTTP/1.1
Host: api.kamy.dev
Authorization: Bearer $KAMY_API_KEY
Content-Type: application/json

{
  "template": "invoice-v3",
  "data": { "invoiceNumber": "INV-2025-001", "total": 4200 },
  "output": { "format": "pdf" }
}

A typical response:

json
{
  "id": "doc_01HQ8K2X9P",
  "url": "https://cdn.kamy.dev/doc_01HQ8K2X9P.pdf?sig=...",
  "expiresAt": "2025-03-14T18:00:00Z",
  "bytes": 184213
}

In a worker, the call looks like this:

ts
// worker.ts
import { kamy } from "@kamydev/sdk";
import { buildInvoicePayload } from "./assemble";
import { uploadToS3 } from "./storage";

export async function renderInvoiceJob(invoiceId: string) {
  const data = await buildInvoicePayload(invoiceId);

  const doc = await kamy.documents.render({
    template: "invoice-v3",
    data,
    idempotencyKey: `invoice:${invoiceId}:v3`,
  });

  const buffer = await fetch(doc.url).then((r) => r.arrayBuffer());
  const s3Key = `invoices/${invoiceId}.pdf`;
  await uploadToS3(s3Key, Buffer.from(buffer));

  await db.invoice.update({
    where: { id: invoiceId },
    data: { pdfKey: s3Key, renderedAt: new Date() },
  });
}

Two details that matter:

• Idempotency keys prevent duplicate documents if the worker retries after a network hiccup.
• Version your templates (invoice-v3, not invoice). When legal asks "what did the invoice look like in March?", you'll want an answer.

Stage 4: Storage

The renderer's signed URL is convenient, but it expires. For anything customers might re-download, store the bytes in your own bucket. Set lifecycle policies based on your retention requirements.

A reasonable schema:

txt
s3://my-app-docs/
  invoices/<invoice_id>.pdf
  contracts/