How We Guarantee Exactly One Workflow Run per Event

By Govind Sharma

The Problem

Stamina's customers build automations of the form "when this happens in my CRM, do these steps": a deal moves stage, a lead replies, a contact is created in HubSpot. Behind that sentence is a pipeline where every hop delivers at least once. The outbox publisher can publish twice. Pub/Sub redelivers. CRM webhooks retry, and their retries are re-signed, so they don't look like duplicates.

The customer's expectation is the opposite: this lead is enrolled once, and if it isn't, the product says why. The engine now runs 5,000+ workflows a day across 10+ integrations. This post covers the three decisions that made "exactly one run per event" hold up: where events come from, how a run is admitted, and how a step that has side effects survives a crash.

Constraints we wrote down first

• No message may be published before the database row it points at is visible.
• Dedup must hold with many consumers admitting events concurrently.
• A record we decide not to run must be explained, and that explanation must not roll back with a failed insert.
• Runs can wait for days, retry, and be replayed, so a run is a durable state machine, not one message handler.
• A CRM create must never silently happen twice.

Where Events Come From: an Outbox, Not Row Diffs

The first version inferred business events from database changes using change data capture. It was elegant and wrong: it fired for transactions that later rolled back, and it made "what happened" a guess reconstructed from column diffs. We removed it and made every domain write commit a typed trigger event in the same transaction, in an outbox table.

A publisher claims a batch of outbox rows in a short transaction by writing a time-boxed hold, forwards them to Pub/Sub with no lock held, then marks them published, fenced on that same hold. A crashed publisher's hold simply expires. Messages carry ids only, never payloads, so a redelivered message always reads the current row.

// 1. Claim: short transaction, no network calls inside it.
const holdId = randomUUID();
const rows = await db.query(
  `UPDATE workflow_trigger_events
      SET publish_hold = $1, hold_expires_at = now() + interval '5 minutes'
    WHERE id IN (
      SELECT id FROM workflow_trigger_events
       WHERE published_at IS NULL
         AND (hold_expires_at IS NULL OR hold_expires_at < now())
       ORDER BY id
       LIMIT 100
       FOR UPDATE SKIP LOCKED)
    RETURNING id`,
  [holdId],
);

// 2. Forward ids only, outside any transaction.
await Promise.all(rows.map((r) => topic.publish({ eventId: r.id })));

// 3. Mark published, but only rows we still hold.
await db.query(
  `UPDATE workflow_trigger_events SET published_at = now()
    WHERE id = ANY($1) AND publish_hold = $2`,
  [rows.map((r) => r.id), holdId],
);

Admitting a Run: Three Layers of Dedup

The matcher never scans saved workflow graphs. Activating a workflow writes one binding row per event type it listens to, so matching is an index lookup. Each matching workflow then goes through admission, inside one transaction:

1. Start key. The run row carries workflow_event:<eventId> under a unique index on (workflowId, startKey). A redelivered event finds the existing run and resumes dispatching it instead of creating a second one.
2. Delivery claim. A row unique on (event, workflow) records that this pairing was handled at all, including when it was refused.
3. Per-trigger dedup mode. once_per_record (the default), not_while_running or every_time, serialized with a transaction-scoped advisory lock on (workflow, record). A partial unique index on active runs per record is the backstop if the application logic is ever wrong.

Check-then-insert was rejected outright: two consumers can both check, both see nothing, and both insert. Every layer here is a constraint or a lock the database enforces, so concurrency can only produce a conflict, never a duplicate.

Refusals are written outside the admission transaction, with a reason: duplicate, already running, rule not matched, insufficient credits, echo suppressed, provider suppressed, or missing record identity. That is what lets support answer "why didn't my workflow run for this lead?" from a table instead of from logs.

Steps That Survive Crashes

The earlier engine executed a whole workflow inside one message handler. It could not wait three days, retry one step, or replay. The replacement stores every step as a row with an explicit state: pending, running, succeeded or skipped, plus separate waiting-for-delay and waiting-for-retry states. Those two are kept apart on purpose: a delay's resume time means "the wait is already served", and a retry parked in the same state would be misread.

• Claim: a conditional UPDATE … WHERE status = 'pending' increments the attempt and sets a 30-minute lease, renewed by a heartbeat every 5 minutes. A sweep reclaims expired leases, fenced on the old lease so a slow worker can't overwrite its successor.
• Version pinning: a run executes the graph version it started with. Publishing a new version rebinds triggers for new runs only.
• Failure classes: 5xx, 408, 425, 429, connection resets, timeouts and Postgres serialization or lock errors are transient. Anything unknown is permanent. Action steps retry up to 3 times with deterministic backoff from 1 to 15 minutes; flow-control steps never retry.
• The next steps' rows are created in the same transaction that settles the current one, so a crash can't leave a run with nothing to do next.

The Trade-off: At-Most-Once CRM Creates

Retries are only safe if the side effect is idempotent. Before shipping CRM blocks we ran a replay test against the integration vendor: complete a create, drop the response, replay the identical request. It created a second HubSpot record. The vendor's normalized create had no idempotency we could rely on.

The first decision record made creates upsert-only, keyed on a field each CRM enforces as unique. A later decision record reversed that, on product evidence: upsert-only wasn't the block users asked for, it needed per-tenant Salesforce setup, and one CRM couldn't support it at all. We kept the finding and changed the disposition. Every CRM create now declares itself at-most-once, and the engine refuses to load a create descriptor that doesn't.

// Before the call: mark that a side effect may have left the process.
const marked = await steps.markSideEffectStarted(step.id); // sets it only WHERE NULL

if (!marked) {
  // A previous attempt got this far and we don't know what happened.
  // Retrying could create a duplicate record; surface it instead.
  return settle(step, { status: "failed", outcome: "OUTCOME_UNKNOWN" });
}

const result = await crm.create(step.input);
return settle(step, { status: "succeeded", output: result });

The cost is visible: a dropped response now shows up as a failed step the customer can see and replay, instead of a silent duplicate in their CRM. That is the right way round for a sales team, whose worst outcome is a prospect receiving two of everything.

Replay Without Double Billing

Every block is billed in credits, so replay had to be cheap to reason about. A replay is a new run linked to the original. It copies the settled steps, resets only the nodes reachable from the first error (following the branch choices the original run recorded), and reuses the idempotency key of the oldest step in the lineage, so the provider and the ledger both see the same operation. A block already charged in the lineage is never charged again. Replays are allowed for 60 days and only when the stored payload wasn't truncated at its 64 KiB limit.

Owning the Port to the Integration Vendor

We had already switched integration vendors once, and it rippled through issues and design docs before any code existed. So the engine talks to a port we own. No vendor type or error crosses it: errors arrive already classified as transient or permanent, and optional capabilities such as write idempotency or event-author identity are declared, so the engine never branches on a vendor's name.

• Webhooks are size-checked and HMAC-verified, stored raw in an inbox table keyed by a hash of the body, and acknowledged immediately. A scheduler drains the inbox every 5 seconds with leases and exponential backoff.
• Shadow records keep the last-seen values of the fields active triggers watch, hashed per record, so an update fires only when a watched field changed, and out-of-order or re-announced events are dropped.
• Echo suppression records a short-lived marker for every write the engine makes, so a workflow's own CRM update doesn't come back as a new trigger.
• Provider subscriptions are computed as the union of what active bindings need and reconciled per connection, with an hourly drift repair.
• The generic HTTP block resolves DNS once, rejects private, loopback, link-local and metadata addresses, pins the connection to the validated IP and refuses redirects.

What I'd Do Next

• Per-connection throttling is still in-process. Moving it onto the same Redis limiter the AI workers use would make the rate a property of the connection, not of how many workers are running.
• Run admission is correct but chatty: several round trips per binding. Batching admission per event would cut database load at peak.
• The refusal reasons are the best debugging tool we have; they deserve a customer-facing view, not just a support one.