Why One Analysis Ran Five Times: Idempotency at the Write

By Govind Sharma

The Problem

A FastBowlingLab analysis is a long, paid, multi-stage job: a pose pass that is CPU-bound for minutes, then two LLM calls, a PDF and notifications. The athlete is charged up front, so every failure must refund exactly once, and work must survive deploys, dead containers, duplicate events and replays. This post is about the incident that taught me where idempotency actually has to live, and the pipeline built around that lesson.

The Incident: Five Paid Runs for One Analysis

The symptom was mild. Analyses finished, but some report rows landed in a failed state, and manual retries seemed to fix them. The cause was a default nobody had chosen.

1. Inngest invokes each step over HTTP through the load balancer, whose idle timeout was the AWS default of 60 seconds.
2. Report generation had become two sequential model calls taking 90 to 300 seconds.
3. The load balancer cut every call at 60 seconds and returned 504. Inngest recorded a failed attempt and invoked the step again, while the original kept running.

The metrics were unambiguous once I looked: response time never exceeded 60 seconds in 36 hours (peak 59.35 s), 504s appeared in every generation window, and duplicate executions were spaced 85 to 125 seconds apart. One analysis had five concurrent executions, each a full pair of paid model calls.

The second defect was worse

Report writes had no status guard, and the slower, failing executions wrote last. An analysis would go ready, render its PDF, notify the athlete and get paid for, and then a losing execution would flip it back to failed minutes later. On the rejection path, a stale "not a bowling action" verdict could demote a delivered, paid report and refund it.

The fixes

• The idle timeout is now 900 seconds, set explicitly in the bootstrap script, because the default is what caused the incident. It stays under Inngest's step ceiling.
• Every outcome write is a compare-and-set: update only where the status isn't already 'ready'. A losing execution concedes: it reads the delivered result back and returns 'already delivered', so the athlete doesn't get a second notification.
• An alarm fires on any 504 from the load balancer within 5 minutes. Regression tests were confirmed to fail without the guard.

export async function writeResultUnlessDelivered(analysisId: string, result: Report) {
  const updated = await db
    .update(analyses)
    .set({ status: "ready", result })
    .where(and(eq(analyses.id, analysisId), ne(analyses.status, "ready")))
    .returning({ id: analyses.id });

  if (updated.length === 0) {
    // Another execution already delivered. Concede instead of overwriting,
    // and tell the caller not to notify the athlete again.
    return { alreadyDelivered: true as const };
  }
  return { alreadyDelivered: false as const };
}

The Pipeline Around That Lesson

The analysis is one durable Inngest function, retried up to 4 times and keyed on the analysis id. Each stage is a memoized step: mark analysing, optionally wake the pose service, a cheap vision precheck, dispatch to the pose worker, wait for its completion event, persist, then run the shared report tail. A catch-all marks the analysis failed and refunds it with an idempotent credit grant keyed on the analysis, so a refund can't happen twice.

Deploys can't break runs in flight

Inngest memoizes steps by id and by position, and a run that is mid-analysis during a deploy replays against the new code. So new flag reads are added as new steps, and the flag must ship off: a run that replays and reads "on" would take the new branch and start a second pose pass. Old step ids keep their historical names, because a step id is a memoization key, not documentation. The rollout runbook is deploy with the flag off, let in-flight runs drain, then flip it.

A claim-check queue for the pose fleet

The original dispatch is HTTP: the worker accepts a job with 202, says 429 when busy, and calls back with an HMAC-signed result. I'm moving to a BullMQ queue behind a flag, after a hand-written Redis Streams broker needed an ack timeout longer than the slowest analysis and silently duplicated pose passes when it wasn't. The queue design:

• The job is a claim check, just the analysis id, so an orchestration retry can't dispatch twice and nothing in the job expires while it waits.
• The worker redeems the claim for freshly presigned storage URLs at claim time. Queue dwell is unbounded, and a retried job would otherwise carry dead URLs.
• The analysis runs in a thread executor, never on the event loop, because BullMQ renews its lock on the loop. Measured with a 2-second lock and a 6-second job: executor, 1 execution; event loop, 2 executions and a failed job.
• One semaphore covers both the HTTP and queue paths, so a pod never runs more than it can during a rollout when both are live.
• Queue names, key prefixes and payload shapes are mirrored between TypeScript and Python, and CI runs a real Node-to-Python round trip against Redis. A mismatched key prefix would be a silent outage.

Paying Only for Work That Happens

• The pose service can scale to zero on Fargate. A run wakes it and probes health with durable sleeps, so no request is held open while the image pulls. A 5-minute reconciler scales it in using analysis state in the database as the source of truth, and re-reads after scaling in to close the race with new work.
• Before the pose pass, a cheap vision model looks at 3 frames per clip (at most 8, no wider than 1024 px) and rejects footage that isn't a bowling action. Frames are seeked, not decoded, because decoding a 4K clip to its middle costs more than the check saves.
• Every AI task has its own model setting, and token cost is priced when read, so correcting a price re-prices history instead of leaving a month of reports priced wrong.

Uploads follow the same pattern: reserve one credit for the whole submission, upload clips straight to storage with retries on the same key, then confirm. A lost confirm response is reconciled by re-reading the row, never by reserving again, and a cron refunds uploads that never finished.