Why Our Credit Ledger Can't Double-Spend

By Govind Sharma

The Problem

Stamina sells usage in credits. One brand's wallet is spent at the same moment by enrichment workers, the AI reply agent, personalization jobs and workflow blocks. Jobs retry. Credit grants expire on different dates. Stripe webhooks arrive late, out of order and more than once. And two billing versions run side by side, chosen per brand.

I designed the credit ledger (billing v3) to make three things impossible: spending a credit twice, reviving one that expired, and letting Stripe and our database disagree for longer than one webhook or one nightly run.

The Data Model

• credit_wallet: one per brand.
• credit_bucket: one row per grant, with its own expiry and remaining amount, stored as numeric(20,4), never floats.
• credit_ledger_entry: an append-only list of signed deltas.
• credit_operation: one row per spend, grant or refund, unique on its idempotency key.

The balance is not a column. It is the sum of remaining amounts over unexpired buckets, so a grant stops counting the instant it expires, without a cron racing a spend.

Spending Exactly Once

Every spend runs in a SERIALIZABLE transaction and starts by inserting its operation row. If the key already exists, the stored result is returned and nothing else happens; that is what makes a retried job safe. Otherwise buckets are locked soonest-expiring first and drained in that order, and the per-user quota is checked in the same transaction.

export async function consume(walletId: string, amount: Decimal, key: string) {
  return withSerializableRetry(async (tx) => {
    const op = await tx.query(
      `INSERT INTO credit_operation (idempotency_key, wallet_id, amount)
       VALUES ($1, $2, $3)
       ON CONFLICT (idempotency_key) DO NOTHING
       RETURNING id`,
      [key, walletId, amount],
    );
    if (op.rowCount === 0) return loadStoredResult(tx, key); // a retry

    const buckets = await tx.query(
      `SELECT id, amount_remaining FROM credit_bucket
        WHERE wallet_id = $1 AND expires_at > now() AND amount_remaining > 0
        ORDER BY expires_at
        FOR UPDATE`,
      [walletId],
    );
    return drainSoonestExpiringFirst(tx, op.rows[0].id, buckets.rows, amount);
  });
}

Serialization failures (SQLSTATE 40001) are retried with the same key, backing off 25 ms, 75 ms, 200 ms, 1 s and 2 s with jitter. A failure rolls the whole transaction back, including the operation row, so an error is never cached under a key as if it were a result.

Why serializable, and not a balance column with a lock

A single balance column with SELECT … FOR UPDATE is simpler, and it would have been enough without expiry. With expiring grants, the question "how much can this brand spend right now" depends on several rows and on the clock. Serializable isolation lets Postgres prove that concurrent spends saw a consistent set of buckets; the cost is retries under contention, which the backoff absorbs.

An Append-Only Ledger, Enforced

"Append-only" is a convention until the database enforces it. A trigger raises on any UPDATE or DELETE of a ledger entry. Refunds are new entries, capped at the original amount minus earlier refunds, each with its own key. A refund goes into a bucket that inherits the original bucket's expiry, so refunding a failed job can't bring expired credit back to life.

The one legitimate exception is deleting a tenant. Rather than disabling the trigger (which needs table ownership and an exclusive lock) or switching replication roles (which needs superuser and turns off foreign keys), the purge sets a transaction-local flag that the trigger checks. Nothing outside that one transaction can delete a ledger row.

Charge, Act, Refund

There are no holds in the ledger. Callers charge first with a key that names the work, do the work, and refund per ledger entry if it fails. The AI reply agent charges with a key derived from the inbound message id, so a redelivered message can't pay twice. Table enrichment checks the worst case up front (rows × columns × unit cost) but charges after each batch for billable cells only, so a cell that found nothing costs nothing. When a wallet runs dry mid-job, a compare-and-set lets exactly one worker fail the job and notify the customer; the remaining batches acknowledge without calling a model.

The workflow engine needed more than that, because a run must not start side effects it can't pay for. It layers its own reservation state machine on top (pending → reserved → committed, or released), and holds a run's first steps until the trigger charge commits.

Staying in Step with Stripe

• Writes go through Stripe. Local subscription and invoice state changes only when the webhook arrives.
• Webhook events are stored once, keyed on Stripe's event id, then processed by workers that claim batches with FOR UPDATE SKIP LOCKED: 6 attempts, backoff from 60 seconds, then a dead letter. A reaper returns stranded rows every 5 minutes.
• Status only moves forward. Subscriptions rank incomplete < live < terminal; invoices rank draft < open < paid, void or uncollectible. A late payment_failed for an invoice we already saw paid is skipped, side effects included, so we never dun a customer who paid.
• Grants are idempotent by construction: the key for a subscription grant is the subscription, its period start and the price, so a redelivered renewal can't grant twice.
• A nightly reconciliation pages through Stripe, creates any invoice we missed, and reports subscription drift for a human rather than fixing it silently.

Trade-offs I Accepted

• Entitlement snapshots are cached without a TTL, so a catalog change reaches existing subscribers at their next billing event, which can be a full quarter away. It is documented, with a manual backfill, because the alternative was a hot-path read on every request.
• Charge-then-refund means a customer can briefly see credits leave and come back. Holds would hide that, at the cost of a second state machine in every caller.
• Metered usage reporting to Stripe isn't wired up yet; usage is rolled up hourly, and the consumer is the next piece of work.