What It Takes to Keep Every Mailbox Inside Its Send Limits

By Govind Sharma

The Problem

A cold-email platform lives or dies on deliverability, and deliverability starts with each mailbox behaving like a person: a daily cap, and a minimum gap between sends. Those limits belong to the mailbox, but a mailbox is shared by every campaign that uses it, and campaigns are spread across many sender processes.

The symptom that started this work was concrete. With a 16-process sender, one mailbox sent 6 emails within 1 to 2 seconds against a 7-minute minimum gap. Every process had read the same "this mailbox is free" snapshot, sent, and recorded the send afterwards. The platform now sends about 30,000 emails a day, and the fixes below are what keep each mailbox inside its limits at that rate.

Constraints

• No lock may be held across the SMTP call, which can take seconds.
• The send-stats table is a TimescaleDB hypertable, so a unique index must include the partition column, which rules out a unique-constraint reservation.
• Campaigns are sharded by a hash of the campaign id, but mailbox state is global.
• When we don't know whether a send went out, we must err towards under-sending.

Generation One: Reserve Before Send, in Postgres

The first fix moved the check and the record into one database function, called before sending. It takes a non-blocking, transaction-scoped advisory lock on the mailbox, re-checks today's count and the last send time on committed data, and inserts an unconfirmed reservation. It returns reserved, skipped-locked, skipped-cooldown or skipped-daily-limit. The sender confirms after a successful send or deletes the reservation on a definite rejection; an unknown outcome keeps it.

• Session-level advisory locks were rejected: they leak on pooled connections.
• A separate status table was rejected: two sources of truth for one fact.
• Redis was rejected at this stage: at the measured database load it solved a problem we didn't have.

We load-tested it before trusting it. 10,000 concurrent reserve calls against 5 mailboxes produced exactly 75 reservations and 0 errors. An end-to-end run sent 660 emails across 60 mailboxes, every pair spaced 60 to 61 seconds apart, never under 60. The test also found the next bottleneck: on the small database, max_connections was 25, and that limit, not the lock, capped throughput.

Generation Two: Atomic Redis Guards for a Sharded Scheduler

The rewritten scheduler runs many shards, and several of them can pick the same mailbox. It reserves in a fixed order (choose an inbox, reserve a campaign slot, reserve an account slot, then claim the job optimistically) and releases whatever it already took if a later step fails. Each reservation is a single Lua script, so the check and the increment can't interleave across shards.

-- KEYS[1] = last-send key for one mailbox
-- ARGV[1] = now (ms), ARGV[2] = minimum gap (ms)
local last = tonumber(redis.call('GET', KEYS[1]) or '0')
local now = tonumber(ARGV[1])
if now - last < tonumber(ARGV[2]) then
  return 0 -- too soon; caller tries another inbox
end
redis.call('SET', KEYS[1], now, 'PX', 172800000)
return 1

The gap guard was added after staging showed 4 to 6 emails within 15 to 30 seconds despite a 5 to 10 minute gap: the daily counters were atomic, but the last-send key was written by several shards. Two smaller decisions mattered as much as the scripts:

• Release is conditional. If the provider says the mailbox hit a quota or a rate limit, the slot stays taken, so a throttled inbox never looks more available than it is.
• Counters are rebuilt from Postgres on boot, and Redis errors fail open. Redis is a fast guard; Postgres remains the record.
• Each scheduler sleeps after its cycle instead of on a fixed interval, so one process's cycles can never overlap.

Never Sending Twice

Pub/Sub redelivers, and a worker can die after Gmail accepts a message but before the database knows. Before sending, the worker saves a send attempt id and a pre-generated Message-ID. A redelivery that finds a claimed job with an attempt id takes the crash-retry path: it finishes the post-send writes and never sends again. The ack contract is explicit: if the email went out but the writes failed, nack so the retry path runs; if nothing went out, ack and return the job to pending. Exactly one component owns each retry.

Blaming the Right Party

Gmail errors are ambiguous, and the wrong reading either duplicates mail or keeps hammering a dead address from many of the customer's domains. Every failure is classified into one of six classes, and each has one response:

• Account quota: park the inbox for the day.
• Rate limited: cool the inbox down for 4 hours (Gmail's own block can last 24 to 48).
• Account auth: disconnect the inbox and reroute the lead without using up one of its attempts.
• Recipient: cancel the job, suppress the address and publish a bounce.
• Content: fail the job.
• Transient: retry.

A per-inbox circuit breaker trips after 5 failures in 30 minutes, but only if the system as a whole had at least 10 recent successes, so a general outage doesn't trip every inbox at once. Two trips in a day escalate to disconnecting the inbox. Bounce reports, which had been silently dropped because replies were matched only on In-Reply-To, are now parsed to recover the original Message-ID, suppress the address and lower the inbox's health score.

Rotating the Sending IP

The submission IP appears in the Received header, where blocklists and per-IP velocity checks read it, and the original hosting gave every brand the same handful of IPs. We moved senders onto hosts with a pool of Elastic IPs, leased atomically from Redis: each IP retires after a random 8 to 15 sends, and a slot is only rotated once no live leases remain. A send is never blocked for lack of an IP; it falls back to the primary address.

Staging then broke the obvious design. AWS kept handing back recently released addresses: 498 redraws returned only 4 distinct IPs, which made a 7-day denylist useless. The fix was to park retired IPs, allocated but unattached, for 15 minutes before releasing them.

What I'd Do Next

• Inbox selection takes the first eligible inbox. Least-used-first would spread volume more evenly and is a one-query change with a measurable deliverability effect.
• The campaign-level random delay collapses to almost nothing at scale; the jitter should move to the mailbox, where it matters.
• List-Unsubscribe headers and open and click tracking are still missing, and the first of those is now a mailbox-provider requirement.