Queues: Decoupling the Request Path

The single most underrated scaling move: move work out of the request path. A signup that takes 2s because it sends email, resizes the avatar, and indexes the user is a signup that fails at 2s — the same work queued finishes in 50ms and the user sees a response immediately.

The delivery guarantee is a lie

No queue guarantees exactly-once. Every queue is at-least-once (the message might be delivered twice — retries, timeouts, consumer crashes) or at-most-once (drop on failure — fine for metrics, wrong for payments). Design consumers to be idempotent: processing the same message twice produces the same result.

def handle(job):
    if db.exists(job.id):      # idempotency check — already processed
        return
    process(job)
    db.mark_done(job.id)

The check-and-mark must be atomic (same transaction as the work) or the idempotency claim is fiction.

Choosing a queue

• SQS / Pub/Sub — managed, pay-per-use, right for most async work.
• RabbitMQ — when you need routing flexibility (topic exchanges, priority queues).
• Postgres (SKIP LOCKED) — for small systems, one less infrastructure piece; fine until it isn't (~10k jobs/sec).
• Kafka — a log, not a queue; right for event streaming, wrong for task processing.

The poison message problem

A message that crashes the consumer gets retried forever (in at-least-once) or dropped (at-most-once). Fix: a dead-letter queue after N retries — failed messages land there for inspection, the consumer moves on. No DLQ means a single bad message takes down your entire job pipeline.

::exercise(idempotent-consumer)

Field notes

• Queue depth is the metric — a queue that grows faster than consumers drain it is an outage in slow motion.
• Timeout on visibility (visibility_timeout) is what makes at-least-once work — a message invisible to other consumers while being processed.
• The queue consumer is the trust boundary — validate the payload again; "the producer sent it" isn't a type guarantee.