Distributed Queue Design: Ordering, Retries, and Throughput

A distributed queue looks simple: producers put messages in, consumers take them out. In reality, it is one of the hardest problems in distributed systems because it sits at the intersection of durability, ordering, latency, and failure recovery — all of which conflict.

Consider what happens when a consumer crashes mid-processing. If the broker already deleted the message, you lose it permanently. If the broker re-delivers it, you may process it twice. The gap between "the consumer received it" and "the consumer finished processing it" is the fundamental tension that shapes every design decision in this space.

Three properties are in conflict:

1. Throughput — partitioned logs (Kafka) can handle 1M+ msgs/sec but sacrifice global ordering.
2. Ordering guarantees — global ordering (single partition) caps throughput at ~50K msgs/sec and creates head-of-line blocking.
3. Delivery semantics — exactly-once delivery requires distributed transactions; at-least-once is the practical default that forces idempotent consumers.

What most candidates get wrong: they conflate the queue system design (Kafka, SQS) with the application-level guarantees. The queue provides at-least-once delivery as a property; exactly-once processing is something your consumer code must implement via idempotency keys, deduplication stores, or transactional outbox patterns. These are orthogonal concerns and conflating them is a senior-level interview trap.

The second common mistake: treating a queue as merely a transport layer rather than a system boundary. A well-designed queue system defines the contract between producer and consumer: what ordering scope exists (global, per-partition, per-key), what the failure recovery story is (retries, DLQ, manual replay), and what the backpressure mechanism is when consumers fall behind.

Two dominant paradigms exist: the partitioned log (Kafka, AWS Kinesis) and the traditional message queue (RabbitMQ, AWS SQS, Azure Service Bus).

Partitioned log: messages are appended to an ordered, immutable log. Consumers read by offset and do not delete messages on read — retention is time-based. This means multiple independent consumer groups can read the same topic concurrently (fan-out is free), you can replay history by resetting offsets, and you can add new consumers later without any producer changes. The cost: consumers must track their own offset; no per-message TTL; throughput is bounded by partition count.

Traditional queue: messages are deleted on acknowledgment. The broker routes messages to available consumers (push model), enabling simple work-queue patterns. Visibility timeout (SQS) or consumer ack (RabbitMQ) determines when a message is re-queued. The cost: replay requires DLQ + requeue scripts; fan-out requires explicit topic subscriptions or SNS fan-out pattern.

Choosing the paradigm:

• Use Kafka when: multiple independent consumers need the same stream, replay/auditing is required, throughput > 50K msgs/sec, or you need log compaction (event sourcing, CDC).
• Use SQS/RabbitMQ when: simple work queue, messages should auto-delete after processing, per-message TTL is needed, or team is small and doesn't want Kafka operational overhead.

Distributed queues offer three delivery guarantees, and the distinction between delivery and processing is the key insight that separates senior from staff-level answers.

At-most-once: commit offset before processing. Message is lost if the consumer crashes during processing. Acceptable for lossy telemetry or metrics where occasional loss is tolerable.

At-least-once: commit offset after processing. If the consumer crashes after processing but before committing, the message is re-delivered and processed again. This is the safe default — it preserves correctness at the cost of duplicate processing. Your consumer must be idempotent.

Exactly-once (the myth): Kafka introduced exactly-once semantics (EOS) via the transactional API (enable.idempotence=true + transactional.id). This prevents duplicates at the broker level — the producer won't write the same message twice even on retry. However, end-to-end exactly-once requires the consumer's side effect to also be idempotent or transactional. If the consumer writes to Postgres and the write succeeds but the Kafka offset commit fails, the message is re-delivered and the write happens again. Kafka EOS solves the producer↔broker leg; you must solve the broker↔consumer leg yourself.

Practical recommendation: use at-least-once delivery + idempotent consumers. Implement idempotency via a deduplication key stored in Redis (TTL = 2× max retry window) or a unique constraint in your database. For financial systems, use the transactional outbox pattern on the consumer side: write the business result and the processed-offset atomically in the same DB transaction.

Partition count is the single most important architectural decision in Kafka — and it is irreversible without reprocessing.

Partition key selection determines ordering scope. Choosing user_id as the key ensures all events for a given user land in the same partition and are processed in order. Choosing random (or null) maximizes throughput but gives no ordering. The rule: pick the smallest entity whose events must be ordered relative to each other.

Partition count drives max parallelism. A consumer group can have at most N consumers where N = number of partitions. If you have 12 partitions and 16 consumers, 4 consumers are idle. If you later need more parallelism, you must increase partitions — which reshuffles key → partition assignments and breaks any downstream ordering guarantees temporarily.

Sizing rule of thumb: plan for 2–3× the throughput you need today. For a 100K msgs/sec stream at 1KB average message size: each partition handles ~10MB/sec comfortably → need ~10 partitions, plan for 30.

Consumer group lag as the SLO: set alerts when consumer group lag exceeds (desired max latency) / (average message size / consumer throughput). A lag of 1M messages with a 10K msgs/sec consumer means 100 seconds of latency — act before it hits this threshold by adding consumers or scaling compute.

Consumer lag spiral is the most common production failure. It happens when consumers are processing slower than producers are writing — the lag grows, consumers under memory pressure OOM, restart, reprocess from their last committed offset (re-reading already-lagged messages), and the lag grows further. The fix is a combination of: autoscaling consumers on lag metric (not CPU), bounded retry with exponential backoff, and circuit-breaking expensive downstream calls.

Retry storm: naive retry with zero backoff causes all failed messages to re-attempt simultaneously, flooding the downstream system. Use exponential backoff with jitter: delay = min(cap, base * 2^attempt) + random(0, base). Implement via a retry topic per attempt level (topic.retry.1, topic.retry.2, topic.retry.3) — each topic has a Kafka consumer with a sleep before re-publishing to the original topic.

Poison messages: messages that always fail processing (bad schema, dependency permanently down) will cause consumers to retry indefinitely, blocking all subsequent messages in that partition. The solution is a dead-letter queue (DLQ): after N failed retries (typically 5), route the message to the DLQ with metadata (original topic, original offset, error reason, timestamp). Alert on DLQ depth. Provide a replay tool that re-publishes DLQ messages to the original topic after the root cause is fixed.

Hot partitions: if one partition key is disproportionately active (e.g., a single power user generating 80% of events), one consumer handles 80% of load while others are idle. Solutions: add a random suffix to the key (user_id:random(0,N)) — this re-fans the load across partitions at the cost of losing strict ordering for that user; or pre-shard high-volume keys explicitly.