Messaging & Event Streaming

A message queue (RabbitMQ, SQS, ActiveMQ) treats each message as a discrete unit of work. Once a consumer acknowledges processing, the message is removed from the queue. Multiple competing consumers pull from the same queue and each message goes to exactly one consumer—classic work distribution.

An event stream (Kafka, Kinesis, Pulsar) appends events to an immutable, partitioned log. Consumers read at their own offset; messages are not deleted on read (retention is time- or size-based). The same event can be replayed by new consumers, reprocessed after a bug fix, or fan-out to many independent consumer groups without duplicating writes.

flowchart LR
  subgraph Queue["Message Queue"]
    Q1[Msg A] --> C1[Consumer 1]
    Q2[Msg B] --> C2[Consumer 2]
    Q3[Msg C] --> C3[Consumer 3]
  end

  subgraph Stream["Event Stream (log)"]
    L[Partition log\noffset 0..N]
    L --> G1[Consumer Group A\noffset 42]
    L --> G2[Consumer Group B\noffset 17]
    L --> G3[New replay consumer\noffset 0]
  end

Pub/sub vs point-to-point

Point-to-point: one producer, one logical queue, N competing consumers—each message processed once. SQS standard queues and RabbitMQ work queues follow this model.

Pub/sub: one producer publishes to a topic; every subscriber receives a copy. RabbitMQ fanout/topic exchanges, SNS→SQS fan-out, and Kafka consumer groups (each group gets all partitions) implement variants of pub/sub. The critical distinction: in Kafka, "subscriber" means a consumer group, not an individual consumer instance.

Work queue (competing consumers)

Producers enqueue tasks; a pool of workers competes to pull the next available message. Horizontal scaling is trivial: add workers until queue depth stabilizes. SQS + Auto Scaling Group and Celery + RabbitMQ are canonical implementations.

flowchart LR
  P1[Producer] --> Q[Task Queue]
  P2[Producer] --> Q
  Q --> W1[Worker 1]
  Q --> W2[Worker 2]
  Q --> W3[Worker 3]

Scaling rule: if queue depth grows linearly with traffic, you are under-provisioned. Target p99 queue wait < 30 seconds for interactive-adjacent tasks; batch jobs can tolerate minutes. At 10K tasks/min and 500 ms average processing time, you need ~84 concurrent workers (10,000 × 0.5 / 60 ≈ 83).

Fan-out

One event triggers multiple downstream actions without the producer knowing subscribers. SNS → multiple SQS queues, RabbitMQ fanout exchange → bound queues, Kafka topic → multiple consumer groups.

SNS + SQS fan-out handles ~30K messages/sec per region with automatic scaling. Each subscriber queue processes independently—slow analytics does not block fast notification delivery.

Priority queue

VIP or time-sensitive messages jump ahead of bulk traffic. Implementations vary:

• Separate queues — high-priority queue with dedicated workers (simplest, most reliable)
• RabbitMQ priority — x-max-priority header (0–255); head-of-line blocking risk
• Weighted polling — workers poll high queue 3× before low queue once
• SQS — no native priority; use separate queues + weighted consumers

Delay queue

Defer processing until a future time—retry backoff, scheduled reminders, order cancellation windows. SQS supports native delay up to 15 minutes per message. For longer delays, store in DynamoDB/Redis with TTL + sweeper, or use Step Functions wait states. RabbitMQ requires TTL + dead-letter exchange trick or delayed-message plugin.

Dead letter queue (DLQ)

Messages that fail processing after N retries route to a DLQ for inspection, manual replay, or alerting. Without a DLQ, poison messages block the queue indefinitely or silently drop.

# SQS redrive policy — send to DLQ after 3 failures
{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789:orders-dlq",
    "maxReceiveCount": 3
  }
}

# Consumer: always extend visibility timeout for long tasks
def process_message(msg):
    try:
        do_work(msg.body)
        sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=msg.receipt_handle)
    except TransientError:
        # Message returns to queue after visibility timeout
        raise
    except PermanentError:
        # Log and delete — or let maxReceiveCount send to DLQ
        logger.error("poison message", extra={"id": msg.message_id})
        sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=msg.receipt_handle)

Request-reply

Async RPC: client publishes request with a reply-to queue/correlation ID; worker responds on the reply queue. RabbitMQ direct reply-to and temporary exclusive queues support this natively. Kafka request-reply uses a dedicated reply topic + correlation ID in headers (higher latency, less common).

sequenceDiagram
  participant Client
  participant ReqQ as Request Queue
  participant Worker
  participant ReplyQ as Reply Queue

  Client->>ReqQ: publish(request, reply_to=ReplyQ, corr_id=abc)
  ReqQ->>Worker: deliver
  Worker->>Worker: process
  Worker->>ReplyQ: publish(response, corr_id=abc)
  ReplyQ->>Client: deliver matching corr_id

Choose Kafka when

• Throughput exceeds 100K events/sec sustained
• Multiple consumer groups need the same event stream (analytics + search + fraud)
• Event replay, CDC, or stream processing (Kafka Streams, Flink) is required
• You can invest in partition design, retention tuning, and broker operations

Choose RabbitMQ when

• Complex routing (topic/direct/fanout exchanges, headers routing)
• Low-latency task dispatch (<5 ms) with moderate throughput (<50K/s)
• Request-reply, priority queues, or per-message TTL needed out of the box
• Team already runs Erlang-friendly infra or uses CloudAMQP

Choose SQS when

• AWS-native stack; zero desire to operate message brokers
• Work queue pattern with Auto Scaling; throughput per queue <3K/s (shard queues if higher)
• Decoupling microservices with DLQ + SNS fan-out; replay not required
• Message size <256 KB; latency tolerance 50–200 ms

flowchart TD
  Start[Need async messaging?] --> Replay{Need replay / stream processing?}
  Replay -->|Yes| Kafka[Kafka / Kinesis / Pulsar]
  Replay -->|No| AWS{All-in on AWS?}
  AWS -->|Yes, simple tasks| SQS[SQS + SNS fan-out]
  AWS -->|No or complex routing| RMQ{Routing complexity?}
  RMQ -->|High| RabbitMQ[RabbitMQ]
  RMQ -->|Low, high throughput| Kafka
  RMQ -->|Low, moderate| SQS or RabbitMQ

Symptoms: growing consumer lag (Kafka), queue depth monotonically increasing (SQS), RabbitMQ memory alarms, OOM kills on consumers, or cascading timeouts as downstream retries amplify load.

Detection metrics

Solutions (in order of preference)

1. Scale consumers — add worker instances until lag stabilizes. Kafka: add consumers up to partition count (max parallelism = partitions). SQS: Auto Scaling on queue depth metric.
2. Rate-limit producers — token bucket at API gateway when downstream saturated. Better to reject requests with 429 than accept and lose them.
3. Increase partition/queue count — horizontal shard for Kafka partitions or multiple SQS queues.
4. Drop or sample — for non-critical telemetry, sample 10% under pressure. Never drop payment or auth events without explicit product approval.
5. Bounded buffers + blocking — reactive streams, Go channels with capacity, Java BlockingQueue. Producer blocks when buffer full—backpressure propagates synchronously.
6. Circuit breaker on publish — stop producing when broker health degrades; spill to local disk buffer (Kafka producer buffer.memory) with timeout.

// Kafka consumer: reduce max.poll.records under backpressure
props.put("max.poll.records", 100);        // default 500
props.put("max.poll.interval.ms", 300_000); // allow long processing

// SQS: reduce batch size + increase visibility timeout
ReceiveMessageRequest req = ReceiveMessageRequest.builder()
    .queueUrl(url)
    .maxNumberOfMessages(1)           // process one at a time when struggling
    .waitTimeSeconds(20)              // long polling — fewer empty receives
    .visibilityTimeout(300)           // 5 min for heavy jobs
    .build();

flowchart TD
  P[Producer] -->|publish rate| B[Broker / Queue]
  B -->|deliver| C[Consumer pool]
  C -->|slow processing| L[Consumer lag grows]
  L --> A1[Scale consumers]
  L --> A2[Rate-limit producers]
  L --> A3[Add partitions / queues]
  L --> A4[Sample non-critical traffic]
  A1 --> B
  A2 --> P

Ordering levels

Kafka partition key strategy

Messages with the same key hash to the same partition and are consumed in produce order. Use user_id, order_id, or account_id as key—not a constant, which forces single-partition bottleneck.

# Preserve order per user across account updates
producer.send(
    topic="account-events",
    key=str(user_id).encode(),  # same user → same partition
    value=event_json.encode()
)

# Anti-pattern: null key or constant key → all messages to partition 0
producer.send(topic="events", key=b"all")  # 1M/s becomes impossible

SQS FIFO constraints

FIFO queues guarantee order within a MessageGroupId and exactly-once processing (dedup window 5 min). Throughput: 300 msg/s per message group, up to 3,000 msg/s per queue with batching. Hot group (single group ID) becomes a serial bottleneck—same problem as single Kafka partition.

The duplicate problem

At-least-once is the production default. Consumer processes message → crashes before ACK → message redelivered → duplicate charge, duplicate email, double inventory decrement. The fix is never "hope duplicates don't happen"— it is idempotent consumers.

Idempotent consumer pattern

1. Every message carries a unique idempotency_key (UUID, event ID, or business key)
2. Before side effects, check a dedup store (Redis SET, DB unique constraint, DynamoDB conditional write)
3. If key seen → ACK and skip; if new → process + record key atomically with side effect

def handle_payment_event(event):
    key = event["idempotency_key"]  # e.g. payment_id

    # Atomic check-and-set — Redis or DB unique index
    if redis.setnx(f"processed:{key}", 1):
        redis.expire(f"processed:{key}", 86400 * 7)
    else:
        return  # duplicate — safe to ACK

    charge_customer(event["amount"], event["customer_id"])
    # If crash here, retry re-enters but setnx fails → no double charge

Kafka exactly-once semantics (EOS)

Kafka 0.11+ offers idempotent producers (enable.idempotence=true) preventing duplicate writes on retry, and transactions for atomic read-process-write across topics. EOS applies within Kafka—your DB write is still at-least-once unless you use transactional outbox or two-phase commit.

sequenceDiagram
  participant App
  participant DB
  participant Outbox as Outbox Table
  participant Relay as Outbox Relay
  participant Kafka

  App->>DB: BEGIN TX
  App->>DB: UPDATE orders SET status='paid'
  App->>Outbox: INSERT event (same TX)
  App->>DB: COMMIT
  Relay->>Outbox: poll unpublished events
  Relay->>Kafka: publish
  Relay->>Outbox: mark published

Transactional outbox

Write business state + outbox row in one DB transaction. Separate relay process publishes to Kafka and marks sent. Guarantees: if DB commit succeeds, event eventually publishes; no orphan publishes without DB state. De facto standard for microservices needing reliable event emission.