Message Brokers Explained: When You Actually Need RabbitMQ or Kafka

Here is a scene every growing app hits. Your checkout endpoint used to just save an order and return. Then you added an email receipt. Then inventory updates. Then a fraud check, an analytics event, and a webhook to the shipping provider. Now a single “place order” call fans out into six things, some of them slow, some of them flaky, and if the email service is down for 20 seconds your customer sees a spinning wheel and a failed checkout.

That pain is what message brokers exist to solve. This article explains what a message broker actually is, the two very different flavors you will meet (RabbitMQ and Kafka), when you genuinely need one, and when you are better off without one. Then we spin up both brokers in Docker and push real messages through them. Every command below was run before publishing, so the output you see is the output you get.

What a message broker actually does

A message broker sits between the code that produces work and the code that handles it. Instead of your checkout endpoint calling the email service directly and waiting, it drops a message (“send receipt for order 1042”) into the broker and moves on. Some other process picks that message up and does the work whenever it is ready.

That one move buys you a lot:

• Decoupling. The producer does not know or care who handles the message, or how many handlers there are.
• Buffering. If work arrives faster than it can be processed, the broker holds the backlog instead of dropping it.
• Resilience. If a consumer crashes mid-job, the message is not lost. It gets redelivered.
• Load spreading. Ten workers can pull from the same queue and share the load automatically.

The trade you make is complexity. You now run another piece of infrastructure, and your system becomes asynchronous, which is harder to reason about. That trade is worth it at a certain scale and a waste of time before it.

Two models: the queue and the log

RabbitMQ and Kafka are both called message brokers, but they work on fundamentally different ideas. Getting this distinction is the whole point.

RabbitMQ is a queue. Think of a to-do list. A producer puts a task on the list, the broker hands it to one available worker, the worker finishes and acknowledges it, and the task is deleted. The broker is smart: it routes messages, tracks acknowledgments, and redelivers failures. Consumers are simple. This is the classic “task queue” pattern, and it is great for distributing jobs to a pool of workers.

Kafka is a log. Think of an append-only ledger. Producers append messages to the end, and the broker keeps them for a configured retention period whether anyone reads them or not. Consumers track their own position (their “offset”) and read forward at their own pace. Nothing is deleted when it is read, so multiple independent consumers can read the same stream, and you can rewind and replay history. The broker is simple, the consumers are smart.

That single difference, delete-on-consume versus keep-and-replay, drives almost every other difference between them.

RabbitMQ vs Kafka at a glance

	RabbitMQ	Kafka
Core model	Message queue	Distributed log
After a message is read	Deleted	Kept until retention expires
Replay old messages	No	Yes
Routing	Rich (exchanges, bindings)	Simple (topic + partition)
Ordering	Per queue	Per partition
Typical throughput	Tens of thousands/sec	Millions/sec
Best at	Task queues, RPC, complex routing	Event streaming, analytics, replay
Mental model	Smart broker, simple consumer	Simple broker, smart consumer

Neither is “better.” They are tools for different jobs. Plenty of companies run both.

When you actually need one (and when you don’t)

Let me be honest, because this is where teams waste the most time. You probably do not need a message broker if:

• Your background jobs are low volume and a database-backed job table plus a worker loop handles them fine.
• A slow downstream call can just be awaited, or retried later with a simple cron.
• You are one service talking to one database. Adding Kafka here is resume-driven development.

You do start needing one when:

• A single user action must trigger several independent side effects and you do not want to wait for all of them.
• Traffic is spiky and you need a buffer so bursts do not knock over slow downstream services.
• You have multiple services that all care about the same events (an order was placed, a user signed up).
• You need to process a firehose of events for analytics, or you need to replay history to rebuild state.

Rough rule: reach for RabbitMQ when you are distributing discrete tasks to workers and want reliable delivery with flexible routing. Reach for Kafka when you are moving a high-volume stream of events that several consumers care about, or when replay matters.

Hands-on: RabbitMQ in 5 minutes

You need Docker. Start RabbitMQ with its management UI:

docker run -d --name rabbit -p 5672:5672 -p 15672:15672 rabbitmq:3-management

Give it about 15 seconds to boot. Now declare a durable queue (durable means it survives a broker restart):

docker exec rabbit rabbitmqadmin declare queue name=task_queue durable=true

queue declared

Publish a message to it:

docker exec rabbit rabbitmqadmin publish routing_key=task_queue payload='Process order #1042'

Message published

Confirm it is sitting in the queue, waiting for a worker:

docker exec rabbit rabbitmqctl list_queues name messages

name        messages
task_queue  1

Now consume it. In real code a worker stays connected and pulls continuously, but we can grab it by hand:

docker exec rabbit rabbitmqadmin get queue=task_queue ackmode=ack_requeue_false

+-------------+----------+---------------+---------------------+---------------+
| routing_key | exchange | message_count |       payload       | payload_bytes |
+-------------+----------+---------------+---------------------+---------------+
| task_queue  |          | 0             | Process order #1042 | 19            |
+-------------+----------+---------------+---------------------+---------------+

The message came back and the queue is now empty (message_count is 0). That ack_requeue_false is the acknowledgment: it tells RabbitMQ the work is done, so delete it. If your worker had crashed before acking, RabbitMQ would have handed the message to another worker instead of losing it. That redelivery guarantee is the heart of why you use a broker for important jobs.

Hands-on: Kafka in 5 minutes

Start a single-node Kafka (modern Kafka runs in KRaft mode, no ZooKeeper needed):

docker run -d --name kafka -p 9092:9092 apache/kafka:latest

Create a topic with 3 partitions. Partitions are how Kafka spreads a topic across machines and gives you parallelism:

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

Created topic orders.

Describe it to see the partition layout:

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --describe --topic orders

Topic: orders   PartitionCount: 3   ReplicationFactor: 1
    Topic: orders   Partition: 0    Leader: 1   Replicas: 1   Isr: 1
    Topic: orders   Partition: 1    Leader: 1   Replicas: 1   Isr: 1
    Topic: orders   Partition: 2    Leader: 1   Replicas: 1   Isr: 1

Produce five messages:

docker exec kafka bash -lc 'printf "order-1001\norder-1002\norder-1003\norder-1004\norder-1005\n" \
  | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders'

Now read them back from the start. Notice we can read the whole history, because Kafka kept every message:

docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders --from-beginning --timeout-ms 8000

order-1001
order-1002
order-1003
order-1004
order-1005

Here is the part that makes Kafka different. Read the topic as a named consumer group, then inspect where that group is:

docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group billing

GROUP    TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
billing  orders  0          5               5               0
billing  orders  1          0               0               0
billing  orders  2          0               0               0

CURRENT-OFFSET is where the group has read up to, LOG-END-OFFSET is the newest message, and LAG is how far behind it is. Lag is the single most important number to monitor on a Kafka system: growing lag means your consumers cannot keep up. The messages themselves are still on disk, so a second, unrelated consumer group could read the exact same five orders without affecting billing at all. That is the replay superpower a queue does not give you.

How to choose, quickly

• Distributing jobs to workers, need acks and retries, moderate volume, complex routing rules → RabbitMQ.
• High-volume event stream, multiple independent consumers, need replay or long retention → Kafka.
• Neither of those describes you yet → skip both. A database job table and a worker will carry you a long way, and you can add a broker the day you actually feel the pain.

Gotchas worth knowing before production

• A broker is infrastructure you now have to run. It needs monitoring, disk, and a plan for when it fills up. Managed options (CloudAMQP, Amazon MQ, Confluent Cloud, MSK) exist for a reason.
• Ordering is narrower than people expect. RabbitMQ orders within a queue, Kafka within a partition. If you need messages for one customer in order, route them by a key so they land on the same partition or queue.
• At-least-once delivery is the default. Messages can be delivered more than once after a failure, so make your consumers idempotent (processing the same message twice should be safe).
• Kafka partition count is not a knob you turn casually. It sets your maximum consumer parallelism and is awkward to change later. Plan it up front.

Wrapping up

A message broker is not a database, and it is not something to add on day one. It is what you reach for when one action needs to trigger many, when bursts need buffering, or when several services care about the same stream of events. RabbitMQ gives you a smart queue for reliable task distribution. Kafka gives you a replayable log for high-volume event streaming. As you just saw, trying each one is a single docker run and a handful of commands away, so when the day comes that you need one, you will already know which shape fits.