Message Queues & RabbitMQ

The big idea

At a busy restaurant, the waiter doesn't stand in the kitchen waiting for each dish. They pin the order ticket on a rail and go back to serving customers. Cooks take tickets from the rail when they're ready.

The ticket rail is a message queue:

• Producer (waiter): puts messages in.
• Queue (rail): holds them safely, in order.
• Consumer (cook): takes messages out and does the work.

A message queue decouples the producer from the consumer

Why use a queue?

Point-to-point vs publish/subscribe

• Work queue: "resize this image". Only one worker should do it.
• Pub/Sub: "a user signed up". Email, analytics and CRM all want to know.

RabbitMQ in one picture

In RabbitMQ, producers never send directly to a queue. They send to an exchange, which routes messages to queues using bindings.

Code: producer and consumer (Node.js, amqplib)

// producer.js
import amqp from "amqplib";

const connection = await amqp.connect("amqp://localhost");
const channel = await connection.createChannel();
await channel.assertQueue("emails", { durable: true }); // survives broker restart

channel.sendToQueue(
  "emails",
  Buffer.from(JSON.stringify({ to: "ana@mail.com", template: "welcome" })),
  { persistent: true, messageId: crypto.randomUUID() }, // write to disk
);

// consumer.js
const channel = await connection.createChannel();
await channel.assertQueue("emails", { durable: true });
channel.prefetch(10); // at most 10 unacknowledged messages per worker

channel.consume("emails", async (msg) => {
  const job = JSON.parse(msg.content.toString());
  try {
    await sendEmail(job);
    channel.ack(msg);                 // ✅ done: remove from the queue
  } catch (error) {
    channel.nack(msg, false, false);  // ❌ failed: send to the dead-letter queue
  }
});

Acknowledgements: don't lose messages

Because a message can be delivered more than once (the worker processed it, then crashed before acking), consumers must be idempotent: processing the same message twice must be harmless.

// Idempotent consumer: remember processed message IDs
if (await db.processedMessages.exists(msg.properties.messageId)) return channel.ack(msg);
await db.transaction(async (tx) => {
  await doTheWork(tx, job);
  await tx.processedMessages.insert(msg.properties.messageId);
});
channel.ack(msg);

Retries and dead-letter queues

Some messages fail temporarily (an API is down); some fail forever (bad data). Don't retry forever, and don't lose them.

• Exponential backoff: wait 30 s, then 2 min, then 10 min… so a struggling service can recover.
• Dead-letter queue (DLQ): messages that keep failing are parked for a human to inspect and replay.
• Poison message: a message that crashes every consumer. The DLQ stops it from blocking the queue forever.

Delivery guarantees

Popular message brokers

Key takeaways

• Queues decouple producers and consumers, smooth traffic spikes and make systems resilient.
• Work queues send each message to one consumer; pub/sub sends it to every subscriber.
• RabbitMQ routes messages through exchanges (direct, topic, fanout) to queues.
• Ack after processing, make consumers idempotent, and use retries with backoff plus a dead-letter queue.
• "Exactly once" in practice = at-least-once delivery + idempotent processing.