What is Message Queue?

Decoupling the sender from the worker

When an online shop accepts an order, stock must be reserved, an invoice generated, the warehouse notified and a confirmation email sent. Doing all of that inside the order request is slow and fragile: if the email provider is down, the order fails too. A message queue breaks the chain. The order service drops a message and moves on; each downstream service picks messages up at its own pace.

• Producer: creates a message and sends it to the queue.
• Message broker: the server or managed service that accepts, stores, routes and hands out messages.
• Consumer: receives a message, processes it and tells the broker it is done.

This separation buys three things. Temporal decoupling: if a consumer is down for maintenance, messages wait rather than disappear. Load levelling: a burst of orders during a sale piles up in the queue and is worked off at a steady rate. Horizontal scalability: when a queue falls behind, you add consumers.

Acknowledgements and at-least-once delivery

A broker does not delete a message the moment it hands it out. The message stays in flight until the consumer sends an acknowledgement (ack). If the consumer crashes or the ack does not arrive in time, the message becomes visible again and goes to another consumer.

msg = queue.receive(visibility_timeout=30)   # hidden from others for 30 s
process_order(msg.body)                      # side effects happen here
queue.ack(msg)                               # only after the work is done

That prevents lost messages, but it has a price: if the work completes and the connection drops just before the ack, the message is processed a second time. Delivery guarantees are usually described in three tiers:

Most real systems deliver at least once. For SQS standard queues, the AWS documentation warns that a message copy can occasionally be received again and tells you to design applications to be idempotent. A consumer that sees the same message twice must not charge the card twice or issue a second invoice. Idempotency is the answer: record each message ID and skip the ones you have already handled.

Queues versus topics

In a point-to-point queue, each message goes to exactly one of the consumers; they are competing workers sharing the load. When a fact such as “order created” must reach billing, the warehouse and analytics separately, you use publish/subscribe instead, where every subscriber gets its own copy. That second model is the backbone of event-driven architecture. Brokers such as RabbitMQ let you build both through exchanges and queue bindings.

Poison messages and backpressure

A malformed message that crashes the consumer every time creates an endless retry loop. The standard remedy is a dead letter queue: after a set number of failed attempts, the message is moved aside for inspection and can be replayed once fixed. Watch two numbers in production: queue depth and the age of the oldest message. A queue that keeps growing means consumers cannot keep up, and you need either more consumers or backpressure that slows the producers down.

When you don't need one

Operations whose result the user must see immediately, such as logging in or checking stock, are simpler as plain request/response. Running a separate broker for a small single-server app is often more infrastructure than the problem deserves; a database-backed job queue may cover the same need. And if an external system only needs to notify you of events, a webhook can be enough.