Queues

A FlowMQ Queue provides a point-to-point messaging model for reliably distributing tasks among a pool of consumers. Unlike a stream, where multiple consumers can read the same message, a message sent to a queue is delivered to exactly one consumer. Once the consumer successfully processes the message and acknowledges it, the message is permanently removed from the queue.

This model is ideal for work-sharing, load balancing, and decoupling services in a distributed system. Common use cases include processing user requests, managing background jobs (like sending emails or resizing images), and offloading tasks to worker services.

The Competing Consumer Model

The core concept behind a queue is the competing consumer pattern. Multiple consumer instances connect to the same queue, and the broker distributes incoming messages among them.

Key behaviors of this model include:

• Load Balancing: The broker automatically distributes messages across all available consumers, spreading the processing load. If you add more consumers, they will immediately start sharing the work.
• Exclusive Delivery: Each message is delivered to only one consumer. This prevents different workers from accidentally processing the same task twice.
• At-Least-Once Delivery: To ensure reliability, a message is not considered fully processed until the consumer sends an acknowledgment (ack) back to the broker.

The Acknowledgment (Ack) Lifecycle

Acknowledgments are critical to ensuring no work is lost.

1. Delivery: The broker delivers a message to a consumer and temporarily marks it as "in-flight."
2. Processing: The consumer receives the message and performs its task.
3. Acknowledgment:
   • On Success: If the task completes successfully, the consumer sends an ack to the broker. The broker then permanently deletes the message from the queue.
   • On Failure: If the consumer crashes, disconnects, or fails to send an ack within a configured timeout, the broker assumes the task failed. It will then make the message visible again and redeliver it to another available consumer. This guarantees that work is eventually completed, even if a worker fails.

Binding to Topics

Similar to streams, queues do not receive messages directly from producers. A queue must be bound to a topic filter. When a producer publishes a message to a topic that matches the queue's filter, a copy of that message is placed in the queue for the next available consumer.

This decoupling allows for powerful and flexible routing. For example, a single message about a new order could be sent to a queue for fulfillment processing and simultaneously to a stream for archival and analytics, all without the producer knowing about the downstream consumers.

Producing to a Queue

Similar to streams, you do not publish messages directly to a queue. The producer's role is simply to send a message to a specific topic.

The process is as follows:

1. A Queue is created and bound to a topic filter (e.g., image.jobs.#).
2. A Producer publishes a message to a matching topic (e.g., image.jobs.resize).
3. The FlowMQ broker sees that the topic matches the filter for the queue and routes a copy of the message to it.

This indirect approach means the producer remains completely decoupled from the queue and its consumers.

Consuming from a Queue

Consuming from a queue involves a client connecting to the queue and processing messages one by one.

1. A Consumer client subscribes to a specific queue by name (e.g., image-processing-jobs).
2. The broker delivers the next available message to the consumer.
3. The consumer processes the message (e.g., resizes the image).
4. Upon successful completion, the consumer sends an acknowledgment (ack) to the broker.
5. The broker receives the ack and permanently deletes the message from the queue.

If multiple consumers subscribe to the same queue, they form a competing consumer group. The broker will automatically load balance messages across all connected consumers, ensuring that work is distributed efficiently.

When to Use a Queue vs. a Stream

Choose a Queue when:

• You need to distribute work among a group of identical workers.
• You want to ensure each message (or task) is processed exactly once by only one consumer.
• You need to load balance incoming requests across multiple service instances.
• You are building a system that needs to be resilient to consumer failures.

Choose a Stream when you need to broadcast messages to multiple independent consumers or when you need a replayable log of events.