Core Concepts
Unified Routing Model
At the heart of FlowMQ is its Unified Routing Model. This model seamlessly combines streams, queues, and pub/sub into a single engine.
A producer sends a message to a Topic. From there, FlowMQ can route that message to a Queue, one or more Subscriptions, and a Stream—all at the same time.
Here’s the flow:
- Producer: This is the application or service that sends a message to a single, logical Topic.
- Topic: In this model, the Topic acts as the central entry and initial distribution point. It's a core concept borrowed from the pub/sub paradigm.
- Unified Routing: From the Topic, the model allows messages to be routed in three different ways, catering to the different paradigms:
- Topic --> Queue: This path represents the message queue paradigm. Messages sent to the Topic are then placed into a Queue.
- Queue: A Queue typically ensures that each message is processed by only one Consumer (point-to-point delivery). It's ideal for task distribution where you want to ensure a message is handled once.
- Topic --> Subscription: This path directly embodies the pub/sub paradigm.
- Subscription: Multiple Consumers can subscribe to a Topic via a Subscription. Each Consumer attached to a Subscription will receive a copy of every message published to the Topic. This is great for broadcasting information to multiple interested parties.
- Topic --> Stream: This path represents the data stream paradigm.
- Stream: A Stream typically handles a continuous flow of data. Consumers can read from different points in the stream, replay messages, and process data in real-time or batches. This is suited for scenarios like event sourcing, real-time analytics, or log aggregation.
- Consumer: This is the application or service that receives and processes the messages from the Queues, Subscriptions, or Streams.
In essence, the "Unified Routing Model" works like this:
A Producer doesn't need to decide upfront whether its message is for a queue, a direct pub/sub broadcast, or a stream. It simply publishes to a Topic. The platform then allows different types of consumers to access these messages according to their needs:
If a Consumer needs traditional load-balanced, one-message-at-a-time processing, it connects to a Queue that is fed by the Topic. If multiple Consumers need to receive every message for independent processing (e.g., notifications, different analytical tasks), they each create a Subscription to the Topic. If Consumers need to process a continuous flow of events, potentially with replay capabilities or complex event processing, they connect to a Stream that is sourced from the Topic.
The Universal Post Office
To understand FlowMQ's Unified Routing Model, let's start with a simple analogy.
Imagine a universal post office that can instantly understand and translate between any postal system in the world. You could send a package using the American ZIP code system, and your recipient could pick it up using British Postcodes, with the post office handling the conversion seamlessly.
This is exactly what FlowMQ does for messaging. It provides a single, consistent way to name and access your message streams, regardless of the protocol your application uses (like MQTT or Kafka). You publish to one logical channel (Topic), and FlowMQ ensures it's available to be consumed by any other supported protocol.
Key Components
The model is built on two simple but powerful components.
Namespace
A Namespace is your private, top-level container for all your messaging resources within FlowMQ.
- Isolation: All your topics, credentials, queues, streams, and subscriptions are securely isolated within your Namespace.
- Endpoint: A Namespace provides a single, unique service URL. All your applications (MQTT, Kafka, or AMQP) will point to this same URL to connect to FlowMQ.
Message
A Message (also known as a record or event) is the unit of data that flows through the system. It consists of a payload (the actual data you send) and optional metadata or attributes.
Topic
A Topic is the core of the routing model. It is a named, logical channel within your Namespace where producers send messages.
The Topic in FlowMQ is more similar to MQTT than Kafka, it not couples with how messages storage, partition, and consumption.
It is like a specific P.O. Box in our universal post office. This P.O. Box has a single, simple address (e.g., orders), but it can accept mail delivered by any courier (MQTT, AMQP, Kafka) and can be opened by a recipient using a key from any of those same couriers.
Producer / Publisher
A Producer (or Publisher) is a client application that creates and sends messages to a specific Topic in FlowMQ.
Consumer / Subscriber
A Consumer (or Subscriber) is a client application that attaches to a Queue, Subscription, or Stream to receive and process messages.
Queue
A Queue provides point-to-point, load-balanced message delivery. When multiple consumers listen to the same queue, FlowMQ ensures each message is delivered to only one of them, effectively distributing the workload.
A Queue in FlowMQ can be consumed by AMQP and MQTT clients.
Subscription
A Subscription delivers a copy of every message from a Topic to its attached consumers. This is a classic publish/subscribe (pub/sub) fan-out pattern.
Stream
A Stream is a durable, ordered, and replayable log of all messages published to a Topic. Consumers can read the stream from any point in time, rewind to re-process old messages, or read in real-time. This is analogous to a Kafka topic.
A Stream in FlowMQ can be consumed by Kafka clients.
Topic Filter
A Topic Filter allows consumers to subscribe to multiple topics using wildcards (e.g., devices/+/temperature or alerts/#). This is a powerful feature common in protocols like MQTT that enables a single consumer to receive messages from a dynamic range of topics.
Benefits of the Model
This unified approach offers significant advantages over managing separate messaging systems.
Drastic Simplicity: Instead of managing separate systems for queues, pub/sub, and streaming, You only need to learn and manage a single platform can handle all these messaging needs.
Ultimate Flexibility: Producers and consumers are fully decoupled. You can easily switch out or add new services using different protocols without re-architecting your entire messaging infrastructure.
Reduced Overhead: Eliminate the need for bridges, connectors, and custom glue code, which saves significant development time and reduces potential points of failure.
Future-Proof: As FlowMQ adds support for new protocols such as AWS SQS/SNS, etc, your existing topics automatically become accessible to them, protecting your investment.
- Flexibility: Producers can publish data without being tightly coupled to the consumption pattern. Consumers can choose the paradigm that best suits their processing needs.
Next Steps
Now that you understand the core concepts and the lifecycle of a message, you're ready to dive deeper into the high-level system design.
- Learn how to mapping the model to different Protocols
- Learn more in our Architecture Overview