Topics

Interoperability is essential for FlowMQ to unify pubsub, queues, and streams in a single platform, while maintaining protocol-specific semantics across multiple messaging protocols. The key to this unification is the decoupling of message producers from consumers through an intermediary routing layer called Topics.

Design

Producers publish messages to topics. Consumers, however, do not consume from topics directly. Instead, a Queue or a Stream subscribes to one or more topics using a flexible topic filter. This architecture allows a single message to be routed to different destinations for different purposes—for example, to a queue for workload processing and simultaneously to a stream for archival.

The topic names and filters are designed following these principles:

• Straightforward: Simple, string-based, and case-sensitive topic names and filters.
• Hierarchical: A hierarchical topic structure using delimiters (/) for logical organization.
• Wildcards: Wildcard subscriptions (+ and #) to efficiently filter and route messages.
• Interoperable: Topics act as an abstraction layer that allows multiple protocols to communicate seamlessly.

Topic Names

A topic name is a hierarchical identifier used to route messages between producers and consumers. It consists of a series of levels separated by forward slashes (/), similar to a file path or URL. For example, sensors/us/california/temperature has four levels that describe the message's origin and type. This hierarchical structure, which follows the MQTT standard, allows for flexible and intuitive message routing while keeping the overhead minimal.

For example, a smart sensor might publish messages to topics like:

• sensors/us/california/sfo/temperature
• sensors/us/california/sfo/humidity
• sensors/eu/germany/berlin/temperature

Producers are only concerned with sending a message to the correct topic. They do not need to know which queues, streams, or consumers will ultimately receive it. Producers can only publish messages to specific topics and cannot use wildcards (+ or #).

Topic Filters

Queues and Streams receive messages by creating a binding to topics using a topic filter. These filters support wildcards, allowing a consumer to subscribe to a pattern of topics. This system is identical to the MQTT protocol's topic matching.

For example, a single filter like orders/# can match multiple topics like orders/eu/new and orders/us/cancelled. This flexible matching system is identical to the MQTT protocol's topic filtering, making it familiar to MQTT users while providing powerful routing capabilities to all FlowMQ clients.

There are two wildcard characters:

1. Single-Level Wildcard: +

The + symbol matches exactly one level in the topic hierarchy.

• Filter: sensors/+/california/+/temperature
• Matches:
  • sensors/us/california/sfo/temperature
  • sensors/ca/california/lax/temperature
• Does NOT Match:
  • sensors/us/arizona/phx/temperature (level 2, arizona, does not match california)
  • sensors/us/california/sfo/humidity (the last level, humidity, does not match temperature)

2. Multi-Level Wildcard: #

The # symbol matches zero or more levels, but it must be the last character in the filter.

• Filter: sensors/us/#
• Matches:
  • sensors/us/california/sfo/temperature
  • sensors/us/arizona/phx
  • sensors/us (matches zero subsequent levels)
• Does NOT Match:
  • sensors/eu/germany/berlin/temperature (does not start with sensors/us/)
  • data/sensors/us/california (does not start with sensors/us/)

Topic Mapping and Translation

FlowMQ transparently translates protocol-specific topic formats into its own canonical, slash-separated topic structure. This enables seamless interoperability between different messaging protocols - a message published from an MQTT client can be consumed by an AMQP or Kafka client without either client needing to be aware of the other's protocol.

MQTT Topic Mapping

Since FlowMQ's native topic filtering is identical to MQTT's, no translation is needed. MQTT topics are used directly within the broker, making the integration seamless and highly performant. An MQTT client publishing to sensors/us/temp is publishing to the exact same topic that a native FlowMQ consumer would subscribe to.

Kafka Topic Mapping

Unlike Apache Kafka's flat topic namespace, FlowMQ maps Kafka topics (which use . as a separator) into its hierarchical structure. For example:

Kafka Topic	Mapped FlowMQ Topic
user-activity	user-activity
prod.invoices	prod/invoices
events.alerts.critical	events/alerts/critical

For example, an MQTT client could subscribe to events/# to receive messages from both events.alerts.critical and events.alerts.warning Kafka topics.

AMQP Routing Key Mapping

AMQP 0-9-1 uses dot-separated routing keys for topic exchanges. FlowMQ translates these directly into its own topic structure by replacing the delimiter, preserving the routing hierarchy.

AMQP Routing Key	Mapped FlowMQ Topic
orders.eu.new.books	orders/eu/new/books
logs.error	logs/error
stock.market.nyse	stock/market/nyse

This direct mapping allows MQTT or other FlowMQ-native clients to seamlessly subscribe to AMQP message streams using familiar wildcard filters like logs/# or stock/market/+.

AMQP Binding Key Mapping

AMQP consumers use a binding key to subscribe to messages from a topic exchange. These keys can contain wildcards: * (star) to match a single level and # (hash) to match zero or more levels. FlowMQ translates these binding keys into equivalent MQTT-style topic filters.

The translation rules are:

• The . delimiter is replaced with /.
• The * wildcard is replaced with +.
• The # wildcard is used directly.

AMQP Binding Key	Mapped FlowMQ Topic Filter
orders.*.new.books	orders/+/new/books
logs.error.#	logs/error/#
stock.market.*	stock/market/+

This translation enables an AMQP consumer to receive messages published from any protocol. For instance, an AMQP client binding with logs.# can receive a message published by an MQTT client to the topic logs/security/critical.

Topic-Based Message Routing

This separation of topic from consumer (Queue/Stream) is what enables FlowMQ's flexibility. Let's consider a single message published to the topic orders/eu/new/books.

Here's how different parts of a system could use it:

In this example, one single publish event serves three distinct purposes:

1. Work Queuing: A book-processing-eu queue, subscribed to the specific topic, gets the message. A pool of workers can process this order. Because it's a queue, only one worker will get this specific message.
2. Global Auditing: An all-orders-log stream, subscribed to all order events with orders/#, receives a copy. This stream acts as a permanent, replayable log of every order in the system.
3. Regional Analytics: An eu-events stream, subscribed to orders/eu/#, gets a copy for real-time sales monitoring in the EU region.

This model allows you to add new integrations and use cases without ever changing your original publisher code. You simply create a new queue or stream with the appropriate topic filter.