Architecture Overview

This chapter provides a high-level look at FlowMQ's architecture. We will start with the core principles that guided our design and then explore how each component of the system embodies those principles.

Design Principles

FlowMQ is built on a foundation of modern, cloud-native design principles. These are not just technical details; they are the reasons FlowMQ is both powerful and easy to operate.

Separation of Compute and Storage

The broker (compute), which processes active connections and routes messages, is a completely independent service from the message log (storage). Benefit: This allows for independent, elastic scaling. You can add more broker nodes to handle a surge in client connections without altering your storage tier, and your storage can grow to petabytes without requiring more brokers.

Separation of Data Plane and Control Plane

The management of the system (the Control Plane) is fully separate from the real-time path a message travels (the Data Plane). Benefit: This provides exceptional reliability. An issue in a background management task or the user interface will not impact the core mission of routing messages at high speed. The planes are isolated to protect message flow.

Stateless Brokers

The broker nodes that make up the data plane do not store any critical, long-term state. All durable message data and meatadata lives in the Storage and Control Planes. Benefit: This provides extreme resilience and rapid scaling. A failed broker node can be replaced instantly by a new one without any data loss or complex state recovery, making the system self-healing.

Separation of Protocols and Routing Core

The logic for handling specific protocols like MQTT, AMQP, and the Kafka protocol is implemented in separate "adapter" layers, away from the central routing engine. Benefit: This design makes the system highly extensible. New protocols can be added as new adapters in the future without requiring a rewrite of the core routing and storage systems.

The Three-Layer Architecture

FlowMQ's architecture is composed of three distinct, independent layers that embody the principles above. A client's request may interact with all three, but each layer has a specialized job, allowing it to be optimized and scaled independently.

The Control Plane (The Brain)

The Control Plane is the authoritative brain of the entire FlowMQ system. It does not handle the high-throughput message traffic itself; instead, it is responsible for coordination, configuration, and ensuring the system runs correctly.

Key Responsibilities:

• Metadata Management: The source of truth for all topics, subscriptions, queues, and their configurations. When you create a topic in the UI or via the CLI, you are talking to the Control Plane.
• Authentication & Authorization: Manages user credentials and security policies. It tells the Broker Layer which clients are allowed to connect and what topics they can read from or write to.
• Cluster & Configuration Management: Coordinates the broker nodes, manages deployments, and handles system-wide configuration changes.

The Broker Layer (The Stateless Nexus)

This is the "workhorse" Data Plane that your clients connect to. It’s responsible for receiving, routing, and delivering messages at high speed. As outlined in our principles, this entire layer is stateless, meaning it can be scaled up or down rapidly to meet demand.

Protocol Adapters

The Protocol Adapters are the front door for your clients. Each adapter is responsible for the specific needs of its protocol.

Responsibilities include:

• Handling the protocol-specific handshake and connection logic.
• Managing connection and session state (like MQTT subscriptions).
• Translating protocol-specific commands (e.g., an AMQP basic_publish) into a normalized format that the Core Routing Engine can understand.

The Core Routing Engine

The Core Routing Engine is the heart of the broker. It receives messages in a standardized format from the adapters and performs the critical routing logic based on the topic and subscription rules managed by the Control Plane. It then interacts with the Storage Layer to write and read the actual message data.

The Storage Layer

This layer is the durable "memory" of FlowMQ. It embodies the principle of decoupling compute from storage and is optimized for both high-throughput writes and cost-effective, long-term retention.

• Durable Log Storage (The Stream)

The immutable, long-term home for all messages is a durable log. This log is built on top of cloud-native object storage (e.g., AWS S3, Google Cloud Storage, Azure Blob Storage), which enables virtually infinite, highly-reliable, and cost-effective storage for your message streams.

• High-Performance Tier (Cache & Hot Storage)

To provide low-latency acknowledgements and fast reads for recent data, the Storage Layer uses a performance-optimized tier built with technologies like distributed key-value databases. This tier acts as a "write-back cache," absorbing high-volume writes and serving reads for "hot" data before it is asynchronously flushed to the durable object store for long-term retention. This gives you the best of both worlds: speed and durability.

Putting It All Together: Tracing a Message Flow

Let's trace a single message to see how all three layers work in concert.

1. Connection: A client application initiates a connection to a Broker in the Broker Layer. The Broker consults the Control Plane to authenticate the client's credentials and fetch the security policies for that user.
2. Publish: The client publishes a message. The appropriate Protocol Adapter on the Broker decodes the message, and the Core Routing Engine takes over.
3. Durable Write: The Core Routing Engine sends the message to the Storage Layer, where it is first written to the High-Performance Tier. The write is acknowledged to the client as soon as it's secured in this tier, ensuring a low-latency response.
4. Asynchronous Flush: In the background, the Storage Layer safely flushes the message from the high-performance tier to the Durable Log Storage (in object storage) for cost-effective, long-term retention.
5. Consumption: When another client connects to consume messages, its request goes to a Broker. The Broker serves the messages from the High-Performance Tier if the data is recent and "hot." If the requested data is older, the Broker efficiently retrieves it directly from the Durable Log Storage.

Next Steps

With this architectural overview, you now have a solid foundation for understanding FlowMQ's behavior and benefits. For a deeper dive on specific components, performance tuning, or API references, consult our Detailed Reference Docs.