Flow SDK Developer Guide

1. Introduction to Flow SDK

1.1. What is Flow SDK?

The Flow SDK is a collection of language-specific libraries designed for seamless interaction with FlowMQ. The core SDK is implemented in the Flow programming language—a C++11 extension featuring actor-based concurrency, famously used in FoundationDB—and communicates with the FlowMQ broker via gRPC for high performance. It abstracts the complexities of FlowMQ's "Unified Routing Model," providing a clean, idiomatic, and high-performance interface for developers.

• Purpose: To simplify the development of applications that produce messages to or consume messages from FlowMQ.
• Benefits:
  • Idiomatic: Feels natural to use within the target programming language.
  • High-performance: Optimized for low latency and high throughput.
  • Feature-rich: Provides access to FlowMQ's core functionalities.
  • Reduces boilerplate: Less code for you to write for common messaging tasks.

1.2. Design Philosophy

The Flow SDK is built with the following principles in mind:

• Ease of use: The APIs are designed to be intuitive and straightforward, enabling developers to get started quickly.
• Performance: Focus on efficient resource utilization, connection management, and message handling to ensure optimal performance.
• Unified Experience: The SDK provides a consistent API surface for all of FlowMQ's messaging models—publish/subscribe, message queues, and data streams. This is a direct reflection of FlowMQ's core "Unified Routing Model", which allows developers to learn one set of concepts and apply them across different use cases. Once you learn to publish to a topic, consuming from a queue or a stream feels immediately familiar.
• Asynchronous Nature: Leverages non-blocking operations to build responsive and scalable applications. This is managed through Futures/Promises or async/await patterns depending on the language.
• Resilience: Includes features like automatic reconnection and configurable retry mechanisms to help build robust applications that can withstand transient network issues.

1.3. Supported Languages

The Flow SDK is officially available for the following programming languages:

• Java (version 8+)
• Go (version 1.18+)
• Python (version 3.7+)
• Rust (version 1.56+)

1.4. Where to Get the SDK

You can add the Flow SDK to your project using standard package managers for each supported language:

• Java: Maven Central (com.flowmq:flowmq-sdk)
• Go: Go modules (go get github.com/flowmq/flowmq-sdk-go)
• Python: PyPI (pip install flowmq-sdk)
• Rust: crates.io (cargo add flowmq-sdk)

The source code for each SDK is available on our official GitHub organization: https://github.com/flowmq/

2. Core Concepts & Architecture

2.1. The FlowClient

The FlowClient is the primary entry point for all interactions with FlowMQ. It represents a logical connection to a FlowMQ cluster and manages the underlying gRPC connections, authentication, and configuration. It is a heavyweight, thread-safe object designed to be long-lived.

• Configuration: You instantiate and configure the FlowClient with details such as:
  • FlowMQ broker address(es).
  • Authentication credentials (e.g., API keys, tokens).
  • Connection options (e.g., timeouts, TLS/SSL settings, reconnection policies).

2.2. Producers

Producers are responsible for sending messages to FlowMQ. The Flow SDK provides a Producer object created from the FlowClient.

• Publishing to Topics: Messages are published to named Topics. FlowMQ's routing engine then handles delivery based on how these topics are configured to interact with queues, streams, or other subscriptions.

2.3. Consumers

Consumers are responsible for receiving and processing messages from FlowMQ. The SDK provides Consumer objects created from the FlowClient.

• Subscribing to Topics (Pub/Sub): Applications can subscribe to topics to receive all messages published to them.
• Consuming from Queues: Applications can consume messages from named Queues. Messages from a queue are typically processed by one consumer in a group.
• Consuming from Streams: Applications can consume messages from Streams, which represent ordered, replayable sequences of records. Consumers can read from specific positions (offsets).

2.4. Message Representation

The SDK defines a Message object that encapsulates the data being sent or received.

• Structure:
  • payload: The actual content of the message (e.g., bytes, string, JSON).
  • headers/properties: Key-value pairs for custom metadata.
  • metadata: SDK or FlowMQ-provided information, such as messageId, timestamp, topic, and offset (for stream messages).

2.5. Asynchronous Operations

Nearly all SDK operations that involve network I/O are performed asynchronously to avoid blocking application threads and to maximize performance, fitting the actor-based model of the underlying Flow implementation.

• Publishing: publish() methods return immediately. The result (success or failure) is communicated via a Future/Promise or async/await.
• Consuming: Message handlers are callback functions that the SDK invokes when a new message arrives.

2.6. Error Handling

The SDK defines a set of exceptions or error types for common conditions (e.g., connection failures, authentication errors, publish timeouts). Your application should be prepared to handle these gracefully.

3. Getting Started: Language-Specific Guides

For detailed instructions, installation steps, and complete, runnable examples for your language of choice, please see the dedicated guides:

• Connect with Java
• Connect with Python
• Connect with Go
• Connect with Rust

Below is a high-level overview of the main operations.

3.1. Installation

The Flow SDK is distributed via standard package managers for each supported language (e.g., Maven Central, PyPI, Go modules, crates.io). Please see the language-specific guides for exact dependency details.

3.2. Initializing the FlowClient

The first step in any application is to configure and create an instance of the FlowClient. This is typically done once per application. You provide the broker address and authentication credentials to a configuration object, which is then used to instantiate the client.

4. Publishing Messages

The process of sending a message generally follows these steps:

1. Create a Producer: A lightweight Producer object is created from the FlowClient instance.
2. Construct a Message: Messages are represented by a Message object that contains the payload (as bytes) and optional headers (as key-value pairs).
3. Publish to a Topic: The primary method for sending a message is publish, which is an asynchronous operation. You provide the topic name and the message object. The operation returns a Future, Promise, or uses a callback/await mechanism to signal completion.

For full code examples, please see the guides linked above.

5. Receiving Messages

5.1. Consumer Concepts

• Message Processing: Consumers run a loop or register a handler that is invoked upon message arrival.
• Acknowledgements (Queues): Acknowledging (ack) a message confirms it has been processed and can be removed from the queue.
• Offset Committing (Streams): Committing an offset saves a consumer's progress in a stream, allowing it to resume from that point.

5.2. Consuming from Queues

To consume from a queue, you create a Consumer, specify the Queue name, and provide a message handler function. Inside your handler, you must call ack() on the message after you have finished processing it to ensure it is removed from the queue.

5.3. Consuming from Streams

Consuming from a stream is similar to consuming from a queue, but instead of acknowledging each message, you commit the offset. This marks your position in the stream, allowing you or another consumer to resume from that exact point. You can specify a StartPosition (e.g., earliest or latest) when you begin consuming.

For full code examples, please see the guides linked above.

6. Advanced Features

• Connection Management: Automatic reconnection with configurable backoff strategies.
• Error Handling: SDK-level retries for transient errors.
• Security: TLS/SSL configuration and pluggable authentication modules.
• Configuration: Granular control over client, producer, and consumer behavior (e.g., batching, timeouts, buffer sizes).

7. Best Practices

• Client Lifecycle: FlowClient instances are heavyweight and thread-safe. Create one per application and reuse it. Ensure it's closed on shutdown.
• Efficient Serialization: Use efficient formats like Protobuf or Avro for performance-critical applications.
• Idempotent Consumers: Design message handlers to be idempotent to safely handle message redeliveries, which are possible with at-least-once semantics.
• Acknowledgement vs. Offset Committing:
  • Use message acknowledgement (ack/nack) for Queues.
  • Use offset committing for Streams.