How Pub/Sub works

Cloudflare Pub/Sub is a powerful way to send (publish) messages to and from remote clients.

There are four major concepts to understand with Pub/Sub:

1. Brokers and namespaces
2. Authentication
3. Topics and subscriptions
4. Messages

​​ Brokers and namespaces

Brokers and namespaces are fundamentally “containers” for organizing clients, topics, and their associated permissions.

• A namespace is a collection of brokers that can be organized by location, end-customer, environment (production vs. staging), or by teams within an organization. When starting out, one namespace is typically all you need. Namespaces are globally unique across all customers.

• A broker is a term commonly used in MQTT to refer to the “server,” but because an MQTT “server” is effectively a relay or proxy that accepts messages from one set of clients and sends them to the next, the term broker is used to distinguish from a typical client-server architecture. Clients – and their credentials – are scoped to a broker, and a broker itself is an addressable endpoint that accepts MQTT connections from clients on a TCP port.

For example, you could create a namespace called acme-telemetry and a broker called dev-broker. Together, these define an endpoint of dev-broker.acme-telemetry.cloudflarepubsub.com that authenticated clients can connect, send (publish), and receive (subscribe) messages against.

​​ Authentication

All clients must authenticate – prove they are allowed to connect – to a broker, and credentials are scoped per broker. A client with credentials for dev-broker.acme-telemetry.cloudflarepubsub.com would need a separate set of credentials to connect to prod-broker.acme-telemetry.cloudflarepubsub.com even if both are in the same account.

• Authentication is based on the MQTT standard, which allows for username and password (often called token auth) authentication, as well as implementation specific Mutual TLS (often called TLS Client Credentials) based authentication.
• With Cloudflare Pub/Sub, the easiest way to get started is to issue per-client tokens. Tokens take the place of the password in the authentication flow. These tokens are signed JSON Web Tokens Open external link , which can only be generated by Cloudflare. Because they are signed, the client ID, permissions, or other claims embedded in the token cannot be changed without invalidating the signature.

For more information about how authentication is handled, refer to Authentication and authorization.

​​ Topics and subscriptions

The topic is the core concept of Pub/Sub and MQTT, and all messages are contained within the topic they are published on. In MQTT, topics are strings that are separated by a / (forward slash) character to denote different topic levels and define a hierarchy for subscribers. Importantly, and one of the benefits of the underlying MQTT protocol, topics do not have to be defined centrally. Clients can publish to arbitrary topics, provided the broker allows that client to do so, which allows you to flexibly group messages as needed.

A Pub/Sub client can be both a publisher and subscriber at once, and can publish and subscribe to multiple topics at once.

As a set of best practices when constructing and naming your topics:

• Define topics as a consistent hierarchy such as location/data-type/data-format/. For example, a client in the EU publishing HTTP metrics data would publish to eu/metrics/request_count so that subscribers can more easily identify the data.
• Ensure that you are consistent with your casing (lower vs. upper case) for topic names. Topics are case-sensitive in the MQTT protocol and a client subscribed to eu/metrics/request_count will never receive a message published to EU/metrics/request_count (note the upper-cased “EU”).
• Avoid overly long topic names (the MQTT specification supports up to 65K bytes). Long topic names will increase your payload size and the cost of message processing on both publishers and subscribers.
• Avoid a leading forward slash when naming topics. /us/metrics/transactions_processed is a different topic from us/metrics/transactions_processed. The leading slash is unnecessary.

​​ Messages

The MQTT standard that Cloudflare Pub/Sub is built on defines a message as the payload within a PUBLISH packet Open external link . Payloads themselves can be UTF-8 strings, which is what most developers are used to dealing with, or a stream of bytes (a “byte array”) for more complex use cases or for cases where you are using other serialized data formats, such as Protobuf or MessagePack.

In Pub/Sub, which is based on MQTT v5.0, you can also set additional fields to indicate whether the payload is a string or a stream of bytes. Both the payloadFormatIndicator (0 for bytes; 1 for strings) property and the contentType property, which accepts a MIME type Open external link , can be used by the client to more clearly define the payload format.

As a set of best practices when sending Pub/Sub messages, you should consider:

• Keep messages reasonably sized. Buffering data on the client up to 1 KB (or every 2-3 seconds) is a good way to optimize for message size, throughput, and overall system latency.
• Set the payloadFormatIndicator property when publishing a message. This gives your subscribers or Workers a hint about how to parse the message.
• Set the contentType property to the MIME type of the payload. For example, application/json or application/x-msgpack as an additional hint, especially if clients are actively publishing messages in different formats.