Pulsar Messaging Model

Pulsar sends and receives messages using the publish-subscribe model.

How does Pulsar manage messages?

Pulsar uses acknowledgements for reliable messaging. Let's return back to the notion of a postal service system. This is similar to applying the postmark in the postal processing center. For example, before a letter moves on to the postal carrier for delivery to the recipient, it must be postmarked. In the same way, a message in Pulsar must be acknowledged before the broker sends it on to the consumer.

1. A producer sends messages to the broker and receives an acknowledgement when the message is persisted. Messages await dispatch to the consumer.
2. The broker sends a message to the consumer.
3. The consumer sends an acknowledgement when it consumes a message. The broker will only dispatch more messages when the consumer has space.

How does Pulsar acknowledge messages?

Pulsar can acknowledge messages in two ways individually or cumulatively. By having both types of acknowledgements, Pulsar can support a broader set of use cases.

Individual acknowledgements

The consumer acknowledges each message and sends an acknowledgement request to the broker. This type of acknowledgement is commonly used in messaging systems for preventing redelivery and to support work queues. It is ideal for messaging systems to reduce message redelivery because acknowledgements are more frequent and more granular.

Cumulative acknowledgements

The consumer only acknowledges the last message it received. This type of acknowledgement is commonly used in streaming systems for ordered delivery or fan-out consumption. Cumulative acknowldgements can:

• Act as a "watermark" or periodic checkpoint in streaming systems
• Increase performance because they are less "chatty"
• Provide more message redelivery in the case of errors

Subscription Model

Subscription modes are the key to understanding how Pulsar unifies messaging and streaming in one platform. In Pulsar, you have flexibility to use four different subscription modes:

• Exclusive
• Failover
• Shared
• Key Shared

Exclusive

Exclusive subscriptions allow only a single consumer to be connected at a time. That consumer is guaranteeing order. If multiple consumers subscribe to a topic using the same subscription, an error occurs.

In diagram below, only consumer A-0 can consume messages.

Failover

Failover subscriptions are similar to exclusive, but if a consumer fails, a second one can pick up where the first one left off. The failover subscription allows multiple consumers to subscribe to the same topic.

A master consumer is selected for non-partitioned topics or for each partition of partitioned topics to receive messages. When the master consumer gets disconnected, all (non-acknowledged and subsequent) messages are delivered to the next consumer in line.

The failover subscription allows multiple consumers to subscribe to the same topic. Among all consumers, one is selected as the master consumer to receive messages. When failures occur to this master consumer, the next consumer in line takes the role. This subscription type guarantees order.

In the diagram below, consumer B-0 is the master consumer. If consumer B-0 loses connection, the next consumer B-1 will receive the messages, including the non-acknowledged and all subsequent ones.

Exclusive and Failover subscriptions are best applied to fan-out or streaming use cases where strict ordering is required. They allow only one consumer per topic partition per subscription. They consume messages in partition order.

Shared

Shared (or round robin) mode allows multiple consumers to attach to the same subscription. Shared subscriptions consume a subset of the messages. There is no ordering guarantee but it allows for distributed work. Messages are delivered in a round robin distribution across consumers, and any given message is delivered to only one consumer.

Shared subscriptions are best for queuing use cases where ordering is not required.

Key Shared

Key Shared mode distributes messages using an ordering key. Messages with the same key are delivered to the same consumer in the order they were received.

Multiple consumers can attach to the same subscription. Messages with same key or same ordering key are delivered to only one consumer. No matter how many times the message is re-delivered, it is delivered to the same consumer.

Key Shared subscriptions make many streaming uses cases easier because they allow preserving order (per key) while distributing work among any number of consumers.

Subscriptions and cursors

A messaging system can be distributed across many machines, racks, or even data centers. At times, things can and will fail. In the case of a failure, you want to be able to start reading from where you left off (once everything recovers), so that you don't miss messages and so you don't have to process messages that have already been processed.

In Pulsar, this "restart" point is called a cursor. Pulsar can go back to a specific message using a cursor. Each subscription for a topic has a cursor. The cursor contains information about message acknowledgements. When a consumer reads and processes a message, it sends an acknowledgment to the Pulsar broker and the cursor is updated. Updating the cursor ensures that the consumer will not receive that message again — even when the consumer crashes, recovers, and reattaches to the subscription.

Next steps

• Learn about Pulsar architecture and design.
• Learn about Pulsar storage model.