Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar
Authored by
Support type
Apache License 2.0

Kafka-on-Pulsar (aka KoP)

KoP (Kafka on Pulsar) brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the KoP protocol handler to your existing Pulsar cluster, you can now migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar’s powerful features, such as:

  • Streamlined operations with enterprise-grade multi-tenancy
  • Simplified operations with a rebalance-free architecture
  • Infinite event stream retention with Apache BookKeeper and tiered storage
  • Serverless event processing with Pulsar Functions

KoP, implemented as a Pulsar protocol handler plugin with the protocol name "kafka", is loaded when Pulsar broker starts. It is believed that providing a native Kafka protocol support on Apache Pulsar helps reduce the barriers for people adopting Pulsar to achieve their business success. By integrating two popular event streaming ecosystems, KoP unlocks new use cases. You can leverage advantages from each ecosystem and build a truly unified event streaming platform with Apache Pulsar to accelerate the development of real-time applications and services.

KoP implements the Kafka wire protocol on Pulsar by leveraging the existing components (such as topic discovery, the distributed log library - ManagedLedger, cursors and so on) that Pulsar already has.

The following figure illustrates how is the Kafka-on-Pulsar protocol handler was implemented within Pulsar.


Check the following requirements before using KoP.

Currently, KoP supports Kafka Client 1.x and 2.x and it is build based on Pulsar 2.6.1.

The pulsar-protocol-handler-kafka-${version}.nar is the KoP binary file which can be loaded by Pulsar broker.

Enable KoP on your existing Apache Pulsar clusters

If you already have a Apache Pulsar cluster running, it is very straightforward to enable Kafka-on-Pulsar on your existing Pulsar cluster by downloading and installing the KoP protocol handler to Pulsar brokers.

Download or build KoP protocol handler

You can download the released KoP protocol handler from here.

Alternatively, you can also build the KoP protocol handler from source code.

  1. clone this project from GitHub to your local.
git clone
cd kop
  1. build the project.
mvn clean install -DskipTests
  1. the nar file can be found at this location.

Install KoP protocol handler

As mentioned previously, the KoP protocol handler is a plugin that can be installed to the Pulsar brokers.

All what you need to do is to configure the Pulsar broker to run the KoP protocol handler as a plugin, that is, add configurations in Pulsar's configuration file, such as broker.conf or standalone.conf.

  1. Set the configuration of the KoP protocol handler.

    Add the following properties and set their values in Pulsar configuration file, such as conf/broker.conf or conf/standalone.conf.

    KoP supports partitioned topics only. Therefore, you had better to set allowAutoTopicCreationType to partitioned. If it is set to non-partitioned by default, the topics that are automatically created by KoP are still partitioned topics. However, topics that are created automatically by Pulsar broker are non-partitioned topics.

    PropertySet it to the following valueDefault value
    protocolHandlerDirectoryLocation of KoP NAR file./protocols


  2. Set Kafka service listeners.


    The hostname in listeners should be the same as Pulsar broker's advertisedAddress.



Restart Pulsar brokers to load KoP

After you have installed the KoP protocol handler to Pulsar broker, you can restart the Pulsar brokers to load KoP.

Run Kafka client to verify

  1. Download the Kafka 2.0.0 release and untar it.

    tar -xzf kafka_2.11-2.0.0.tgz
    cd kafka_2.11-2.0.0
  2. Use a Kafka producer and a Kafka consumer to verify.

    In Kafka’s binary, there is a command-line producer and consumer.

    Run the command-line producer and send a few messages to the server.

    > bin/ --broker-list [pulsar-broker-address]:9092 --topic test
    This is a message
    This is another message

    Kafka has a command-line consumer dumping out messages to standard output.

    > bin/ --bootstrap-server [pulsar-broker-address]:9092 --topic test --from-beginning
    This is a message
    This is another message

Configure KoP

For details, see Configure KoP.

Secure KoP

KoP supports TLS encryption and integrates with Pulsar's authentication and authorization providers seamlessly.

For details, see Secure KoP.


See Implementation for the implementation details, including some difference of basic concepts between Kafka and Pulsar, and how the conversion is done.