Pulsar Schema Registry

Pulsar messages are stored as unstructured byte arrays, and structure is applied to that data only when it’s read. Producers and consumers must therefore agree on the shape of a message, including its fields and their types. A Pulsar schema is the metadata that defines how to translate raw message bytes into a structured type. It acts as a contract between the applications that produce messages and the applications that consume them: data is serialized into bytes before it’s published to a topic, and deserialized back into a typed object before it’s delivered to a consumer. Every Pulsar cluster on StreamNative Cloud runs a schema registry inside its brokers. The registry stores registered schema information centrally, so producers and consumers can coordinate the schema of a topic’s messages through the broker rather than out of band.

A Pulsar cluster with the Kafka protocol enabled also runs the Kafka Schema Registry. The two registries are separate systems and are not interoperable—a schema registered in one is not visible to the other. See Data governance overview before you build.

Why use a schema

Type safety matters in any system built around messaging and streaming. Raw bytes are flexible, but that flexibility has a cost: every application has to layer its own type checking and serialization on top to guarantee that what goes in can be read back out. A Pulsar schema addresses this by:

Enforcing type safety. Once a topic has a schema, producers and consumers connect only if they use a compatible schema.
Centralizing schema information. One location holds the schemas used across your organization, which makes sharing them between teams straightforward.
Acting as a single source of truth for the message schemas used across your services.
Keeping versions compatible. When a new schema is uploaded, compatibility rules govern whether older consumers can still read the data.
Reusing existing storage. Schemas live in the cluster’s existing storage layer. No extra system to operate.

How it works

Pulsar schemas are applied and enforced at the topic level. Both producers and consumers can upload schemas to the broker.

Producer side

The application builds a producer from a schema instance. That instance defines the schema for the data the producer sends. With Avro, for example, Pulsar extracts the schema definition from the POJO class and constructs a SchemaInfo.
The producer connects to the broker, passing the SchemaInfo from the schema instance.
The broker looks the schema up in the registry. If it’s already registered, the broker returns the schema version to the producer and the flow ends here.
If the schema isn’t registered, the broker checks whether schemas on this topic can be updated automatically. If not, the schema can’t be registered and the broker rejects the producer.
Otherwise the broker runs the compatibility check configured for the topic. If the schema passes, the broker stores it and returns the schema version, and every message this producer sends is tagged with that version. If it fails, the broker rejects the producer.

Consumer side

The application builds a consumer from a schema instance.
The consumer connects to the broker, passing the SchemaInfo from that instance.
The broker checks whether the topic is in use—that is, whether it already has a schema, data, an active producer, or an active consumer.
If the topic isn’t in use, the broker checks whether schemas can be updated automatically. If they can, it registers the schema and connects the consumer. If not, it rejects the consumer.
If the topic is in use, the broker runs the compatibility check and connects the consumer only if the schema passes.

What it looks like in code

With a schema, you work in your language’s own types instead of hand-rolling serialization. Take a User class:

public class User {
   public String name;
   public int age;

   User() {}

   User(String name, int age) {
      this.name = name;
      this.age = age;
   }
}

Without a schema, a producer can send only byte[], so you serialize the object yourself:

Producer<byte[]> producer = client.newProducer()
        .topic(topic)
        .create();
User user = new User("Tom", 28);
byte[] message = // serialize the user yourself
producer.send(message);

With a schema, you send the object directly:

// Send with a JSON schema
Producer<User> producer = client.newProducer(JSONSchema.of(User.class))
        .topic(topic)
        .create();
User user = new User("Tom", 28);
producer.send(user);

// Receive with a JSON schema
Consumer<User> consumer = client.newConsumer(JSONSchema.of(User.class))
   .topic(schemaTopic)
   .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
   .subscriptionName("schema-sub")
   .subscribe();
Message<User> message = consumer.receive();
User user = message.getValue();
assert user.age == 28 && user.name.equals("Tom");

Client support

Pulsar schemas are available in the Java, Go, Python, Node.js, C++, and C# clients. Support for individual schema types varies by client—see Schema types for the details.

What’s next

Schema types

Primitive types, complex types, and auto schemas.

Compatibility

Compatibility strategies, versioning, and safe evolution.

Manage schemas

Upload, retrieve, and delete schemas with the CLI and REST API.

Use with clients

Produce and consume typed messages from your application.

Parts of this page are adapted from the Apache Pulsar documentation, licensed under the Apache License 2.0.

Get Started

Clusters

Data Streams

Process

Connect

Lakehouse

Governance

Pulsar Clients

MQTT Clients

Tools

Pulsar Changelogs

Why use a schema

How it works

Producer side

Consumer side

What it looks like in code

Client support

What’s next

Schema types

Compatibility

Manage schemas

Use with clients

​Why use a schema

​How it works

​Producer side

​Consumer side

​What it looks like in code

​Client support

​What’s next

Schema types

Compatibility

Manage schemas

Use with clients

Why use a schema

How it works

Producer side

Consumer side

What it looks like in code

Client support

What’s next