Pulsar Schema Evolution and Compatibility

Schemas change as the business changes. The Pulsar Schema Registry versions every schema on a topic and checks each new version against the old ones, so that a schema change doesn’t silently break the applications already reading that topic.

Schema evolution applies to Avro, JSON, Protobuf, and ProtobufNative schemas. Other schema types use a default checker that disables evolution entirely.

Two mechanisms govern evolution: the compatibility check, which decides whether a new schema is allowed at all, and AutoUpdate, which decides whether a passing schema is registered automatically.

Schema versioning

Every SchemaInfo stored with a topic carries a version, and versions are what make schema change manageable within a topic. Messages produced with a SchemaInfo are tagged with its schema version. When a client consumes one, it uses that version to retrieve the matching SchemaInfo and deserialize the data with the right schema. Once a producer is assigned a version, every subsequent message it sends carries that version. Suppose you create a producer like this:

PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

Producer<SensorReading> producer = client.newProducer(JSONSchema.of(SensorReading.class))
        .topic("sensor-data")
        .sendTimeout(3, TimeUnit.SECONDS)
        .create();

Three things can happen when that producer connects:

Scenario	Result
No schema exists for the topic.	The producer is created with the given schema, the schema is sent to the broker and stored, and any consumer using the same schema can read from `sensor-data`.
A schema exists and the producer connects with the same one.	The schema is sent to the broker, the broker finds it compatible, and—on discovering it’s already stored—reuses it to tag produced messages.
A schema exists and the producer connects with a new, compatible one.	The schema is sent to the broker, which finds it compatible and stores it as the current version under a new version number.

Schema compatibility check

A compatibility check exists to guarantee that consumers already running can still process the messages a new schema introduces. When a broker receives a SchemaInfo from a producer, it identifies the schema type and runs that type’s compatibility checker against the topic’s existing schema, applying the configured strategy. Avro, JSON, and Protobuf each have their own checker; every other schema type falls back to a default checker that disables evolution.

Compatibility strategies

Suppose a topic holds three schema versions—V1 is the oldest, V3 the newest.

Strategy	Definition	Changes allowed	Checked against
`ALWAYS_COMPATIBLE`	Disables the compatibility check.	All changes	All previous versions
`ALWAYS_INCOMPATIBLE`	Disables schema evolution—every change is rejected.	None	N/A
`BACKWARD`	Consumers on V3 can read data written by producers on the last version, V2.	Add optional fields; delete fields	Latest version
`BACKWARD_TRANSITIVE`	Consumers on V3 can read data written by producers on all previous versions, V2 and V1.	Add optional fields; delete fields	All previous versions
`FORWARD`	Consumers on the last version V2 can read data written by producers on the new schema V3, though they may not use everything the new schema offers.	Add fields; delete optional fields	Latest version
`FORWARD_TRANSITIVE`	Consumers on any previous version, V2 or V1, can read data written by producers on V3.	Add fields; delete optional fields	All previous versions
`FULL`	Both backward and forward compatible between V2 and V3—consumers on either can read data written under the other.	Modify optional fields	Latest version
`FULL_TRANSITIVE`	Both backward and forward compatible across V3, V2, and V1.	Modify optional fields	All previous versions

The default strategy depends on the schema type: FULL for Avro and JSON, and ALWAYS_INCOMPATIBLE for everything else. To change it, see Set the compatibility strategy.

A Pulsar cluster with the Kafka protocol enabled runs both registries, and they do not share a default. Pulsar defaults to FULL for Avro and JSON; the Kafka Schema Registry defaults to BACKWARD for every format. If you operate both on the same cluster, set the strategy explicitly on each rather than relying on either default.

For deeper background on what constitutes a compatible change in each format, see the Avro schema resolution rules and the Protobuf field rules.

Schema validation enforcement

Schema validation enforcement lets brokers reject producers and consumers that connect without a schema. It’s disabled for producers by default (isSchemaValidationEnforced is false), which means:

A producer with no schema can write anything to a topic that has one, so a misconfigured application can put garbage on a schematized topic.
Clients with no schema support can produce to a topic that has a schema.

Turn it on when a topic’s schema is a contract you need enforced. See Manage schema validation.

Schema AutoUpdate

AutoUpdate is enabled by default. When a schema passes the compatibility check, the producer registers it against the topic automatically.

On the producer side

If the topic has no schema—the data is raw bytes—Pulsar registers the schema automatically.
If the topic has a schema but the producer carries none, it produces raw bytes. Whether it’s allowed depends on schema validation enforcement: with enforcement off the producer connects and writes; with it on the producer is rejected.
If the topic has a schema and the producer carries one, the broker follows the producer-side workflow.

On the consumer side

A consumer connecting to a topic without a schema consumes raw bytes and connects with no compatibility check.
A consumer connecting to a topic with a schema follows the consumer-side workflow.

Order of upgrading clients

Which side you upgrade first is determined by the compatibility strategy—get this backwards and you break consumers in production even though every schema passed its check.

Strategy	Upgrade order	Why
`ALWAYS_COMPATIBLE`	Any order	The compatibility check is disabled.
`ALWAYS_INCOMPATIBLE`	N/A	Schema evolution is disabled.
`BACKWARD`, `BACKWARD_TRANSITIVE`	Consumers first	Nothing guarantees that a consumer on the old schema can read data written under the new one. Upgrade every consumer, then start producing new data.
`FORWARD`, `FORWARD_TRANSITIVE`	Producers first	Nothing guarantees that a consumer on the new schema can read data written under the old one. Upgrade every producer, let the old data age out of reach, then upgrade the consumers.
`FULL`, `FULL_TRANSITIVE`	Any order	Consumers on either schema can read data written under the other.

What’s next

Manage schemas

Set the compatibility strategy and manage schema versions.

Schema types

Which types support evolution, and how they’re defined.

Parts of this page are adapted from the Apache Pulsar documentation, licensed under the Apache License 2.0.

Get Started

Clusters

Data Streams

Process

Connect

Lakehouse

Governance

Pulsar Clients

MQTT Clients

Tools

Pulsar Changelogs

Schema versioning

Schema compatibility check

Compatibility strategies

Schema validation enforcement

Schema AutoUpdate

On the producer side

On the consumer side

Order of upgrading clients

What’s next

Manage schemas

Schema types

​Schema versioning

​Schema compatibility check

​Compatibility strategies

​Schema validation enforcement

​Schema AutoUpdate

​On the producer side

​On the consumer side

​Order of upgrading clients

​What’s next

Manage schemas

Schema types

Schema versioning

Schema compatibility check

Compatibility strategies

Schema validation enforcement

Schema AutoUpdate

On the producer side

On the consumer side

Order of upgrading clients

What’s next